CN102332090A

CN102332090A - Compartmentalizing focus area within field of view

Info

Publication number: CN102332090A
Application number: CN2011101790441A
Authority: CN
Inventors: S·麦克尔道尼; J·A·塔迪夫; J·克拉维恩; D·科恩; G·叶海弗
Original assignee: Microsoft Corp
Current assignee: Microsoft Technology Licensing LLC
Priority date: 2010-06-21
Filing date: 2011-06-20
Publication date: 2012-01-25
Anticipated expiration: 2031-06-20
Also published as: US8654152B2; CN102332090B; US20110310125A1

Abstract

A system and method are disclosed for selectively focusing on certain areas of interest within an imaged scene to gain more image detail within those areas. In general, the present system identifies areas of interest from received image data, which may for example be detected areas of movement within the scene. The system then focuses on those areas by providing more detail in the area of interest. This may be accomplished by a number of methods, including zooming in on the image, increasing pixel density of the image and increasing the amount of light incident on the object in the image.

Description

Be divided in the focus area in the visual field

Technical field

The application relates to focusing technology, particularly a kind of technology that is divided in the focus area in the visual field.

Background technology

In the past, use controller, telepilot, keyboard, mouse or the like to allow other aspects of user's operate game personage or application program such as the computing application program of computer game and multimedia application etc.Recently, computer game and multimedia application have brought into use camera and software gesture recognition engine that nature user interface (" NUI ") is provided.By means of NUI, other aspects of game character or application program are surveyed, explained and be used to control to user's posture.

Sometimes, activity can occur in the little part of whole visual field.For example, unique user may stand in the big room of fixed object.Perhaps, the user can only use his hand to make gesture, for example controls user interface or carries out sign language.Yet conventional NUI system handles all information from scene in an identical manner, does not consider that its is static or dynamic.Therefore, need the system that the dynamic area in the visual field is compared the static region focusing more attention in the visual field.

Summary of the invention

Disclosed herein is to be used for focusing on selectively interested some zone in the imaging scene so that obtain the system and method for the more image details in those zones.Usually, native system is from the view data that received sign interesting areas, the moving area that these zones can for example be in scene to be surveyed.Then, this system focuses on those zones through the more details in the interesting areas are provided.This can accomplish through many methods, such as for example to the picture element density in this regional mechanical zoom or digital zoom, the increase zone, reduce this extra-regional picture element density and be added to the amount that is mapped to the light on this zone.In order to handle the view data in given frame per second, the zone of the image outside interesting areas can be stored in impact damper and reuse when needed with the view data from interesting areas, so that present the image of (render) scene.

In one embodiment, present technique relates to the method for the image detail in the one or more interesting areas that are increased in the scene of being caught by capture device.This method may further comprise the steps: a) receive the information from scene; B) be identified at the interior one or more interesting areas of this scene; C) obtain about the one or more interesting areas in this scene, with respect to the regional more images details outside said one or more interesting areas in this scenery; And d) at least periodically keeps watch on the information outside one or more interesting areas in this scene so that judge whether to define again one or more interesting areas.

In a further embodiment; Present technique relates to the method for the image detail in the one or more interesting areas that are increased in the scene of being caught by capture device; May further comprise the steps: a) be defined in the focus area in the scene, this focus area is defined as corresponding with the one or more desired interesting areas in this scene; And b) obtain about the focus area in this scene, with respect to the regional more images details outside one or more interesting areas in this scene.

In a further embodiment, present technique relates to the method for the image detail in the one or more interesting areas that are increased in the scene of being caught by capture device.This method may further comprise the steps: a) receive the information from scene; B) be identified at the interior one or more users of this scene; C) obtain about at least one body part of the one or more users in this scene, with respect to the more images details in the zone except that said one or more users in this scene; D) use the more images details in said step c), obtained to identify the posture of carrying out by one or more users about the one or more users' in this scene said at least body part; And e) at least periodically keeps watch on information outside one or more users in this scene, so that judge whether in said group of one or more users of more images details that in step c), obtains about it, to add or deduct a user.

This general introduction is provided so that the selected works of the notion that below the reduced form introduction, in detailed description, further describes.This general introduction is not intended to identify the key feature or the essential feature of required for protection theme, and also it is not intended to the scope of helping confirm required for protection theme.In addition, required for protection realization that theme is not limited to solve any or all shortcoming of mentioning in the present disclosure.

Description of drawings

Figure 1A explains the example embodiment of Target Recognition, analysis and tracker.

Figure 1B explains the further example embodiment of Target Recognition, analysis and tracker.

The example embodiment of the capture device that Fig. 2 explaination can be used in Target Recognition, analysis and tracker.

Fig. 3 A explaination can be used in Target Recognition, analysis and tracker, explain the example embodiment of the computing environment of one or more postures.

Fig. 3 B explaination can be used in Target Recognition, analysis and tracker, explain another example embodiment of the computing environment of one or more postures.

The user's that Fig. 4 explaination has generated from Target Recognition, analysis and the tracker of Figure 1A-Fig. 2 bone mapping.

Fig. 5 is the explaination by the pixelation image of the scene of capture device seizure.

Fig. 6 is the explaination of the pixelation image of the more focus on the interesting areas that is illustrated in the scene of being caught by capture device.

Fig. 6 A is the explaination of the pixelation image that substitutes of the more focus on the interesting areas that is illustrated in the scene of being caught by capture device

Fig. 7 is the explaination of the pixelation image of the more focus on the interesting areas that substitutes that is illustrated in the scene of being caught by capture device.

Fig. 8 is the explaination of a pair of pixelation image that illustrates two interesting areas of the scene that free capture device catches.

Fig. 9 is the explaination that is illustrated in by the pixelation image of the picture element density that has increase in the interesting areas in the scene of capture device seizure.

Figure 10 is the explaination that is illustrated in the focus area in the image of the scene of being caught by capture device.

Figure 11 is the process flow diagram that is used for the operation of a kind of embodiment of the present technique of image focusing on the interesting areas in scene.

Figure 12 is the process flow diagram of operation of further embodiment that is used for through the picture element density that increases the zone image being gathered in the present technique of the interesting areas in scene.

Figure 13 is the process flow diagram of operation of another embodiment of the present technique present technique that is used for increasing passively in the focus area on the interesting areas of scene focus.

Figure 14 is the process flow diagram of operation of further embodiment that is used for being added to the present technique of the amount that is mapped to the light on the object of scene in interesting areas.

Figure 15 is the process flow diagram of operation of another embodiment that is used for being added to passively the present technique of the amount that is mapped to the light on the object of scene in interesting areas.

Figure 16 is the block diagram that the gesture recognition engine that is used to discern posture is shown.

Figure 17 is the process flow diagram of operation of the gesture recognition engine of Figure 16.

Embodiment

Describe each embodiment of present technique referring now to Figure 1A-Figure 17, these embodiment relate generally tos are used for focusing on selectively interested some zone in the imaging scene so that obtain the system and method for the more image details in those zones.Usually, the view data sign interesting areas of native system from being received, these zones are the moving area of being surveyed in this scene for example.Then, this system focuses on those zones through the more details that are provided in the interesting areas.This can accomplish through many methods, and these methods are included in the picture element density of convergent-divergent on the image, increase image and are added to the amount of the light on the object that is mapped in the image.

Referring to Figure 1A-Fig. 2, be used to realize that the hardware of present technique comprises Target Recognition, analysis and tracker 10 at first, this system can be used to discern, analyze and/or follow the tracks of the people's class targets such as user 18 etc.Each embodiment of Target Recognition, analysis and tracker 10 comprises the computing environment 12 that is used to carry out recreation or other application programs.Computing environment 12 can comprise makes computing environment 12 can be used to carry out nextport hardware component NextPort and/or the component software such as the application program of recreation and non-game application etc.In one embodiment, computing environment 12 can comprise that such as standard processor, application specific processor, microprocessor or the like processor this processor can be carried out and be stored in the instruction on the readable memory device of processor so that carry out process described here.

System 10 also comprises and is used to catch one or more users and/or the image of object and the capture device 20 of voice data that relates to by the capture device perception.In each embodiment, capture device 20 can be used to catch the information that relates to the moving of one or more users, posture and speech, and this information receives and be used to appear by computing environment, the each side of mutual and/or control recreation or other application programs with it.Example calculations environment 12 will be explained in more detail with capture device 20 below.

Each embodiment of Target Recognition, analysis and tracker 10 can be connected to the audio/video devices 16 with display 14.Equipment 16 can for example be recreation or the TV of application program video and/or audio, monitor, high-definition television (HDTV) or the like to be provided to the user.For example, computing environment 12 can comprise can provide and play or the audio/video signal of other application-associated such as the EGA of video card etc. and/or such as the audio frequency adapter of sound card etc.Audio/video devices 16 can receive the audio/video signal from computing environment 12, and the recreation or the application program video and/or audio that can be associated with audio/video signal to user's 18 outputs then.According to a kind of embodiment, audio/video devices 16 can be connected to computing environment 12 via for example S vision cable, concentric cable, HDMI cable, DVI cable, VGA cable, component video cable or the like.

In each embodiment, computing environment 12, A/V equipment 16 and capture device 20 can be cooperated and on display 14, presented incarnation or screen personage 19.In each embodiment, incarnation 19 analog subscribers 18 moving in real world space makes user 18 can carry out moving of incarnation 19 and the moving and posture of action on the control display 14.

In Figure 1A, capture device 20 is used to just the playing NUI system of Association football recreation of wherein for example a pair of user 18.In this example, computing environment 12 can use audiovisual display 14 to provide with the visual representation by two incarnation 19 of the Association football sportsman form of each user 18 controls.User 18 can move or carry out to play to move in physical space and in gamespace, move or play Association football so that cause their coupled movements person's incarnation 19.Thereby; According to example embodiment; Computing environment 12 can be used to discern and analysis user 18 moving and posture in physical space with capture device 20, and so mobile and posture can be interpreted as user's game control or the action of related incarnation 19 in gamespace.

The embodiment of Figure 1A is in the many different application program that can on computing environment 12, move, and the application program of operation on computing environment 12 can be multiple other recreation and non-game application.In addition, can also system 10 be moved as be used for interpreting user 18 at the field of play of operation on the computing environment 12 or the operating system (OS) outside the concrete application program and/or application program control.Figure 1B illustrates an example, and wherein user's 18 rollings and control have the user interface 21 that is present in each menu option on the display 14.In fact any controllable aspect of operating system and/or application program can be controlled by moving of user 18.

Figure 1A and 1B also illustrate stationary body 23, for example chair and plant.These are the objects in this scene (that is, by the zone of capture device 20 seizure), but between frame and frame, can not change.Except shown chair and plant, stationary body can be any object that is picked up by the image camera in the capture device 20.Additional stationary body in this scene can comprise any wall, floor, ceiling, window, door, wall fitting thing or the like

The suitable example of system 10 and assembly thereof is shown in following co-pending patented claim; All these co-pending patented claims all are herein incorporated by reference clearly: the title of submitting on May 29th, 2009 is No. the 12/475th, 094, the U.S. Patent application of " Environment And/Or Target Segmentation (environment and/or target are cut apart) "; The title of submitting on July 29th, 2009 is the U.S. Patent application of " Auto Generating a Visual Representation (generating visual expression automatically) " No. the 12/511, No. 850; The title of submitting on May 29th, 2009 is No. the 12/474th, 655, the U.S. Patent application of " Gesture Tool (posture instrument) "; The title of submitting on October 21st, 2009 is No. the 12/603rd, 437, the U.S. Patent application of " Pose Tracking Pipeline (Attitude Tracking pipeline) "; The title of submitting on May 29th, 2009 is No. the 12/475th, 308, the U.S. Patent application of " Device for Identifying and Tracking Multiple Humans Over Time (being used for identifying in time and following the tracks of a plurality of people's equipment) "; The title of submitting on October 7th, 2009 is No. the 12/575th, 388, the U.S. Patent application of " Human Tracking System (human tracker) "; The title of submitting on April 13rd, 2009 is No. the 12/422nd, 661, the U.S. Patent application of " Gesture Recognizer System Architecture (gesture recognizers system architecture) "; The title of submitting on February 23rd, 2009 is No. the 12/391st, 150, the U.S. Patent application of " Standard Gestures (standard posture) "; And be No. the 12/474th, 655, the U.S. Patent application of " Gesture Tool (posture instrument) " in the title that on May 29th, 2009 submitted to.

Fig. 2 explaination can be used to the example embodiment of the capture device 20 of Target Recognition, analysis and tracker 10.In an example embodiment, capture device 20 can be configured to catch the video with depth image, and depth image can comprise the depth value via any suitable technique, and these technology comprise for example transit time, structured light, stereo image or the like.According to a kind of embodiment, capture device 20 can be organized as the depth information that is calculated " Z layer " or can be perpendicular to each layer of Z axle expansion from depth cameras along its light.

As shown in Fig. 2, capture device 20 can comprise image camera assembly 22.According to an example embodiment, image camera assembly 22 can be the depth cameras that can catch the depth image of scene.Depth image can comprise two dimension (2-D) pixel region of the scene of being caught; Wherein each pixel in this 2-D pixel region can be represented depth value, and depth value is such as length or distance with for example centimetre, millimeter or the like from the object in the scene of being caught of this camera.

As shown in Fig. 2, according to an example embodiment, image camera assembly 22 can comprise IR optical assembly 24, three-dimensional (3D) camera 26 and RGB camera 28 of the depth image that can be used to catch scene.For example; During transit time analyzed, the IR optical assembly 24 of capture device 20 can be transmitted into infrared light on the scene and can use the sensor (not shown) for example to use 3D camera 26 and/or RGB camera 28 to survey one or more targets and the backward scattered light of object surfaces from scene then.

In certain embodiments, can use pulsed infrared light so that can measure target or the physical distance of the ad-hoc location on the object being used at output optical pulse with corresponding to the time between the input optical pulse and with it confirming from capture device 20 to scene.In addition, in each embodiment of other examples, can the phase place of output light-wave be compared with the phase place of importing light wave, so that confirm phase shift.Then, phase shift can be used to confirm from capture device 20 to target or the physical distance of the ad-hoc location of object.

According to another example embodiment; Through via comprising that for example shutter light pulse is imaged on the interior time dependent institute of various technical Analysis beam reflected brightness, can use the transit time to analyze and confirm from capture device 20 to target indirectly or the physical distance of the ad-hoc location on the object.

In another example embodiment, capture device 20 can be caught depth information by utilization structure light.In such analysis, can patterning light (that is, be shown as such as the known pattern of lattice or candy strip etc. light) be projected on the scene via IR optical assembly 24 for example.In case run into one or more targets or object surfaces in the scene, as response, this pattern can be out of shape.Can catch such pattern deformation by for example 3D camera 26 and/or RGB camera 28, and can analyze pattern deformation then to judge from capture device 20 to target or the physical distance of the ad-hoc location on the object.

According to another embodiment, capture device 20 can comprise and can check scene so that obtain two or more physically separated cameras of visual stereo data from different angles, can resolve visual stereo data to generate depth information.In another example embodiment, capture device 20 can use cloud data and target number technology to survey user 18 characteristic.

Capture device 20 can also comprise microphone 30.Microphone 30 can comprise the converter or the sensor that can receive sound and convert tones into electric signal.According to a kind of embodiment, microphone 30 can be used in Target Recognition, analysis and tracker 10, reduce the feedback between capture device 20 and computing environment 12.In addition, microphone 30 can be used to received audio signal, the user also can sound signal be provided in case control such as the game application that can carry out by computing environment 12, the application program of non-game application or the like.

In an example embodiment, capture device 20 can also comprise the processor 32 that operation is upward communicated by letter with image camera assembly 22.Processor 32 can comprise the standardization device that can execute instruction, application specific processor, microprocessor or the like; These instructions can comprise and be used to receive depth image, judge whether to comprise suitable target at depth image, and the Target Transformation that this is suitable becomes bone to represent or instruction or any other suitable instruction of object module.

Capture device 20 can also comprise memory assembly 34, and memory assembly 34 can store the instruction that can be carried out by processor 32, by the frame of the image of 3D camera or RGB captured by camera or image or any other appropriate information, image or the like.According to an example embodiment, memory assembly 34 can comprise random-access memory (ram), ROM (read-only memory) (ROM), high-speed cache, flash memory, hard disk or any other suitable storage assembly.As shown in Fig. 2, in one embodiment, memory assembly 34 can be and the assembly that separates of image camera assembly 22 and processor 32 communications.According to another embodiment, memory assembly 34 can be integrated in processor 32 and/or the image camera assembly 22.

As shown in Fig. 2, capture device 20 can be communicated by letter with computing environment 12 via communication link 36.Communication link 36 can be comprise that USB for example connects, live wire connects, wired connection that Ethernet cable connects or the like and/or such as wireless 802.11b, the wireless connections that g, a or n connect etc.According to a kind of embodiment, computing environment 12 can provide to capture device 20 via communication link 36 and can be used to determine when and catch the for example clock of scene.

In addition, capture device 20 can provide the depth information of being caught by for example 3D camera 26 and/or RGB camera 28 and image and can be by the skeleton model of capture device 20 generations via communication link 36 to computing environment 12.Exist and to be used to judge that target that capture device 20 is detected or object are whether corresponding to the various known technologies of people's class targets.Then, can use the bone mapping techniques to confirm all places in the place that bone, joint and the pelvis of user's hand, wrist, elbow, knee, nose, ankle, shoulder contact with spine.Other technologies comprise to be represented image transformation to represent with the grid model that image transformation is become the individual for individual's phantom type.

Computing environment then, can skeleton model be offered computing environment 12, so that can be carried out exercises.Computing environment can also be based on for example confirming to carry out which control the application program that computer environment is carried out from the user's of skeleton model identification posture.For example, as shown, in Fig. 2, computing environment 12 can comprise and is used for confirming when the user carries out the gesture recognizers engine 190 of predetermined gestures.Computing environment 12 can also comprise the focus engine 192 that is used for focusing on from scene interesting areas, like following explanation.Partly or entirely can residing on the capture device 20 and by processor 32 of focus engine 192 carried out.

Fig. 3 A explaination can be used to the example embodiment of computing environment of one or more positions and the action of interpreting user in Target Recognition, analysis and tracker.Such as above computing environment with respect to described computing environment 12 grades of Figure 1A-Fig. 2 can be multimedia console 100, for example game console.Shown in Fig. 3 A, multimedia console 100 has CPU (CPU) 101, and CPU (CPU) 101 has 102,2 grades of high-speed caches 104 of 1 grade of high-speed cache and flash ROM 106.1 grade of high-speed cache 102 and 2 grades of high-speed caches 104 are stored data provisionally and are therefore reduced the quantity of memory access cycle, improve processing speed and handling capacity thus.CPU 101 may be provided with more than one core and thereby has additional 1 grade and 2 grades of high-speed caches 102 and 104.Flash ROM 106 can be stored in the executable code that during the starting stage of bootup process, loads when multimedia console 100 powers on.

The Video processing pipeline that GPU (GPU) 108 and video encoder/video codec (encoder/decoder) 114 are formed at a high speed and high graphics is handled.Via bus data are transported to video encoder/video codec 114 from GPU 108.The Video processing pipeline to A/V (audio/video) port one 40 output datas so that send TV to or other displays.Memory Controller 110 is connected to GPU 108 so that promote the various types of storeies 112 of processor access, such as but not limited to RAM.

Multimedia console 100 comprises the I/O controller 120 that preferably is implemented on the module 118, System Management Controller 122, audio treatment unit 123, network interface controller 124, USB host's controller 126, the 2nd USB host's controller 128 and front panel I/O assembly parts 130.

USB controller

126 and 128 serves as the host who is used for peripheral controllers 142 (1)-142 (2), wireless adapter 148 and external memory devices 146 (for example, flash memory, external CD/DVD ROM driver, removable medium or the like).Network interface 124 and/or wireless adapter 148 provide the access of network (for example, the Internet, home network or the like) and can be to comprise any in the various wired or wireless adapter assembly of Ethernet card, modulator-demodular unit, bluetooth module, cable modem or the like.

System storage 143 is provided so that the application data that loads during being stored in bootup process.Media drive 144 is provided, and it can comprise DVD/CD driver, hard disk drive or other removable media drives or the like.Media drive 144 can be internal or external at multimedia console 100.Can be via media drive 144 access application data so that carry out, play or the like by multimedia console 100.Media drive 144 is connected to I/O controller 120 via the bus that connects (for example, IEEE 1394) etc. such as serial ATA bus or other at a high speed.

System Management Controller 122 provides the various service functions relevant with the availability of guaranteeing multimedia console 100.Audio treatment unit 123 forms with audio codec 132 has the corresponding audio processing pipeline of high fidelity and stereo processing.Between audio treatment unit 123 and audio codec 132, deliver voice data via communication link.The Audio Processing pipeline reproduces for external audio player with audio capability or equipment to A/V port one 40 output datas.

Front panel I/O assembly parts 130 supports to be exposed on power button 150 and the function of ejector button 152 and any LED (light emitting diode) or other indicators of the outside surface of multimedia console 100.System power supply module 136 is to the assembly power supply of multimedia console 100.Circuit in the fan 138 cooling multimedia consoles 100.

CPU 101 in multimedia console 100, GPU 108, Memory Controller 110 and multiple other assemblies interconnect via one or more buses, and these buses comprise any universal serial bus and parallel bus, memory bus, peripheral bus and processor bus or the local bus that uses in the various bus architectures.As an example, such architecture can comprise periphery component interconnection (PCI) bus, PCI-Express bus or the like.

When multimedia console 100 powers on, can application data be loaded into storer 112 and/or the high-speed cache 102,104 and at CPU 101 from system storage 143 and carry out.Application program can be presented on the graphic user interface of the user experience that provides consistent when navigating to available different medium type on the multimedia console 100.In operation, can or play application program and/or other media that is comprised in the media drive 144 from media drive 144 startups, so that additional function is provided to multimedia console 100.

Through simply system being connected to TV or other displays, multimedia console 100 can be used as one-of-a-kind system and operates.In this single cpu mode, multimedia console 100 allows one or more users and system interaction, watches film or music appreciating.Yet, integrated by means of the broadband connection property that provides through network interface 124 or wireless adapter 148, multimedia console 100 can also be operated as the participant in the bigger Web Community.

When multimedia console 100 powered on, the hardware resource of set amount was used for multimedia console operating system by system's reservation.These resources can comprise to be reserved storer (for example, 16MB), CPU and GPU cycle (for example, 5%), networking bandwidth is (for example, 8kbs), or the like.Because when system bootstrap, keep these resources, there is not institute's resources reserved from the angle of application program.

Especially, storer keeps preferably even as big as comprising startup kernel, concurrent system application program and driver.CPU keeps preferably constant, if so that the CPU purposes that is kept is not used by system application, then idle thread will consume any untapped cycle.

Keep about GPU, so that the scheduling code is ready-made coverage diagram with the ejection window, show the lightweight message (for example, ejecting window) that generates by system application through using GPU to interrupt.The amount of the desired storer of coverage diagram depends on the overlay area size, and coverage diagram is preferably with the screen resolution convergent-divergent.When the concurrent system application program is used complete user interface, the preferred resolution that is independent of application program resolution of using.Scaler can be used to be provided with this resolution and change frequency and cause the needs that TV is synchronous again to eliminate.

After multimedia console 100 guiding and system resource were retained, the concurrent system application program was carried out so that systemic-function to be provided.Systemic-function is encapsulated within the group system application program of carrying out in the system resource that is kept described above.Operating system nucleus sign is system application thread but not the thread of game application thread.System application preferably is scheduled as in the predetermined moment and at interval operation on CPU 101, so that to application program consistent system resource view is provided.Scheduling is to interrupt for the high-speed cache that is minimized in the game application of moving on the control desk.

When the concurrent system application program requires audio frequency, because time sensitivity is dispatched Audio Processing to game application asynchronously.When system application is movable, multimedia console application manager (being described below) control game application audible level (for example, quiet, decay).

Game application and system application are shared input equipment (for example, controller 142 (1) and 142 (2)).Input equipment is not institute's resources reserved, but should between system application and game application, switch, so that each all will have the focus of equipment.The knowledge that application manager preferably need not game application is just controlled the switching of inlet flow, and driver is kept the status information of switching about

focus.Camera

26,28 and capture device 20 can define the additional input equipment that is used for control desk 100.

Fig. 3 B explaination can be another example embodiment of the computing environment 220 of the computing environment 12 of in Target Recognition, analysis and tracker, explaining one or more positions and action of being used to shown in Figure 1A-Fig. 2.Computingasystem environment 220 only is an example of suitable computing environment, and is not intended to hint to the usable range of disclosed theme or any restriction of function at present.Computing environment 220 should not be interpreted as yet have with the assembly of in exemplary operations environment 220, being explained in relevant any dependence or the requirement of any one or combination.In certain embodiments, the various computing elements of narrating can comprise the circuit of the concrete aspect that is configured to the instantiation present disclosure.For example, the terms circuit that is used to present disclosure can comprise the specialized hardware components that is configured to carried out by firmware or switch function.In other example embodiment, terms circuit can comprise by the General Porcess Unit of the software instruction configuration that realizes being operable as the logic of carrying out function, storer or the like.Circuit comprises that in the example embodiment of combination of hardware and software, the implementer can write the source code of realizing logic therein, and source code can be compiled into the machine readable code that can be handled by General Porcess Unit.Because those of skill in the art can understand; The state of this area has evolved to and wherein between the combination of hardware, software or hardware/software, has had the seldom point of difference, and selecting hardware still is that to implement concrete function be the design alternative of leaving the implementer for to software.More specifically, those of skill in the art can understand that software process can be transformed into the hardware configuration of equivalence, and hardware configuration itself can be transformed into the software process of equivalence.Thereby, select hardware to realize that still the software realization is a design alternative and leaves the implementer for.

In Fig. 3 B, computing environment 220 comprises computing machine 241, and computing machine 241 generally includes various computer-readable mediums.Computer-readable medium can be can be by any available medium of computing machine 241 visit, and comprises volatibility and non-volatile media, removable and removable medium not.System storage 222 comprises with such as ROM 223 and the volatibility of RAM 260 grades and/or the computer-readable storage medium of nonvolatile memory form.Comprising the basic input/output 224 (BIOS) that transmits the basic routine of information between each element that for example between the starting period, helps in computing machine 241 is stored among the ROM 223 usually.RAM 260 comprises data and/or program module addressable immediately and/or that operated by processing unit 259 at present usually.And unrestricted, Fig. 3 B explains operating system 225, application program 226, other program modules 227 and routine data 228 as an example.Fig. 3 B also comprises and is used for handling and the graphics processor unit of storing with related VRAM 230 (GPU) 229 with high graphics at a high speed.GPU 229 can be connected to system bus 221 through graphic interface 231.

Computing machine 241 can comprise that also other are removable/can not move, volatile/nonvolatile computer storage media.Only as an example, Fig. 3 B explaination is read or to its hard disk drive that writes 238, read or read or to its CD drive that writes 240 to its disc driver that writes 239 and from the movably non-volatile CD 253 such as CD ROM or other optical mediums etc. from non-volatile magnetic disk 254 movably from immovable non-volatile magnetic medium.Other that can be used for the exemplary operations environment are removable/and not removable, volatile/nonvolatile computer storage media includes but not limited to tape cassete, flash memory card, digital versatile dish, digital recording band, solid-state RAM, solid-state ROM or the like.Hard disk drive 238 is connected to system bus 221 through the not removable memory interface such as interface 234 grades usually, and disc driver 239 and CD drive 240 are connected to system bus 221 by the removable memory interface such as interface 235 etc. usually.

Each driver of discussing in the above and in Fig. 3 B, explaining with they related computer-readable storage medium be the storage that computing machine 241 provides computer-readable instruction, data structure, program module and other data.In Fig. 3 B, for example, hard disk drive 238 is illustrated as storage operating system 258, application program 257, other program modules 256 and routine data 255.Notice that these assemblies can be identical or different with operating system 225, application program 226, other program modules 227 and routine data 228.Operating system 258, application program 257, other program modules 256 are given different numerals here with routine data 255, and they are different copies at least so that explain.The user can be through ordering such as the input equipment of keyboard 251 and the pointing apparatus 252 that is commonly called mouse, tracking ball or Trackpad etc. and information is input to computing machine 241.Other input equipment (not shown) can comprise microphone, operating rod, game mat, satellite dish, scanner or the like.These usually are connected to processing unit 259 through the user's input interface 236 that is coupled to system bus with other input equipments, but can be connected with bus structure by other interfaces such as parallel port, game port or USB (USB) etc.Camera 26,28 and capture device 20 can be used to define the additional input equipment of control desk 100.The display device of monitor 242 or other types also is connected to system bus 221 via the interface such as video interface 232 etc.Except monitor, computing machine also can comprise can be through other peripheral output devices such as loudspeaker 244 and printer 243 grades of output peripheral interface 233 connections.

The logic that computing machine 241 can use such as one or more remote computers of remote computer 246 grades connects in networked environment, to operate.Remote computer 246 can be personal computer, server, router, network PC, peer device or other common network node; And generally include many or whole in the above element of describing with respect to computing machine 241, although in Fig. 3 B, only explained memory storage device 247.The logic of in Fig. 3 B, narrating connects and comprises Local Area Network 245 and wide area network (WAN) 249, but also can comprise other networks.Such networked environment is in the computer network of office, enterprise-wide, Intranet and the Internet, to be general.

When being used to the LAN networked environment, computing machine 241 is connected to LAN 245 through network interface or adapter 237.When being used to the WAN networked environment, computing machine 241 generally includes modulator-demodular unit 250 or is used for setting up on such as the WAN 249 of the Internet etc. other devices of communication.Modulator-demodular unit 250 that can be internal or external can be connected to system bus 221 via user's input interface 236 or other suitable mechanism.In networked environment, program module or its part narrated with respect to computing machine 241 can be stored in the remote memory storage device.And unrestricted, Fig. 3 B is illustrated as remote application 248 and resides on the memory devices 247 as an example.Should understand that it is exemplary that shown network connects, and can use other devices of between computing machine, setting up communication link.

The user's that Fig. 4 narration can generate from capture device 20 example bone mapping.In this embodiment, identify various joints and bone: top 326 and the end 328 and the waist 330 of each hand 302, each forearm 304, each elbow 306, each biceps 308, each shoulder 310, each hip 312, each thigh 314, each knee 316, each front foot 318, each pin 320,322, trunk 324, spine.Following the tracks of the more occasion of multiple spot, can identify additional characteristic, for example the bone and the joint of finger or toe, the perhaps personal feature of face, for example nose and eye.

As indicated in the background parts, hope to need not the stand-by period is incorporated into the just object of some from the scene acquisition detailed image data that appear of image sometimes.According to the each side of present technique, can in scene, identify interesting areas, and can focus on those zones so that therefrom obtain the more images details.

Fig. 5 illustrates the image of being surveyed by the sensor of the image camera assembly 22 of capture device 20.It can for example be the user who goes up user interface interaction with screen shown in Figure 1B.This image is divided into the dot matrix (wherein some are in the accompanying drawings by numbering) of the pixel 350 of horizontal line and vertical row.At image camera assembly 22 are occasions of depth cameras 26; Each pixel in the dot matrix is caught x, y and the z position of the object in the scene; Wherein the z axle is defined as from camera lens directly to come out, and x axle and y axle are respectively horizontal-shift and the vertical shifts of leaving the z axle.At camera assembly 22 are occasions of RGB camera 28, and each pixel in the dot matrix is caught the rgb value of the object in the scene.The RGB camera is registered to depth cameras, makes by mutually synchronization mutually on camera 24 and each frame time that 26 are caught.The scene of Fig. 5 illustrates the user 18 that caught by pixel 350 and such as the fixed objects 23 of chair and plant etc.Image detail in the scene is crossed over all pixels 350 and is distributed equably.

With reference now to Fig. 6, the details of the embodiment of the present technique of explaining to the explaination of Figure 10 and Figure 11 to the process flow diagram of Figure 15.Figure 11 illustrate be referred to herein as from the active focus on the interesting areas of scene (below being different from respect to Figure 13 explain by oving foci) first embodiment.In Figure 11, can start-up system 10 in the step 400.After this, in step 402, system can catch and appear from the many frame n in the view data of depth cameras 26 and RGB camera 28 (explaining below).Seizure is from the data of both all pixels 350 of depth cameras 26 and RGB camera 28 and send it to computing environment for processing.In each embodiment, processor can be analyzed data and confirm user's existence from the identification of bone pattern.System also can be presented on image on the display based on the view data of being surveyed in the step 402.In the example of Fig. 6, view data is used to survey and is used to control computing environment 12 or the User Interface of operation on computing environment 12 mutual.

In each embodiment, focus engine 192 is analyzed the n frame data in step 404, and is identified at the one or more interesting areas in this scene.In various embodiment, can define interesting areas in a different manner.In one embodiment, interesting areas can be in this scene, to detect such as for example one or more zones of moving of mobile grade of the user 18 in this scene.Replace be whole user's interesting areas, interesting areas can be limited to user's specific body part in each embodiment, for example their head, hand, pin or wherein to interested other body part of detailed view data of acquisition body part.

In a further embodiment, interesting areas can be by the application program definition of operation on computing environment 12.For example, be the occasion of sign language application program in application program, interesting areas can be around user's the left hand and/or the right hand, no matter whether those hands are moving.In a further embodiment, some characteristic of scene can always be counted as interesting areas, and does not consider application program or moving of being surveyed.For example, a kind of embodiment can always regard user's head as interesting areas, so that can survey the facial expression and/or the details of speech.

In each embodiment, the stationary body in the scene such as chair and plant 23 etc. and any institute seizure wall, floor, ceiling, window, door and wall fitting thing, will not be considered interesting areas.The zone that is not included in the interesting areas of scene is referred to herein as static region.Object except that the user also can be included in the interesting areas from scene.For example, the user hold or can be interesting areas (or being included in the interesting areas) with other mode by the object that the user moves in real world space everywhere.In addition; In each embodiment; The stationary body that the user does not move also can be included in the interesting areas, and wherein this object relates to the application program of operation on computing environment 12 with certain mode, and has details or the information that need be used with programs by camera assembly 22 perception.

As indicated, in step 402, the quantity n of frame can and discern the quantity of the needed frame of interesting areas in the scene for focus engine 192 analysis of image data in step 404.Focus engine 192 can be carried out this action according to various methods.In a kind of embodiment described above, computing environment receives view data and can be from user's skeletal structure identification he or she from capture device 20, and his position in scene.In case the position of computing environment 12 identification users and Ta, focus engine 192 can be the interesting areas in this scene with this location definition.In this type of embodiment, focus engine 192 can be made this judgement from one or two Frame.In such example, in step 402, n can equal one or two Frame.

Can judge except one or more body part from the n frame data in computing environment; The people is in the occasion of sitting down or not moving with other mode, and system judges that which body part is moving and focus engine 192 is defined as the interesting areas in the scene with those one or more body part.Computing environment can be made this judgement in every way, for example comprises the view data of first frame and the view data of second frame are compared.In this type of embodiment, in step 402, n can equal the data of two or three frames.Should understand that n can be focus engine 192 makes a distinction mobile object (or body part) and non-moving object (or body part) needed frame in scene a any amount.

In step 404, the interesting areas in the focus engine 192 sign scenes.In step 406, the focus engine can be stored in the static region of scene in impact damper or other storeies, for using like following explanation.

In step 410, the focus engine focuses on defined interesting areas.Accumulate in and mean the more images details that provides about interesting areas here.In one embodiment, this can mean the amplification interesting areas, simultaneously static region is got rid of outside view.Focus engine 192 can focus on interesting areas so that the height in the visual field of the complete filling camera assembly 22 of acquisition interesting areas and/or the maximum zoom of width.In a further embodiment, the focus engine can be enlarged into interesting areas and be less than possible maximum zoom.

Fig. 6 is the image of the scene identical with Fig. 5, but in Fig. 6, the focus engine has caused that camera assembly 22 amplifies interesting areas 360, and interesting areas 360 is users 18 in Fig. 5 and Fig. 6.Go out as shown in Figure 6,, compare the more pixels 350 of use with Fig. 5 and represent user 18, thereby more image details of user 18 are provided in case amplify.Fig. 6 illustrates an embodiment, and wherein whole pixel-matrix is used to catch interesting areas 360.In a further embodiment, only the part of the pixel 350 around interesting areas 360 is used to catch interesting areas.Focus engine 192 can judge which pixel 350 is used to catch interesting areas in Fig. 6 and 6A, and the focus engine can be ignored the data from the pixel that is not used to catch interesting areas.This can further reduce processing speed and the stand-by period when being avoided presenting frame.

In each embodiment, depth cameras 26 is as one man operated with RGB camera 28 so that amplify identical object to identical degree.Like following explanation, in a further embodiment, they do not need zoom together.Fig. 7 is similar to Fig. 6, but interesting areas is user's a head.In the embodiment of Fig. 7; Be used to view data so that catch user's face and head than the pixel of the much bigger quantity of the panoramic view of Fig. 5, so that user's facial expression is provided and user's lip and the much more details how tongue moves when the user speaks.

Camera assembly 22 zooms described above can be optics (machinery) zooms of camera lens, and perhaps it can be the digital zoom of wherein accomplishing with software.The mechanical zoom system and the digital zoom system of camera are known, and are operating as and change focal length (can be surface go up or in fact change focal length) so that increase the size of the image in the visual field of camera lens.For example be the United States Patent (USP) 7th of the title of issue on January 13rd, 2009 for " Combined Optical And Digital Zoom (optical zoom of combination and digital zoom) "; 477; Disclose the example of digital zoom system in No. 297, this patent is herein incorporated with its integral body by reference.

In step 412, camera assembly 22 can be caught the next frame view data.View data is focused in interesting areas, and for example Fig. 6 and shown in Figure 7 goes out.Can be from the view data of step 412, catching static region in major part or the Ignore All view.

Replace the zoom or in addition in one or two camera assembly 22 described above, in step 416, on view data, carry out various known algorithms so that strengthen the view data of interesting areas by focus engine 192.In using each embodiment of these algorithm for image enhancement, can be to from each algorithm of execution on the view data of the interesting areas of depth cameras and RGB camera.Alternatively, first algorithm for image enhancement can be carried out the depth cameras data, and the second different images enhancement algorithms can be carried out the RGB camera data.In a further embodiment; One in depth cameras and the RGB camera can be optical zoom or digital zoom as stated; Strengthen from another the view data in depth cameras and the RGB camera with algorithm for image enhancement simultaneously, and the result is mated the data with the line focus that interesting areas is provided.Can omit step 416 (and therefore shown in broken lines in Figure 11) among each embodiment of present technique.

Fig. 8 illustrates its scene and comprises that two interesting areas are the embodiment of user and user's hand.For example use the digital zoom technology and use algorithm for image enhancement, focus engine 192 can focus on this two zones independently possibly.The image of the user's on the left side line focus provides and compares more user's details with panoramic view, and the image of the line focus on the right provides and compares with the view of the line focus on the panoramic view or the left side even the details of more user's hand.Should be understood that in a further embodiment, in the given moment, can be from the view of single scene generation more than two activity.

The view data of the line focus that obtains from interesting areas can be used to various purposes.In step 418, the view data of line focus can be used to controlling application program or operation system function.Alternatively or in addition, the view data of line focus can be used for gesture recognition by the following gesture recognition engine of explaining 190.Especially, the view data of line focus allows the meticulousr and trickleer posture of identification, and this requires how to carry out about the user high degree of detail of posture.

For example, in the gesture application program, the nuance of hand position can mean different things and corresponding to different predetermined gestures.In this type of embodiment, interesting areas can be user's a hand, and the view data of line focus provides the level of detail that allows gesture recognition engine 190 to distinguish different gestures.Further in the example, application program can be explained facial expression, and carries out different actions based on the different facial expressions of being surveyed.In this type of embodiment, interesting areas can be user's head or a face, and the view data of line focus provides the level of detail that allows gesture recognition engine 190 to distinguish different facial expressions.

The Another Application program can seek to realize or strengthen speech recognition through analyzing and resolve mouth and tongue moving when formation word and the sound.In this type of embodiment, interesting areas can be user's head or face or user's a mouth specifically, and the view data of line focus provides the level of detail that allows gesture recognition engine 190 to distinguish the different mouth/tongue position when forming word with sound.Purposes above these of view data of line focus only is provided as an example.Should be understood that can the user moves by being used for wherein according to the view data of the line focus of present technique (trickle or with other mode) is used to any other purpose of gesture recognition or application program/OS control.

In step 424, except gesture recognition and application program/OS control, the view data of line focus can be used to appear image.In many cases, the view data of line focus will be used to there to be kind mode of (the monkey see monkey do) that imitate on display 14, to make the animation of user's avatar 19.The accurate reproduction that the view data of line focus can especially be used to provide the user in the interesting areas that the view data by line focus covers to move.

If need be from any element of the static region in the scene in rendering step, then in the step 422 before rendering step 424, computing environment can be from those elements of memory search.In step 406, stationary region image data is stored in the storer as stated.Present technique is utilized the fact that can not change and not need each frame all to form images such as some objects of for example chair and plant 19 etc. in the scene.Thereby another feature of present technique be not to each frame resampling and the still image of handling scene to save the processing time.

Not needing to catch again static region allows focus engine 192 to focus on interesting areas particularly.Yet, the situation that the zone of possible occurrence scene outside interesting areas changes frequently.For example, other users can come in or withdraw from scene.Therefore, focus engine 192 can periodically retract so that obtain the current images data of whole scene.In step 426, whether the inspection of focus engine it focused on the frame that interesting areas reaches a certain quantity m.If not, then focus engine 192 can turn back to step 412 so that obtain the next frame view data from the capture device 20 that focuses on interesting areas.

On the other hand, if focus engine 192 has focused on interesting areas m frame arranged, then the focus engine turns back to step 402 so that obtain the live image data of whole scene.Can select the quantity m of frame based on different competition factors.The value of m is big more, few more time of waste just in sampling and on handling from the view data of the static region of scene.Yet the value of m is big more, and another object such as another user etc. that has got into scene equipment 20 that just might not be captured is more caught.In each embodiment, m can be between 2 frames and 300 or more frame.

For bigger m value,, then need new user be presented on the display with existing user if when turning back to step 402, just in scene, find new user.In this instance, new user's avatar can appear on the display simply.Alternatively, new user's avatar can with the scene integrator, perhaps computing environment 12 can be depicted as new incarnation from one moving by side to the scene (this mobile will be not based on new user's actual image data).

In a further embodiment, system 10 can adopt two capture devices 20.First capture device is operating as and focuses on interesting areas as stated.Second capture device 20 can keep aiming at whole scene.In such embodiment, the focus engine need not turn back to step 402 at every m frame, but can keep focusing on interesting areas, detects the moment that new object gets into scene up to second capture device 20.If second capture device detects the change in the scene, then first capture device can turn back to step 402 so that catch the view data from whole scene.Alternatively, can use from second capture device from the whole scene view data, and with itself and view data combination from the line focus of first capture device.As long as the position of two capture devices is known each other, use known transformation matrix that two views from different capture devices are resolved to the identical visual angle view of first capture device (for example from).

Among the embodiment of two different capture devices of superincumbent use, from presenting purpose, possibly hope can the element from the view of a capture device be sewn onto the view of other capture devices.Such stitching can be accomplished with level and smooth and seamless mode; For example open in following patent: the title of announcing May 17 in 2007 is No. the 2007/0110338th, the U.S. Patent Publication of " Navigating Images Using Image Based Geometric Alignment and Object Based Controls (use geometric alignment and object-based control based on image to come navigation picture ", and this announcement is used to the Photosynth that Microsoft releases ^TMThe technology of image recognition software, and this announcement is herein incorporated with its integral body by reference.

In described each embodiment up to the present, picture element density has remained constant, and the details that is increased obtains through amplifying interesting areas.In a further embodiment, the picture element density of the pixel of the interesting areas of seizure in scene can increase with respect to the static region around the image.Explain the example of such embodiment below with reference to the explaination of the process flow diagram of Figure 12 and Fig. 9.In step 430, start-up system 10, and in step 434, capture device 20 obtains the n frame image data that appears as stated.In step 438, focus engine 192 identifies interesting areas in the scene as stated.As indicated, such zone can be the zone of wherein detecting in the scene of moving.

In case in step 438, detect one or more interesting areas, the picture element density of catching the pixel of the one or more interesting areas in the scene increases with respect to the static region around the image.This can accomplish with many modes.In one embodiment, camera itself can have in a zone ability that increases picture element density with respect to another zone selectively.In this type of embodiment; In step 434; Camera (degree of depth and/or RGB) will be caught view data; Focus engine 192 will identify interesting areas, and the focus engine is relayed back this information and is increased in the picture element density around one or more interesting areas that identify for camera for camera then.Fig. 9 illustrate comprise aforesaid pixel 350 and around the interesting areas second group more high density pixel 352 at interior image.

In a further embodiment; Pixelation from camera keeps constant, but other Processing Algorithm in focus engine 192 or capture device 20 or the computing environment 12 can be in case receive with regard to the view data of processing in one or more interesting areas so that increase the picture element density in those zones.The sub-pixel technology can be used to the pixel in the interesting areas is divided into smaller units.In a further embodiment, can obtain all images data with high relatively picture element density.In case the acquisition view data, can be for example through combination of pixels (pixel binning) technology handle image static region view data in case the combination of pixels that will close on together.In such example, can in the frame per second (for example 30Hz) of computing environment, handle scene static region do not have the stand-by period through the more high density pixel of the interesting areas of pixel and the scene of combination.

In a further embodiment, can use two different capture devices 20 as stated; Catch view data with high relatively resolution for one, and second is caught data with second low resolution.In case obtain view data from two capture devices, just can be used as the view data of interesting areas from the more high-resolution data of first camera.Can be used to the view data of the static region of scene from the data of the low resolution of second camera.As indicated above, can be transformed into common view (the for example view of first camera) and appeared from the different views of each camera.If necessary, two views can be sewn to together from presenting purpose, such as No. the 2007/0110338th, the disclosed in the above U.S. Patent Publication that merges for example.

In step 442, can obtain to comprise the next frame data of the more highdensity view data of interesting areas then.In step 444, can use this view data to discern user's posture as stated, be controlled at the each side of the application program of moving on the computing environment and/or the each side of the operating system on the control computing environment 12.In step 446,, also can use the view data of the more highdensity view data that comprises interesting areas to be used for appearing image like top explanation.

Among this embodiment that relative picture element density between interesting areas and static region changes therein, described in the embodiment of zoom, do not need periodically to turn back to the full view of scene as top.This is because the panorama that capture device 20 is caught in all frames.Thereby focus engine 192 can detect when object gets into or leaves.In step 448, focus engine 192 gets into or leaves any object of scene from new Frame inspection.If do not detect the object that gets into or leave scene, then interesting areas remains unchanged.This system turns back to step 440, increase (or keeping said increase) around interesting areas picture element density and catch the next frame data.On the other hand, if detect the object that gets into or leave scene, then the focus engine turns back to step 434, makes to identify interesting areas once more.

In the example described above, interesting areas is initiatively to identify, and interesting areas can change with new frame data.In the further embodiment of present technique, interesting areas can be set passively.That is to say, one or more predefined focus areas can be set in scene, and be that the interior view data in those zones provide more focus.These zones do not change with moving to the user or other objects that perhaps leave scene in the scene.Explain such embodiment with reference to the process flow diagram of Figure 13 and the explaination of Figure 10 below.

In the embodiment of Figure 10 and Figure 13, one or more predefined focus areas can be positioned at the optional position of scene.Figure 10 illustrates along the y axle and along the focus area 356 of the intermediate altitude of whole x axle.Although in Figure 10, do not indicate, focus area can be along whole z axle expansion.This focus area 356 is as an example, and is set up as any 3D zone definitions in scene is become focus area.In each embodiment, to given capture device 20, this zone can be fix with lasting, and do not consider on computing environment 12 application program of operation.

In a further embodiment, replacement is lasting situation, and the application program of operation can be provided with focus area 356 based on this application program on computing environment 12.In this type of embodiment, in the duration of application program operation, focus area will be fixed.For example, if application program is the Association football recreation, then focus area can so that catch detailed the moving of athletic pin, also can be at the top of y axle, so that catch detailed the moving of athletic head along the bottom of y axle.For the lip-read application program, focus area can be along the top of y axle with along the part of x axle, so that catch the more details of user's face.Shown in Figure 10, focus area 356 also can be set at along the center section of y axle, so that in the more details of catching user's hand with the custom menu of application program or operating system or other control function when mutual.Those of skill in the art should understand that application program or the operation of carrying out depended in various other positions of focus area 356.

In the above embodiments, in the duration of application program operation, focus area 356 can be positioned at single position.In a further embodiment, can imagine that focus area 356 moves under the control of application program.For example, the cycle very first time of application program maybe the detailed view data relevant with user's pin, and second time cycle of application program maybe the detailed view data relevant with user's hand.In such example, when application program was transformed into for second time cycle from the cycle very first time, application program can the moving focal point zone.

Figure 10 and Figure 13 by oving foci embodiment in, the zone of the scene that the focus in the focus area can be outer than focus area is big.Thereby user in focus area 356 or subject image data will be bigger than the view data outside focus area.Can be through the above details that increases the view data in the focus area with respect to the described any method of interesting areas that initiatively identifies.Such method includes but not limited to the picture element density in optical zoom and digital zoom, enhancement algorithms, the increase focus area and reduces the picture element density (for example passing through combination of pixels) in the outer zone of focus area.Also can use the capture device 20 of catching two separation of view data to define the focus area in the scene with different resolution.

In addition, be lasting embodiment for the focus area of wherein given capture device, lens that can be through the camera in the shaping capture device or control lens peculiarity with other mode and create focus area.For example, lens can be formed creates the flake effect, and wherein the object at the center of image is bigger than the object of side.Such effect cause in image image in the heart object have than the object of side and more many image of pixel and more details.Can change camera lens through additive method, so that focus area is moved to any desired zone in scene.

Figure 13 illustrates the operation of the embodiment of passive focus area.In step 450, can start-up system 10.In step 454, can come the zone outer to increase the focus in the focus area through any method described above with respect to focus area.Result as lens peculiarity creates among the embodiment that increases focus area automatically therein, and step 454 will be skipped (this is because the focusing that is increased will be intrinsic, and does not need initiatively to carry out separation steps) concerning this system.Thereby in Figure 13, be shown in broken lines step 454.

After this, by the operation of oving foci embodiment as carrying out the normal picture capture operation.In step 465, catch view data.In step 460, this view data can be used to gesture recognition, application program control and/or operating system control.As stated, the details through increasing of given view data from focus area can be used to distinguish trickle moving and posture from the data in this zone.In step 462, comprise that the view data of the more highdensity view data of interesting areas also can be used to appear image, and then, to subsequently Frame repeating step 456 to step 462.

Up to the present among described each embodiment, in camera or through handling, accomplish the details that is increased from scene from the view data of camera.Yet the illumination of scene also can have remarkable result to deriving how many details from irradiated zone.Usually, the NUI system adopts the illuminator of the All Ranges of attempting the uniform irradiation scene.Yet like top explanation, it is useful obtaining more details from some zones of scene, and to obtain details from other zones be more unessential.Thereby IR light source 24 can be controlled to be some interesting areas that better is radiated in this scene on one's own initiative or passively.Below refer to figs. 14 and 15 explaining initiatively illumination and passive illumination embodiment.

At first referring to Figure 14, to step 478, start-up system 10, obtain the interesting areas in n frame image data and the sign scene, as stated in step 470.In step 480, IR light source 24 can focus on this zone.This can accomplish with many modes.In one embodiment, can come focused light source by mechanical hook-up, such as for example making emission light narrow down and light source 24 is supported on the support such as 2 shaft universal-joints etc., this 2 shaft universal-joint allows in the x-y plane of scene the controlled of light source to make an accurate selection of around axle.Other mechanical systems are known.

In a further embodiment, can filter light, so that stress one or more wavelength with respect to other wavelength from source 24.Character based on the object in the interesting areas is selected these wavelength.Especially, which wavelength will the object in interesting areas to have maximum reflectivity based on and select one or more wavelength.

In arbitrary embodiment, the better illumination of the interesting areas through focused light source will increase information and the details that receives from the pixel of the light of interesting areas reflection.In addition, any minimizing from the light of the static region of scene reflection will reduce the optical noise in the pixel of interesting areas.

The embodiment of the light of line focus can operate with any photosystem that is used for IR light source 24 described above, comprises for example pulsed light, phase measurement, transit time and structured light.Pulsed frequency, light pattern and the phase place of the light that possibly hope to send from IR light source 24 for the one or more adjustment in these photosystems one, this depends on where light is directed in the scene.

In step 482, capture the next frame data.As stated, in step 484, this view data can be used to gesture recognition, application program control and/or operating system control.Like top explanation, in step 486, view data also can be used to appear image.In step 488,192 inspections of focus engine get into or leave any object of scene from new Frame.If do not detect the object that gets into or leave scene, then interesting areas remains unchanged.This system turns back to step 482 and uses the light of identical line focus to catch the next frame data.On the other hand, if detect the object that gets into or leave scene, then focus engine 192 turns back to step 474, so that can identify interesting areas once more.

Figure 15 illustrates the embodiment that wherein can be directed to passive focus area from the light of light source.In such embodiment, light source keep to be aimed at the specific region, and does not consider from move to scene or leave scene.Can start-up system 10 in step 490, and light source is focused in specific optional zone in step 492.By means of the light source that is focused in the specific region, in step 494, obtain the next frame data.Then, in step 496, can be used to discern user's posture, be controlled at the each side of the application program of moving on the computing environment and/or the each side of the operating system on the control computing environment 12 from the view data of image.In step 498, can present image, and then, to Frame repeating step 494 to 498 subsequently.

Focus on and passive focusing to the active on the described image detail of Figure 13 with respect to Figure 11 above replacing, perhaps combine with it, the active with respect to the described light source of Figure 14 to 15 above can using focuses on and passive focusing.

As stated, being provided at more details in the view data can promote posture preferably to survey and the detection of meticulousr, trickleer posture.Figure 15 illustrates the block diagram of gesture recognition engine 190, and Figure 16 illustrates the block diagram of operation of the gesture recognition engine 190 of Figure 15.In step 550, gesture recognition engine 190 receives attitude information 500.Attitude information can comprise and the user's who in view data, is surveyed the body part and the position and/or the relevant various parameters of motion in joint.

In step 554, whether the attitude information 500 that gesture recognition engine 190 analysis is received matees and is stored in any predefined regular 542 in the gesture library 540 so that understand this attitude information.When ad-hoc location and/or action that the rule 542 of being stored is described by attitude information 500 indications should be interpreted as predefined posture.In each embodiment, each posture can have different, unique rule or rule set 542.Each rule can have the one or more many parameters (change of joint position vector, maximum/minimum position, position or the like) in the body part shown in Fig. 4.For each body part 302 to 330 shown in each parameter and Fig. 4, the rule of being stored can define the parameter and the incoherent indication of judging by this rule covering of posture of single value, codomain, maximal value, minimum value or this body part.Each rule can be created by the host or the user oneself of recreation author, gaming platform.

Gesture recognition engine 190 can be exported posture and the level of confidence through sign, and level of confidence is corresponding to position/mobile possibility corresponding to this posture of user.Especially, except the desired parameter of definition posture, rule can also be included in attitude information 500 and be interpreted as posture desired threshold values level of confidence before.Some postures can have than system command or the more influence power of game command, and thereby before attitude is explained this posture, require higher level of confidence.The accumulation level of confidence that relatively obtains whether indicating posture of attitude information and the parameter of the rule of being stored about this attitude information.

In case confirmed whether satisfy the regular level of confidence of given posture about given attitude or motion, then, in step 556, gesture recognition engine 190 just judges whether level of confidence is higher than the predetermined threshold values of the rule of being considered.The threshold values level of confidence can be stored with the rule association of being considered.If level of confidence is lower than threshold values, does not then detect posture (step 560) and do not take action.On the other hand, if level of confidence is higher than threshold values, then the posture rule of being considered is satisfied in the motion of judges, and gesture recognition engine 190 returns the posture that is identified in step 564.

The more images details that system provided of given present technique; Gesture library 540 can comprise than posture definition trickleer and meticulousr in the conventional system; And the attitude information 500 that is received can comprise more detailed information, makes this system can determine whether to have carried out these trickleer and meticulousr postures.

Be in the foregoing detailed description that explaination and purpose of description have presented system of the present invention.It is not intended to is detailed, and also not being intended to system constraint of the present invention is disclosed precise forms.According to top instruction, many modifications and change are possible.Selecting described embodiment is principle and practical application thereof in order to explain system of the present invention best, allows in this area other to come to utilize best system of the present invention with the various modifications that are suitable for desired special-purpose in various embodiments thus.The scope of stipulating system of the present invention is defined by accompanying claims.

Claims

1. a system (10) that comprises the computing environment (12) that is coupled to the capture device (20) that is used for catching motion; The method of the image detail in a kind of one or more interesting areas (360) that are increased in the scene of being caught by capture device (20) comprises:

A) receive (step 402) information from said scene;

B) be identified at said one or more interesting areas (360) of (step 404) in the said scene;

C) obtain (step 410) about the said one or more interesting areas (360) in said scene, with respect to the regional more images details outside said one or more interesting areas in said scene; And

D) at least periodically keep watch on (step 402) information outside said one or more interesting areas (360) in said scene, so that judge whether to define again said one or more interesting areas (360).

2. the method for claim 1, the said step b) that is identified at the said one or more interesting areas in the said scene comprises the step that is identified at the moving area in the said scene.

3. the method for claim 1, said acquisition about the step c) of the more images details of the said one or more interesting areas in said scene comprise carry out mechanical zoom or digital zoom one of them so that focus on the step of at least one interesting areas in said one or more interesting areas.

4. the method for claim 1, said acquisition comprises through view data carries out image enhancement algorithms being strengthened the step of view data about the step c) of the more images details of the said one or more interesting areas in said scene.

5. the method for claim 1, said acquisition comprise the step that is increased in the picture element density around said one or more interesting areas about the step c) of the more images details of the said one or more interesting areas in said scene.

6. the method for claim 1, said acquisition comprises that about the step c) of the more images details of the said one or more interesting areas in said scene the applied light source of change is so that with the step of said light-resource fousing at least one interesting areas of said one or more interesting areas.

7. the method for claim 1, said acquisition comprises the combination of pixels step together with the view data in the zone outside said one or more interesting areas about the step c) of the more images details of the said one or more interesting areas in said scene.

8. the method for claim 1, the said step b) that is identified at the said one or more interesting areas in the said scene comprises the step that is identified at the interested 3D region in the said scene.

9. a system (10) that comprises the computing environment (12) that is coupled to the capture device (20) that is used for catching motion; The method of the image detail in a kind of one or more interesting areas (360) that are increased in the scene of being caught by capture device (20) comprises::

A) be defined in the interior focus area (356) of said scene, said focus area (356) is defined as corresponding with the one or more desired interesting areas (360) in said scene; And

B) obtain about the focus area in said scene (356), with respect to the regional more images details outside said one or more interesting areas (360) in said scene.

10. method as claimed in claim 9; The said step a) that is defined in the focus area in the said scene comprises the step of the focus area that definition application is specific, and wherein said application program is defined as said focus area corresponding with desired one or more interesting areas in said application program.

11. method as claimed in claim 9, the said step a) that is defined in the focus area in the said scene is included in the step of the permanent focus area of definition in the said scene.

12. method as claimed in claim 9, said acquisition comprises the step through the pixel of the incompatible expansion of pixel groups outside said focus area about the step b) of the more images details of said focus area.

13. at a games system (10) that comprises the computing environment (12) that is coupled to the capture device (20) that is used for catching motion; The method of the image detail in a kind of one or more interesting areas (360) that are increased in the scene of being caught by capture device (20) comprises:

A) receive (step 402) information from said scene;

B) sign (step 404) one or more users (360) in said scene;

C) obtain (step 410) about at least one body part of the one or more users in said scene, with respect to the regional more images details except that said one or more users in said scene;

D) the said more images details about the said one or more users' in said scene said at least body part of using that (418) obtained in said step c) identifies the posture of being carried out by said one or more users; And

E) at least periodically keep watch on (step 402) information said one or more users (360) outside in said scene, so as to judge whether in said step c), obtain about its more images details one or more users said group of interpolation or deduct a user.

14. method as claimed in claim 13, said acquisition comprises the step of acquisition about the more images details of two different objects about the step c) of the more images details of at least one body part of said one or more users.

15. comprising, method as claimed in claim 13, the step d) that the said more images details of said use identifies posture use said more images details to identify the posture of carrying out by said user's sufficient, hand, face or mouth.