JP2015510169A - Feature improvement by contrast improvement and optical imaging for object detection - Google Patents

Abstract

Improved contrast between the background surface visible in the image and the object is given by the use of controlled illumination directed at the object. To take advantage of the attenuation of light intensity with distance, a light source (or multiple light sources), such as an infrared light source, can be used to illuminate an object while the camera (s) capture an image. Can be placed near. The captured image can be analyzed to distinguish object pixels from background pixels. [Selection] FIG.

Description

  This application claims and benefits from US Serial No. 61 / 724,068 filed Nov. 8, 2012, the entire disclosure of which is incorporated herein by reference. . In addition, this application claims priority to US patent application Ser. Nos. 13/414485 (filed Mar. 7, 2012) and 13/724357 (filed Dec. 21, 2012). No. 61/724091 (filed on Nov. 8, 2012) and 61/588554 (filed on Jan. 17, 2012) also claim priority and benefit. These applications are hereby incorporated by reference in their entirety.

  The present application relates to imaging systems, and in particular to feature evaluation using three-dimensional (3D) object detection, tracking and optical imaging.

  The motion capture system is used in various scenes for acquiring information on the movement and structure of various objects including objects having joint portions such as human hands and human bodies. Such systems typically include a computer that analyzes images to reconstruct the volume, position, and motion of an object, and a camera that captures a series of images of moving objects. At least two cameras are typically used for 3D motion capture.

  Image motion capture systems rely on the ability to distinguish the object of interest from the background. This is often accomplished by using an image analysis algorithm that detects edges (typically detecting sudden changes in color and / or brightness by pixel comparison). However, such conventional systems have many common situations, for example, when the contrast between the object of interest in the background and the background and / or background pattern is low and can be erroneously detected as the edge of the object. Under performance will decrease.

  In some cases, for example, a person is wearing a reflector or a light source mesh while performing a movement, such as “instrumenting” the target object to distinguish between the object and the background. Can be promoted. Special illumination conditions (e.g., low light) can be used to make the reflectors and light sources stand out in the image. However, subject instrumentation is not always a convenient or desirable option.

An embodiment of the present invention relates to an imaging system that improves object recognition by improving the contrast between a background surface visible in the image used and the object. This can be achieved, for example, by means of control of the illumination directed at the object. For example, in a motion capture system that targets an object such as a human hand that is significantly closer to the camera than any background surface, the attenuation of light intensity with respect to distance (1 / r ² for a point light source) Utilized by the arrangement of the light source (or light sources) in the vicinity of the other imaging device (s) and the illumination of light on the object. Source light reflected near the object of interest can be expected to be much brighter than reflected light from farther backgrounds and farther backgrounds (compared to objects), more pronounced effect Can occur. Thus, in some embodiments, a cut-off threshold for the luminance of pixels in the captured image can be used to distinguish “object” pixels from “background” pixels. While a broadband ambient light source can be used, in various embodiments, a limited wavelength range of light and a camera adapted to detect such light are used. For example, light from an infrared light source can be used with one or more cameras that sense infrared frequencies.

  Accordingly, in a first aspect, the present invention relates to an image capturing and analyzing system for identifying a target object from a digitally displayed image scene. In various embodiments, the system includes at least one camera directed to a field of view, at least one light source disposed on the same field side as the camera and directed to illuminate the field of view, and the camera And an image analysis device coupled to the light source (s). The image analysis device operates the camera (s) to capture a series of images, including a first image captured at the same time as the light source (s) illuminate the field of view, and not the background Identify a pixel corresponding to the object, and build a 3D model of the object including the position and shape of the object based on the identified pixel and determine whether it corresponds to the object of interest Can be configured to determine. In one embodiment, the image analysis device corresponds to (i) a foreground image component corresponding to the object located in a near region of the field of view, and (ii) corresponding to the object located in a remote region of the field of view. And the proximity region extends from the camera (s) and the predicted maximum distance between the camera (s) and the object corresponding to the foreground image component The remote area exists at a position beyond the proximity area with respect to at least one of the cameras. For example, the proximity region may have a depth that is at least four times the predicted maximum distance.

  In another embodiment, the image analysis apparatus operates the camera (s) to capture the second and third images when the light source (s) are not illuminating the field of view, Identifying a pixel corresponding to the object based on a difference between the first and second images and a difference between the first and third images, the second image being captured before the first image, The third image is captured after the second image.

  For example, the light source (s) may be a diffuse emitter (eg, an infrared light emitting diode, in which case the camera (s) is an infrared sensitive camera). Two or more of the light sources may be adjacent to the camera (s) and they may be substantially in the same plane. In various embodiments, the camera (s) and the light source (s) are pointing vertically upward. To improve contrast, the camera is operated such that the exposure time is as high as 100 microseconds, and the light source (s) are driven at a power level of at least 5 watts during the exposure time. You may do it. In one implementation, a holographic diffraction grating is placed between each camera lens and the field of view (ie, in front of the camera lens).

  The image analysis device identifies the ellipse that defines the candidate object in terms of volumetric analysis, discards the object segment that is geometrically inconsistent with the definition based on the ellipse, and whether the candidate object corresponds to the target object. With an ellipse-based decision on, it can be determined geometrically whether an object corresponds to the object of interest.

  In another aspect, the present invention relates to an image capturing analysis method. In various embodiments, the method is by driving at least one light source that illuminates a field of view containing the object of interest, and using the camera (or cameras) simultaneously with driving the light source (s). Imaging a series of digital images of the field of view and identifying pixels corresponding to the object rather than the background, and based on the identified pixels, a 3D model of the object including the position and shape of the object And geometrically determine whether it corresponds to the object of interest.

  The light source (s) may be arranged such that the object of interest is located within a proximity area, the proximity area being a predicted maximum distance between the camera and the object of interest from the camera. It spreads to a distance that is at least twice as long. For example, the proximity region may have a depth that is at least four times the predicted maximum distance. The light source (s) may be, for example, a diffuse emitter (eg, an infrared light emitting diode), in which case the camera is an infrared sensitive camera. At least two or more of the light sources may be adjacent to the camera and they may be substantially in the same plane. In various embodiments, the camera and the light source (s) are pointing vertically upward. To improve contrast, the camera is operated such that the exposure time is as high as 100 microseconds, and the light source (s) are driven at a power level of at least 5 watts during the exposure time. You may do it.

  Further, the object pixel includes a first image when the light source (s) are not driven, a second image when the light source (s) are driven, and the light source (s) being driven. A pixel corresponding to the object based on the difference between the second and first images and the difference between the second and third images, which may be identified by imaging the third image when not Is identified.

  The geometric decision as to whether an object corresponds to the object of interest is the identification of an ellipse that defines the candidate object volumetrically and an object segment that is geometrically inconsistent with the ellipse-based definition , And an ellipse-based decision as to whether the candidate object corresponds to the target object.

  In yet another aspect, the invention relates to a method for positioning a round object in a digital image. In various embodiments, the method includes a series of driving at least one light source that illuminates a field of view that includes the object of interest, and a first image that is imaged at the same time as the at least one light source illuminates the field of view. And a step of operating the camera to capture an image and analyzing the image to detect a Gaussian luminance attenuation pattern indicating a round object in the field of view. In some embodiments, the round object is detected without identifying its edges. The method may further comprise tracking the movement of the round object detected through a plurality of captured images.

  In another aspect, the invention relates to an imaging analysis system for positioning a round object in a field of view. In various embodiments, the system includes at least one camera directed to a field of view, at least one light source disposed on the same field side as the camera and directed to illuminate the field of view, and the camera And an image analysis device coupled to the light source. The image analysis device operates at least one of the cameras to capture a series of images, including a first image captured at the same time as at least one of the light sources illuminates the field of view, and has a round shape within the field of view. It may be configured to analyze the image to detect a Gaussian luminance attenuation pattern indicative of the object. A round object may be detected without identifying its edge in some embodiments. The system may track the movement of the round object detected through a plurality of captured images.

  The phrase “substantially” or “approximately” as used herein means ± 10% (eg, weight or volume), and in some embodiments ± 5%. The phrase “consisting essentially of” means that it does not include other materials that contribute to function unless otherwise defined herein. Throughout this specification, references to “one example”, “an example”, “one embodiment”, or “an embodiment” are described with respect to that example. The particular feature, structure or feature made is meant to be included in at least one example of the present technology. As such, the phrases “in one example”, “in an example”, “one embodiment” or “an embodiment” in various places throughout this specification. The description of embodiment) ”does not necessarily refer to the same example. Furthermore, the particular features, structures, routines, steps or characteristics may be combined in any suitable manner in one or more examples of the technology. The headings defined herein are for convenience only and are not intended to limit or interpret the scope or meaning of the claimed technology.

  The following detailed description in conjunction with the accompanying drawings will provide a better understanding of the nature and advantages of the present invention.

1 shows a system for capturing image data according to an embodiment of the present invention. 1 is a simplified block diagram of a computer system that realizes an image analysis apparatus according to an embodiment of the present invention. 6 is a graph of luminance data of pixel rows that can be obtained according to an embodiment of the present invention. 6 is a graph of luminance data of pixel rows that can be obtained according to an embodiment of the present invention. 6 is a graph of luminance data of pixel rows that can be obtained according to an embodiment of the present invention. The flowchart of the process for identifying the position of the object in the image which concerns on embodiment of this invention. 4 shows a time series of pulsed light sources that are turned on at regular intervals according to an embodiment of the present invention. 4 shows a time series of pulse driving and image capturing of a light source according to an embodiment of the present invention. The flowchart of the process which identifies the edge of an object using a series of images which concern on embodiment of this invention. The top view of the computer system containing the motion detector which is a user input device concerning the embodiment of the present invention. The front view of the tablet computer which shows another example of the computer system containing the motion detector which concerns on embodiment of this invention. 1 illustrates a goggle system including a motion detector according to an embodiment of the present invention. The flowchart of the process which uses motion information as a user input for controlling the computer system which concerns on embodiment of this invention, or another system. 3 shows a system for imaging image data according to another embodiment of the present invention. 6 shows a system for capturing image data according to still another embodiment of the present invention.

  Reference is first made to FIG. 1 illustrating a system 100 for imaging image data according to an embodiment of the present invention. System 100 includes a pair of cameras 102 and 104 coupled to an image analysis system 106. Cameras 102 and 104 include any camera that senses the entire visible spectrum or, more typically, a camera with enhanced sensitivity in a limited wavelength band (eg, infrared (IR) or ultraviolet band). Such a type of camera may be used. More generally, the phrase “camera” herein refers to any device (or combination of devices) capable of taking an image of an object and displaying the image in the form of digital data. For example, a line sensor or a line camera may be used instead of a conventional apparatus that captures a two-dimensional (2D) image. The phrase “light” may or may not be within the visible spectrum and may be broadband (eg, white light) or narrowband (eg, single wavelength or narrow wavelength band). Generally used to imply electromagnetic emission.

  The heart of a digital camera is an image sensor that includes a grid of photosensitive image elements (pixels). The lens condenses light on the surface of the image sensor, and an image is formed when light of various intensities hits the pixels. Each pixel converts the light into a charge that reflects the detected light intensity and collects the charge so that it can be measured. Both CCD and CMOS image sensors perform the same function, but differ in signal measurement and transmission methods.

  In a CCD, the charge from each pixel is conveyed to a single structure that converts the charge to a measurable voltage. This is done by each pixel moving its charge sequentially to its neighboring pixels in a “bucket relay” manner for each row and column until the measurement structure is reached. In contrast, a CMOS sensor has a measurement structure at each pixel location. The measurement results are transferred directly from each position to the sensor output.

  Although the cameras 102 and 104 are preferably capable of capturing video images (ie, a series of image frames at a constant rate of at least 15 frames per second), a specific frame rate is not required. The function of the cameras 102, 104 is not critical to the present invention, and the camera can be used for frame rate, image resolution (eg, number of pixels per image), color or intensity resolution (eg, bits of intensity data per pixel), There may be various variations in lens focal length, depth of field, and the like. In general, any camera capable of focusing on an object in the target spatial volume may be used for a particular application. For example, to image the movement of a person's hand with other parts stationary, the volume of interest can be defined as a cube that is approximately 1 meter on a side.

  The system 100 further includes a pair of light sources 108 and 110 disposed on both sides of the cameras 102 and 104 and controlled by the image analysis system 106. The light sources 108 and 110 may be an infrared light source, such as an infrared light emitting diode (LED), which is a typical conventional design, and the cameras 102 and 104 may be capable of sensing infrared light. Filters 120, 122 can be placed in front of cameras 102, 104 so that visible light is removed and only infrared light is recorded in images captured by cameras 102, 104. In some embodiments where the object of interest is a human hand or body, the use of infrared light allows the motion capture system to operate under a wide range of lighting conditions, Interferences that may be associated with the incidence of visible light within the moving area can be avoided. However, specific wavelengths and electromagnetic spectrum regions are required.

  It should be emphasized that the above arrangement is exemplary and not limiting. For example, a laser or other light source can be used in place of the LED. For laser setup, additional optics (eg, a lens or diffuser) may be used to expand the laser beam (and create a field of view similar to that of the camera). Useful configurations may further include short wide angle illuminators for different ranges. The light source is typically a diffusive point source rather than a specular reflection. For example, LEDs packaged by light diffusion encapsulation are suitable.

  In operation, the cameras 102, 104 are pointed toward a target area 112 where a target object 114 (in this example, a hand) and one or more background objects 116 may be present. The light sources 108 and 110 are arranged so as to irradiate the region 112. In some embodiments, the one or more light sources and the one or more cameras 102, 104 are below the detected movement (eg, directly below the spatial region in which the movement occurs if hand movement is detected). Be placed. The amount of information recorded for a hand is proportional to the number of pixels it occupies in the camera image, and the hand occupies more pixels if the camera's angle to the “pointing direction” of the hand is as vertical as possible. Therefore, the above position is optimal. For a user, pointing the palm against the screen is cramped, so either looking up from the bottom, looking down from the top or looking up diagonally from the screen bezel or looking down is the optimal position. When looking up, the possibility of confusion with background objects (for example, scattered objects on the user's desk) is low, and if looking straight up, the possibility of confusion with other people outside the field of view is reduced. (Furthermore, privacy is improved by not imaging the face). For example, the image analysis system 106, which can be a computer system or the like, can control the operation of the light sources 108, 110 and the cameras 102, 104 to capture an image of the region 112. Based on this captured image, the image analysis system 106 determines the position and / or movement of the object 114.

For example, as a step in determining the position of the object 114, the image analysis system 106 may determine pixels of various images captured by the cameras 102, 104 that include a portion of the object 114. In some embodiments, any pixel in the image may be classified as an “object” pixel or a “background” pixel based on whether it is a pixel that includes a portion of the object 114. Classification of object or background pixels using light sources 108, 110 may be performed based on pixel brightness. For example, the distance (r _O ) between the target object 114 and the cameras 102, 104 is expected to be smaller than the distance (r _B ) between the background object 116 and the cameras 102, 104. Since the intensity of the light from the light sources 108, 110 decreases by 1 / r ² , the object 114 is illuminated brighter compared to the background 116, and the pixel that contains a portion of the object 114 (ie, the object pixel) Corresponding to the pixel 116 (i.e., the background pixel) including a part of the background 116. For example, if r _B / r _O = 2, it is assumed that the object 114 and the background 116 similarly reflect light from the light sources 108 and 110, and further the entire illumination of the region 112 (at least the frequency imaged by the cameras 102 and 104. Assuming that (in-band) is dominated by the light sources 108, 110, the object pixel will be approximately four times brighter than the background pixel. These assumptions are generally maintained in the proper selection of cameras 102, 104, light sources 108, 110, filters 120, 122 and the objects normally encountered. For example, the light sources 108 and 110 can be infrared LEDs that can emit strong radiation in a narrow frequency band, and the filters 120 and 122 can match the frequency bands of the light sources 108 and 110. Thus, a human hand or body, or a heat source or other object in the background, can emit infrared light, but the response of the cameras 102, 104 still originates from the light sources 108, 110 and the object 114 and / or background. It can be dominated by the light reflected by 116.

In this configuration, the image analysis system 106 can quickly and accurately distinguish the target pixel from the background pixel by applying a luminance threshold value to each pixel. For example, the brightness of a pixel in a CMOS sensor or similar device can be measured in a range having several gradations between 0.0 (dark) and 1.0 (fully saturated) based on the sensor design. The luminance encoded by the camera pixel is typically due to accumulated charge or diode voltage and corresponds to a standard (linear) with respect to the brightness of the subject. In some embodiments, the light sources 108, 110 cause light reflected from an object at a distance r _O to produce a luminance level of 1.0 while light reflected from an object at a distance r _B = 2r _O. Is bright enough to produce a brightness level of 0.25. The target pixel can thus be easily distinguished from the background pixel based on the luminance. Furthermore, the edges of the object can also be easily detected based on the difference in brightness between adjacent pixels, allowing the position of the object in each image to be determined. The association of the position of the object between the images from the cameras 102, 104 enables the determination of the position of the object 114 in 3D space in the image analysis system 106, and the analysis of the series of images is performed by the image analysis system 106. Allows the reconstruction of 3D motion of the object 114 using conventional motion algorithms in FIG.

  Of course, the system 100 is exemplary, and changes and modifications are possible. For example, the light sources 108 and 110 are shown as being disposed on both sides of the cameras 102 and 104. This may facilitate illumination of the edges of the object 114 viewed from the perspective of both cameras. However, a specific arrangement of cameras and lights is not necessary. (Examples of other configurations are described below.) As long as the object is significantly closer to the camera than the background, improved contrast as described herein can be achieved.

  The image analysis system 106 (also referred to as an image analysis device) may be included in or constitute any device or device component capable of imaging and processing image data using, for example, the techniques described herein. obtain. FIG. 2 is a simplified block diagram of a computer system 200 that implements the image analysis apparatus 106 according to the embodiment of the present invention. The computer system 200 includes a processor 202, a memory 204, a camera interface 206, a display 208, a speaker 209, a keyboard 210 and a mouse 211.

  Memory 204 may be used to store not only instructions executed by processor 202 but also input and / or output data associated with execution of the instructions. In particular, the memory 204 stores instructions that control the operation of the processor 202 and interaction with other hardware components, conceptually illustrated as a group of modules described in detail below. The operating system directs low-level execution, which is a basic system function such as memory allocation, file management, and mass storage operation. The operating systems are Microsoft Windows (registered trademark) operating system, Unix (registered trademark) operating system, Linux (registered trademark) operating system, Xenix operating system, IBM AIX operating system, Hewlett Packard UX operating system, Novell NETWARE operating system. , Sun Microsystems SOLARIS operating system, OS / 2 operating system, BeOS operating system, MACINTOSH operating system, APACHE operating system, OPENSTEP operating system or platform It can consist of or include various operating systems, such as another operating system.

  The computing environment may include other removable / non-removable, volatile / nonvolatile computer storage media. For example, hard disk drives can read or write to non-removable, non-volatile magnetic media. A magnetic disk drive can read or write to a removable and non-volatile magnetic disk, and an optical disk drive can read or write to an optical disk such as a non-volatile CD-ROM or other optical media . Other removable / non-removable, volatile / nonvolatile computer storage media include those used in the exemplary operating environment, including magnetic tape cassettes, flash memory cards, digital versatile disks, digital The video tape, solid state RAM, solid state ROM, etc. are not limited to these. The storage medium is typically connected to the system bus via a removable or non-removable memory interface.

  The processor 202 may be a general-purpose microprocessor, but instead, depending on the implementation, a microcontroller, peripheral integrated circuit elements, CSIC (Customer Specific Integrated Circuit), ASIC (Application-Specific Integrated Circuit), logic circuit, Digital signal processor, programmable logic device such as FPGA (Field Programmable Gate Array), PLD (Programmable Logic Device), PLA (Programmable Logic Array), RFID processor, smart chip or processing steps of the present invention can be executed It can be any other device or device configuration.

  The camera interface 206 is not only related to hardware and / or software that enables communication between a camera such as the cameras 102 and 104 shown in FIG. 1 and the computer system 200, but also related to the light sources 108 and 110 shown in FIG. A light source may also be included. Thus, for example, the camera interface 206 may be a conventional motion capture (“Morcap”) program that executes on the processor 202 data signals received from the camera as well as one or more data ports 216, 218 to which the camera is connected. A hardware and / or software signal processor may also be included for modification (eg, noise reduction or data reformatting) prior to being provided as an input to 214. In some embodiments, the camera interface 206 may also send signals to the camera, eg, to drive or stop the camera, control camera settings (frame rate, image quality, sensitivity, etc.), etc. . Such signals can be generated in turn in response to user input or other detected events and can be transmitted in response to a control signal from processor 202, for example.

  The camera interface 206 may also include controllers 217, 219 that can be connected to light sources (eg, light sources 108, 110). In some embodiments, the controllers 217, 219 provide operating current to the light source, eg, in response to an instruction from the processor 202 executing the morphcap program 214. In other embodiments, the light source may draw operating current from an external power source (not shown), and the controllers 217, 219 may generate control signals for the light source that indicate, for example, turning on or off the light source or changing the brightness. . In some embodiments, one controller can be used to control multiple light sources.

  Instructions that define the motocap program 214 are stored in the memory 204. When these instructions are executed, motion capture analysis is performed on an image provided from a camera connected to the camera interface 206. In one embodiment, the mocap program 214 includes various modules such as the object detection module 222 and the object analysis module 224. Furthermore, both of these modules are conventional and well characterized in the art. The object detection module 222 may analyze the image (eg, an image taken via the camera interface 206) to detect other information about the edge of the object and / or the position of the object in the image. Object analysis module 224 may analyze the object information provided by object detection module 222 to determine the 3D position and / or movement of the object. Examples of operations that can be performed by the code modules of the motocap program 214 are described below. Memory 204 may also include other information and / or code modules used by motocap program 214.

  Display 208, speaker 209, keyboard 210, and mouse 211 may facilitate user interaction with computer system 200. These components can be of a general conventional design or modified to make it desirable to provide any type of user interaction. In some embodiments, the results of motion capture using camera interface 206 and mcap program 214 may be interpreted as user input. For example, the user can make a hand gesture that is analyzed using the morphcap program 214 and the results of this analysis are executed on the processor 200 (eg, a web browser, word processor or other application). It can be interpreted as instructions to other programs. Thus, by way of example, the user may rotate upper or lower swipe gestures to “scroll” the current web page displayed on display 208, or increase or decrease the volume of audio output from speaker 209. You can use gestures, etc.

  Of course, the computer system 200 is an example, and changes and modifications are possible. The computer system can be implemented in various form factors including a server system, desktop system, laptop system, tablet, smartphone or personal digital assistant. Particular implementations may include other functions not described herein, such as wired and / or wireless network interfaces, media playback and / or recording functions, for example. In some embodiments, one or more cameras may be incorporated into a computer rather than being provided as a separate component. Further, the image analysis device is a component of a computer system (eg, a processor that executes program code, an ASIC or a fixed function digital signal processor with an appropriate I / O interface for receiving image data and output analysis results). Can be implemented using only a subset of

  Although the computer system 200 is described herein with reference to particular blocks, the blocks are defined for convenience of explanation and refer to particular physical arrangements of components. It should be understood that it is not intended to. Furthermore, the blocks need not correspond to physically separate components. Where physically separate components are used, the connections between components (eg, for data communication, etc.) can be wired and / or wireless as desired.

  The execution of the object detection module 222 by the processor 202 causes the processor 202 to operate the camera interface 206 in order to capture an image of the object, or to distinguish the target pixel from the background pixel by analyzing the image data. 3A-3C are three different graphs of pixel row luminance data that may be obtained according to various embodiments of the invention. Each graph illustrates one pixel row, but it should be understood that an image typically includes a number of pixel rows and that a row may include any number of pixels. For example, an HD video image may include 1080 rows each having 1920 pixels.

  FIG. 3A shows luminance data 300 for a pixel row of an object having a single cross section, such as a cross section of a palm. Pixels in the region 302 corresponding to the object have high brightness, whereas pixels in the regions 304 and 306 corresponding to the background have significantly low brightness. As can be seen from the figure, the position of the object is readily apparent and the edge positions (position 308, position 310) of the object can be easily identified. For example, a pixel having a luminance higher than 0.5 can be regarded as a target pixel, and a pixel having a luminance lower than 0.5 can be regarded as a background pixel.

  FIG. 3B shows luminance data 320 for pixel rows of an object having a plurality of different cross sections, such as a cross section of a finger of an open hand. The regions 322, 323 and 324 corresponding to the object have high brightness, but the pixels in the regions 326 to 329 corresponding to the background have low brightness. Again, a simple cut-off threshold for brightness (e.g., 0.5) is sufficient to distinguish the target pixel from the background pixel, and the edge of the object can be easily determined.

  FIG. 3C shows pixel row luminance data 340 in which the distance to the object varies, such as a cross section of a hand with two fingers spread toward the camera. Regions 342 and 343 corresponding to the open finger have the highest brightness. Regions 344 and 345 corresponding to other parts of the hand have slightly less brightness. This may be due to one being farther away and one being shadowed by an open finger. Regions 348 and 349 corresponding to the background are background regions and are significantly darker than regions 342 to 345 that include hands. A cutoff threshold for luminance (eg 0.5) is still sufficient to distinguish the target pixel from the background pixel. Further analysis of the target pixel can also be performed to detect the edges of regions 342 and 343, giving additional information regarding the shape of the object.

Of course, the data shown in FIGS. 3A-3C are exemplary. In some embodiments, an object at the expected distance (eg, r _O in FIG. 1) is overexposed (ie, many if not all of the target pixels are at a brightness level of 1.0). It may be desirable to adjust the intensity of the light sources 108, 110. (The actual brightness of the object can actually be high.) The background pixels can also be a little brighter, but the intensity of the light intensity with respect to the distance is still reduced 1 / r unless the intensity is set high enough that the background pixels also approach the saturation level. ² is a state in which an object and a background pixel can be distinguished. As shown in FIGS. 3A-3C, the use of illumination directed at the object to create a strong contrast between the object and the background uses a simple and fast algorithm to distinguish the background and target pixels. And can be particularly useful in real-time motion capture systems. Simplification of the task of distinguishing background and object pixels may free up computer resources for other motion capture tasks (eg, reconstruction of object position, shape and / or motion).

  Reference is made to FIG. 4 showing a flow diagram of a process for identifying the position of an object in an image according to an embodiment of the invention. Process 400 may be implemented, for example, in system 100 of FIG. In block 402, the light sources 108, 110 are turned on. At block 404, one or more images are captured using the cameras 102,104. In some embodiments, one image from each camera is taken. In other embodiments, a series of images are taken from each camera. The two images from the two cameras can be closely correlated in time (eg, simultaneously within a few milliseconds) so that the correlation images from the two cameras can be used to determine the 3D position of the object. .

  At block 406, a pixel brightness threshold is applied to distinguish the target pixel from the background pixel. Block 406 may also include locating the edge of the object based on a transition point between the background and the object pixel. In some embodiments, each pixel is initially classified as either an object or background based on whether a luminance cutoff threshold is exceeded. For example, as shown in FIGS. 3A-3C, a cutoff at a saturation level of 0.5 may be used. Once the pixels are classified, an edge can be detected by finding the location where the background pixel is adjacent to the pixel object. Some embodiments require that the background and object pixel regions on either side of the edge have a certain minimum size (eg, 2, 4 or 8 pixels) to avoid noise defects. obtain.

In other embodiments, the edge may be detected without first classifying whether the pixel is an object or a background. For example, Δβ can be defined as the luminance difference between adjacent pixels, and | Δβ | is above a threshold (eg, 0.3 or 0.5 in units of saturation range). A transition from background to object or from object to background may be indicated. (The sign of Δβ may indicate the direction of the transition.) If the edge of the object is actually the center of the pixel, there may be a pixel with an intermediate value at the boundary. For example, for pixel i, two luminance values (βL = (β _i + β _i−1 ) / 2 and βR = (β _i + β _{i + 1} ) / 2, pixel (i−1) is the left side of pixel i, pixel ( i + 1) can be detected by calculating the right side of pixel i). In general, | βL-βR | is close to zero when pixel i is not near the edge, and | βL-βR | is close to 1 when pixel is near the edge, and | βL-βR | A threshold can be used to detect edges.

  In some examples, a part of an object may partially occlude another object in the image. For example, in the case of a hand, the finger may partially block the palm or another finger. Shielding edges that result from partly obscuring another object may also be detected based on small but obvious changes in brightness if background pixels are removed. FIG. 3C shows an example of such partial occlusion, where the location of the occlusion edge is clear.

  The detected edges can be used for many purposes. For example, as described above, the edge of an object viewed from two cameras can be used to determine the approximate position of the object in 3D space. The position of the object in the 2D plane across the optical axis of the camera can be determined from one image, and the offset (parallax) between the positions of the objects in the time-correlated images from two different cameras is If the spacing is known, it can be used to determine the distance to the object.

  Furthermore, the position and shape of the object can be determined based on the position of its edge in the time-correlated images from two different cameras, and the motion of the object (including joints) can be derived from a series of analysis of a pair of images. Can be determined. As an example of a technique that can be used to determine the position, shape, and motion of an object based on the position of the edge of the object, the entire disclosure of co-pending serial number 13/414485 (filed March 7, 2012 US) is , Incorporated herein by reference. Those skilled in the art accessing the present disclosure will recognize other techniques that may also be used to determine the position, shape, and movement of an object based on information about the position of the edge of the object.

  Based on the 13/414485 application, the movement and / or position of the object is reconstructed using a small amount of information. For example, the shape of an object or the outline of a silhouette viewed from a particular viewpoint can be used to define a tangent to the object from that viewpoint in various planes (referred to herein as a “slice”). With only two different viewpoints, four (or more) tangents from the viewpoint to the object can be obtained within a given slice. From these four (or more) tangents, it is possible to determine the position of the object in the slice and approximate its cross-section in the slice using, for example, one or more ellipses or other simple closed curves. It is possible. As another example, the position of a point on the surface of an object in a particular slice can be determined directly (eg, using a time-of-flight camera) and the position and shape of the cross-section of the object in that slice is Can be approximated by fitting an ellipse or other simple closed curve to the point. Position and cross-sectional determinations for different slices can be correlated to build a 3D model of the object including its position and shape. The series of images can be analyzed using the same techniques that model the movement of the object. The movement of complex objects (eg, human hands) with multiple independent joints can be modeled using these techniques.

More specifically, the ellipse in the xy plane has five parameters: center x and y coordinates (X _C , Y _C ), major radius, minor radius, and rotation angle (eg, major radius angle with respect to the x axis). Can be characterized by With only four tangents, the ellipse is undetermined. However, despite this fact, an efficient process for estimating an ellipse is the establishment of an initial working hypothesis (or “guess”) for one of the parameters and additional information collected during the analysis. Including reexamination of hypotheses. This additional information may include physical constraints based on, for example, the nature of the camera and / or object. In some situations, for example, two or more viewpoints are available, so four or more tangents to the object may be available for some or all of the slices. The elliptical cross-section is still determinable and the process in some examples is slightly simplified so that it is not necessary to assume parameter values. In some examples, additional tangents can result in additional complexity. In some situations, four or more tangents to the object may be available for some or all of the slices, for example because the edge of the object is outside the field of view of one camera or no edge was detected. A slice with three tangents can be analyzed. For example, using two parameters from an ellipse that fits an adjacent slice (eg, a slice that had at least four tangents), the simultaneous equations for that ellipse and three tangents can be solved. It is fully determined that there is. Another option is a circle that can fit three tangents. Since only three parameters (center coordinates and radius) are needed to determine the circle in the plane, the three tangents fit the circle sufficiently. Slices with tangents less than 3 can be discarded or combined with adjacent slices.

  One method for geometrically determining whether an object corresponds to a target object is to define the object by determining the volume of a series of ellipses, and to define an object based on an ellipse. And discarding object segments that are geometrically inconsistent (eg, discarding excessively cylindrical, excessively straight, excessively thin, excessively small, or excessively far segments). If a sufficient number of ellipses remain to characterize the object and it matches the object of interest, it can be identified as such and tracked from frame to frame.

  In some embodiments, each of the plurality of slices is analyzed individually to determine the size and position of the elliptical cross section of the object within that slice. This provides an initial 3D model (specifically, a stack of elliptical cross sections) that can be improved by correlating cross sections across different slices. For example, the surface of the object is expected to be continuous, and discontinuous ellipses can be ignored as a result. Further improvements can be obtained, for example, by correlating their 3D model over time based on expectations related to the continuity of movement and deformation. Referring back to FIGS. 1 and 2, in some embodiments, the light sources 108, 110 may be operated in a pulsed mode rather than being turned on continuously. For example, this can be useful if the light sources 108, 110 have the ability to generate light that is pulsed and brighter than steady state operation. FIG. 5 shows a time series in a pulse form in which the light sources 108 and 110 are turned on at regular intervals as indicated by 502. As shown at 504, the shutters of the cameras 102, 104 can be opened to capture an image at a timing that matches the light pulse. In this way, the target object can be illuminated brightly during the time that the image is captured. In some embodiments, object silhouettes are extracted from one or more object images that show information about the object viewed from different viewpoints. Although the silhouette can be obtained using a number of different techniques, in some embodiments, the silhouette is obtained by using a camera to capture an image of the object and analyzing the image to detect the edges of the object.

  In some embodiments, pulsed driving of the light sources 108, 110 can be used to further improve the contrast between the object of interest and the background. In particular, in the case of scenes that contain light that is emitted or that are highly reflective, the ability to distinguish between related and unrelated (eg, background) objects in the scene can be compromised. The problem is that the exposure time of the camera is set to a very short time (eg 100 microseconds or less) and very high power (ie 5-20 watts, or in some cases more than 40 watts, for example). Can be addressed by pulsing the illumination at a high level). At this time, the most common ambient illumination light source (eg, fluorescent lamp) is very dark compared to such bright short-time illumination. That is, in microseconds, non-pulsed light sources are dim even if they appear with exposure times of milliseconds or longer. In fact, this method increases the contrast of the object of interest with respect to other objects even though they emit in the same general band. Thus, discrimination by luminance under such conditions allows for irrelevant objects to be ignored for image reconstruction and processing purposes. Average power consumption is also reduced. For 20 watts and 100 microseconds, the average power consumption is below 10 milliwatts. In general, the light sources 108, 110 are operated to be on during the exposure time of the entire camera (ie, the pulse width is equal to and aligned with the exposure time).

  It is also possible to adjust the pulses of the light sources 108 and 110 for the purpose of comparing images captured with the light sources 108 and 110 turned on and images taken with the light sources 108 and 110 turned off. FIG. 6 illustrates a pulsed state in which the light sources 108 and 110 are turned on at regular intervals as indicated by 602 while the shutters of the cameras 102 and 104 are open to capture an image as indicated by 604. Indicates the series. In this case, the light sources 108 and 110 are “on” for every other image. When the target object is significantly closer to the light sources 108 and 110 than the background region, the difference in light intensity with respect to the target pixel is stronger than the difference in light intensity with respect to the background pixel. Thus, a comparison of pixels in a series of images can help distinguish between object and background pixels.

  FIG. 7 is a flow diagram of a process 700 for identifying an object edge using a series of images according to an embodiment of the invention. In block 702, the light source is turned off, and in block 704, the first image (A) is captured. Next, in block 706, the light source is turned on, and in block 708, the second image (B) is captured. In block 710, a “difference” image B-A is calculated, for example, by subtracting the luminance value of each pixel of image A from the luminance value of the corresponding pixel of image B. Since the image B is captured with the light turned on, B-A is expected to be positive in most pixels.

  The difference image is used to distinguish the background and foreground by applying thresholds or other per-pixel criteria. In block 712, a threshold is applied to the difference image to identify the object pixel, with a threshold value (BA) being associated with the target pixel and a threshold value (BA) being associated with the background pixel. It is done. The edge of the object can then be defined by the identification of the target pixel adjacent to the background pixel as described above. The edge of the object may be used for purposes such as position and / or motion detection as described above.

In an alternative embodiment, object edges are identified using triplet image frames instead of pairs. For example, in one implementation, a first image (image 1) is obtained with the light source turned off, a second image (image 2) is obtained with the light source turned on, and a third image (image 3) is obtained. ) Is imaged with the light source turned off again. Two difference images Image 4 = abs (image 2 -image 1) and image 5 = abs (image 2 -image 3)
Is defined by subtracting the luminance value of the pixel. The final image, image 6, is defined based on two images, image 4 and image 5. In particular, the value of each pixel in image 6 is the smaller of the two corresponding pixel values in images 4 and 5. In other words, for each pixel, image 6 = min (image 4, image 5). Image 6 represents a difference image with improved accuracy, and most of its pixels are positive. Again, thresholds or other criteria may be used for each pixel to distinguish foreground and background pixels.

  The contrast-based object detection described herein is for any situation where the target object is expected to be much closer (eg, half the distance) to the light source (s) than the background object. Can be applied. One such application for using motion detection is user input for interacting with a computer system. For example, when a user points to the screen or makes a gesture with another hand, it can be interpreted as input to the computer system.

  A computer system 800 including a motion detector that is a user input device according to an embodiment of the present invention is shown in FIG. The computer system 800 includes a desktop box 802 that may contain various computer system components, such as a processor, memory, fixed or removable disk drives, video drivers, audio drivers, network interface components, and the like. The display 804 is connected to the desktop box 802 and arranged so that the user can browse. The keyboard 806 is disposed within a range that can be easily reached by the user. The motion detector unit 808 is located near the keyboard 806 (eg, back or one side as shown) in which it is natural for the user to make a gesture toward the display 804 (eg, on the keyboard). It is directed to the upper space (in front of the monitor). Cameras 810, 812 (eg, which may be similar to or the same as the cameras 102, 104 described above) are generally arranged to face upwards, and light sources 814, 816 (similar to or similar to the light sources 108, 110 described above). (Which may be the same) are placed on either side of the cameras 810, 812 to illuminate the area above the motion detector unit 808. In a typical implementation, cameras 810, 812 and light sources 814, 816 are substantially in the same plane. This configuration prevents, for example, the appearance of shadows that can interfere with edge detection (similar to the case where the light source is not adjacent to the camera and is located in between). A filter not shown is above the top surface of the motion detector unit 808 (or just above the apertures of the cameras 810, 812) to remove all light that falls outside the band near the peak frequency of the light sources 814, 816. ).

  In the illustrated configuration, when a user moves a hand or other object (eg, a pencil) within the field of view of the cameras 810, 812, the background may possibly consist of a ceiling and / or various fixings provided on the ceiling. The user's hand can be 10-20 centimeters above the motion detector unit 808, while the ceiling can be 5-10 times (or more) that distance. The illumination from the light sources 814, 816 is therefore much stronger to the user's hand compared to the ceiling, and the techniques described herein can be used from background pixels in the image captured by the cameras 810, 812. It can be used to reliably distinguish object pixels. When infrared light is used, the user is not distracted or disturbed by the light.

  The computer system 800 may utilize the structure shown in FIG. For example, the cameras 810, 812 of the motion detector unit 808 can provide image data to the desktop box 802, and image analysis and subsequent interpretation uses a processor and other components contained in the desktop box 802. Can be done. The motion detector unit 808 may also include a processor or other component for performing some or all stages of image analysis and interpretation. For example, motion detector unit 808 may include a processor (programmable or fixed function) that performs one or more of the processes described above for distinguishing between object pixels and background pixels. In this case, the motion detector unit 808 may send a reduced display of the captured image (eg, a display with all background pixels zeroed) to the desktop box 802 for further analysis and interpretation. No special division of computational tasks between the processor in the motion detector unit 808 and the processor in the desktop box 802 is necessary.

It is not always necessary to distinguish between an object pixel and a background pixel based on an absolute luminance level. For example, if there is knowledge of the object shape, a luminance decay pattern can be used to detect the object in the image without obvious detection of the object edge. For a rounded object (such as a hand or finger), for example, the 1 / r ² relationship produces Gaussian or near-Gaussian brightness distributions near the center of the object. When a cylinder that is illuminated by the LED and is arranged perpendicular to the camera is imaged, the image has a bright center line corresponding to the cylinder axis and the brightness is attenuated on each side (around the cylinder). The finger is approximately cylindrical and by identifying these Gaussian peaks, the background is close and the edges are not visible due to the relative brightness of the background (due to proximity or aggressive infrared light can be emitted. For this reason, the finger can be placed even in the situation. The phrase “Gaussian” is used broadly herein to imply a negative second derivative curve. In many cases, such curves are bell-shaped and symmetric, but not necessarily. For example, if the object is more specular or if the object is at an extreme angle, the curve may shift in a particular direction. Therefore, the term “Gaussian” as used herein is not limited to only curves that clearly fit a Gaussian function.

  FIG. 9 shows a tablet computer 900 including a motion detector according to an embodiment of the present invention. The tablet computer 900 has a housing including a display screen 902 surrounded by a bezel 904 on the front surface. One or more control buttons 906 may be included in the bezel 904. The tablet computer 900 may have various conventional computer components (processor, memory, network interface, etc.) within the housing (eg, behind the display screen 902). The motion detector unit 910 is provided within the bezel 904 and is directed to the front to capture the movement of a user located in front of the tablet computer 900 (eg, with the cameras 102, 104 of FIG. 1). Similar or identical) and light sources 916, 918 (eg, similar or identical to light sources 108, 110 of FIG. 1) may be implemented.

  As the user moves a hand or other object within the field of view of the cameras 912, 914, motion is detected as described above. In this case, the background is probably the user's own body and is a distance of approximately 25-30 centimeters from the tablet computer 900. A user may hold a hand or other object at a short distance, such as 5-10 centimeters, from the display 902. As long as the user's hand is significantly closer to the light source 916, 918 than the user's body (eg, half the distance), the illumination-based contrast improvement techniques described herein distinguish the target pixel from the background pixel. Can be used. Image analysis and subsequent interpretation as input gestures is performed within the tablet computer 900 (eg, utilizing the main processor to run an operating system or other software to analyze data obtained from the cameras 912, 914). Can be broken. The user can thereby interact with the tablet 900 using gestures in 3D space.

  The goggles system 1000 shown in FIG. 10 may also include a motion detector according to an embodiment of the present invention. Goggle system 1000 may be used in connection with, for example, virtual reality and / or augmented reality environments. Goggles system 1000 includes goggles 1002 that a user can wear, similar to conventional glasses. The goggles 1002 include eyepieces 1004 and 1006 that may include small display screens that provide images (eg, virtual reality environment images) to the left and right eyes of the user. These images may be provided by a base unit 1008 (eg, a computer system) that communicates with the goggles 1002 via either a wired or wireless channel. Cameras 1010, 1012 (eg, similar or identical to cameras 102, 104 of FIG. 1) may be provided in the frame portion of goggles 1002 so that they do not obscure the user's view. The light sources 1014 and 1016 can be provided on both sides of the cameras 1010 and 1012 in the frame portion of the goggles 1002. Images collected by the cameras 1010, 1012 may be sent to the base unit 1008 for analysis and interpretation as gestures that indicate user interaction with a virtual or extended environment. (In some embodiments, the virtual or extended environment presented via eyepieces 1004, 1006 may include a display of the user's hand, and the display may be based on images collected by cameras 1010, 1012. .)

  When the user makes a gesture using the hand or other object within the field of view of the cameras 1010 and 1012, motion is detected as described above. In this case, the background is probably the wall of the room where the user is, and the user is probably sitting or standing some distance from the wall. As long as the user's hand is significantly closer to the light source 1014, 1016 than the user's body (eg, half the distance), the illumination-based contrast improvement techniques described herein can distinguish the target pixel from the background pixel. make it easier. Image analysis and subsequent interpretation as input gestures may be performed within the base unit 1008.

  Of course, the motion detector implementations shown in FIGS. 8-10 are exemplary and can be changed or modified. For example, the motion detector or its components can be incorporated into a single housing with other user input devices such as a keyboard and trackpad. In another example, the motion detector may have, for example, an upward facing camera and light source incorporated in the same plane as the laptop keyboard (eg, one side of the keyboard, or in front or behind) or a forward facing camera and light source. Included in a laptop computer incorporated in a bezel that surrounds a laptop display screen. In yet another example, the wearable motion detector may be implemented as, for example, a headband or headset that does not include an active display or optical components.

  As shown in FIG. 11, motion information may be used as user input to control a computer system or other system according to embodiments of the present invention. Process 1100 may be performed, for example, on a computer system as shown in FIGS. At block 1102, an image is captured using the motion detector light source and camera. As mentioned above, imaging an image may involve the use of a light source to illuminate the camera's field of view such that objects close to the light source (and the camera) are illuminated brighter than objects farther away.

  At block 1104, the captured image is analyzed to detect the edge of the object based on the change in brightness. For example, as described above, this analysis includes comparison between the luminance of each pixel and a threshold value, detection of a luminance transition from a low level to a high level in an adjacent pixel, and / or the presence or absence of illumination by a light source A comparison of a series of images taken at can be included. In block 1106, an edge-based algorithm is used to determine the position and / or motion of the object. This algorithm can be, for example, any tangent-based algorithm described in the aforementioned 13/414485 application. Other algorithms can also be used.

  At block 1108, a gesture is identified based on the position and / or movement of the object. For example, a library of gestures may be defined based on the position and / or movement of the user's finger. A “tap” may be defined based on a fast movement of a finger extending toward the display screen. “Trace” may be defined as an extended finger movement in a plane generally parallel to the display screen. An inner pinch can be defined as two extended fingers that move closer together, and an outer pinch can be defined as two extended fingers that move further open. A swipe gesture can be defined based on the movement of the entire hand in a particular direction (eg, up, down, left, right), while another swipe gesture can be defined as the number of extended fingers (eg, 1, 2 , All) can be further defined. Other gestures can also be defined. By comparing the motion detected in the library, a specific gesture associated with the detected position and / or motion can be determined.

  At block 1110, the gesture is interpreted as user input that can be processed by the computer system. The specific processing typically depends on the application programs currently running on the computer system and how they are configured to respond to specific inputs of these programs. For example, a tap in a browser program can be interpreted as a link selection indicated by a finger. A tap in a document processing program can be interpreted as placing a cursor at a position indicated by a finger or as a selection of menu items or other graphic control elements that can be seen on the screen. The specific gesture and interpretation can be determined at the level of the operating system and / or the required application, and no specific interpretation of any gesture is necessary.

  Whole body motion can be captured and used for similar purposes. In such an embodiment, analysis and reconstruction is conveniently performed in approximately real time (eg, time comparable to human reaction time) so that the user experiences natural interactions with the device. In other applications, motion capture may be used for digital rendering that is not performed in real time (eg, computer animated movies, etc.). In such cases, the analysis can take the required length.

  The embodiments described herein provide an efficient distinction between objects and background in a captured image by utilizing a decrease in light intensity as a function of distance. Illuminating an object brightly using one or more light sources that are significantly closer to the object than the background (eg, twice or more) can increase the contrast between the object and the background. In some examples, a filter may be used to remove light from light sources other than the intended light source. Using infrared light can reduce unwanted "noise" and possibly bright spots from visible light sources that are present in the environment where the image is captured, and the user (will not be able to see the infrared light) It can also reduce distraction of people.

  The embodiment described above comprises two light sources, one arranged on each side of the camera used to take an image of the object of interest. This arrangement can be particularly useful when the position and motion analysis relies on object edge information viewed from the respective cameras and the light source illuminates those edges. However, other arrangements can be used. For example, FIG. 12 shows a system 1200 having a single camera 1202 and two light sources 1204, 1206 disposed on either side of the camera 1202. This arrangement can be used to capture an image of the object 1208 and the object 1208 is shaded against the flat background region 1210. In this embodiment, the target pixel and the background pixel can be easily distinguished. Furthermore, the background 1210 is not so far from the object 1208, but is still provided with sufficient contrast to allow discrimination between both shadowed and non-shadowed background area pixels. ing. The algorithm for detecting the position and motion using the image of the object and its shadow is described in the aforementioned 13/414485 application, and the system 1200 uses the input information including the position of the object and its shadow edge as its input information. Can be given to such an algorithm.

  A single camera implementation 1200 may benefit from including a holographic diffraction grating 1215 disposed in front of the lens of the camera 1202. The grating 1215 creates a fringe pattern that appears as a ghost silhouette and / or a tangent to the object 1208. In particular, when separable (ie, when the overlap is not excessive), these patterns have a high contrast that facilitates distinguishing objects from the background. See, for example, the Grating Handbook (Newport Corporation, January 2005, available at http://gratings.newport.com/library/handbook/handbook.asp), the entire disclosure of which is incorporated herein by reference. Incorporated.

  FIG. 13 shows another system 1300 having two cameras 1302, 1304 and one light source 1306 disposed between the cameras. System 1300 may capture an image of object 1308 against background 1310. System 1300 is generally less reliable for edge illumination than system 100 of FIG. However, the algorithm for determining all positions and motions does not rely on accurate information on the edges of the object. Thus, the system 1300 can use an edge-based algorithm, for example, in situations where less accuracy is required. In system 1300, non-edge based algorithms may also be used.

  Although the present invention has been described with respect to particular embodiments, those skilled in the art will recognize that many variations are possible. The number and arrangement of cameras and light sources can be varied. Camera capabilities including frame rate, spatial resolution, and intensity resolution can also be varied as needed. The light source can operate in continuous or pulsed mode. The system described herein provides an image with improved contrast between the two to facilitate the distinction between object and background, and this information can be used for a number of purposes, such as position and / or Motion detection is just one of many possibilities.

Cut-off thresholds and other specific criteria for identifying objects from the background can be adapted to specific cameras and specific environments. As mentioned above, the contrast is expected to increase as the ratio r _B / r _O increases. In some embodiments, the system can be calibrated to a particular environment, for example, by adjusting light source brightness, threshold criteria, and the like. The use of simple criteria that can be implemented in a fast algorithm can free up processing power in a given system for other applications.

  Any type of object can be targeted for motion capture using these techniques, and various aspects of the implementation can be optimized for a particular object. For example, the type and position of the camera and / or light source may be optimized based on the size of the object for which motion is to be captured and / or the size of the space for which motion is to be captured. The analysis techniques according to embodiments of the present invention can be implemented as any suitable computer language algorithm and run on a programmable processor. Alternatively, some or all of the algorithms can be implemented in fixed function logic circuits, and such circuits can be designed and manufactured using conventional or other tools.

  A computer program containing various features of the present invention may be encoded on various computer readable storage media. Suitable media include optical storage media such as magnetic disks or tapes, compact disks (CDs) or DVDs (digital versatile discs), flash memory and any other non-transitory media that holds data in a computer readable form. Etc. A computer readable storage medium encoded with the program code may be provided separately from the package or other device together with a compatible device. Further, the program code may be transmitted over an optical wired and / or wireless network that complies with various protocols (eg, including the Internet that can be distributed via Internet download).

  Although the invention has been described with reference to specific embodiments, it will be understood that the invention is intended to cover all modifications and equivalents within the scope of the following claims.

Claims (24)

An image imaging analysis system for identifying a target object from a digitally displayed image scene,
At least one camera directed at the field of view;
At least one light source disposed on the same field side as the camera and directed to illuminate the field;
An image analyzer coupled to the camera and at least one of the light sources;
The image analysis device includes:
Operating at least one of the cameras to capture a series of images including a first image captured at the same time as the at least one light source illuminates the field of view;
Identify pixels corresponding to the object, not the background,
Based on the identified pixels, it is configured to construct a 3D model of the object including the position and shape of the object and geometrically determine whether it corresponds to the object of interest. system. The image analysis device includes: (i) a foreground image component corresponding to the object located in a proximity region of the visual field; and (ii) a background image component corresponding to the object located in a remote region of the visual field. Is to distinguish
The proximity region extends from at least one of the cameras and has a depth that is at least twice the predicted maximum distance between at least one of the cameras and the object corresponding to the foreground image component;
The system according to claim 1, wherein the remote area is located beyond the proximity area with respect to at least one of the cameras.   The system of claim 2, wherein the proximity region has a depth that is at least four times the predicted maximum distance.   The system of claim 1, wherein the at least one light source is a diffuse emitter.   The system of claim 4, wherein at least one of the light sources is an infrared light emitting diode and at least one of the cameras is an infrared sensitive camera.   The system of claim 1, wherein at least two of the light sources are adjacent to at least one of the cameras and are substantially in the same plane.   The system of claim 1, wherein at least one of the cameras and at least one of the light sources are oriented vertically upward.   The at least one camera is operated to have an exposure time as high as 100 microseconds, and the at least one light source is driven at a power level of at least 5 watts during the exposure time. System.   The system of claim 1, further comprising a holographic diffraction grating disposed between at least one camera lens and the field of view. The image analysis apparatus operates at least one of the cameras when at least one of the light sources is not illuminating the field of view to capture the second and third images, and the difference between the first and second images. Identifying a pixel corresponding to the object based on the difference between the first and third images;
The system according to claim 1, wherein the second image is captured before the first image, and the third image is captured after the second image. A method of image capture analysis, the driving of at least one light source for illuminating a field of view including a target object;
Taking a series of digital images of the field of view by using a camera simultaneously with driving at least one of the light sources;
Identifying the pixels corresponding to the object rather than the background, and
A method of constructing a 3D model of the object including the position and shape of the object based on the identified pixels and geometrically determining whether it corresponds to the object of interest. At least one of the light sources is arranged such that the object of interest is located within a proximity region;
The method of claim 11, wherein the proximity region extends from the camera to a distance that is at least twice a predicted maximum distance between the camera and the object of interest.   The method of claim 12, wherein the proximity region has a depth that is at least four times the predicted maximum distance.   The method of claim 11, wherein at least one of the light sources is a diffuse emitter.   The method of claim 11, wherein at least one of the light sources is an infrared light emitting diode and the camera is an infrared sensitive camera.   12. The method of claim 11, wherein two light sources that are adjacent to and substantially in the same plane as the camera are driven.   The method of claim 11, wherein the camera and the at least one light source are oriented vertically upward. A first image when at least one light source is not driven, a second image when at least one light source is driven, and a third image when at least one light source is not driven. Further comprising imaging of
The method of claim 11, wherein a pixel corresponding to the object is identified based on a difference between the second and first images and a difference between the second and third images. A method for positioning a round object in a digital image, driving at least one light source that illuminates a field of view including the object of interest;
Operation of the camera to capture a series of images, including a first image captured at the same time as the at least one light source illuminates the field of view;
Analysis of the image to detect a Gaussian luminance decay pattern indicating a round object in the field of view;
A method comprising the steps of:   The method of claim 19, wherein the round object is detected without identifying its edges.   The method of claim 19, further comprising tracking movement of the round object detected through a plurality of captured images. An image capturing and analyzing system for positioning a round object in a field of view,
At least one camera directed at the field of view;
At least one light source disposed on the same field side as the camera and directed to illuminate the field;
An image analyzer coupled to the camera and at least one of the light sources;
The image analysis device includes:
Operating at least one of the cameras to capture a series of images including a first image captured at the same time as the at least one light source illuminates the field of view;
A system configured to analyze the image to detect a Gaussian luminance decay pattern indicative of a round object in the field of view.   24. The system of claim 22, wherein the round object is detected without identifying its edges.   23. The system of claim 22, further tracking the movement of the round object detected through a plurality of captured images.

Images