WO2018165148A1

WO2018165148A1 - Foreground object detection

Info

Publication number: WO2018165148A1
Application number: PCT/US2018/021132
Authority: WO
Inventors: Gautam Singh; Glenn E. Casner; Robert W. Shannon; Jonathan D. Gandrud; Blake R. Shamla; Alexander P. DOMEK
Original assignee: 3M Innovative Properties Company
Priority date: 2017-03-06
Filing date: 2018-03-06
Publication date: 2018-09-13

Abstract

This disclosure is directed to techniques for performing image segmentation that include capturing an image of an illuminated object, and applying an object region identification algorithm to a difference image based on the image of the illuminated object. In one example, a method includes receiving, by a computing device, a first image of an environment; receiving, by the computing device, a second image of the environment, wherein the second image includes an object not present in the first image of the environment; generating, by the computing device, a difference image from the first image and the second image; identifying, by the computing device, an object region of the difference image; generating, by the computing device and based on the object region of the difference image, a mask; and applying, by the computing device, the mask to the second image of the environment to generate an image of the object.

Description

FOREGROUND OBJECT DETECTION

TECHNICAL FIELD

[0001] This disclosure relates to image processing, and, more specifically, to image segmentation.

BACKGROUND

[0002] Fingerprints, palm prints, or other biometric data may be used by a variety of government, commercial, residential or industrial entities for verifying the identity of an individual. Example applications include identifying individuals at border locations, confirming an identity for purposes of access control or authentication, or identifying an unknown individual during a forensic investigation. Some techniques for capturing fingerprints are time-consuming and inefficient, and may require special equipment or materials. Further, some techniques require physical contact by a hand or finger on a surface used by many people, thereby potentially spreading contagions.

SUMMARY

[0003] This disclosure is directed to techniques, devices, and systems related to performing image processing. For example, a technique may include capturing an image of an illuminated object and applying an object region identification algorithm to a difference image derived, in part, from the captured image of the illuminated object. In accordance with one or more aspects of the present disclosure, a computing system may be configured to, in some examples, capture two images of an environment, one image with an object included in the environment, and one image with the object not present. The computing system may generate a difference image from the two images. The computing system may further process the difference image by applying a median filter, performing an object region identification algorithm, and/or carrying out other operations to reduce or eliminate noise in the resulting image. The resulting image may be used as, or as the basis for, a binary mask. The computing system may apply the binary mask to the image of the illuminated object to perform foreground segmentation and/or background subtraction.

[0004] In some examples, systems or techniques in accordance with one or more aspects of the present disclosure may be used to capture fingerprints in a non-contact fingerprint capture system. For instance, a fingerprint capture system may capture an image of an environment that includes an illuminated hand, and the system may capture another image of the environment without the hand. The system may process the two images to carry out segmentation of the hand (or fingers of the hand) to isolate the hand from other objects of the environment that would have otherwise also been present in the image. The system may analyze the resulting image to process the fingerprints included within the image.

[0005] The details of one or more examples of the disclosure are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the disclosure will be apparent from the description and drawings, and from the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

[0006] FIG. 1 is a conceptual diagram illustrating an example computing device configured to perform image segmentation, in accordance with one or more aspects of the present disclosure.

[0007] FIG. 2 is a block diagram illustrating an example computing device configured to perform image segmentation, in accordance with one or more aspects of the present disclosure.

[0008] FIG. 3 is a conceptual diagram illustrating example images resulting from respective image processing operations performed by an example computing device, in accordance with one or more aspects of the present disclosure.

[0009] FIG. 4 is a conceptual diagram illustrating application of an example binary mask to an example object image by an example computing device, in accordance with one or more aspects of the present disclosure.

[0010] FIG. 5 is a flow diagram illustrating example operations performed by an example computing device, in accordance with one or more aspects of the present disclosure.

DETAILED DESCRIPTION

[0011] Techniques of disclosure include techniques relating to image processing that, in some examples, may be used to capture fingerprints. Historically, systems for capturing fingerprints rely on physical contact between a subject's hands and a glass or other surface. Systems in which physical contact is required tend to be less efficient, and collecting fingerprints from many people may be slow and time-consuming with low throughput. Further, systems in which the same surface is used to collect fingerprints from many people may result in the spread of contagions. Accordingly, a system in which fingerprints can be collected without contact may be more efficient, and also may reduce the risk of spreading contagions.

[0012] In accordance with one or more aspects of the present disclosure, an algorithm is disclosed for performing an object region identification algorithm on a difference image, and/or performing segmentation of fingers by eliminating most of the irrelevant background pixels in an image. By eliminating irrelevant background pixels in an image, segmentation of finger pixels and background subtraction may be performed simultaneously. [0013] By performing an object region identification algorithm on a difference image, a more effective and/or accurate mask can be generated that can be applied to an image of an object. By applying a more effective and/or accurate mask, a computing device may more accurately identify, detect, and/or segment one or more objects in the image, and may do so without requiring physical contact, or the use of a screen, lid, or other object to eliminate background objects.

[0014] Further, by capturing an image of an object while that object is illuminated by a light source, a computing device may more accurately perform an object region identification algorithm on the resulting difference image, because using an image with the object illuminated creates a greater contrast between the object and any items in the background. By creating a greater contrast, the difference image exhibits characteristics that enable the object region identification algorithm to more accurately identify the object.

[0015] FIG. 1 is a conceptual diagram illustrating an example computing device for performing image segmentation, in accordance with one or more aspects of the present disclosure. In the example of FIG. 1, computing device 200 includes one or more cameras 212 and one or more processing modules 230. Computing device 200 may be configured to capture and process one or more images of environment 105. In the example of FIG. 1, such images may include non-object image 110 and object image 120. In FIG. 1, object image 120 includes object 106, which in the examples shown herein, corresponds to fingers from a hand. The functionality of computing device 200 may be implemented in hardware or in a combination of software and hardware, where requisite hardware may be provided to store and execute software instructions. Computing device 200 may include a variety of devices configured for processing and/or manipulating data. In one example, computing device 200 may be incorporated into an apparatus for capturing and/or processing fingerprints.

[0016] While illustrated as a single computing device 200 for purposes of illustration, it should be understood that computing device 200 may include a plurality of individual components. For example, cameras 212 may be included in a device that is separate from the device that includes or implements processing module 230. In some examples, processing module 230 may be housed in a portable computing device, workstation, or even networked server that receives images (e.g., data) from one or more cameras 212. Moreover, many other examples of computing device 200 may be used in other instances and may include a subset of the components included in the example computing device 200 or may include additional components not shown in the example computing device 200 of FIG. 1.

[0017] One or more cameras 212 may refer to any appropriate type of image acquisition device, such as a camera. Camera 212 may be configured to capture images within a field of view of the camera, which may correspond to or be a part of environment 105. In some examples, the field of view of camera 212 may extend upward, as shown in FIG. 1, so that an image of a hand may be captured by positioning the hand above camera 212 of computing device 200. Camera 212 may operate in conjunction with a light source, so that objects within the field of view of camera 212 may be illuminated while an image is being captured. Camera 212 may include an array of sensing elements that detect and/or convey light, images, movement, or other information.

Camera 212 may include one or more infrared cameras, thermographic cameras, thermal imaging cameras, light-sensitive cameras, range sensors, tomography devices, radar devices, or ultrasonic cameras. In some examples, one or more cameras 212 may include, but are not limited to, arrays of charge-coupled devices (CCD), active pixel sensors in complementary metal-oxide- semiconductor (CMOS) devices, N-type metal-oxide-semiconductor technologies, or other sensing elements. Any appropriate device whether now known or hereafter devised that is capable of detecting and/or conveying information constituting an image, sequence of images, or a video may appropriately serve as camera 212.

[0018] Processing module 230 may perform functions relating to image processing, including image capture, mask creation, mask application or processing, and image segmentation and/or foreground detection. Processing module 230 may include hardware, such as processing circuitry, and/or software modules configured to perform the processes described herein. In some examples, processing module 230 may control and/or interact with camera 212 to capture and store images within a field of view of camera 212. Processing module 230 may cause camera 212 to capture an image of an environment that includes an illuminated object, and processing module 230 may further cause camera 212 to capture an image of the same environment without the object.

Processing module 230 may perform image processing operations, such as generating a difference image from two images captured by camera 212. Processing module 230 may apply an edge detection algorithm, a thresholding algorithm, or other object region identification algorithm to the difference image, and also apply various filters to generate a binary mask that can be used to identify, isolate, or segment the object within the image. Processing module 230 may perform operations described herein using software, hardware, firmware, or a mixture of hardware, software, and firmware residing in and/or executing at one or more computing devices. Processing module 230 may be implemented in various ways. For example, processing module 230 may be implemented as a software application, as downloadable or pre-installed application or "app," as a system service of computing device 200, or as part of an operating system executing on computing device 200.

[0019] In the example of FIG. 1, and in accordance with one or more aspects of the present disclosure, computing device 200 may capture multiple images of environment 105. For example, environment 105 may correspond to the field of view of camera 212 and/or the physical area capable of being captured by camera 212. Camera 212 of computing device 200 may capture non- object image 110 of environment 105. After camera 212 has captured non-object image 110, an object (e.g., object 106) may move into the field of view of camera 212 within environment 105. In the example of FIG. 1, object 106 is a hand or a portion of a hand. When the hand is within environment 105, camera 212 of computing device 200 may capture object image 120.

Accordingly, non-object image 110 is an image of environment 105 without the object, and object image 120 is an image of environment 105 with the object. Although non-object image 110 is described as being captured by computing device 200 before object image 120, in other examples, object image 120 may be captured by computing device 200 before non-object image 110.

[0020] Computing device 200 may process non-object image 110 and object image 120. For example, processing module 230 of computing device 200 may generate a difference image based on non-object image 110 and object image 120. In some examples, generating a difference image may involve processing module 230 subtracting the pixel values of object image 120 from non- object image 110. In such an example, processing module 230 may compute the absolute differences, on a pixel-wise or pixel-by-pixel basis, between non-object image 110 and object image 120, converting one or more negative values to a corresponding absolute value. For pixels that are the same in both non-object image 110 and object image 120, the resulting difference image may have a corresponding pixel with a value of zero. For pixels that are not the same, the resulting difference image may have a corresponding pixel with a value equal to the absolute difference of the values of the pixels within non-object image 110 and object image 120.

[0021] Processing module 230 may, prior to performing the image differencing process, align non-object image 110 and object image 120. For example, for both non-object image 110 and object image 120, camera 212 may capture the same environment 105, so that the presence of the object within object image 120 accounts for primary difference between non-object image 110 and object image 120. However, in some cases, camera 212 may move between the time that non- object image 110 and object image 120 are captured. Accordingly, processing module 230 may analyze non-object image 110 and object image 120 and adjust for any movement of camera 212 that may have occurred between the times that non-object image 110 and object image 120 were captured. This process may involve identifying a common object (e.g., group or pattern of similar pixels) between the non-object image 110 and object image 120. In some examples, processing module 230 may determine an offset that may be applied to coordinates within object image 120 to align object image 120 with coordinates of non-object image 110. Alternatively, processing module 230 may determine an offset that may be applied to coordinates within non-object image 110 to alight non-object image 110 with coordinates of object image 120. [0022] Processing module 230 may generate a filtered difference image. For example, processing module 230 may, in some examples, apply a median filter to the difference image to reduce or eliminate noise in the difference image. The median filter may operate by replacing the value of each pixel in the difference image with the median of neighboring pixels. As a result of creating the difference image and applying a median filter to the resulting difference image, processing module 230 may generate median-filtered difference image 130, as illustrated in FIG. 1.

[0023] Processing module 230 may process median-filtered difference image 130 to obtain the foreground of object image 120 and isolate the hand within object image 120. For example, processing module 230 may apply an object region identification algorithm to median-filtered difference image 130. In some examples, the object region identification algorithm may comprise an edge detection algorithm. In other examples, the object region identification algorithm may comprise a thresholding algorithm. Processing module 230 may apply a sum filter to the image resulting from the object region identification algorithm or process, refining the image resulting from the object region identification process. Processing module 230 may further perform morphological operations, such as hole filling and image closing, to the image resulting from application of the sum filter. Processing module 230 may, as a result of such operations, generate binary mask 150. Processing module 230 may obtain the foreground of object image 120 by applying binary mask 150 to object image 120, thereby isolating the foreground (i.e., performing foreground segmentation). In the example of FIG. 1, the foreground corresponds to the hand shown within object image 120.

[0024] By performing an object region identification or edge detection algorithm to a difference image, computing device 200 may be able to generate a more effective and/or accurate mask that can be applied to an image of an object. By applying a more effective and/or accurate mask, computing device 200 may more accurately identify, detect, and/or segment one or more objects in the image, and may do so without requiring the use of a screen, lid, or other object to eliminate background objects. Therefore, aspects of this disclosure may improve the function of computing device 200 because performing an appropriate object region identification algorithm to a difference image may have the effect of more accurately identifying, detecting, and/or segmenting an object in the image, without physically screening background objects when the image is captured.

[0025] Further, by capturing an image while illuminating the object in the field of view of camera 212, computing device 200 may more accurately perform an object region identification or edge detection algorithm on the difference image, because using an image with the object illuminated creates a greater contrast between the object in the object image 120 and any items in the background of non-object image 110 for the difference image. By creating a greater contrast, the difference image exhibits characteristics that enable the object region identification or edge detection algorithm to more accurately identify the object. Therefore, aspects of this disclosure may improve the function of computing device 200 because performing object region

identification or edge detection more accurately may result in a more accurate binary mask, and a more accurate binary mask may lead to more accurate foreground segmentation.

[0026] Further, by applying one or more aspects of the present disclosure, computing device 200 may be used to advantageously collect images of fingerprints. In such a system, computing device 200 may collect fingerprint images without requiring physical contact between a finger and another surface. Systems in which physical contact is required tend to be less efficient, so collecting fingerprints from many people may be slow and time-consuming. Further, systems in which the same surface is used to collect fingerprints from many people may result in the spread of contagions. Accordingly, a system in which fingerprints can be collected without contact may be more efficient, and also may reduce the risk of spreading contagions.

[0027] FIG. 2 is a block diagram illustrating an example computing device for performing image segmentation, in accordance with one or more aspects of the present disclosure. Computing device 200 of FIG. 2 may be described as an example or alternate implementation of computing device 200 of FIG. 1. One or more aspects of FIG. 2 may be described herein within the context of computing device 200 of FIG. 1. FIG. 2 illustrates one example or alternate implementation of computing device 200. Other example or alternate implementations of computing device 200 may be appropriate in other instances. Such implementations may include a subset of the devices and/or components included in the example of FIG. 2 and/or may include additional devices and/or components not shown in FIG. 2.

[0028] Computing device 200 may be implemented in a number of different ways. In some examples, one or more devices of computing device 200 that are illustrated as separate devices may be implemented as a single device; one or more components of computing device 200 that are illustrated as separate components may be implemented as a single component. Also, in some examples, one or more devices of computing device 200 that are illustrated as a single device may be implemented as multiple devices; one or more components of computing device 200 that are illustrated as a single component may be implemented as multiple components. Each of the multiple devices may be directly coupled via wired or wireless communication and/or remotely coupled via one or more networks. Further, one or more devices or components of computing device 200 that are illustrated in FIG. 2 may be implemented as part of another device or component not shown in FIG. 2. In this manner, some of the functions described herein may be performed via distributed processing by two or more devices. [0029] Computing device 200 may include power source 201, one or more processors 203, one or more interfaces 205, one or more input/output devices 207, one or more user interface devices 211, one or more cameras 212, one or more light sources 214, one or more sensors 216, one or more displays 218, and one or more storage devices 220. Storage device 220 may include user interface module 221, images data store 222, and processing module 230. Processing module 230 may include image capture module 232, mask creation module 234, and foreground detection module 236. One or more of the devices, modules, storage areas, or other components of computing device 200 may be interconnected to enable inter-component communications (physically, communicatively, and/or operatively). In some examples, such connectivity may be provided by through communication channels (e.g., communication channels 202), a system bus, a network connection, an inter-process communication data structure, or any other method for

communicating data.

[0030] One or more cameras 212 may be implemented in a manner consistent with the description provided in connection with FIG. 1, and may include the same capabilities as described in connection with FIG. 1. In other examples, camera 212 of FIG. 2 may be implemented in a manner different than that described in connection with FIG. 1. For example, camera 212 of FIG. 2 may include more capabilities than those described in connection with FIG. 1, less capabilities than those described in connection with FIG. 1, or capabilities different than those described in connection with FIG. 1.

[0031] Power source 201 may provide power to one or more components of computing device 200. Power source 201 may be a battery. In other examples, power source 201 may receive power from the primary alternative current (AC) power supply in a building, home, or other location. In still further examples, computing device 200 and/or power source 201 may receive power from another source. One or more of the devices or components illustrated within computing device 200 may be connected to power source 201, and/or may receive power from power source 201. Power source 201 may have intelligent power management or consumption capabilities, and may such features may be controlled, accessed, or adjusted by one or more modules of computing device 200 and/or by one or more processors 203 to intelligently consume, allocate, supply, or otherwise manage power.

[0032] One or more processors 203 of computing device 200 may implement functionality and/or execute instructions associated with computing device 200 or associated with one or more modules illustrated herein and/or described below. Examples of processors 203 include processing circuitry such as microprocessors, application processors, display controllers, auxiliary processors, one or more sensor hubs, and any other hardware configured to function as a processor, a processing unit, or a processing device. Computing device 200 may use one or more processors 203 to perform operations in accordance with one or more aspects of the present disclosure using software, hardware, firmware, or a mixture of hardware, software, and firmware residing in and/or executing at computing device 200.

[0033] One or more interfaces 205 of computing device 200 may communicate with devices external to computing device 200 by transmitting and/or receiving data, and may operate, in some respects, as both an input device and an output device. In some examples, interface 205 may communicate with other devices over a network. In other examples, interfaces 205 may send and/or receive radio signals on a radio network such as a cellular radio network. In other examples, interfaces 205 of computing device 200 may transmit and/or receive satellite signals on a satellite network such as a Global Positioning System (GPS) network. Examples of interfaces 205 include a network interface card (e.g. such as an Ethernet card), an optical transceiver, a radio frequency transceiver, a GPS receiver, or any other type of device that can send and/or receive information. Other examples of interfaces 205 may include devices capable of communicating over Bluetooth®, GPS, NFC, ZigBee, and cellular networks (e.g., 3G, 4G, 5G), and Wi-Fi® radios found in mobile devices as well as Universal Serial Bus (USB) controllers and the like. Such communications may adhere to, implement, or abide by appropriate protocols, including

Transmission Control Protocol/Internet Protocol (TCP/IP), Ethernet, Bluetooth, NFC, or other technologies or protocols.

[0034] One or more input/output devices 207 may represent any input or output devices of computing device 200 not otherwise separately described herein. One or more input/output devices 207 may generate, receive, and/or process input from any type of device capable of detecting input from a human or machine. For example, one or more input/output devices 207 may generate, receive, and/or process input in the form of physical, audio, image, and/or visual input (e.g., keyboard, microphone, camera). One or more input/output devices 207 may generate, present, and/or process output through any type of device capable of producing output. For example, one or more input/output devices 207 may generate, present, and/or process output in the form of tactile, audio, visual, and/or video output (e.g., haptic response, sound, flash of light, and/or images). Some devices may serve as input devices, some devices may serve as output devices, and some devices may serve as both input and output devices.

[0035] User interface device 211 may function as an input and/or output device or set of input/output devices for computing device 200, and may be implemented using various devices, components, and/or technologies. User interface device 211 may include one or more keypads or keyboards, one or more pointing devices, presence-sensitive input panel technologies, microphone technologies, voice activation and/or recognition technologies, cameras, sensor technologies (e.g., infrared, image, location, motion, accelerometer, gyrometer, magnetometer), or other input device technology for use in receiving user input. User interface device 211 may include display devices, speaker technologies, haptic feedback technologies, tactile feedback technologies, light emitting technologies, or other output device technologies for use in outputting information to a user. In some examples, one or more components of user interface device 211 may be integrated together, so that one component is or appears to be a built-in component of another. For example, a display device and a presence-sensitive panel may be used together to implement a touch-screen display.

[0036] Light source 214 may include light emitting diodes (LEDs) or other components for illuminating objects prior to capturing images of the objects with camera 212. In some examples, light source 214 may illuminate a hand or a portion of the hand when an image of the hand is being captured by camera 212. Human skin has been found to have higher reflectivity in the green and blue portions of the visible light spectrum, thus emitting light with wavelengths peaking in the blue and green portions of the visible light spectrum can help to more clearly illuminate details of a user's hand (e.g., a friction ridge surface of a user's hand). Accordingly, in some examples, light source 214 may include LEDs that emit light peaking in the blue wavelength (e.g., in the range of 440 to 570 nanometers (nm)).

[0037] Light emitted by light source 214 may be of varying power levels. Light source 214 may be, in some instances, paired with one or more light guides to direct the output of light source 214 in a uniform manner. In one instance, light guides are made of a polycarbonate tube lined with enhanced specular reflector (ESR) film and a turning film. In some instances, the light guides may collimate the emitted light so that each ray is parallel, reducing light scattering and undesired reflections.

[0038] One or more sensors 216 may include any of a variety of types of sensors. For instance, sensor 216 may include motion sensors capable of detecting movement of an object near computing device 200. In some examples, sensors 216 may be capable of determining whether an object has moved within a field of view of camera 212, or determining whether an object has moved out of a field of view of camera 212. Such sensors 216 may include proximity sensors that generate proximity data. Proximity data may indicate whether an object is within proximity to sensor 216. In some examples, proximity data may indicate how close an object is to sensor 216.

[0039] One or more displays 218 may refer to any appropriate type of display device, such as a display associated with any type of computing device, such as a desktop computer, laptop computer, tablet, mobile phone, watch, or any other type of wearable, non-wearable, mobile, or non-mobile computing device. Display 218 may be used for displaying images captured by camera 212, or for displaying images derived from those captured by camera 212. Display 218 may function as one or more output (e.g., display) devices using technologies including liquid crystal displays (LCD), dot matrix displays, light emitting diode (LED) displays, organic light- emitting diode (OLED) displays, e-ink, or similar monochrome or color displays capable of generating tactile, audio, and/or visual output. Display 218 may include a cathode ray tube (CRT) monitor, liquid crystal display (LCD), Light-Emitting Diode (LED) display, or any other type of display device.

[0040] One or more storage devices 220 within computing device 200 may store information for processing during operation of computing device 200. Storage devices 220 may store program instructions and/or data associated with one or more of the modules described in accordance with one or more aspects of this disclosure. One or more processors 203 and one or more storage devices 220 may provide an operating environment or platform for such modules, which may be implemented as software, but may in some examples include any combination of hardware, firmware, and software. One or more processors 203 may execute instructions and one or more storage devices 220 may store instructions and/or data of one or more modules. The combination of processors 203 and storage devices 220 may retrieve, store, and/or execute the instructions and/or data of one or more applications, modules, or software. Processors 203 and/or storage devices 220 may also be operably coupled to one or more other software and/or hardware components, including, but not limited to, one or more of the components of computing device 200 and/or one or more devices or systems illustrated as being connected to computing device 200.

[0041] In some examples, one or more storage devices 220 are temporary memories, meaning that a primary purpose of the one or more storage devices is not long-term storage. Storage devices 220 of computing device 200 may be configured for short-term storage of information as volatile memory and therefore not retain stored contents if deactivated. Examples of volatile memories include random access memories (RAM), dynamic random access memories (DRAM), static random access memories (SRAM), and other forms of volatile memories known in the art. Storage devices 220, in some examples, also include one or more computer-readable storage media. Storage devices 220 may be configured to store larger amounts of information than volatile memory. Storage devices 220 may further be configured for long-term storage of information as non-volatile memory space and retain information after activate/off cycles.

Examples of non-volatile memories include magnetic hard disks, optical discs, floppy disks, Flash memories, or forms of electrically programmable memories (EPROM) or electrically erasable and programmable (EEPROM) memories.

[0042] User interface module 221 may manage user interactions with user interface device 211 and other components of computing device 200. User interface module 221 may cause user interface device 211 to output various user interfaces for display or presentation or otherwise, as a user of computing device 200 views, hears, or otherwise senses output and/or provides input at user interface device 211. User interface device 211 may detect input, and may output to user interface module 221 one or more indications of input as a user of computing device 200 interacts with a user interface presented at user interface device 211. User interface module 221 and user interface device 211 may interpret inputs detected at user interface device 211 and may relay information about the inputs detected at user interface device 211 to one or more associated platforms, operating systems, applications, and/or services executing at computing device 200 to cause computing device 200 to perform one or more functions.

[0043] User interface module 221 may receive information and instructions from a platform, operating system, application, and/or service executing at computing device 200 and/or one or more remote computing systems. In addition, user interface module 221 may act as an intermediary between a platform, operating system, application, and/or service executing at computing device 200 and various output devices of computing device 200 (e.g., speakers, LED indicators, audio or electrostatic haptic output devices, light emitting technologies, displays, etc.) to produce output (e.g., a graphic, a flash of light, a sound, a haptic response, etc.). User interface module 221 may perform operations described using software, hardware, firmware, or a mixture of hardware, software, and firmware residing in and/or executing at computing device 200.

Computing device 200 may execute user interface module 221 with multiple processors or multiple devices, as a virtual machine executing on underlying hardware, as one or more services of an operating system or computing platform, and/or as one or more executable programs at an application layer of a computing platform.

[0044] Images data store 222 may include information captured by camera 212, including images, videos, or other information. In some examples, images data store 222 may include processed images derived from images captured by camera 212 or derived from other processed images. Images data store 222 may be created or updated by image capture module 232, mask creation module 234, and/or foreground detection module 236. Image capture module 232 may store captured images within images data store 222. Mask creation module 234 may store processed images within images data store 222. Foreground detection module 236 may access both captured images within images data store 222 and processed images within images data store 222, and generate an image with the foreground segmented or isolated from the background of an image.

[0045] Image capture module 232 may perform functions relating to controlling and/or interacting with camera 212 to capturing images within a field of view of camera 212. Image capture module 232 may receive information about input detected by user interface device 211, and image capture module 232 may determine that the input corresponds to a request to capture an image. Image capture module 232 may cause light source 214 to illuminate a field of view for camera 212. Image capture module 232 may cause camera 212 to capture multiple images, such as, for example, an image of an environment that includes a hand, and an image of the same environment without the hand. Image capture module 232 store images within images data store 222. Image capture module 232 may also receive indications of input from sensor 216 and determine that an object has moved within or moved outside of a field of view of camera 212. Image capture module 232 may interact with and/or operate in conjunction with one or more modules of device 200.

[0046] Mask creation module 234 may perform functions relating to processing images captured by camera 212. For example, mask creation module 234 of computing device 200 may generate a difference image from two images captured by camera 212. Mask creation module 234 may also apply various filters, including a median filter and a sum filter, to images captured by camera 212 or to processed images. The median filter may operate by replacing the value of each pixel in the difference image with the median of neighboring pixels. The sum filter may operate to refine an image and reduce or eliminate some of holes along blob boundaries present in the image. Mask creation module 234 may apply an edge detection algorithm to an image, including to a difference image. Mask creation module 234 may apply one or more filters to images captured by camera 212 or to processed images to reduce or eliminate holes along blob boundaries present in one or more images. Mask creation module 234 may apply morphological operations like contour hole filling and image closing.

[0047] Foreground detection module 236 may perform functions relating to detecting or isolating the foreground of an image captured by camera 212 and/or an image processed by mask creation module 234. In some examples, foreground detection module 236 may apply a binary mask to an object image to perform foreground segmentation. In some examples, such an operation may result in an image in which a foreground object is segmented from other parts of the image.

Foreground detection module 236 may present the image at a display, or communicate the image to one or more other devices.

[0048] With reference to FIG. 1 and FIG. 2, and in accordance with one or more aspects of the present disclosure, computing device 200 may capture non-object image 110 and object image 120 in response to user input. For example, user interface device 211 may detect input and output to user interface module 221 an indication of input. User interface module 221 may output to image capture module 232 information about the input. Image capture module 232 may determine that the input corresponds to a request to capture an image. Image capture module 232 may cause light source 214 to illuminate environment 105 and cause camera 212 to capture an image of environment 105 (e.g., non-object image 110). Image capture module 232 may store the image within images data store 222. User interface device 211 may detect additional input, and output to user interface module 221 an indication of the additional input. User interface module 221 may output to image capture module 232 information about the input. Image capture module 232 may determine that the input corresponds to a request to capture a second image. Image capture module 232 may cause light source 214 to illuminate environment 105 and cause camera 212 to capture a second image of environment 105 (e.g., object image 120). Image capture module 232 may store the second image within images data store 222.

[0049] In another example, computing device 200 may automatically capture non-object image 110 and object image 120. For example, sensor 216 may detect input and output to image capture module 232 an indication of input. Image capture module 232 may determine that the input corresponds to an object being moved within the field of view of camera 212. For instance, sensor 216 may detect movement resulting from a hand being moved within the field of view of camera 212. Image capture module 232 may cause light source 214 to illuminate the object within the view of camera 212, and cause camera 212 to capture an image of the object within the field of view of camera 212 (e.g., object image 120). Image capture module 232 may store the image (the "object image") within images data store 222. Sensor 216 may detect further input and output to image capture module 232 an indication of input. Image capture module 232 may determine that the input corresponds to the object being removed from the field of view of camera 212. Image capture module 232 may cause camera 212 to capture a "non-object image" (e.g., non-object image 110) after the object has been removed from the field of view. Image capture module 232 may store the "non-object image" within images data store 222.

[0050] Modules illustrated in FIG. 2 (e.g., processing module 230, image capture module 232, mask creation module 234, and foreground detection module 236) and/or illustrated or described elsewhere in this disclosure may perform operations described using software, hardware, firmware, or a mixture of hardware, software, and firmware residing in and/or executing at one or more computing devices. For example, a computing device may execute one or more of such modules with multiple processors or multiple devices. A computing device may execute one or more of such modules as a virtual machine executing on underlying hardware. One or more of such modules may execute as one or more services of an operating system or computing platform. One or more of such modules may execute as one or more executable programs at an application layer of a computing platform. In other examples, functionality provided by a module could be implemented by a dedicated hardware device. Although certain modules, data stores, components, programs, executables, data items, functional units, and/or other items included within one or more storage devices may be illustrated separately, one or more of such items could be combined and operate as a single module, component, program, executable, data item, or functional unit. For example, one or more modules or data stores may be combined or partially combined so that they operate or provide functionality as a single module. [0051] Further, one or more modules may operate in conjunction with one another so that, for example, one module acts as a service or an extension of another module. Also, each module, data store, component, program, executable, data item, functional unit, or other item illustrated within a storage device may include multiple components, sub-components, modules, sub-modules, data stores, and/or other components or modules or data stores not illustrated. Further, each module, data store, component, program, executable, data item, functional unit, or other item illustrated within a storage device may be implemented in various ways. For example, each module, data store, component, program, executable, data item, functional unit, or other item illustrated within a storage device may be implemented as a downloadable or pre-installed application or "app." In other examples, each module, data store, component, program, executable, data item, functional unit, or other item illustrated within a storage device may be implemented as part of an operating system executed on a computing device.

[0052] FIG. 3 is a conceptual diagram illustrating example image processing operations performed by an example computing device, in accordance with one or more aspects of the present disclosure. One or more aspects of FIG. 3 may be described herein within the context of computing device 200 of FIG. 2. FIG. 3 illustrates one example set of operations performed by computing device 200. Other operations may be appropriate in other implementations. Such implementations may include a subset of the operations included in the example of FIG. 3 and/or may include additional operations not shown in FIG. 3.

[0053] Referring now to FIG. 2 and FIG. 3, computing device 200 may generate median-filtered difference image 130 using the "object image" and the "non-object image" described above in connection with FIG. 1 and FIG. 2. For example, mask creation module 234 of computing device 200 may generate a difference image by computing a pixel-by -pixel or pixel-wise difference between the "object image" (e.g., object image 120 of FIG. 1) and the "non-object image" (e.g., non-object image 110 of FIG. 1). Mask creation module 234 may use a difference value derived from each pair of pixels within each image to generate a pixel for the difference image. In some examples, the "object image" and the "non-object image" of are of substantially the same environment or field of view of camera 212, but in the former image, an object is present within the environment, and in the latter image, the object is absent from the environment. Mask creation module 234 may also apply a median filter to eliminate noise in the difference image.

Accordingly, mask creation module 234 may generate median-filtered difference image 130, as illustrated in FIG. 3, as a result of computing the pixel-wise difference image and applying a median filter to the image.

[0054] Computing device 200 may generate binary mask 150 that can be used to identify an object in the originally captured "object image." For example, mask creation module 234 may apply an edge detection algorithm to median-filtered difference image 130 to generate image 140. Mask creation module 234 may apply a sum filter to image 140, to refine image 140 and reduce or eliminate some of the holes along blob boundaries present in image 140. Mask creation module 234 may generate image 142 as a result of applying the sum filter to image 140. Mask creation module 234 may perform morphological operations on image 142, such as contour hole filling and image closing. Mask creation module 234 may, as a result of such operations, generate image 144. Mask creation module 234 may perform further morphological image closing operations, and eliminate contours with low areas. Mask creation module 234 may, as a result of such operations, generate image 146. Mask creation module 234 may analyze image 146 and identify any holes along boundaries in image 146. Mask creation module 234 may address such areas by closing the mask along the boundary holes and performing contour hole filling. Mask creation module 234 may, as a result of such operations, generate binary mask 150. In some examples, the operations described in FIG. 3 may be performed in a different order and still result in generation of the binary mask 150.

[0055] FIG. 4 is a conceptual diagram illustrating application of an example binary mask to an example object image by an example computing device, in accordance with one or more aspects of the present disclosure. One or more aspects of FIG. 4 may be described herein within the context of computing device 200 of FIG. 1 and FIG. 2. FIG. 4 illustrates one example set of operations performed by computing device 200. Other operations may be appropriate in other instances.

[0056] With reference to FIG. 2 and FIG. 4, computing device 200 may generate object-isolated image 160 by combining object image 120 and binary mask 150. For example, foreground detection module 236 of computing device 200 may apply binary mask 150 to object image 120 to generate object-isolated image 160. In some examples, foreground detection module 236 may perform a bit-wise logical "and" operation to perform background subtraction and isolate the foreground (i.e., the hand) within object image 120. Foreground detection module 236 may, as a result of such operations, generate object-isolated image 160. In some examples, foreground detection module 236 may cause display 218 to present object-isolated image 160 for display. Alternatively, or in addition, foreground detection module 236 may cause interface 205 to transmit or otherwise communicate one or more images, such as object-isolated image 160, to one or more other devices for analysis. For example, one or more other devices may analyze object-isolated image 160 as part of a system for processing and/or analyzing fingerprint data from images such as object-isolated image 160.

[0057] FIG. 5 is a flow diagram illustrating operations performed by an example computing device, in accordance with one or more aspects of the present disclosure. FIG. 5 is described below within the context of computing device 200 of FIG. 2. In other examples, operations described in FIG. 5 may be performed by one or more other components, modules, systems, or devices, such as processor 203 of computer device 200. Further, in other examples, operations described in connection with FIG. 5 may be merged, performed in a difference sequence, or omitted.

[0058] In the example of FIG. 5, and in accordance with one or more aspects of the present disclosure, computing device 200 may capture a first image of an environment (501). For example, image capture module 232 may control camera 212 to capture an image of the field of view of camera 212.

[0059] Computing device 200 may capture a second image of the environment, wherein the second image includes an object not present in the first image of the environment, and wherein capturing the second image includes illuminating the object within the environment (502). For example, sensor 216 of computing device 200 may detect input and output to image capture module 232 an indication of input. Image capture module 232 may determine that the input corresponds to an object being moved within the field of view of camera 212. Image capture module 232 may cause light source 214 to illuminate the object within the view of camera 212, and cause camera 212 to capture an image of the object within the field of view of camera 212.

[0060] Computing device 200 may generate a difference image from the first image and the second image (503). For example, mask creation module 234 may generate an image by computing the pixel-wise difference between the first image and second image.

[0061] Computing device 200 may identify an object region of the difference image (504). For example, mask creation module 234 may apply an edge detection algorithm to the difference image, or to a median-filtered version of the difference image.

[0062] Computing device 200 may generate, based on the object region of the difference image, a mask (505). For example, mask creation module 234 may apply a sum filter to the image created as a result of applying the edge detection algorithm, to refine the image and reduce or eliminate some of the holes that might be present along blob boundaries in the image. Mask creation module 234 may perform morphological operations, such as contour hole filling and image closing, to thereby create a binary mask.

[0063] Computing device 200 may apply the mask to the second image of the environment to generate an image of the object (506). For example, foreground detection module 236 of computing device 200 may apply the binary mask to the second image. In some examples, applying the binary mask may involve a bit-wise logical "and" operation using the mask to perform background subtraction and isolate the foreground (i.e., the hand) within the second image. [0064] For processes, apparatuses, and other examples or illustrations described herein, including in any flowcharts or flow diagrams, certain operations, acts, steps, or events included in any of the techniques described herein can be performed in a different sequence, may be added, merged, or left out altogether (e.g., not all described acts or events are necessary for the practice of the techniques). Moreover, in certain examples, operations, acts, steps, or events may be performed concurrently, e.g., through multi-threaded processing, interrupt processing, or multiple processors, rather than sequentially. Further certain operations, acts, steps, or events may be performed automatically even if not specifically identified as being performed automatically. Also, certain operations, acts, steps, or events described as being performed automatically may be alternatively not performed automatically, but rather, such operations, acts, steps, or events may be, in some examples, performed in response to input or another event.

[0065] In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored, as one or more instructions or code, on and/or transmitted over a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media, which corresponds to a tangible medium such as data storage media, or communication media including any medium that facilitates transfer of a computer program from one place to another (e.g., pursuant to a communication protocol). In this manner, computer-readable media may correspond to (1) tangible computer-readable storage media, which is non-transitory or (2) a communication medium such as a signal or carrier wave. Data storage media may be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, code and/or data structures for implementation of the techniques described in this disclosure. A computer program product may include a computer- readable medium.

[0066] By way of example, and not limitation, such computer-readable storage media can include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage, or other magnetic storage devices, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Also, any connection is properly termed a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. It should be understood, however, that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but are instead directed to non-transient, tangible storage media. Disk and disc, as used, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, where disks usually reproduce data magnetically, while discs reproduce data optically with lasers.

Combinations of the above should also be included within the scope of computer-readable media.

[0067] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Accordingly, the term "processor" or "processing circuitry" as used may refer to any of the foregoing structure or any other structure suitable for implementation of the techniques described. In addition, in some aspects, the functionality described may be provided within dedicated hardware and/or software modules. Also, the techniques could be fully implemented in one or more circuits or logic elements.

[0068] The techniques of this disclosure may be implemented in a wide variety of devices or apparatuses, including a wireless handset, a mobile or non-mobile computing device, a wearable or non-wearable computing device, an integrated circuit (IC) or a set of ICs (e.g., a chip set). Various components, modules, or units are described in this disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, but do not necessarily require realization by different hardware units. Rather, as described above, various units may be combined in a hardware unit or provided by a collection of interoperating hardware units, including one or more processors as described above, in conjunction with suitable software and/or firmware.

Claims

WHAT IS CLAIMED IS:

1. A method comprising:

receiving, by a computing device, a first image of an environment;

receiving, by the computing device, a second image of the environment, wherein the second image includes an object not present in the first image of the environment;

generating, by the computing device, a difference image from the first image and the second image;

identifying, by the computing device, an object region of the difference image;

generating, by the computing device and based on the object region of the difference image, a mask; and

applying, by the computing device, the mask to the second image of the environment to generate an image of the object.

2. The method of claim 1, wherein the object is a hand, and wherein the object is illuminated in the second image.

3. The method of claim 1, further comprising:

before identifying the object region of the difference image, applying a median filter to the difference image.

4. The method of claim 1, wherein receiving the first image includes receiving the first image from a sensor, and wherein receiving the second image includes receiving the second image from the sensor, the method further comprising:

capturing, by the sensor, the first image; and

capturing, by the sensor, the second image.

5. The method of claim 1, wherein identifying the object region of the difference image comprises performing edge detection on the difference image to generate an edge detection image.

6. The method of claim 5,

wherein generating the difference image comprises computing, on a pixel-by -pixel basis, absolute differences between the first image and the second image; and

wherein generating the mask comprises applying morphological operations to the edge detection image.

7. The method of claim 1, wherein receiving the second image comprises receiving the second image prior to receiving the first image, and wherein the method further comprises:

detecting that the object has moved out of the environment, and wherein receiving the first image comprises receiving the first image in response to detecting that the object has moved out of the environment.

8. A computing system comprising:

processing circuitry; and

at least one storage device that stores instructions that, when executed, configure the processing circuitry to:

receive a first image of an environment,

receive a second image of the environment, wherein the second image includes an object not present in the first image of the environment,

generate a difference image from the first image and the second image,

identify an object region of the difference image,

generate, based on the object region of the difference image, a mask, and

apply the mask to the second image of the environment to generate an image of the object.

9. The computing system of claim 8, the computing system further comprising:

a light source; and

wherein the instructions, when executed, further configure the processing circuitry to: illuminate, with the light source, the object within the environment.

10. The computing system of claim 8, wherein the instructions, when executed, further configure the processing circuitry to:

before identifying the object region of the difference image, apply a median filter to the difference image.

1 1. The computing system of claim 8, wherein receiving the first image includes receiving the first image from a sensor, wherein receiving the second image includes receiving the second image from the sensor, and wherein the instructions, when executed, further configure the processing circuitry to:

capture, by the sensor, the first image; and

capture, by the sensor, the second image.

12. The computing system of claim 8, wherein the instructions that configure the processing circuitry to identify the object region of the difference image include instructions that, when executed, configure the processing circuitry to perform edge detection on the difference image to generate an edge detection image.

13. The computing system of claim 12,

wherein the instructions that configure the processing circuitry to generate the difference image include instructions that, when executed, configure the processing circuitry to compute, on a pixel-by-pixel basis, absolute differences between the first image and the second image; and

wherein the instructions that configure the processing circuitry to generate the mask include instructions that, when executed, apply morphological operations to the edge detection image.

14. The computing system of claim 8, wherein the second image is received prior to receiving the first image, and wherein the instructions, when executed, further configure the processing circuitry to:

detect that the object has moved out of the environment, and wherein the instructions that configure the processing circuitry to receive the first image include instructions that, when executed, configure the processing circuitry to receive the first image in response to detecting that the object has moved out of the environment.

15. A computer-readable storage medium comprising instructions that, when executed, configure one or more processors of a computing system to:

receive a first image of an environment;

receive a second image of the environment, wherein the second image includes an object not present in the first image of the environment;

generate a difference image from the first image and the second image;

identify an object region of difference image;

generate, based on the object region of the difference image, a mask; and

16. The computer-readable storage medium of claim 15, wherein the instructions, when executed, further configure the one or more processors to:

control a light source to illuminate the object within the environment.

17. The computer-readable storage medium of claim 15, wherein the instructions, when executed, further configure the one or more processors to:

18. The computer-readable storage medium of claim 15, wherein the object is a hand.

19. The computer-readable storage medium of claim 15, wherein the instructions that configure the one or more processors to identify the object region of the difference image include instructions that, when executed, configure the one or more processors to perform edge detection on the difference image to generate an edge detection image.

20. The computer-readable storage medium of claim 19,

wherein the instructions that configure the one or more processors to generate the difference image comprise instructions that, when executed, configure the one or more processors to compute, on a pixel-by-pixel basis, absolute differences between the first image and the second image; and

wherein the instructions that configure the one or more processors to generate the mask comprise instructions that, when executed, configure the one or more processors to apply morphological operations to the edge detection image.