Classifications

• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
  • H04N13/20—Image signal generators
  • H04N13/204—Image signal generators using stereoscopic image cameras
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/20—Analysis of motion
  • G06T7/254—Analysis of motion involving subtraction of images
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T19/00—Manipulating 3D models or images for computer graphics
  • G06T19/006—Mixed reality
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/10—Segmentation; Edge detection
  • G06T7/194—Segmentation; Edge detection involving foreground-background segmentation
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/50—Depth or shape recovery
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/70—Determining position or orientation of objects or cameras
  • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/10—Image acquisition modality
  • G06T2207/10028—Range image; Depth image; 3D point clouds

Definitions

• the present disclosure is generally related to augmented reality applications for mobile devices.
• wireless computing devices such as portable wireless telephones, personal digital assistants (PDAs), and paging devices that are small, lightweight, and easily carried by users.
• portable wireless telephones such as cellular telephones and internet protocol (IP) telephones
• IP internet protocol
• wireless telephones can communicate voice and data packets over wireless networks.
• many such wireless telephones include other types of devices that are incorporated therein.
• a wireless telephone can also include a digital still camera, a digital video camera, a digital recorder, and an audio file player.
• such wireless telephones can process executable instructions, including software applications, such as a web browser application, that can be used to access the Internet. As such, these wireless telephones can include significant computing capabilities.
• Some electronic devices use augmented reality applications to add computer- generated images to reality -based images or video. For example, when a camera of a mobile device is pointed at a target area, a game might reproduce an image of the target area on a display of the mobile device while augmenting the image by adding a computer-generated "virtual" object.
• Such applications may consume significant processing resources of the electronic device and may present challenges for detection of interaction between the user and the virtual object.
• Some electronic devices utilize "markers" to enable location detection.
• a user of a mobile device may print an image having a size and pattern recognizable by an application of the mobile device.
• the application may "view” the marker to estimate the distance between the marker and the mobile device.
• markers may be inconvenient and bothersome for users.
• the mobile device may be moving with respect to the marker while the application is being processed, the mobile device may need to repetitively reevaluate the marker to re-estimate the location of the ground or other surface.
• a mobile device in accordance with the present disclosure may utilize one or more techniques to enable efficient processing of augmented reality applications.
• the one or more techniques enable interactions between a user and a "virtual object" without utilizing a marker.
• the mobile device may use an integral image technique to efficiently segment (e.g., subtract) a background associated with an image to be displayed.
• the mobile device may utilize sensor data to determine a location of a ground surface relative to the mobile device, thus enabling placement of virtual objects on a ground surface without use of a marker.
• the mobile device may use a stereo camera to determine depth information (e.g., a depth profile) associated with a scene of interest. The depth information may be used to determine a position of the virtual object (e.g., whether the virtual object is displayed in front of or occluded by a foreground object).
• a method includes evaluating, at a mobile device, a first area of pixels to generate a first result. The method further includes evaluating, at the mobile device, a second area of pixels to generate a second result. Based on comparing a threshold with a difference between the first result and the second result, a determination is made that the second area of pixels corresponds to a background portion of a scene or a foreground portion of the scene.
• a mobile device includes an evaluator
• the mobile device further includes logic configured to determine that the second area of pixels corresponds to a background portion of a scene or a foreground portion of the scene based on comparing a threshold with a difference between the first result and the second result.
• a mobile device includes means for evaluating a first area of pixels to generate a first result and further for evaluating a second area of pixels to generate a second result.
• the mobile device further includes means for determining that the second area of pixels corresponds to a background portion of a scene or a foreground portion of the scene based on comparing a threshold with a difference between the first result and the second result.
• a computer-readable non-transitory medium stores instructions executable by a processor of a mobile device to evaluate, at the mobile device, a first area of pixels to generate a first result and to evaluate, at the mobile device, a second area of pixels to generate a second result.
• the instructions are further executable by the processor to determine that the second area of pixels corresponds to a background portion of a scene or a foreground portion of the scene based on comparing a threshold with a difference between the first result and the second result.
• a method in another particular embodiment, includes determining, using a sensor of a mobile device, an angle of a longitudinal extent of the mobile device with respect to a ground surface. The method further includes estimating a first distance with respect to the ground surface. The first distance is associated with a first projection from a center of the mobile device to the ground surface, where the first projection is perpendicular to the longitudinal extent of the mobile device. A second distance is estimated based on the angle and the first distance. The second distance is associated with a second projection from the center of the mobile device to the ground surface, where the second projection is perpendicular to the ground surface.
• a mobile device in another particular embodiment, includes a sensor configured to determine an angle of a longitudinal extent of the mobile device with respect to a ground surface.
• the mobile device further includes an estimator.
• the estimator estimates a first distance associated with a first projection from a center of the mobile device to the ground surface, where the first projection is perpendicular to the longitudinal extent of the mobile device.
• the estimator further estimates, based on the angle and the first distance, a second distance associated with a second projection from the center of the mobile device to the ground surface.
• the second projection is perpendicular to the ground surface.
• a mobile device includes means for
• the mobile device further includes means for estimating a first distance and for estimating a second distance based on the angle and the first distance.
• the first distance is associated with a first projection from a center of the mobile device to the ground surface, where the first projection is perpendicular to the longitudinal extent of the mobile device.
• the second distance is associated with a second projection from the center of the mobile device to the ground surface. The second projection is
• a computer-readable non-transitory medium stores instructions executable by a processor of a mobile device to determine, using a sensor of the mobile device, an angle of a longitudinal extent of the mobile device with respect to a ground surface.
• the instructions are further executable by the processor to estimate a first distance and to estimate a second distance based on the angle and the first distance.
• the first distance is associated with a first projection from a center of the mobile device to the ground surface, where the first projection is perpendicular to the longitudinal extent of the mobile device.
• the second distance is associated with a second projection from the center of the mobile device to the ground surface. The second projection is perpendicular to the ground surface.
• a mobile device uses a pixel intensity sum technique, such as an integral image technique, to efficiently identify foreground and background portions of the image.
• a pixel intensity sum technique such as an integral image technique
• at least some of the disclosed embodiments enable use of augmented reality applications that do not require use of a marker, thereby making the augmented reality applications more convenient for users.
• FIG. 1 is a diagram of a particular embodiment of a scene and a mobile device to generate an image of the scene
• FIG. 2A is a block diagram of a particular embodiment of the mobile device of
• FIG. 1 is a diagrammatic representation of FIG. 1 ;
• FIG. 2B is a block diagram of another particular embodiment of the mobile device of FIG. 1;
• FIG. 2C is a block diagram of another particular embodiment of the mobile device of FIG. 1;
• FIG. 3 A is a flow diagram of a particular embodiment of a method of operation of the mobile device of one or more of FIGS. 1 and 2A-2C, or a combination thereof;
• FIG. 3B is a flow diagram of another particular embodiment of a method of operation of the mobile device of one or more of FIGS. 1 and 2A-2C, or a combination thereof;
• FIG. 4 is a block diagram of a particular embodiment of a mobile device to estimate a location of a ground surface;
• FIG. 5 is a flow diagram of a particular embodiment of a method of operation of the mobile device of one or more of FIGS. 1, 2A-2C, and 4, or a combination thereof;
• FIG. 6 is a block diagram of a particular embodiment of a mobile device to generate an image based on a depth profile associated with stereo camera data
• FIG. 7 is a block diagram of a particular example illustrating increasing
• FIG. 8 is a flow diagram of a particular embodiment of a method of operation of a mobile device, such as the mobile device of one or more of FIGS. 1, 2A-2C, 4, and 6, or a combination thereof; and
• FIG. 9 is a block diagram of a particular embodiment of a mobile device, such as the mobile device of one or more of FIGS. 1, 2A-2C, 4, and 6, or a combination thereof.
• FIG. 1 depicts a particular illustrative embodiment of a mobile device 104 and a scene 108.
• the mobile device 104 includes a camera 1 10, an evaluator 120, a background/foreground analyzer 140, and a display 150.
• the scene 108 may include a background object 160, a foreground object 170, and a location 180.
• the location 180 may correspond to a virtual object to be displayed at the display 150, for example in connection with an augmented reality application, as described further below with reference to operation of the display 150.
• the display 150 may display an image 152 that corresponds to the scene 108.
• the camera 110 may capture images of the scene 108.
• the camera 1 10 may generate a first area of pixels 1 12 (e.g., a plurality of pixels, such as a matrix or a table) at a first time.
• the camera 110 may generate a second area of pixels 1 14 at a different time than the first time (e.g., at a second time subsequent to the first time).
• the first area of pixels 1 12 may be associated with an orientation (e.g., positioning) of the mobile device 104 and the scene 108 at the first time.
• the second area of pixels 114 may be associated with an orientation (e.g., positioning) of the mobile device 104 and the scene 108 at the second time.
• the areas of pixels 112, 114 are each used to determine a background portion of the image 152 corresponding to the background object 160, as described further below with reference to at least FIG. 2A. In at least another embodiment, one or more of the areas of pixels 1 12, 114 are used to determine a foreground portion of the image 152 corresponding to the foreground object 170, as described further below with reference to at least FIGS. 2B and 2C.
• the evaluator 120 may evaluate the first area of pixels 1 12 according to an
• the operation is a pixel intensity sum operation that sums pixel intensities of the first area of pixels 112.
• the operation may be an integral image operation (also known as a summed area operation) and the first result 122 may be an integral image (also known as a summed area table) associated with the first area of pixels 1 12.
• the integral image at position (m, n) of the first area of pixels 1 12 may be:
• the integral image operation may generate a plurality of values (e.g., a table) in which each value has a position corresponding to a particular pixel of the first area of pixels 112 and indicates a sum of those pixel intensity values of the first area of pixels 1 12 "up and to the left" of the particular pixel (including the particular pixel).
• values e.g., a table
• the evaluator 120 may also evaluate the second area of pixels 114 according to the operation to generate a second result 124.
• the second result 124 may be an integral image associated with the second area of pixels 1 14.
• the background/foreground analyzer 140 may be responsive to the evaluator
• the background/foreground analyzer 140 compares results, such as the results 122, 124, to generate one or more of a background portion of the image 152 and a foreground portion of the image 152. For example, the results 122, 124, to generate one or more of a background portion of the image 152 and a foreground portion of the image 152. For example, the results 122, 124, to generate one or more of a background portion of the image 152 and a foreground portion of the image 152. For example, the results 122, 124.
• background/foreground analyzer 140 may compare the results 122, 124 to determine a difference between the results 122, 124. In at least one embodiment, the background/foreground analyzer 140 compares the difference to a threshold (e.g., a predetermined threshold) to determine whether the result 124 corresponds to one or more of the background object 160 and the foreground object 170. In at least one embodiment, the threshold used is a percentage (e.g., a percentage difference between the results 122, 124).
• the threshold used by the background/foreground analyzer 140 may be any threshold used by the background/foreground analyzer 140.
• the threshold is "adaptive" relative to the images being captured. For example, when a background and a foreground are more different (e.g., when the foreground and background are of different colors), then the threshold may be increased. When the background and foreground are less different (e.g., when the background and foreground are of similar colors), then the threshold may be decreased. In a particular illustrative embodiment, the threshold is variable between a "minimum" value and a "maximum” value, such as between 1 and 255 (e.g., the threshold may be set to 20).
• the background/foreground analyzer 140 may track differences between foregrounds and backgrounds and adjust the threshold accordingly.
• the background/foreground analyzer 140 may also utilize multiple thresholds, such as a first or "background” threshold used for determining a background portion of the image 152 and a second or “foreground” threshold for determining a foreground portion of the image 152, as described further with reference to FIGS. 2A- 2C.
• the first area of pixels 1 12 corresponds to a first region of the scene 108 (e.g., one of a plurality of regions of an image captured by the camera 1 10) at a first time and the second area of pixels 114 corresponds to the first region of the scene at a second time, then if the difference is determined to be low (e.g., does not exceed the threshold), the first region of the scene may be determined to be relatively stationary or stable between the first time and the second time.
• the first region of the scene 108 may be a selected portion of a captured image and the selected portion may be changed during operation to evaluate each portion of the captured image.
• the first region may be identified as a background portion (e.g., as a portion of the background object 160) of the scene 108.
• An example of such a stationary object may be the sun depicted in FIG. 1.
• the first region of the scene 108 may be determined to have moved or changed between the first time and the second time.
• the first region may be identified as a foreground portion (e.g., as a portion of the foreground object 170) of the scene 108.
• a user engaged in a game of "virtual soccer" (e.g., in connection with an augmented reality game) is an example of an application in which a foreground object may exhibit movement that may exceed the threshold.
• each region of a captured image of the scene 108 may be compared to a corresponding region of at least one prior captured image of the scene 108 in order to identify background and/or foreground portions of a captured image.
• the display 150 may be responsive to the background/foreground analyzer 140 to display the image 152.
• the image 152 may correspond to the scene 108.
• the image 152 includes a background portion (i.e., the sun) corresponding to the background object 160, a foreground portion (i.e., a user) corresponding to the foreground object 170, and a virtual object (i.e., a ball generated in connection with an augmented reality application, such as a game) corresponding to the location 180.
• Example operations of the mobile device 104 are described further with reference to FIGS. 2A-2C.
• FIG. 1 may enable simplified background/foreground analysis for the mobile device 104.
• object motion may be determined based on a degree of change of pixel intensities in a region of an image.
• the mobile device 104 may avoid repetitively reevaluating orientation and location data while the mobile device 104 is in motion and is processing an augmented reality application.
• FIG. 2 A depicts a particular illustrative embodiment of the mobile device 104 described with reference to FIG. 1.
• the mobile device 104 of FIG. 2A may include the camera 110, the evaluator 120, the background/foreground analyzer 140, and the display 150.
• the evaluator 120 includes a circuit 204 to perform an operation, such as an integral image operation, as explained further below.
• the background/foreground analyzer 140 includes a comparison circuit 130, which may be configured to determine a background model 212 based on comparing at least one difference between successive integral images to a background threshold, as explained further below.
• the evaluator 120 may be responsive to a first area of pixels 1 12A and to a second area of pixels 1 14A each generated by the camera 110.
• the circuit 204 may perform an operation based on the areas of pixels 1 12A, 114A to generate a first result 122A and a second result 124A, respectively.
• the areas of pixels 1 12 A, 114A and the results 122A, 124A correspond to the areas of pixels 1 12, 114 and the results 122, 124 of FIG. 1, respectively.
• the operation performed by the circuit 204 is an integral image operation
• the first result 122A is an integral image associated with the first area of pixels 1 12A
• the second result 124A is an integral image associated with the second area of pixels 1 14A.
• the integral image at position (m, n) of the first area of pixels 1 12A is
• I(x, y) is a pixel intensity value having position (x, y) in the matrix of pixel intensity values.
• I(x, y) is a pixel intensity value having position (x, y) in the matrix of pixel intensity values.
• the circuit 204 may determine the corresponding integral image to be: 3 8 9
• the background/foreground analyzer 140 is
• the comparison circuit 130 may determine a difference between the results 122 A, 124A. If the difference does not exceed a threshold associated with determining background models (i.e., a "background threshold"), then a determination may be made that the areas of pixels 112 A, 1 14A represent a substantially “stable” or "steady" image of a scene (e.g., the background object 160 of the scene 108 of FIG. 1).
• a threshold associated with determining background models i.e., a "background threshold”
• one or more of the areas of pixels 1 12 A, 1 14A and the results 122A, 124A may be used to construct the background model 212, to render a background portion at the display 150 using the background model 212 (e.g., in connection with an augmented reality application), or a combination thereof.
• the background model 212 is stored at the mobile device 104 (e.g., stored by the background/foreground analyzer 140, as depicted in the example of FIG. 2A) and used in subsequent calculations, as described further with reference to at least FIGS. 2B and 2C.
• the result 124A may be stored at the mobile device 104 (e.g., as part of a background model, such as the background model 212) and is used to determine a foreground portion of an image, as described further with reference to at least FIGS. 2B and 2C.
• background models may be sent and received by mobile devices, such as the mobile device 104 of FIG. 2A.
• background models may be generated based on a "user designation" technique, as described further with reference to FIG. 8.
• a substantially “stable” or "steady” set of pixels e.g., background model
• the background/foreground analyzer 140 may determine that the second area of pixels 1 14A corresponds to a foreground portion. Determination of foreground portions is described further with reference to at least FIGS. 2B and 2C.
• the integral image technique described with reference to FIG. 2A is computationally robust and may enable simplified background/foreground analysis for the mobile device 104.
• object motion may be determined based on a degree of change of pixel intensities in a region of an image (e.g., whether the degree of change exceeds a threshold) instead of based on a location and position of a marker.
• the mobile device 104 of FIG. 2A may avoid repetitively reevaluating orientation and location of the marker while the mobile device is in motion and while processing an augmented reality application, computational complexity of generating the background model 212 may be reduced.
• FIG. 2B depicts a particular illustrative embodiment of the mobile device 104 described with reference to FIG. 1.
• the mobile device 104 of FIG. 2B may include the camera 1 10, the evaluator 120, the background/foreground analyzer 140, and the display 150.
• the evaluator 120 includes the circuit 204 of FIG. 2A.
• the background/foreground analyzer 140 includes the comparison circuit 130 of FIG. 2A, which may be configured to determine a foreground/background portion by comparing a difference between an integral image and a background model to a foreground threshold, as explained further below.
• the evaluator 120 may be responsive to a first area of pixels 1 12B and to a second area of pixels 1 14B, each generated by the camera 110.
• the circuit 204 may perform an operation based on the areas of pixels 1 12B, 1 14B to generate a first result 122B and a second result 124B, respectively.
• the areas of pixels 1 12B, 1 14B and the results 122B, 124B correspond to the areas of pixels 1 12, 114 and the results 122, 124 of FIG. 1, respectively.
• the operation performed by the circuit 204 is an integral image operation
• the first result 122B is an integral image associated with the first area of pixels 1 12B
• the second result 124B is an integral image associated with the second area of pixels 1 14B.
• the comparison circuit 130 of FIG. 2B is configured to compare results of operations (e.g., the results 122B, 124B) to the background model 212 to determine foreground models.
• the comparison circuit 130 may compare the first result 122B to the background model 212 (or to a portion of the background model 212) to determine a first difference (e.g., a difference between two image integrals).
• the background/foreground analyzer 140 may determine, based on the first difference, that the first result 122B does not correspond to a foreground portion of a scene (e.g., the foreground object 170 of the scene 108 of FIG. 1, or a portion thereof).
• a threshold associated with foreground models i.e., a "foreground threshold”
• the backgr ound/foreground analyzer 140 may determine that the first area of pixels 1 12B does not correspond to a foreground object of the scene (e.g., does not correspond to a "dynamic" object, such as a user in motion, or a portion thereof).
• the background/foreground analyzer 140 may cause the camera 1 10 to capture the second area of pixels 114B.
• the comparison circuit may be responsive to the second result 124B generated by the circuit 204 based on the second area of pixels 1 14B.
• the comparison circuit 140 may compare the second result 124B to the background model 212 to generate a second difference. If the second difference exceeds the foreground threshold, then the
• background/foreground analyzer 140 may determine, based on the second difference, that the second result 124B does not substantially "match” or is not “similar” to the background model 212.
• the background/foreground analyzer 140 may thus determine that the second area of pixels 1 14B corresponds to a foreground object of the scene (e.g., corresponds to a "dynamic" object, such as a user in motion, or a portion thereof).
• One or more of the second area of pixels 1 14B and the second result 124B may be used to render a representation of the foreground object at the display 150, for example in connection with an augmented reality application.
• object motion may be determined based on a degree of change of pixel intensities in a region of an image (e.g., whether the degree of change exceeds a threshold) instead of based on a location and position of a marker.
• the mobile device may avoid repetitively reevaluating orientation and location of the marker while the mobile device is in motion and while processing an augmented reality application, computational complexity of background/foreground analysis may be reduced.
• the first result 122B in response to determining that the first difference does not exceed the threshold, may also be "subdivided" into portions (e.g., sub-blocks). One or more of the subdivided portions may be compared to the background model 212. Such techniques are described further with reference to at least FIG. 2C.
• FIG. 2C depicts a particular illustrative embodiment of the mobile device 104 described with reference to FIG. 1.
• the mobile device 104 of FIG. 2C may include the camera 1 10, the evaluator 120, the background/foreground analyzer 140, and the display 150.
• the evaluator 120 includes the circuit 204 of FIGS. 2A and 2B and further includes a storage device 216.
• the background/foreground analyzer 140 may include the comparison circuit 130 of FIGS. 2A and 2B and may further include a fast integral image calculator 220.
• the fast integral image calculator 220 may include circuitry configured to perform a "fast integral image operation" using one addition operation and two subtraction operations, as described further below.
• the camera 110 may capture images of a scene, such as the scene
• a first area of pixels 1 12C may correspond to an image of the scene captured at a first time.
• the circuit 204 may perform an operation on the first area of pixels 1 12C to generate a first result 122C.
• the first area of pixels 112C and the first result 122C may correspond to the first area of pixels 112 and the first result 122 of FIG. 1, respectively.
• the first area of pixels 1 12C corresponds to pixels of an entire image captured by the camera 1 10 and the first result 122C corresponds to an integral image associated with the entire image (e.g., the first result 122C may be a "high-level" integral image).
• the background/foreground analyzer 140 may be responsive to the high-level integral image generated by the circuit 204.
• the fast integral image calculator 220 may divide the first result 122C into portions (e.g., sub-blocks) and calculate a respective integral image associated with each of the portions.
• the first area of pixels 1 12C is a four- by-four matrix:
• the circuit 204 may compute the first result 122C to be a "high-level"
• the background/foreground analyzer 140 may compare the first result 122C to the background model 212 using an appropriate technique, such as one or more of the techniques described with reference to FIGS. 1, 2A, and 2B (e.g., using a threshold), to determine whether the first result 122C "matches” the background model 212. If the comparison circuit 130 does not determine that the first result 122C "matches” the background model 212, the fast integral image calculator 220 may generate one or more sub-portions or "sub-integral images" (e.g., an integral image of a subset of pixels of the first area of pixels 1 12C) based on the first result 122C and may compare the one or more sub-integral images to the background model 212. In a particular embodiment, the fast integral image calculator calculates the one or more sub-integral images according to:
• One or more such sub-integral images may be compared to the background model 212 to determine a "match.”
• the sub-integral images are compared to "neighbor portions” (e.g., portions within a predetermined range of the sub-integral images) of the background model 212.
• the "upper left" two-by-two sub-integral image may be compared the "upper left” two-by-two portion of the background model 212. If one or more of the sub-integral images match the background model 212, then the one or more sub-integral images may be determined to correspond to background.
• the first result 122C can be further subdivided into smaller sub-integral images. As described further with reference to FIG. 3B, if none of the portions is determined to "match" the background model 212, then the first result 122C may be determined to correspond to a foreground portion (e.g., a "dynamic" object in motion). [0071] It should be appreciated that the techniques described in connection with
• FIG. 2C may enable fast comparison of pixels (e.g., the first area of pixels 112C) with a background model (e.g., the background model 212).
• a background model e.g., the background model 212
• only a "high-level" integral image is stored (e.g., stored at the storage device 216), which may be used to generate any sub-integral images, such as using the fast integral image calculator 220.
• the first area of pixels 1 12C is not used for additional integral image generation (e.g., since sub-integral images may be generated based instead on the high-level integral image). Accordingly, due to the
• augmented reality applications may be processed and results rendered (e.g., at the display 150) more rapidly, thus providing users of the mobile device 104 a more realistic augmented reality experience.
• the method 300 includes evaluating, at the mobile device 104, a first area of pixels (e.g., the first area of pixels 1 12) to generate a first result (e.g., the first result 122), at 304.
• the method 300 further includes evaluating, at the mobile device, a second area of pixels (e.g., the second area of pixels 1 14) to generate a second result (e.g., the second result 124), at 308.
• the determination is made based on whether the difference exceeds or does not exceed a threshold.
• FIG. 3B another particular illustrative embodiment of a method of operation of the mobile device 104 of one or more of FIGS. 1, 2A, 2B, and 2C is depicted and generally designated 350.
• the method 350 includes capturing a plurality of initial images of a scene, at 302.
• the initial images may include the areas of pixels 1 12, 1 14.
• the scene may be the scene 108.
• the initial images may be captured by the camera 1 10.
• the comparison circuit 130 determines that the at least one difference does not exceed the first threshold using one or more techniques described with reference to FIG. 2A.
• a background model (e.g., the background model 212) is generated based on at least one of the plurality of initial images.
• the background model may include a first integral image associated with the scene.
• the background model may be generated according to user input, as described further with reference to FIG. 8.
• the method 350 further includes capturing (e.g., by the camera 1 10) a second image associated with the scene and determining a second integral image associated with the second image, at 314.
• the second integral image may be any of the results 122B, 124B, and 122C described with reference to FIGS. 2B and 2C.
• the second integral image is partitioned (e.g., divided) into one or more portions having a size n and each of the one or more portions is compared to neighbor portions (e.g., portions within a predetermined range) of the first integral image.
• the size n may be a length (e.g., number of columns) of the one or more portions, a height (e.g., a number of rows) of the one or more portions, or a combination thereof.
• the second integral image is a four-by-four matrix, then the second integral image may be partitioned into one or more two-by-two portions.
• the second integral image is partitioned according to the "fast integral image” technique described with reference to FIG. 2C.
• the one or more portions may be the "sub-integral images" described with reference to FIG. 2C and may be each generated using one addition operation and two subtraction operations, such as according to a + d - b - c, as described with reference to FIG. 2C.
• the determination may be made according to one or more techniques described with reference to FIGS. 1 and 2A-2C, such as by comparing the one or more portions to a second threshold to generate a second difference and determining whether the second difference exceeds a second threshold. If any of the one or more portions matches the first integral image, then a determination may be made that the portion corresponds to a background portion of the scene (e.g., corresponds to the background object 160, or a portion thereof, of the scene 108).
• a background portion of the scene e.g., corresponds to the background object 160, or a portion thereof, of the scene 108.
• n may be reduced, at 330, and the second integral image may be re-partitioned, at 318. That is, smaller (e.g., "finer") portions of the second integral image may be generated and compared to neighbor portions of the first integral image.
• the first integral image may be partitioned according to the "fast integral image” technique to generate the neighbor portions.
• the second integral image may be determined to correspond to a foreground portion of the scene (e.g., corresponds to the foreground object 170, or a portion thereof, of the scene 108), at 338.
• FIG. 4 depicts a particular illustrative embodiment of a mobile device 404
• the mobile device 404 may be the mobile device 104 of one or more of FIGS. 1 and 2A-2C (or another mobile device).
• the mobile device 404 may include a display, such as the display 150 of FIGS. 1 and 2A-2C.
• the mobile device 404 may further include a camera 402 (e.g., a stereo camera), a sensor 412, an indication 416 of a height of a user of the mobile device 404, and an estimator 420.
• the camera 402 is the camera 1 10 described with reference to FIG. 1.
• the camera 402 may include an auto focus feature 424 (e.g., logic to cause the camera 402 to automatically focus on objects within a field of view of the camera 402).
• the sensor 412 may be a sensor configured to detect position and/or motion of the mobile device 404, such as a gyroscope, an accelerometer, a motion sensor, or a combination thereof. As shown in the example of FIG. 4, a longitudinal extent 428 of the mobile device 404 may be positioned according to an angle ⁇ relative to the ground surface 408 (e.g., while the mobile device 404 is being handled by the user).
• the senor 412 may determine the angle ⁇ of the longitudinal extent
• the estimator 420 may generate a first estimate 444 of a first distance L ⁇ associated with a first projection 432 from a center 436 of the mobile device 404 (or from a center of the camera 402) to the ground surface 408.
• the first projection 432 is perpendicular to the longitudinal extent 428.
• the first estimate 444 may be generated using stereo camera data provided by the camera 402 of the mobile device 404, using the autofocus feature 424 of the mobile device, or a combination thereof.
• the estimator 420 may further generate, based on the angle ⁇ and the first
• the estimator 420 may generate the second estimate 448 based on the indication 416 of the height of the user of the mobile device 404, and the first estimate 444 may be determined based on the second estimate 448. For example, for a mobile device that does not include a stereo camera (not shown in FIG.
• the second estimate 448 may be generated based on the indication 416 of the height of the user of the mobile device 404 and the first estimate 444 may be determined according to Z 2 /(cos(#)).
• the estimator 420 may generate an estimated location 452 of the ground surface
• the display 150 may be responsive to the estimated location 452 and may display an image 456 that includes a ground surface portion that is based on the estimated location 452 of the ground surface 408.
• the techniques described with reference to FIG. 4 may enable generation of the estimated location 452 of the ground surface 408 without utilizing a marker.
• operation of augmented reality applications may be simplified and made more convenient for users of the mobile device 404.
• a flow diagram of a method of operation of the mobile device 404 of FIG. 4 is depicted and generally designated 500.
• the method 500 includes determining, using a sensor (e.g., the sensor 412) of the mobile device, an angle (e.g., the angle ⁇ ) of a longitudinal extent (e.g., the longitudinal extent 428) of the mobile device (e.g., the mobile device 404) with respect to a ground surface (e.g., the ground surface 408), at 504.
• the method 500 further includes estimating a first distance (e.g., the first
• the first distance L ⁇ associated with a first projection (e.g., the first projection 432) from a center (e.g., the center 436) of the mobile device to the ground surface, at 508.
• the first projection is perpendicular to the longitudinal extent of the mobile device.
• the first distance may be estimated using a stereo camera of the mobile device, an auto focus feature (e.g., the auto focus feature 424) of the mobile device, or a combination thereof.
• a second distance (e.g., the second distance L 2 ) is estimated, at 512.
• the second distance is associated with a second projection (e.g., the second projection 440) from the center of the mobile device to the ground surface.
• the second projection is perpendicular to the ground surface.
• the method 500 may further include determining a location of a virtual object based at least in part on the estimated second distance, at 516. For example, based on distance and location of the ground surface, the virtual object can be rendered as being on the ground (e.g., a ball at rest) or as being above the ground (e.g., the ball after being kicked). [0090] Referring to FIG.
• the mobile device 600 may include a stereo camera 610 and a depth evaluator 620 coupled to the stereo camera 610.
• the stereo camera 610 may include multiple lenses, such as a first lens 616 and a second lens 614.
• the depth evaluator 620 may include a detail generator 622. It should be appreciated that one or more features and operations of the mobile device 600 of FIG. 6 may be described with reference to the features and operations described with reference to FIGS. 1-5. For example, although not shown in FIG. 6 for clarity of illustration, the mobile device 600 may include the display 150 of FIGS. 1, 2A-2C, and 4.
• the stereo camera 610 may generate image depth data 612 related to a scene (e.g., the scene 108 of FIG. 1).
• the image depth data 612 may indicate object depths.
• the depth evaluator 620 may be responsive to the image depth data 612 to generate a depth profile 624.
• the depth profile 624 may include depth information related to one or more images (e.g., related to one or more of the areas of pixels 1 12, 1 14 of FIG. 1).
• the depth evaluator 620 is configured to determine, based on the depth profile 624, whether a foreground portion 632 is to be rendered in front of or behind virtual objects. For example, as shown in FIG.
• the depth evaluator 620 may determine, based on the depth profile 624, that a virtual object 634 is to be rendered in front of the foreground portion 632. As another example, the depth evaluator 620 may determine, based on the depth profile 624, that a virtual object 636 is to be occluded by the foreground portion 632.
• the detail generator 622 may be configured to
• the detail generator 622 may determine that the object corresponds to image background and is to be rendered as a background portion 638. As another example, in response to the depth profile indicating that the object has a small depth relative to the mobile device 600, the detail generator 622 may determine that the object corresponds to image foreground and is to be rendered as a foreground portion 642. In at least one embodiment and as described further with reference to FIG.
• a boundary of the foreground portion 642 (e.g., a boundary of the foreground portion 642 relative to the background portion 638) may be determined based on the depth profile 624.
• resolution of a portion of the image 630 (e.g., a portion corresponding to the boundary of the foreground portion 642) may be increased in response to determining the boundary of the foreground portion 642 based on the depth profile 624.
• the techniques described in connection with FIG. 6 enable efficient generation of depth information.
• use of the stereo camera 610 may enable generation of depth information without analyzing a location and position of a marker.
• the depth information can be used to selectively change image resolution and quality. That is, as described further with reference to FIG. 7, once a boundary of a foreground object of an image has been determined (e.g., using the stereo camera 610), portions outside the foreground object can be reduced in resolution and portions within the boundary can be increased in resolution.
• additional processing resources can be allocated to portions of more interest (e.g., foreground portions) rather than to other portions (e.g., background portions).
• FIG. 7 a portion of the image 630 of FIG. 6 prior to and subsequent to selectively changing image resolution based on the depth profile 624 is depicted and generally designated 630A and 630B, respectively.
• the portion 630A includes the background portion 638 and the foreground portion 642.
• the foreground portion 642 includes a boundary 712.
• the boundary 712 may be determined based on the depth profile 624. In at least one embodiment, the boundary 712 is determined based on a depth difference between the foreground portion 642 and the background portion 638 exceeding a threshold.
• portion 642 may be selectively changed. For example, a first portion 716 of the foreground portion 642 may be decreased in resolution, or reclassified as a background portion, based on the boundary 712. A second portion 720 of the foreground portion 642 may be increased in resolution based on the boundary 712. For example, in at least one embodiment, the second portion 720 is improved from a block-based resolution to a pixel-based resolution (e.g., from coarse resolution to fine resolution). In at least one embodiment, one or more low resolution image portions are generated based on a high- level integral image stored at the storage device 216, as described with reference to FIG. 2C. Accordingly, selectively changing resolution of the foreground portion 642 based on the boundary 712 may improve contrast of the foreground portion 642 relative to the background portion 638, thus improving image quality.
• FIG. 8 an example operation of a mobile device is depicted and generally designated 800.
• the operations 800 may be performed by the mobile device 104 of one or more of FIGS. 1 and 2A-2C, the mobile device 404 of FIG. 4, the mobile device 600 of FIG. 6, or a combination thereof.
• images may be captured at a camera of the mobile device, at 804.
• the camera may be the camera 1 10 of one or more of FIGS. 1 and 2A-2C, the camera 402 of FIG. 4, the stereo camera 610 of FIG. 6, or a combination thereof.
• the images may correspond to the areas of pixels 112, 1 14.
• the operations 800 further include generating a virtual object, at 808.
• virtual object may be the virtual object 634 of FIG. 6, the virtual object 636 of FIG. 6, or a combination thereof.
• the virtual object may be a computer-generated virtual object that is generated using a processor of the mobile device, as described further with reference to FIG. 9.
• the operations 800 further include generating a background model of an image to be displayed, at 812.
• the background model is generated based on user input. For example, according to a particular example, a user may point the camera 1 10 at a background object (e.g., the background object 108) to designate the background model, for example in connection with an augmented reality game. Such a background model may also be sent and received between mobile devices, for example in connection with the augmented reality game.
• the background model may include a ground surface portion generated according to the techniques described with reference to FIGS. 4 and 5. The background model may be generated prior to generating the first result 122 as described with reference to FIGS. 1 and 2A-2C.
• the background model is updated according to a pixel intensity sum technique (e.g., the integral image technique described with reference to FIGS. 1-3).
• the pixel intensity sum technique may be utilized to determine that data corresponds to a foreground portion (e.g., the foreground portion 632 of FIG. 6, the foreground portion 642 of FIG. 6, or a combination thereof) based on whether a difference exceeds a threshold.
• the operations 800 further include inserting the virtual object into the image to be displayed, at 818.
• the virtual object may be inserted in front of or behind the foreground portion based on the depth profile 624 of FIG. 6.
• the operations 800 further include detecting collisions, user input, or a combination thereof, at 820. For example, a collision between a foreground object (e.g., a user) and a location (e.g., the location 180 of FIG. 1) corresponding to a virtual object may be detected.
• the operations 800 may further include performing sequence recognition, at
• an output is generated.
• an image e.g., the image 152 of FIG. 1, the image 630 of FIG. 6, or a combination thereof
• FIG. 8 may enable efficient processing of augmented reality applications. Further, the augmented reality applications may be processed without utilizing a marker, which may provide convenience and improved performance for users.
• FIG. 9 a block diagram of a particular embodiment of a mobile device is depicted and generally designated 900.
• the mobile device 900 includes a processor 910 (e.g., a digital signal processor) that includes the evaluator 120 of FIG. 1, the comparison circuit 130 of FIG. 1, the background/foreground analyzer 140 of FIG. 1, the estimator 420 of FIG. 4, and the depth evaluator 620 of FIG. 6.
• a processor 910 e.g., a digital signal processor
• Each of the evaluator 120, the comparison circuit 130, the background/foreground analyzer 140, the estimator 420, and the depth evaluator 620 may include hardware, instructions executable by the processor 910, or a combination thereof.
• FIG. 9 further depicts a memory 932 coupled to the processor 910. The memory
• the 932 is a computer-readable non-transitory (e.g., tangible) medium configured to store instructions 954 and data 956.
• the instructions 954 may include image processing instructions, such as instructions executable by the processor 910 to perform one or more operations described with reference to FIGS. 1-8.
• the instructions 954 may include an augmented reality application (e.g., an application including instructions executable by the processor 910 to generate virtual objects).
• the data 956 may include the areas of pixels 112, 1 14 of one or more of FIGS. 1 and 2A-2C, the results 122, 124 of one or more of FIGS. 1 and 2A-2C, the depth profile 624 of FIGS. 6 and 7, or a combination thereof.
• the mobile device 900 may include a display, such as the display 150 described with reference to one or more of FIGS. 1, 2A-2C, and 4.
• the mobile device 900 may further include a camera 946 coupled to a camera controller 990.
• the camera 946 may be the camera 1 10 of one or more of FIGS. 1 and 2A-2C, the camera 402 of FIG. 4, the stereo camera 610 of FIG. 6, or a combination thereof.
• FIG. 9 also shows a display controller 926 that is coupled to the processor 910 and to the display 150.
• a coder/decoder (CODEC) 934 can also be coupled to the processor 910.
• a speaker 936 and a microphone 938 can be coupled to the CODEC 934.
• FIG. 9 further indicates that a wireless controller 940 can be coupled to the processor 910 and to transceiver 950.
• the transceiver 950 may be coupled to an antenna 942.
• the processor 910, the display controller 926, the camera controller 990, the memory 932, the CODEC 934, the wireless controller 940, and the transceiver 950 are included in a system-in-package or system-on-chip device 922.
• an input device 930, a power supply 944, and a sensor are each coupled to the system-on-chip device 922.
• the display 150, the input device 930, the speaker 936, the microphone 938, the antenna 942, the power supply 944, and the sensor 412 are external to the system-on-chip device 922.
• each of the display 150, the input device 930, the speaker 936, the microphone 938, the antenna 942, the power supply 944, and the sensor 412 can be coupled to a component of the system-on-chip device 922, such as an interface or a controller.
• a mobile device in conjunction with the disclosed embodiments, includes means for evaluating (e.g., the evaluator 120) a first area of pixels to generate a first result and further for evaluating a second area of pixels to generate a second result.
• the mobile device further includes means for determining (e.g., the background/foreground analyzer 140) that the second area of pixels corresponds to one of a background portion of a scene and a foreground portion of the scene based on comparing a threshold with a difference between the first result and the second result.
• a mobile device that includes means for determining (e.g., the sensor 412) an angle (e.g., the angle ⁇ ) of a longitudinal extent (e.g., the longitudinal extent 428) of the mobile device with respect to a ground surface (e.g., the ground surface 408).
• the mobile device further includes means for estimating (e.g., the estimator 420) a first distance (e.g., the first distance and for estimating a second distance (e.g., the second distance L 2 ) based on the angle and the first distance.
• the first distance is associated with a first projection (e.g., the first projection 432) from a center (e.g., the center 436) of the mobile device to the ground surface, where the first projection is perpendicular to the longitudinal extent of the mobile device.
• the second distance is associated with a second projection (e.g., the second projection 440) from the center of the mobile device to the ground surface. The second projection is perpendicular to the ground surface.
• a software module may reside in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disk, a removable disk, a compact disc read-only memory (CD-ROM), or any other form of storage medium known in the art.
• An exemplary non-transitory (e.g. tangible) storage medium is coupled to the processor such that the processor can read information from, and write information to, the storage medium.
• the storage medium may be integral to the processor.
• the processor and the storage medium may reside in an application-specific integrated circuit (ASIC).
• the ASIC may reside in a computing device or a user terminal.
• the processor and the storage medium may reside as discrete components in a computing device or user terminal.

Landscapes

• Engineering & Computer Science (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Theoretical Computer Science (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• Computer Graphics (AREA)
• Computer Hardware Design (AREA)
• General Engineering & Computer Science (AREA)
• Software Systems (AREA)
• Processing Or Creating Images (AREA)
• Image Analysis (AREA)
• Image Processing (AREA)

Priority Applications (3)

Applications Claiming Priority (4)

Publications (2)

Family

ID=50232882

Family Applications (1)

Country Status (5)

Cited By (2)

Families Citing this family (57)

Family Cites Families (21)

• 2012
  • 2012-10-12 US US13/651,098 patent/US9466121B2/en not_active Expired - Fee Related
• 2013
  • 2013-08-09 CN CN201380044121.XA patent/CN104584079B/zh not_active Expired - Fee Related
  • 2013-08-09 JP JP2015531095A patent/JP6077121B2/ja not_active Expired - Fee Related
  • 2013-08-09 KR KR1020157004940A patent/KR101788945B1/ko active Active
  • 2013-08-09 WO PCT/US2013/054423 patent/WO2014042795A2/en not_active Ceased
• 2016
  • 2016-09-01 US US15/255,121 patent/US10038892B2/en not_active Expired - Fee Related

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events