US9883155B2 - Methods and systems for combining foreground video and background video using chromatic matching - Google Patents

Methods and systems for combining foreground video and background video using chromatic matching Download PDF

Info

Publication number
US9883155B2
US9883155B2 US15/181,653 US201615181653A US9883155B2 US 9883155 B2 US9883155 B2 US 9883155B2 US 201615181653 A US201615181653 A US 201615181653A US 9883155 B2 US9883155 B2 US 9883155B2
Authority
US
United States
Prior art keywords
video data
foreground
pixels
color space
background
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
US15/181,653
Other versions
US20170359559A1 (en
Inventor
Sanjay Patel
Elad Yarkony
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Personify Inc
Original Assignee
Personify Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Personify Inc filed Critical Personify Inc
Priority to US15/181,653 priority Critical patent/US9883155B2/en
Assigned to PERSONIFY, INC. reassignment PERSONIFY, INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: YARKONY, ELAD, PATEL, SANJAY
Publication of US20170359559A1 publication Critical patent/US20170359559A1/en
Application granted granted Critical
Publication of US9883155B2 publication Critical patent/US9883155B2/en
Assigned to HONGFUJIN PRECISION INDUSTRY (WUHAN) CO. LTD. reassignment HONGFUJIN PRECISION INDUSTRY (WUHAN) CO. LTD. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: PERSONIFY, INC.
Assigned to PERSONIFY, INC. reassignment PERSONIFY, INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: HONGFUJIN PRECISION INDUSTRY WUHAN
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N9/00—Details of colour television systems
    • H04N9/64—Circuits for processing colour signals
    • H04N9/73—Colour balance circuits, e.g. white balance circuits or colour temperature control
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/222—Studio circuitry; Studio devices; Studio equipment
    • H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
    • H04N5/272—Means for inserting a foreground image in a background image, i.e. inlay, outlay
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/222—Studio circuitry; Studio devices; Studio equipment
    • H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
    • H04N5/272—Means for inserting a foreground image in a background image, i.e. inlay, outlay
    • H04N5/275—Generation of keying signals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N9/00—Details of colour television systems
    • H04N9/12—Picture reproducers
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N9/00—Details of colour television systems
    • H04N9/64—Circuits for processing colour signals
    • H04N9/74—Circuits for processing colour signals for obtaining special effects
    • H04N9/76—Circuits for processing colour signals for obtaining special effects for mixing of colour signals

Definitions

  • FIG. 1 depicts an example method, in accordance with some embodiments.
  • FIG. 2 depicts an architectural view of an example computing device, in accordance with some embodiments.
  • FIG. 3 depicts an example graphical representation of example chromatically high-dimensional background video data in the ⁇ L,a,b ⁇ color space, in accordance with some embodiments.
  • FIG. 4 depicts an example graphical representation of an example subset of the example chromatically high-dimensional background video data of FIG. 3 , in accordance with some embodiments.
  • FIG. 5 depicts an example graphical representation of example chromatically low-dimensional background video data in the ⁇ L,a,b ⁇ color space, in accordance with some embodiments.
  • FIG. 6 depicts an example graphical representation of an example subset of the example chromatically low-dimensional background video data of FIG. 5 , in accordance with some embodiments.
  • FIG. 7 depicts an example idealization of the example subset of FIG. 6 , in accordance with some embodiments.
  • FIG. 8 depicts an example idealization of the example chromatically low-dimensional background video data of FIG. 5 that incorporates the example idealization of FIG. 7 , in accordance with some embodiments.
  • FIG. 9 depicts an example L-to- ⁇ a,b ⁇ look-up table that corresponds to the example idealization of FIG. 8 , in accordance with some embodiments.
  • FIG. 10 depicts an example table representation of example foreground video data in the ⁇ R,G,B ⁇ color space, in accordance with some embodiments.
  • FIG. 11 depicts an example table representation of the example foreground video data of FIG. 10 after having been converted to the ⁇ L,a,b ⁇ color space, in accordance with some embodiments.
  • FIG. 12 depicts an example graphical representation of the example converted foreground video data of FIG. 11 in the ⁇ L,a,b ⁇ color space, in accordance with some embodiments.
  • FIG. 13 depicts an example graphical representation that shows both the example chromatically low-dimensional background video data of FIG. 5 and the example converted foreground video data of FIG. 12 , both in the ⁇ L,a,b ⁇ color space, in accordance with some embodiments.
  • FIG. 14 depicts an example table representation of the example converted foreground video data of FIGS. 11-12 after having been transformed in the ⁇ L,a,b ⁇ color space using a chromatic-replacement technique, in accordance with some embodiments.
  • FIG. 15 depicts an example graphical representation of the transformed foreground video data of FIG. 14 in the ⁇ L,a,b ⁇ color space, in accordance with some embodiments.
  • FIG. 16 depicts an example graphical representation that shows both the example chromatically low-dimensional background video data of FIG. 5 and the example transformed foreground video data of FIG. 15 , both in the ⁇ L,a,b ⁇ color space, in accordance with some embodiments.
  • FIG. 17 depicts an example table representation of the example transformed foreground video data of FIGS. 14-15 after having been converted to the ⁇ R,G,B ⁇ color space, in accordance with some embodiments.
  • One embodiment takes the form of a method that includes (i) obtaining foreground video data; (ii) obtaining background video data; (iii) determining a color-distribution dimensionality of the background video data to be either high-dimensional chromatic or low-dimensional chromatic; (iv) selecting a chromatic-adjustment technique from a set of chromatic-adjustment techniques based on the determined color-distribution dimensionality of the background video data; (v) adjusting the foreground video data using the selected chromatic-adjustment technique; (vi) generating combined video data at least in part by combining the background video data with the adjusted foreground video data; and (vii) outputting the combined video data for display.
  • Another embodiment takes the form of a system that includes a communication interface, a processor, and a non-transitory computer-readable medium storing instructions executable by the processor for causing the system to perform at least the functions listed in the preceding paragraph.
  • any of the variations and permutations described herein can be implemented with respect to any embodiments, including with respect to any method embodiments and with respect to any system embodiments. Furthermore, this flexibility and cross-applicability of embodiments is present in spite of the use of slightly different language (e.g., process, method, steps, functions, set of functions, and the like) to describe and or characterize such embodiments.
  • determining the color-distribution dimensionality of the background video data to be either high-dimensional chromatic or low-dimensional chromatic includes (i) converting pixels of the background video data from an ⁇ R,G,B ⁇ color space to an ⁇ L,a,b ⁇ color space; (ii) calculating an ⁇ a,b ⁇ variance of the converted background pixels; (iii) comparing the calculated ⁇ a,b ⁇ variance to an ⁇ a,b ⁇ -variance threshold; (iv) determining the color-distribution dimensionality of the background video data to be high-dimensional chromatic if the calculated ⁇ a,b ⁇ variance exceeds the ⁇ a,b ⁇ -variance threshold; and (v) determining the color-distribution dimensionality of the background video data to be low-dimensional chromatic if the calculated ⁇ a,b ⁇ variance does not exceed the ⁇ a,b ⁇ -variance threshold.
  • calculating the ⁇ a,b ⁇ variance of the converted background pixels includes determining how many luminance levels in the converted background pixels have more than a luminance-level-specific degree of ⁇ a,b ⁇ variance.
  • the ⁇ a,b ⁇ -variance threshold is a threshold number of luminance levels; in at least one such embodiment, the threshold number of luminance levels is zero; in at least one other such embodiment, the threshold number of luminance levels is greater than zero.
  • calculating the ⁇ a,b ⁇ variance of the converted background pixels includes (i) determining a respective luminance-level-specific ⁇ a,b ⁇ variance for each of a plurality of luminance levels that are represented in the converted background pixels and (ii) calculating the ⁇ a,b ⁇ variance of the converted background pixels to be a sum of the determined luminance-level-specific ⁇ a,b ⁇ variances.
  • calculating the ⁇ a,b ⁇ variance of the converted background pixels includes (i) determining a respective luminance-level-specific ⁇ a,b ⁇ variance for each luminance level represented in the converted background pixels and (ii) calculating the ⁇ a,b ⁇ variance of the converted background pixels to be a sum of the determined luminance-level-specific ⁇ a,b ⁇ variances.
  • determining the color-distribution dimensionality of the background video data to be either high-dimensional chromatic or low-dimensional chromatic includes (i) determining the color-distribution dimensionality of the background video data to be low-dimensional chromatic if a background-color distribution of the background video data in an ⁇ L,a,b ⁇ color space is supported by a relationship defined by: ⁇ ( L,a,b )
  • the set of chromatic-adjustment techniques includes a white-balancing technique and a chromatic-replacement technique and (ii) selecting a chromatic-adjustment technique based on the determined color-distribution dimensionality includes (a) selecting the white-balancing technique when the color-distribution dimensionality of the background video data is determined to be high-dimensional chromatic and (b) selecting the chromatic-replacement technique when the color-distribution dimensionality of the background video data is determined to be low-dimensional chromatic.
  • adjusting the foreground video data using the white-balancing technique includes (i) determining a foreground average of pixels of the foreground video data in an ⁇ R,G,B ⁇ color space; (ii) determining a background average of pixels of the background video data in the ⁇ R,G,B ⁇ color space; (iii) converting the foreground average and the background average from the ⁇ R,G,B ⁇ color space to a second color space; (iv) determining a transform matrix in the second color space from the converted foreground average to the converted background average; (v) converting the pixels of the foreground video data from the ⁇ R,G,B ⁇ color space to the second color space; (vi) transforming the converted foreground pixels in the second color space using the determined transform matrix; and (vii) converting the transformed foreground pixels from the second color space to the ⁇ R,G,B ⁇ color space.
  • the determined transform matrix includes dimension-wise ratios in the second color space of the converted background average to the converted foreground average.
  • the second color space is an ⁇ L,a,b ⁇ color space.
  • the second color space is an ⁇ X,Y,Z ⁇ color space.
  • converting the foreground pixels from the ⁇ R,G,B ⁇ color space to the second color space includes converting the foreground pixels from the ⁇ R,G,B ⁇ color space to an ⁇ L,a,b ⁇ color space and then from the ⁇ L,a,b ⁇ color space to the ⁇ X,Y,Z ⁇ color space.
  • the method also includes converting pixels of the background video data from an ⁇ R,G,B ⁇ color space to an ⁇ L,a,b ⁇ color space, and adjusting the foreground video data using the chromatic-replacement technique includes: (i) generating an L-to- ⁇ a,b ⁇ lookup table based on the converted background pixels; (ii) converting pixels of the foreground video data from the ⁇ R,G,B ⁇ color space to the ⁇ L,a,b ⁇ color space; (iii) transforming the converted foreground pixels at least in part by (a) using the respective L values of the respective converted foreground pixels to select respective replacement ⁇ a,b ⁇ values for the respective converted foreground pixels based on the L-to- ⁇ a,b ⁇ lookup table and (b) replacing the respective ⁇ a,b ⁇ values of the respective converted foreground pixels with the corresponding respective selected replacement ⁇ a,b ⁇ values; and (iv) converting the transformed foreground pixels from the ⁇ L,a
  • using the respective L values of the respective converted foreground pixels to select the respective replacement ⁇ a,b ⁇ values for the respective converted foreground pixels based on the L-to- ⁇ a,b ⁇ lookup table includes retrieving the respective replacement ⁇ a,b ⁇ values from the L-to- ⁇ a,b ⁇ lookup table in cases where the respective L value of the respective converted foreground pixel is listed in the L-to- ⁇ a,b ⁇ lookup table.
  • using the respective L values of the respective converted foreground pixels to select the respective replacement ⁇ a,b ⁇ values for the respective converted foreground pixels based on the L-to- ⁇ a,b ⁇ lookup table further includes using interpolated ⁇ a,b ⁇ values based on one or more entries in the L-to- ⁇ a,b ⁇ lookup table as the respective replacement ⁇ a,b ⁇ values in cases where the respective L value of the respective converted foreground pixel is not listed in the L-to- ⁇ a,b ⁇ lookup table.
  • the interpolated ⁇ a,b ⁇ values are copied from a nearest L value that is listed in the L-to- ⁇ a,b ⁇ lookup table; in at least one other such embodiment, the interpolated ⁇ a,b ⁇ values are average ⁇ a,b ⁇ values of two or more proximate entries in the L-to- ⁇ a,b ⁇ lookup table.
  • the method also includes (i) obtaining second foreground video data and (ii) adjusting the second foreground video data using the selected chromatic-adjustment technique, and generating the combined video data includes combining the background video data with both the adjusted foreground video data and the adjusted second foreground video data.
  • FIG. 1 depicts an example method, in accordance with some embodiments.
  • FIG. 1 depicts a method 100 that is described below by way of example as being carried out by the example computing device (or system) 200 that is described in connection with FIG. 2 .
  • the method 100 could be carried out by any computing device that is suitably equipped, programmed, and configured.
  • the method 100 includes steps 102 - 114 , which are described briefly in connection with FIG. 1 and more fully in connection with the ensuing figures.
  • the computing device 200 obtains foreground video data.
  • the computing device 200 obtains background video data.
  • the computing device determines a color-distribution dimensionality of the background video data to be either high-dimensional chromatic or low-dimensional chromatic.
  • the computing device 200 selects a chromatic-adjustment technique from a set of chromatic-adjustment techniques based on the determined color-distribution dimensionality of the background video data.
  • the computing device 200 adjusts the foreground video data using the selected chromatic-adjustment technique.
  • the computing device 200 generates combined video data at least in part by combining the background video data with the adjusted foreground video data.
  • the computing device 200 outputs the combined video data for display.
  • FIG. 2 depicts an architectural view of an example computing device, in accordance with some embodiments.
  • the example computing device 200 may be configured to carry out the functions described herein, and as depicted includes a communications interface 202 , a processor 204 , data storage 206 (that contains program instructions 208 and operational data 210 ), a user interface 212 (that contains a display 220 ), peripherals 214 (that contains a camera 216 ), and a communication bus 218 .
  • This arrangement is presented by way of example and not limitation, as other example arrangements could be described here.
  • the communication interface 202 may be configured to be operable for communication according to one or more wireless-communication protocols, some examples of which include Long-Term Evolution (LTE), IEEE 802.11 (Wi-Fi), Bluetooth, and the like.
  • the communication interface 202 may also or instead be configured to be operable for communication according to one or more wired-communication protocols, some examples of which include Ethernet and USB.)
  • the communication interface 202 may include any necessary hardware (e.g., chipsets, antennas, Ethernet interfaces, etc.), any necessary firmware, and any necessary software for conducting one or more forms of communication with one or more other entities as described herein.
  • the processor 204 may include one or more processors of any type deemed suitable by those of skill in the relevant art, some examples including a general-purpose microprocessor and a dedicated digital signal processor (DSP).
  • processors of any type deemed suitable by those of skill in the relevant art, some examples including a general-purpose microprocessor and a dedicated digital signal processor (DSP).
  • DSP dedicated digital signal processor
  • the data storage 206 may take the form of any non-transitory computer-readable medium or combination of such media, some examples including flash memory, read-only memory (ROM), and random-access memory (RAM) to name but a few, as any one or more types of non-transitory data-storage technology deemed suitable by those of skill in the relevant art could be used.
  • the data storage 206 contains program instructions 208 executable by the processor 204 for carrying out various functions described herein, and further is depicted as containing operational data 210 , which may include any one or more data values stored by and/or accessed by the computing device 200 in carrying out one or more of the functions described herein.
  • the user interface 212 may include one or more input devices (a.k.a. components and the like) and/or one or more output devices (a.k.a. components and the like.) With respect to input devices, the user interface 212 may include one or more touchscreens, buttons, switches, microphones, and the like. With respect to output devices, the user interface 212 may include one or more speakers, light emitting diodes (LEDs), and the like. In some embodiments, including the one depicted by way of example in FIG. 2 , the user interface 212 includes a display 220 . Moreover, one or more components (e.g., the display 220 could be an interactive touchscreen and display) of the user interface 212 could provide both user-input and user-output functionality.
  • the display 220 could be an interactive touchscreen and display
  • all or part of the user interface 212 may be virtually provided by the computing device 200 over a network connection; in other embodiments, the computing device 200 provides some or all of the user interface 212 via one or more connected devices such as keyboards, mice, touchpads, monitors, and/or the like. And certainly other configurations could be described here as well.
  • the peripherals 214 may include any computing device accessory, component, or the like, that is accessible to and useable by the computing device 200 during operation.
  • the peripherals 214 includes a camera 216 for, perhaps among other functions, capturing video data.
  • the camera 216 could be an optical camera, and may also have depth-camera functionality.
  • the camera 216 could be arranged to captured still images in addition to video. And certainly other example peripherals could be listed.
  • the computing device 200 obtains foreground video data. This step may be performed in several different ways. In at least one embodiment, the computing system 200 obtains foreground video data by capturing the foreground video data using the camera 216 . In at least one embodiment, the computing device 200 obtains foreground video data by receiving the foreground video data via a network connection using the communication interface 202 . And certainly other examples of ways in which the computing system 200 could obtain foreground video data at step 102 could be listed here.
  • the computing device 200 obtains background video data, which the computing device 200 may do in any of the ways described above in connection with the computing device 200 obtaining the foreground video at step 102 , or in any other manner deemed suitable in a given context by one of skill in the art.
  • a user is using the computing device 200 , which in this example is a laptop computer.
  • the computing device 200 carries out step 102 by capturing video of the user, and then extracts from that captured video a persona of the user.
  • the “foreground video” is the extracted persona.
  • the computing device 200 carries out step 104 by receiving a video feed via a network connection.
  • the “background video” is that received video feed, which could be a slide-based presentation, as one example.
  • both the foreground video and the background video are in the form of frames of pixels; furthermore, at the time these frames are obtained, those pixels are expressed in what is known as the ⁇ R,G,B ⁇ color space.
  • a pixel is an independently changeable and addressable display-data element that has properties such as color and location (e.g., Cartesian coordinates in a rectangular image).
  • properties such as color and location (e.g., Cartesian coordinates in a rectangular image).
  • higher-resolution images include greater number of pixels, and thus the potential for a higher amount of detail, then do lower-resolution images.
  • pixels can contain color information in any of a variety of different color spaces. Some examples that are discussed herein include the ⁇ R,G,B ⁇ color space, the ⁇ L,a,b ⁇ color space, and the ⁇ X,Y,Z ⁇ color space, though certainly many others could be listed here and discussed herein, as those three are listed here and discussed herein by way of example. These three example color spaces are briefly discussed below though, as stated, they are known to those of skill in the art, as are the manners of converting pixels from being expressed in one of those color spaces to being expressed in another.
  • the ⁇ R,G,B ⁇ color space expresses the color of a given pixel using a red (R) value, a green (G) value, and a blue (B) value, each of which can range from 0-255.
  • a related color space is the ⁇ R,G,B,A ⁇ color space, which adds a fourth value, the alpha (A) value, which can also range from 0-255 and is a measure of the transparency (and equivalently then the opacity) at which the given pixel should be displayed.
  • the ⁇ R,G,B ⁇ color space does not express color in a way that requires a separate brightness value. Instead, the resulting color of the pixel is the result of some amount of red-color intensity, some amount of green-color intensity, and some amount of blue-color intensity, where one or more of those amounts could be zero.
  • the ⁇ L,a,b ⁇ color space expresses the color of a given pixel using one value (L) to represent luminance (i.e., brightness) and the other two values (a and b) to represent color.
  • L can range between values of 0 and 100
  • each of ‘a’ and ‘b’ can range between ⁇ 128 and +128.
  • the combination of the ‘a’ value and the ‘b’ value identify a particular color
  • the ‘L’ value indicates the luminance level at which that particular color should be displayed.
  • the combination of the ‘a’ value and the ‘b’ value specify what is known as the chromaticity of the pixel, where chromaticity is an objective specification of the quality of a color independent of luminance, and where the combination of the chromaticity of a pixel and the luminance of the pixel specify the visible color of that pixel.
  • this disclosure considers (i) the ‘L’ value of a given pixel in the ⁇ L,a,b ⁇ color space to represent the luminance level of that pixel and (ii) the ⁇ a,b ⁇ values of a given pixel in the ⁇ L,a,b ⁇ color space to represent the color of that pixel.
  • the ⁇ X,Y,Z ⁇ color space is known as a tristimulus color space, and is based on the fact that the human eye has three types of cone cells, which are the types of cells that are responsible for color vision in medium-brightness and high-brightness situations. Each type of cone cell is essentially a filter having its peak sensitivity at a different respective wavelength of light.
  • the ⁇ X,Y,Z ⁇ color space is similar to the ⁇ L,a,b ⁇ color space in at least one respect: one of the three values (Y) is used to represent luminance and the other two values (X and Z) in combination are used to specify chromaticity (and therefore color, given a level of luminance).
  • this disclosure considers (i) the ‘Y’ value of a given pixel in the ⁇ X,Y,Z ⁇ color space to represent the luminance level of that pixel and (ii) the ⁇ X,Z ⁇ values of a given pixel in the ⁇ X,Y,Z ⁇ color space to represent the color of that pixel.
  • the “color-distribution dimensionality” of a given set of image data, video data, pixels, and/or the like is an expression of how varied (or not varied) the color values are across the particular data set.
  • a given data set that is determined to have a color-distribution dimensionality that is “high-dimensional chromatic” as that term is used herein is one that has been determined to have a relatively widely varying collection of color values across that given data set (i.e., a relatively wide variety of different colors); conversely, a given data set that is determined to have a color-distribution dimensionality that is “low-dimensional chromatic” as that term is used herein is one that has been determined to have only a relatively narrowly varying collection of color values across that given data set (i.e., a relatively narrow concentration of only a relatively small number of different colors).
  • FIG. 3 depicts an example graphical representation of example chromatically high-dimensional background video data in the ⁇ L,a,b ⁇ color space, in accordance with some embodiments.
  • FIG. 3 depicts a graph 300 having an ‘L’ axis 302 , an ‘a’ axis 304 , and a ‘b’ axis 306 that all intersect at an origin point 308 that corresponds to the point ( ⁇ L,a,b ⁇
  • ⁇ 50,0,0 ⁇ )). Passing through the point 318 are (i) a supplemental ‘a’ axis 314 that is parallel to the ‘a’ axis 304 and (ii) a supplemental ‘b’ axis 316 that is parallel to the ‘b’ axis 306 . Together, the supplemental ‘a’ axis 314 and the supplemental ‘b’ axis 316 define a plane 310 that is parallel to the plane 301 . On the plane 310 , L 50.
  • the graph 300 also includes a scattering of pixels (solid black dots) that are collectively representative of the pixels in an example background video (e.g., a frame of background video) that is deemed in this disclosure to be high-dimensional chromatic.
  • an example background video e.g., a frame of background video
  • the various pixels in the graph 300 are scattered widely with respect to the a-b plane.
  • the pixels that are numbered 320 - 326 are located in the plane 310 ; i.e., each of the pixels 320 - 326 has an ‘L’ value equal to 50.
  • the dashed circle 312 around the pixels 320 is an example graphical representation of the overall degree of variance of the pixels in the plane 310 .
  • FIG. 4 depicts an example graphical representation of an example subset of the example chromatically high-dimensional background video data of FIG. 3 , in accordance with some embodiments.
  • FIG. 4 depicts a graph 400 that is a subset of the graph 300 of FIG. 3 from a different perspective.
  • the elements that are common to both graphs are the plane 310 , the circle 312 , the supplemental ‘a’ axis 314 , the supplemental ‘b’ axis 316 , the point 318 , the pixels 320 inside the circle 312 , and the outlier pixels 322 - 326 .
  • FIG. 5 depicts an example graphical representation of example chromatically low-dimensional background video data in the ⁇ L,a,b ⁇ color space, in accordance with some embodiments.
  • FIG. 5 depicts a graph 500 that is somewhat similar to the graph 300 of FIG. 3 other than that the pixels in the graph 500 are meant to illustrate those of an example background video (e.g., a frame of background video) that is deemed in this disclosure to be low-dimensional chromatic (as opposed to high-dimensional chromatic in FIGS. 3-4 ).
  • an example background video e.g., a frame of background video
  • the graph 500 includes an ‘L’ axis 502 , an ‘a’ axis 504 , and a ‘b’ axis 506 that all intersect at an origin point 508 that corresponds to the point ( ⁇ L,a,b ⁇
  • a supplemental ‘a’ axis 514 that is parallel to the ‘a’ axis 504 and (ii) a supplemental ‘b’ axis 516 that is parallel to the ‘b’ axis 506 .
  • the supplemental ‘a’ axis 514 and the supplemental ‘b’ axis 516 define a plane 510 that is parallel to the plane 501 .
  • L 50.
  • the graph 500 also includes a (generally narrower) scattering of pixels that are collectively representative of the pixels in an example background video (e.g., a frame of background video) that is deemed in this disclosure to be low-dimensional chromatic (unlike the high-dimensional chromatic nature that is depicted in FIGS. 3-4 ).
  • an example background video e.g., a frame of background video
  • the various pixels in the graph 500 are not scattered particularly widely with respect to the a-b plane.
  • the pixels that are numbered 520 - 524 are located in the plane 510 ; i.e., each of the pixels 520 - 524 has an ‘L’ value equal to 50.
  • the dashed circle 512 around the pixels 520 is an example graphical representation of the overall degree of variance of the pixels in the plane 510 .
  • FIG. 6 depicts an example graphical representation of an example subset of the example chromatically low-dimensional background video data of FIG. 5 , in accordance with some embodiments.
  • the graph 600 of FIG. 6 is to the graph 500 of FIG. 5 as the graph 400 of FIG. 4 is to the graph 300 of FIG. 3 .
  • FIG. 6 depicts a graph 600 that is a subset of the graph 500 of FIG. 5 from a different perspective.
  • the elements that are common to both graphs are the plane 510 , the circle 512 , the supplemental ‘a’ axis 514 , the supplemental ‘b’ axis 516 , the point 518 , the pixels 520 inside the circle 512 , and the outlier pixels 522 - 524 .
  • the computing device 200 determines the color-distribution dimensionality of the background video data to be either high-dimensional chromatic or low-dimensional chromatic.
  • FIGS. 3-4 pertain to an example set of pixels that is considered high-dimensional chromatic in the terms of this disclosure
  • FIGS. 5-6 pertain to an example set of pixels that is considered to be low-dimensional chromatic in the terms of this disclosure.
  • the computing device 200 carries out step 106 at least in part by carrying out a series of sub-steps.
  • the computing device 200 converts the obtained background video data, which in this example is a single frame of background-video data (though in other examples the background-video data could be from or based on multiple frames of background-video data), from the ⁇ R,G,B ⁇ color space to the ⁇ L,a,b ⁇ color space.
  • the computing device 200 calculates the ⁇ a,b ⁇ variance of the converted background pixels.
  • the computing device 200 compares the calculated ⁇ a,b ⁇ variance to an ⁇ a,b ⁇ -variance threshold.
  • the computing device 200 determines the color-distribution dimensionality of the background video data to be (i) high-dimensional chromatic if the calculated ⁇ a,b ⁇ variance exceeds the ⁇ a,b ⁇ -variance threshold or (ii) low-dimensional chromatic if the calculated ⁇ a,b ⁇ variance does not exceed the ⁇ a,b ⁇ -variance threshold.
  • the computing device 200 could calculate the ⁇ a,b ⁇ variance of the converted background pixels in a number of different ways.
  • the ⁇ a,b ⁇ variance of a set of pixels in this case the converted background pixels—is essentially an expression of how spread out or concentrated the pixels in that set are with respect to the ‘a’ and ‘b’ dimensions.
  • the computing device 200 determines an ⁇ a,b ⁇ variance at each of a plurality of different ‘L’ values (perhaps all ‘L’ values that are represented in the converted background pixels), and on a luminance-level-by-luminance-level basis decides whether the ⁇ a,b ⁇ variance at each such luminance level exceeds a single-luminance-level threshold or degree of ⁇ a,b ⁇ variance.
  • the computing device 200 may then count up how many levels have an ⁇ a,b ⁇ variance that exceeds that single-luminance-level threshold, and then compare that count to a threshold number of levels.
  • the ⁇ a,b ⁇ -variance threshold would take the form of a certain threshold number of levels. That number could be zero or greater than zero in various different embodiments; that number could also be expressed as a fraction or percentage of the total number of luminance levels that are represented in the converted background pixels, the total number of possible luminance levels, or some other possibility.
  • the computing device 200 similarly determines an ⁇ a,b ⁇ variance at each of a plurality of different ‘L’ values (perhaps all ‘L’ values that are represented in the converted background pixels), sums those luminance-level-specific ⁇ a,b ⁇ -variance values, and then compares that sum with an ⁇ a,b ⁇ -variance threshold, which in such embodiments would take the form of a threshold aggregate amount of ⁇ a,b ⁇ variance as opposed to a number of luminance levels as described in the previous example. And certainly numerous other manners of calculating an aggregate ⁇ a,b ⁇ variance of the converted background pixels and comparing the result of that calculation to one or more threshold values could be listed here. And in this context and in others, the use of the ⁇ L,a,b ⁇ color space is by way of example and not limitation. The ⁇ X,Y,Z ⁇ color space or any other suitable color space could be used.
  • step 106 can be characterized as (i) determining the color-distribution dimensionality of the background video data to be low-dimensional chromatic if the background-color distribution of the background video data in the ⁇ L,a,b ⁇ color space can be expressed as: ⁇ ( L,a,b )
  • the background video data can be characterized as being low-dimensional chromatic; and if not, the background video data can be characterized as being high-dimensional chromatic.
  • the computing device 200 selects a chromatic-adjustment technique from a set of chromatic-adjustment techniques based on the determined color-distribution dimensionality of the background video data.
  • the computing device 200 adjusts the foreground video data using the selected chromatic-adjustment technique. The following paragraphs describe a number of different ways that the computing device 200 may carry out steps 108 and 110 .
  • the set of chromatic-adjustment techniques includes what is referred to herein as a white-balancing technique. In at least one embodiment, the computing device 200 selects the white-balancing technique when the color-distribution dimensionality of the background video data is determined to be high-dimensional chromatic. In at least one embodiment, the set of chromatic-adjustment techniques includes what is referred to herein as a chromatic-replacement technique. In at least one embodiment, the computing device 200 selects the chromatic-replacement technique when the color-distribution dimensionality of the background video data is determined to be low-dimensional chromatic.
  • the computing device 200 adjusts the foreground video data using the white-balancing technique at least in part by carrying out a number of sub-steps.
  • the computing device 200 determines an average of the pixels of the foreground video data and an average of the pixels in the background video data.
  • the computing device 200 determines both the foreground average pixel and the background average pixel in the ⁇ R,G,B ⁇ color space.
  • the computing device 200 converts both the foreground average pixel and the background average pixel from the ⁇ R,G,B ⁇ color space to a second color space, which could be ⁇ L,a,b ⁇ , ⁇ X,Y,Z ⁇ , or some other color space.
  • ⁇ L,a,b ⁇ is used by way of example in this part of the disclosure.
  • this transform matrix includes three separate dimension-wise ratios: the ratio of the background-average-pixel ‘L’ value to the foreground-average-pixel ‘L’ value, the ratio of the background-average-pixel ‘a’ value to the foreground-average-pixel ‘a’ value, and the ratio of the background-average-pixel ‘b’ value to the foreground-average-pixel ‘b’ value.
  • this transform matrix could be used.
  • the computing device 200 also converts all or substantially all of the pixels of the obtained foreground video data to the second color space, which again is ⁇ L,a,b ⁇ in this example, and transforms the converted foreground pixels in the second color space using the determined transform matrix.
  • the computing device 200 would transform the converted foreground pixels at least in part by, for each foreground pixel, multiplying the foreground pixel's ‘L’ value by the average-background-L-to-average-foreground-L ratio, (ii) multiplying the foreground pixel's ‘a’ value by the average-background-a-to-average-foreground-a ratio, and (iii) multiplying the foreground pixel's ‘b’ value by the average-background-b-to-average-foreground-b ratio.
  • the computing device 200 may then convert the now-transformed foreground pixels back to the ⁇ R,G,B ⁇ color space.
  • the computing device 200 carries out the step of converting all or substantially all of the pixels of the obtained background video data from the ⁇ R,G,B ⁇ color space to the ⁇ L,a,b ⁇ color space.
  • this conversion step is not necessarily part of carrying out the chromatic-replacement technique, since the computing device 200 may have already converted the background pixels from ⁇ R,G,B ⁇ to ⁇ L,a,b ⁇ in order to make the determination as to whether the color-distribution dimensionality of the background video data is high-dimensional chromatic or low-dimensional chromatic. If, however, the computing device 200 had not previously converted the background pixels to ⁇ L,a,b ⁇ , the computing device 200 may do so as part of—or as a necessary precursor to—carrying out the chromatic-replacement technique. In any event, the chromatic-replacement technique is described below in a manner that presumes that the background pixels have already been converted to ⁇ L,a,b ⁇ .
  • the computing device 200 may do so at least in part by carrying out a number of sub-steps.
  • One such sub-step involves generating an L-to- ⁇ a,b ⁇ lookup table based on the converted background pixels.
  • FIGS. 5-9 An example way of doing this is now described in connection with FIGS. 5-9 .
  • FIG. 5 shows an example low-dimensional-chromatic set of pixels.
  • FIGS. 5-7 are considered to represent the set of background pixels after having been converted to ⁇ L,a,b ⁇ as described above.
  • the specific example described herein is with respect to an ‘L’ value of 50, which as described above is selected purely by way of example.
  • FIG. 7 depicts an example idealization of the example subset of FIG. 6 , in accordance with some embodiments.
  • the graph 700 of FIG. 7 includes a number of elements of the graph 600 of FIG. 6 .
  • the outlier points 522 - 524 have been removed, and the pixels 520 inside the bounding circle 512 have been reduced to a single characteristic pixel 702 having a particular ‘a’ value and a particular ‘b’ value.
  • the characteristic pixel 702 could be determined by averaging the ⁇ a,b ⁇ values of a plurality of pixels (such as the pixels 520 or perhaps the pixels 520 - 524 , as examples). Another way of determining the characteristic pixel 702 could be to randomly select any one of the pixels 520 to represent the group. And certainly other approaches could be used.
  • FIG. 8 depicts an example idealization of the example chromatically low-dimensional background video data of FIG. 5 that incorporates the example idealization of FIG. 7 , in accordance with some embodiments.
  • the column 526 of pixels in the graph 500 (which encompasses all but the outliers) has been idealized into a string (or single-pixel-wide column) of pixels by way of the idealization process described above. This information can then be populated into an L-to- ⁇ a,b ⁇ lookup table.
  • FIG. 9 depicts an example L-to- ⁇ a,b ⁇ look-up table that corresponds to the example idealization of FIG. 8 , in accordance with some embodiments.
  • the a L and b L notation is used for each L in the table 900 , and also some example values for the various characteristic ‘a’ values and characteristic ‘b’ values are included as well.
  • Such extrapolation could involve simply copying the ⁇ a,b ⁇ data from a nearest ‘L’ entry, or perhaps averaging the ⁇ a,b ⁇ values from some number of proximate ‘L’ entries that have valid ⁇ a,b) data, etc. And certainly other example approaches could be listed as well.
  • the computing device 200 converts all or substantially all of the pixels of the foreground video data from the ⁇ R,G,B ⁇ color space to the ⁇ L,a,b ⁇ color space.
  • the computing device 200 may do this before or after the generation of the L-to- ⁇ a,b ⁇ look-up table 900 , which may or may not have been done in the above-described manner.
  • the computing system 200 transforms the converted foreground pixels at least in part by (a) using the respective L values of the respective converted foreground pixels to select respective replacement ⁇ a,b ⁇ values for the respective converted foreground pixels based on the L-to- ⁇ a,b ⁇ lookup table 900 and (b) replacing the respective ⁇ a,b ⁇ values of the respective converted foreground pixels with the corresponding respective selected replacement ⁇ a,b ⁇ values.
  • the computing device 200 converts the transformed foreground pixels from ⁇ L,a,b ⁇ back to ⁇ R,G,B ⁇ . An example of carrying out these last two sub-steps is described below in connection with FIGS. 10-17 .
  • FIG. 10 depicts an example table representation of example foreground video data in the ⁇ R,G,B ⁇ color space, in accordance with some embodiments.
  • FIG. 10 depicts a foreground- ⁇ R,G,B ⁇ table 1000 having four columns: a foreground-pixel index number, an ‘R’ value, a ‘G’ value, and a ‘B’ value.
  • the foreground- ⁇ R,G,B ⁇ table 1000 is also depicted as having a row for each of 307 , 200 different foreground pixels, corresponding as an example to the total number of pixels in a 640 ⁇ 480 image.
  • Some example ‘R,’ ‘G,’ and ‘B’ values are shown in the foreground- ⁇ R,G,B ⁇ table 1000 .
  • Pixel FG-2 has ⁇ R,G,B ⁇ values of ⁇ 181,95,66 ⁇ , corresponding to a light brown.
  • Pixel FG-156000 has ⁇ R,G,B ⁇ values of ⁇ 230,24,102 ⁇ , corresponding to a pinkish red.
  • pixel FG-307199 has ⁇ R,G,B ⁇ values of ⁇ 153,110,70 ⁇ , corresponding to more of a milk-chocolate brown.
  • FIG. 11 depicts an example table representation of the example foreground video data of FIG. 10 after having been converted to the ⁇ L,a,b ⁇ color space, in accordance with some embodiments.
  • FIG. 11 depicts a foreground- ⁇ L,a,b ⁇ table 1100 showing the same foreground pixels but after having been converted to ⁇ L,a,b ⁇ .
  • the table 1100 is quite similar to the table 1000 , other than that the pixels are of course expressed in the table 1100 in the ⁇ L,a,b ⁇ color space instead of in the ⁇ R,G,B ⁇ color space as is the case with table 1000 .
  • FIG. 12 depicts an example graphical representation of the example converted foreground video data of FIG. 11 in the ⁇ L,a,b ⁇ color space, in accordance with some embodiments.
  • FIG. 12 depicts a graph 1200 that shows a relatively scattered set of foreground pixels, which are meant to correspond to the pixels of FIGS. 10 and 11 (although of course expressed in the ⁇ L,a,b ⁇ color space in FIG. 12 as they are in FIG. 11 ).
  • the graph 1200 includes an ‘L’ axis 1202 , an ‘a’ axis 1204 , and a ‘b’ axis 1206 that all intersect at an origin point 1208 that corresponds to the point ( ⁇ L,a,b ⁇
  • Each foreground pixel is represented in the graph 1200 by an empty triangle.
  • the relatively high breadth (i.e., variance) of the foreground pixels collectively is shown using a brace 1210 .
  • FIG. 13 depicts an example graphical representation that shows both the example chromatically low-dimensional background video data of FIG. 5 and the example converted foreground video data of FIG. 12 , both in the ⁇ L,a,b ⁇ color space, in accordance with some embodiments.
  • FIG. 13 depicts a graph 1300 that is essentially the graphs 500 and 1200 overlaid on one another, albeit without the outlier pixels 522 - 524 from the graph 500 .
  • the degree of variety and scattering of the foreground pixels (as represented graphically by the brace 1200 ) significantly exceeds that of the background pixels (as represented graphically by the much narrower brace 526 ).
  • the graph 1300 is a depiction of the fact that, in this example, the foreground pixels as a group have significantly higher color-distribution dimensionality than do the background pixels as a group.
  • the ⁇ L,a,b ⁇ color space (as does the ⁇ X,Y,Z ⁇ color space) is defined such that luminance is in a single dimension and color (or chromaticity) is in the other two dimensions, this color space provides a useful visualization tool of the color variance of a given set of pixels at various different luminance levels and also the relative color variance of two different sets of pixels.
  • the computing system 200 transforms the converted foreground pixels at least in part by (a) using the respective L values of the respective converted foreground pixels to select respective replacement ⁇ a,b ⁇ values for the respective converted foreground pixels based on the L-to- ⁇ a,b ⁇ lookup table 900 and (b) replacing the respective ⁇ a,b ⁇ values of the respective converted foreground pixels with the corresponding respective selected replacement ⁇ a,b ⁇ values.
  • An example result of this sub-step of the chromatic-replacement technique is shown in FIG. 14 .
  • FIG. 14 depicts an example table representation of the example converted foreground video data of FIGS. 11-12 after having been transformed in the ⁇ L,a,b ⁇ color space using a chromatic-replacement technique, in accordance with some embodiments.
  • FIG. 14 depicts an example transformed-foreground- ⁇ L,a,b ⁇ table 1400 that is quite similar in layout to the foreground- ⁇ L,a,b ⁇ table 1100 of FIG. 11 .
  • What is different about the content is that the actual foreground-pixel ⁇ a,b ⁇ values that are present in the foreground- ⁇ L,a,b ⁇ table 1100 of FIG.
  • the computing device 200 in cases where the respective ‘L’ values of the foreground pixels are present in the L-to- ⁇ a,b ⁇ lookup table 900 , the computing device 200 simply replaces the corresponding ⁇ a,b ⁇ values in the table 1100 with the ⁇ a,b ⁇ values from that ‘L’ entry in the L-to- ⁇ a,b ⁇ lookup table 900 ; and in cases where the respective ‘L’ values of the foreground pixels are not present in the L-to- ⁇ a,b ⁇ lookup table 900 , the computing device 200 replaces the corresponding ⁇ a,b ⁇ values in the table 1100 with ⁇ a,b ⁇ values derived by interpolation, perhaps in one of the manners described above, from one or more ‘L’ entries that are present in the L-to- ⁇ a,b ⁇ lookup table 900 .
  • FIG. 15 depicts an example graphical representation of the transformed foreground video data of FIG. 14 in the ⁇ L,a,b ⁇ color space, in accordance with some embodiments.
  • FIG. 15 depicts a graph 1500 that plots the transformed foreground pixels that are listed in tabular form in the transformed-foreground- ⁇ L,a,b ⁇ table 1400 of FIG. 14 .
  • the graph 1500 includes an ‘L’ axis 1502 , an ‘a’ axis 1504 , and a ‘b’ axis 1506 that all intersect at an origin point 1508 that corresponds to the point ( ⁇ L,a,b ⁇
  • the plot of foreground pixels in the graph 1500 appears quite similar to the idealization of the background pixels that is shown in the graph 800 of FIG. 8 .
  • the graph 1500 could be viewed as a plot of only the unique pixels in the transformed foreground pixels of FIG. 14 .
  • the narrow color-distribution dimensionality of the transformed foreground pixels is shown visually by the relatively narrow brace 1510 .
  • FIG. 16 depicts an example graphical representation that shows both the example chromatically low-dimensional background video data of FIG. 5 and the example transformed foreground video data of FIG. 15 , both in the ⁇ L,a,b ⁇ color space, in accordance with some embodiments.
  • the graph 1600 of FIG. 16 is quite similar to the graph 1500 of FIG. 15 , other than that the pixel column and corresponding brace 526 from FIG. 5 have been added. It is apparent from inspection of the graph 1600 that the color-distribution dimensionality of the transformed foreground pixels (as shown be the narrower brace 1510 ) is less than that of the original background pixels (minus the outliers) (as shown by the wider brace 526 ).
  • the post-foreground-transformation situation that is depicted in the graph 1600 of FIG. 16 illustrates that the transformed foreground pixels are now well within the color range of the original background pixels.
  • FIG. 17 depicts an example table representation of the example transformed foreground video data of FIGS. 14-15 after having been converted to the ⁇ R,G,B ⁇ color space, in accordance with some embodiments.
  • the table 1700 has the same content as the table 1400 of FIG. 14 , albeit with the ⁇ L,a,b ⁇ expressions of the foreground pixels from the table 1400 having been replaced by the ⁇ R,G,B ⁇ expressions of the foreground pixels in the table 1700 .
  • the rows 1702 , 1704 , and 1706 show respectively that the pixels FG-2, FG-156000, and FG-307199 each now have the ⁇ R,G,B ⁇ values of ⁇ 201,79,67 ⁇ , which corresponds to a dark orange.
  • the computing device 200 generates combined video data at least in part by combining the background video data with the adjusted foreground video data.
  • This step may be performed in several different ways.
  • the computing device 200 combines the adjusted foreground video data with the background video data using what is known is an alpha mask, which is essentially a mapping of all of the pixel locations that will be in the combined video, where in that mapping, a ‘1’ value indicates that the corresponding pixel location should be filled with the corresponding pixel from the foreground video and a ‘0’ value indicates that the corresponding pixel location should be filled with the corresponding pixel from the background video.
  • an alpha mask is essentially a mapping of all of the pixel locations that will be in the combined video, where in that mapping, a ‘1’ value indicates that the corresponding pixel location should be filled with the corresponding pixel from the foreground video and a ‘0’ value indicates that the corresponding pixel location should be filled with the corresponding pixel from the background video.
  • other techniques could be used,
  • generating the combined video at least in part by combining the adjusted foreground video data with the background video data includes performing a smoothing operation to a boundary between the adjusted foreground video data and the background video data.
  • the smoothing operation includes a Poisson image-blending technique.
  • the method 100 also includes (i) obtaining second foreground video data and (ii) adjusting the second foreground video data using the selected chromatic-adjustment technique, and generating the combined video data includes combining the background video data with both the adjusted foreground video data and the adjusted second foreground video data.
  • foreground videos e.g., personas of respective participants in a given collaborative online communication session
  • the computing device 200 outputs the combined video data for display. This step may be performed in several different ways. In at least one embodiment, the computing device 200 outputs the combined video data for display via the display 220 and/or one or more other displays. In at least one embodiment, the computing device 200 outputs the combined video data via a network connection for display on one or more other devices. And certainly other examples could be listed.
  • one or more videos are chromatically adjusted based on another video and then combined with that other video; in some cases, one or more videos that are to be combined could each be chromatically adjusted based on a common reference point (e.g., a given white point, grey point, or the like).
  • a “mood” or “style” or other setting could be available to chromatically adjust one or more videos (one or more “foreground” videos, a “background” video, and/or one or more other videos) prior to combination using chromatic profiles such as sepia, high brightness, pastels, earth tones, and/or one or more others deemed suitable by those in the art for a given implementation.

Abstract

Disclosed herein are methods and systems for combining foreground video and background video using chromatic matching. In an embodiment, a system obtains foreground video data. The system obtains background video data. The system determines a color-distribution dimensionality of the background video data to be either high-dimensional chromatic or low-dimensional chromatic. The system selects a chromatic-adjustment technique from a set of chromatic-adjustment techniques based on the determined color-distribution dimensionality of the background video data. The system adjusts the foreground video data using the selected chromatic-adjustment technique. The system generates combined video data at least in part by combining the background video data with the adjusted foreground video data. The system outputs the combined video for display.

Description

BACKGROUND
The combination of multiple images, e.g., a foreground image and a background image, into composite images is an important task in various different commercial industries and other contexts. If two images, which could be partial or complete frames of video data, are combined inartfully, the result may be a visually unpleasant and artificial-looking composition.
BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 depicts an example method, in accordance with some embodiments.
FIG. 2 depicts an architectural view of an example computing device, in accordance with some embodiments.
FIG. 3 depicts an example graphical representation of example chromatically high-dimensional background video data in the {L,a,b} color space, in accordance with some embodiments.
FIG. 4 depicts an example graphical representation of an example subset of the example chromatically high-dimensional background video data of FIG. 3, in accordance with some embodiments.
FIG. 5 depicts an example graphical representation of example chromatically low-dimensional background video data in the {L,a,b} color space, in accordance with some embodiments.
FIG. 6 depicts an example graphical representation of an example subset of the example chromatically low-dimensional background video data of FIG. 5, in accordance with some embodiments.
FIG. 7 depicts an example idealization of the example subset of FIG. 6, in accordance with some embodiments.
FIG. 8 depicts an example idealization of the example chromatically low-dimensional background video data of FIG. 5 that incorporates the example idealization of FIG. 7, in accordance with some embodiments.
FIG. 9 depicts an example L-to-{a,b} look-up table that corresponds to the example idealization of FIG. 8, in accordance with some embodiments.
FIG. 10 depicts an example table representation of example foreground video data in the {R,G,B} color space, in accordance with some embodiments.
FIG. 11 depicts an example table representation of the example foreground video data of FIG. 10 after having been converted to the {L,a,b} color space, in accordance with some embodiments.
FIG. 12 depicts an example graphical representation of the example converted foreground video data of FIG. 11 in the {L,a,b} color space, in accordance with some embodiments.
FIG. 13 depicts an example graphical representation that shows both the example chromatically low-dimensional background video data of FIG. 5 and the example converted foreground video data of FIG. 12, both in the {L,a,b} color space, in accordance with some embodiments.
FIG. 14 depicts an example table representation of the example converted foreground video data of FIGS. 11-12 after having been transformed in the {L,a,b} color space using a chromatic-replacement technique, in accordance with some embodiments.
FIG. 15 depicts an example graphical representation of the transformed foreground video data of FIG. 14 in the {L,a,b} color space, in accordance with some embodiments.
FIG. 16 depicts an example graphical representation that shows both the example chromatically low-dimensional background video data of FIG. 5 and the example transformed foreground video data of FIG. 15, both in the {L,a,b} color space, in accordance with some embodiments.
FIG. 17 depicts an example table representation of the example transformed foreground video data of FIGS. 14-15 after having been converted to the {R,G,B} color space, in accordance with some embodiments.
 DETAILED DESCRIPTION
One embodiment takes the form of a method that includes (i) obtaining foreground video data; (ii) obtaining background video data; (iii) determining a color-distribution dimensionality of the background video data to be either high-dimensional chromatic or low-dimensional chromatic; (iv) selecting a chromatic-adjustment technique from a set of chromatic-adjustment techniques based on the determined color-distribution dimensionality of the background video data; (v) adjusting the foreground video data using the selected chromatic-adjustment technique; (vi) generating combined video data at least in part by combining the background video data with the adjusted foreground video data; and (vii) outputting the combined video data for display.
Another embodiment takes the form of a system that includes a communication interface, a processor, and a non-transitory computer-readable medium storing instructions executable by the processor for causing the system to perform at least the functions listed in the preceding paragraph.
Moreover, any of the variations and permutations described herein can be implemented with respect to any embodiments, including with respect to any method embodiments and with respect to any system embodiments. Furthermore, this flexibility and cross-applicability of embodiments is present in spite of the use of slightly different language (e.g., process, method, steps, functions, set of functions, and the like) to describe and or characterize such embodiments.
In at least one embodiment, determining the color-distribution dimensionality of the background video data to be either high-dimensional chromatic or low-dimensional chromatic includes (i) converting pixels of the background video data from an {R,G,B} color space to an {L,a,b} color space; (ii) calculating an {a,b} variance of the converted background pixels; (iii) comparing the calculated {a,b} variance to an {a,b}-variance threshold; (iv) determining the color-distribution dimensionality of the background video data to be high-dimensional chromatic if the calculated {a,b} variance exceeds the {a,b}-variance threshold; and (v) determining the color-distribution dimensionality of the background video data to be low-dimensional chromatic if the calculated {a,b} variance does not exceed the {a,b}-variance threshold.
In at least one embodiment, calculating the {a,b} variance of the converted background pixels includes determining how many luminance levels in the converted background pixels have more than a luminance-level-specific degree of {a,b} variance. In at least one such embodiment, the {a,b}-variance threshold is a threshold number of luminance levels; in at least one such embodiment, the threshold number of luminance levels is zero; in at least one other such embodiment, the threshold number of luminance levels is greater than zero.
In at least one embodiment, calculating the {a,b} variance of the converted background pixels includes (i) determining a respective luminance-level-specific {a,b} variance for each of a plurality of luminance levels that are represented in the converted background pixels and (ii) calculating the {a,b} variance of the converted background pixels to be a sum of the determined luminance-level-specific {a,b} variances.
In at least one embodiment, calculating the {a,b} variance of the converted background pixels includes (i) determining a respective luminance-level-specific {a,b} variance for each luminance level represented in the converted background pixels and (ii) calculating the {a,b} variance of the converted background pixels to be a sum of the determined luminance-level-specific {a,b} variances.
In at least one embodiment, determining the color-distribution dimensionality of the background video data to be either high-dimensional chromatic or low-dimensional chromatic includes (i) determining the color-distribution dimensionality of the background video data to be low-dimensional chromatic if a background-color distribution of the background video data in an {L,a,b} color space is supported by a relationship defined by:
{(L,a,b)|a=f a(L),b=f b(L)}  (Eq. 1)
where fa and fb are functions and (ii) otherwise determining the color-distribution dimensionality of the background video data to be high-dimensional chromatic.
In at least one embodiment: (i) the set of chromatic-adjustment techniques includes a white-balancing technique and a chromatic-replacement technique and (ii) selecting a chromatic-adjustment technique based on the determined color-distribution dimensionality includes (a) selecting the white-balancing technique when the color-distribution dimensionality of the background video data is determined to be high-dimensional chromatic and (b) selecting the chromatic-replacement technique when the color-distribution dimensionality of the background video data is determined to be low-dimensional chromatic.
In at least one embodiment, adjusting the foreground video data using the white-balancing technique includes (i) determining a foreground average of pixels of the foreground video data in an {R,G,B} color space; (ii) determining a background average of pixels of the background video data in the {R,G,B} color space; (iii) converting the foreground average and the background average from the {R,G,B} color space to a second color space; (iv) determining a transform matrix in the second color space from the converted foreground average to the converted background average; (v) converting the pixels of the foreground video data from the {R,G,B} color space to the second color space; (vi) transforming the converted foreground pixels in the second color space using the determined transform matrix; and (vii) converting the transformed foreground pixels from the second color space to the {R,G,B} color space.
In at least one embodiment, the determined transform matrix includes dimension-wise ratios in the second color space of the converted background average to the converted foreground average. In at least one embodiment, the second color space is an {L,a,b} color space. In at least one embodiment, the second color space is an {X,Y,Z} color space. In at least one such embodiment, converting the foreground pixels from the {R,G,B} color space to the second color space includes converting the foreground pixels from the {R,G,B} color space to an {L,a,b} color space and then from the {L,a,b} color space to the {X,Y,Z} color space.
In at least one embodiment, the method also includes converting pixels of the background video data from an {R,G,B} color space to an {L,a,b} color space, and adjusting the foreground video data using the chromatic-replacement technique includes: (i) generating an L-to-{a,b} lookup table based on the converted background pixels; (ii) converting pixels of the foreground video data from the {R,G,B} color space to the {L,a,b} color space; (iii) transforming the converted foreground pixels at least in part by (a) using the respective L values of the respective converted foreground pixels to select respective replacement {a,b} values for the respective converted foreground pixels based on the L-to-{a,b} lookup table and (b) replacing the respective {a,b} values of the respective converted foreground pixels with the corresponding respective selected replacement {a,b} values; and (iv) converting the transformed foreground pixels from the {L,a,b} color space to the {R,G,B} color space.
In at least one embodiment, using the respective L values of the respective converted foreground pixels to select the respective replacement {a,b} values for the respective converted foreground pixels based on the L-to-{a,b} lookup table includes retrieving the respective replacement {a,b} values from the L-to-{a,b} lookup table in cases where the respective L value of the respective converted foreground pixel is listed in the L-to-{a,b} lookup table.
In at least one embodiment, using the respective L values of the respective converted foreground pixels to select the respective replacement {a,b} values for the respective converted foreground pixels based on the L-to-{a,b} lookup table further includes using interpolated {a,b} values based on one or more entries in the L-to-{a,b} lookup table as the respective replacement {a,b} values in cases where the respective L value of the respective converted foreground pixel is not listed in the L-to-{a,b} lookup table. In at least one such embodiment, the interpolated {a,b} values are copied from a nearest L value that is listed in the L-to-{a,b} lookup table; in at least one other such embodiment, the interpolated {a,b} values are average {a,b} values of two or more proximate entries in the L-to-{a,b} lookup table.
In at least one embodiment, the method also includes (i) obtaining second foreground video data and (ii) adjusting the second foreground video data using the selected chromatic-adjustment technique, and generating the combined video data includes combining the background video data with both the adjusted foreground video data and the adjusted second foreground video data.
Before proceeding with this detailed description, it is noted that the entities, connections, arrangements, and the like that are depicted in—and described in connection with—the various figures are presented by way of example and not by way of limitation. As such, any and all statements or other indications as to what a particular figure “depicts,” what a particular element or entity in a particular figure “is” or “has,” and any and all similar statements—that may in isolation and out of context be read as absolute and therefore limiting—can only properly be read as being constructively preceded by a clause such as “In at least one embodiment, . . . .” And it is for reasons akin to brevity and clarity of presentation that this implied leading clause is not repeated ad nauseam in this detailed description.
FIG. 1 depicts an example method, in accordance with some embodiments.
In particular, FIG. 1 depicts a method 100 that is described below by way of example as being carried out by the example computing device (or system) 200 that is described in connection with FIG. 2. In general, the method 100 could be carried out by any computing device that is suitably equipped, programmed, and configured. The method 100 includes steps 102-114, which are described briefly in connection with FIG. 1 and more fully in connection with the ensuing figures.
At step 102, the computing device 200 obtains foreground video data. At step 104, the computing device 200 obtains background video data. At step 106, the computing device determines a color-distribution dimensionality of the background video data to be either high-dimensional chromatic or low-dimensional chromatic. At step 108, the computing device 200 selects a chromatic-adjustment technique from a set of chromatic-adjustment techniques based on the determined color-distribution dimensionality of the background video data. At step 110, the computing device 200 adjusts the foreground video data using the selected chromatic-adjustment technique. At step 112, the computing device 200 generates combined video data at least in part by combining the background video data with the adjusted foreground video data. At step 114, the computing device 200 outputs the combined video data for display.
FIG. 2 depicts an architectural view of an example computing device, in accordance with some embodiments. The example computing device 200 may be configured to carry out the functions described herein, and as depicted includes a communications interface 202, a processor 204, data storage 206 (that contains program instructions 208 and operational data 210), a user interface 212 (that contains a display 220), peripherals 214 (that contains a camera 216), and a communication bus 218. This arrangement is presented by way of example and not limitation, as other example arrangements could be described here.
The communication interface 202 may be configured to be operable for communication according to one or more wireless-communication protocols, some examples of which include Long-Term Evolution (LTE), IEEE 802.11 (Wi-Fi), Bluetooth, and the like. The communication interface 202 may also or instead be configured to be operable for communication according to one or more wired-communication protocols, some examples of which include Ethernet and USB.) The communication interface 202 may include any necessary hardware (e.g., chipsets, antennas, Ethernet interfaces, etc.), any necessary firmware, and any necessary software for conducting one or more forms of communication with one or more other entities as described herein.
The processor 204 may include one or more processors of any type deemed suitable by those of skill in the relevant art, some examples including a general-purpose microprocessor and a dedicated digital signal processor (DSP).
The data storage 206 may take the form of any non-transitory computer-readable medium or combination of such media, some examples including flash memory, read-only memory (ROM), and random-access memory (RAM) to name but a few, as any one or more types of non-transitory data-storage technology deemed suitable by those of skill in the relevant art could be used. As depicted in FIG. 2, the data storage 206 contains program instructions 208 executable by the processor 204 for carrying out various functions described herein, and further is depicted as containing operational data 210, which may include any one or more data values stored by and/or accessed by the computing device 200 in carrying out one or more of the functions described herein.
The user interface 212 may include one or more input devices (a.k.a. components and the like) and/or one or more output devices (a.k.a. components and the like.) With respect to input devices, the user interface 212 may include one or more touchscreens, buttons, switches, microphones, and the like. With respect to output devices, the user interface 212 may include one or more speakers, light emitting diodes (LEDs), and the like. In some embodiments, including the one depicted by way of example in FIG. 2, the user interface 212 includes a display 220. Moreover, one or more components (e.g., the display 220 could be an interactive touchscreen and display) of the user interface 212 could provide both user-input and user-output functionality. In some cases, such as in the case of the computing device 200 being a networked server, all or part of the user interface 212 may be virtually provided by the computing device 200 over a network connection; in other embodiments, the computing device 200 provides some or all of the user interface 212 via one or more connected devices such as keyboards, mice, touchpads, monitors, and/or the like. And certainly other configurations could be described here as well.
The peripherals 214 may include any computing device accessory, component, or the like, that is accessible to and useable by the computing device 200 during operation. In some embodiments, including the one depicted by way of example in FIG. 2, the peripherals 214 includes a camera 216 for, perhaps among other functions, capturing video data. The camera 216 could be an optical camera, and may also have depth-camera functionality. The camera 216 could be arranged to captured still images in addition to video. And certainly other example peripherals could be listed.
Returning to FIG. 1, at step 102, the computing device 200 obtains foreground video data. This step may be performed in several different ways. In at least one embodiment, the computing system 200 obtains foreground video data by capturing the foreground video data using the camera 216. In at least one embodiment, the computing device 200 obtains foreground video data by receiving the foreground video data via a network connection using the communication interface 202. And certainly other examples of ways in which the computing system 200 could obtain foreground video data at step 102 could be listed here.
At step 104, the computing device 200 obtains background video data, which the computing device 200 may do in any of the ways described above in connection with the computing device 200 obtaining the foreground video at step 102, or in any other manner deemed suitable in a given context by one of skill in the art.
Thus, in an example scenario, a user is using the computing device 200, which in this example is a laptop computer. The computing device 200 carries out step 102 by capturing video of the user, and then extracts from that captured video a persona of the user. In this example, then, the “foreground video” is the extracted persona. For an example procedure for persona extraction from a video feed, see U.S. Pat. No. 8,818,028, which issued Aug. 26, 2014 and is entitled “Systems and Methods for Accurate User Foreground Video Extraction,” the entire contents of which are hereby incorporated herein by reference.
Further to this example scenario, the computing device 200 carries out step 104 by receiving a video feed via a network connection. In this example, then, the “background video” is that received video feed, which could be a slide-based presentation, as one example. Further to this example scenario, at the time they are obtained by the computing device 200, both the foreground video and the background video are in the form of frames of pixels; furthermore, at the time these frames are obtained, those pixels are expressed in what is known as the {R,G,B} color space.
As is known in the art, a pixel is an independently changeable and addressable display-data element that has properties such as color and location (e.g., Cartesian coordinates in a rectangular image). Generally speaking, higher-resolution images include greater number of pixels, and thus the potential for a higher amount of detail, then do lower-resolution images. As is further known in the art, pixels can contain color information in any of a variety of different color spaces. Some examples that are discussed herein include the {R,G,B} color space, the {L,a,b} color space, and the {X,Y,Z} color space, though certainly many others could be listed here and discussed herein, as those three are listed here and discussed herein by way of example. These three example color spaces are briefly discussed below though, as stated, they are known to those of skill in the art, as are the manners of converting pixels from being expressed in one of those color spaces to being expressed in another.
The {R,G,B} color space expresses the color of a given pixel using a red (R) value, a green (G) value, and a blue (B) value, each of which can range from 0-255. Moreover, a related color space is the {R,G,B,A} color space, which adds a fourth value, the alpha (A) value, which can also range from 0-255 and is a measure of the transparency (and equivalently then the opacity) at which the given pixel should be displayed. The {R,G,B} color space, then, does not express color in a way that requires a separate brightness value. Instead, the resulting color of the pixel is the result of some amount of red-color intensity, some amount of green-color intensity, and some amount of blue-color intensity, where one or more of those amounts could be zero.
The {L,a,b} color space expresses the color of a given pixel using one value (L) to represent luminance (i.e., brightness) and the other two values (a and b) to represent color. Typically, for a given pixel, ‘L’ can range between values of 0 and 100, while each of ‘a’ and ‘b’ can range between −128 and +128. Thus, the combination of the ‘a’ value and the ‘b’ value identify a particular color, and the ‘L’ value indicates the luminance level at which that particular color should be displayed. More particularly, the combination of the ‘a’ value and the ‘b’ value specify what is known as the chromaticity of the pixel, where chromaticity is an objective specification of the quality of a color independent of luminance, and where the combination of the chromaticity of a pixel and the luminance of the pixel specify the visible color of that pixel. For simplicity of explanation, this disclosure considers (i) the ‘L’ value of a given pixel in the {L,a,b} color space to represent the luminance level of that pixel and (ii) the {a,b} values of a given pixel in the {L,a,b} color space to represent the color of that pixel.
The {X,Y,Z} color space is known as a tristimulus color space, and is based on the fact that the human eye has three types of cone cells, which are the types of cells that are responsible for color vision in medium-brightness and high-brightness situations. Each type of cone cell is essentially a filter having its peak sensitivity at a different respective wavelength of light. The {X,Y,Z} color space is similar to the {L,a,b} color space in at least one respect: one of the three values (Y) is used to represent luminance and the other two values (X and Z) in combination are used to specify chromaticity (and therefore color, given a level of luminance). For simplicity of explanation, this disclosure considers (i) the ‘Y’ value of a given pixel in the {X,Y,Z} color space to represent the luminance level of that pixel and (ii) the {X,Z} values of a given pixel in the {X,Y,Z} color space to represent the color of that pixel.
Moreover, in general, as the term is used herein, the “color-distribution dimensionality” of a given set of image data, video data, pixels, and/or the like is an expression of how varied (or not varied) the color values are across the particular data set. Moreover, a given data set that is determined to have a color-distribution dimensionality that is “high-dimensional chromatic” as that term is used herein is one that has been determined to have a relatively widely varying collection of color values across that given data set (i.e., a relatively wide variety of different colors); conversely, a given data set that is determined to have a color-distribution dimensionality that is “low-dimensional chromatic” as that term is used herein is one that has been determined to have only a relatively narrowly varying collection of color values across that given data set (i.e., a relatively narrow concentration of only a relatively small number of different colors).
FIG. 3 depicts an example graphical representation of example chromatically high-dimensional background video data in the {L,a,b} color space, in accordance with some embodiments. In particular, FIG. 3 depicts a graph 300 having an ‘L’ axis 302, an ‘a’ axis 304, and a ‘b’ axis 306 that all intersect at an origin point 308 that corresponds to the point ({L,a,b}|{0,0,0}). As depicted in FIG. 3, the ‘a’ axis 304 and the ‘b’ axis 306 together define a plane 301 at which L=0. The graph 300 also includes a point 318 on the ‘L’ axis at which L=50 (i.e., the point ({L,a,b}|{50,0,0})). Passing through the point 318 are (i) a supplemental ‘a’ axis 314 that is parallel to the ‘a’ axis 304 and (ii) a supplemental ‘b’ axis 316 that is parallel to the ‘b’ axis 306. Together, the supplemental ‘a’ axis 314 and the supplemental ‘b’ axis 316 define a plane 310 that is parallel to the plane 301. On the plane 310, L=50.
The graph 300 also includes a scattering of pixels (solid black dots) that are collectively representative of the pixels in an example background video (e.g., a frame of background video) that is deemed in this disclosure to be high-dimensional chromatic. As can be seen in FIG. 3, the various pixels in the graph 300 are scattered widely with respect to the a-b plane. Moreover, the pixels that are numbered 320-326 are located in the plane 310; i.e., each of the pixels 320-326 has an ‘L’ value equal to 50. As is explained more fully below in connection with FIG. 4, the dashed circle 312 around the pixels 320 is an example graphical representation of the overall degree of variance of the pixels in the plane 310.
It is further noted that the number of pixels that is depicted in FIG. 3 and each of the other figures that graphically display pixels is by way of example only; in actual implementations, any number of pixels could be present in any one of the figures. Furthermore, as explained below, some of the figures have corresponding elements; for example, the circle 312 and the pixels 320 are shown in both FIG. 3 and FIG. 4; in such instances, these elements are meant to correspond to one another, but minor differences such as different numbers of pixels 320 being displayed in the circle 312 in FIG. 3 vs. FIG. 4 should not be interpreted as mistakes; these figures are meant to be explanatory and helpful in nature, and are used herein to illustrate various aspects of this disclosure. Moreover, the selection of L=50 as an illustrative and explanatory ‘L’ value is purely by way of example and not limitation, as any one or more other ‘L’ values could have used instead or in addition for purposes of illustration and exposition.
FIG. 4 depicts an example graphical representation of an example subset of the example chromatically high-dimensional background video data of FIG. 3, in accordance with some embodiments. Essentially, FIG. 4 depicts a graph 400 that is a subset of the graph 300 of FIG. 3 from a different perspective. The graph 400, then, depicts all of the points from the graph 300 for which L=50. The elements that are common to both graphs are the plane 310, the circle 312, the supplemental ‘a’ axis 314, the supplemental ‘b’ axis 316, the point 318, the pixels 320 inside the circle 312, and the outlier pixels 322-326. The circle 312 is meant to be a graphical illustration of the degree of variance in the a-b plane (i.e., the “{a,b} variance”) of the pixels from the graph 300 at the luminance level L=50. If conceptualized as a minimum bounding circle (around all pixels or around all pixels except for a few outliers), then various different dimensions of the circle 312 could be used as proxies for the degree of {a,b} variance in the plane 310. Some such dimensions of the circle 312 that could be used as an {a,b}-variance proxy include circumference, area, radius, and diameter.
FIG. 5 depicts an example graphical representation of example chromatically low-dimensional background video data in the {L,a,b} color space, in accordance with some embodiments. In particular, FIG. 5 depicts a graph 500 that is somewhat similar to the graph 300 of FIG. 3 other than that the pixels in the graph 500 are meant to illustrate those of an example background video (e.g., a frame of background video) that is deemed in this disclosure to be low-dimensional chromatic (as opposed to high-dimensional chromatic in FIGS. 3-4). As depicted in FIG. 5, the graph 500 includes an ‘L’ axis 502, an ‘a’ axis 504, and a ‘b’ axis 506 that all intersect at an origin point 508 that corresponds to the point ({L,a,b}|{0,0,0}). As depicted in FIG. 5, the ‘a’ axis 504 and the ‘b’ axis 506 together define a plane 501 at which L=0. The graph 500 also includes a point 518 on the ‘L’ axis at which L=50 (i.e., the point ({L,a,b}|{50,0,0})). Passing through the point 518 are (i) a supplemental ‘a’ axis 514 that is parallel to the ‘a’ axis 504 and (ii) a supplemental ‘b’ axis 516 that is parallel to the ‘b’ axis 506. Together, the supplemental ‘a’ axis 514 and the supplemental ‘b’ axis 516 define a plane 510 that is parallel to the plane 501. On the plane 510, L=50.
As alluded to above, the graph 500 also includes a (generally narrower) scattering of pixels that are collectively representative of the pixels in an example background video (e.g., a frame of background video) that is deemed in this disclosure to be low-dimensional chromatic (unlike the high-dimensional chromatic nature that is depicted in FIGS. 3-4). As can be seen in FIG. 5, the various pixels in the graph 500 are not scattered particularly widely with respect to the a-b plane. Moreover, the pixels that are numbered 520-524 are located in the plane 510; i.e., each of the pixels 520-524 has an ‘L’ value equal to 50. As is explained more fully below in connection with FIG. 5, the dashed circle 512 around the pixels 520 is an example graphical representation of the overall degree of variance of the pixels in the plane 510.
FIG. 6 depicts an example graphical representation of an example subset of the example chromatically low-dimensional background video data of FIG. 5, in accordance with some embodiments. On a general level, the graph 600 of FIG. 6 is to the graph 500 of FIG. 5 as the graph 400 of FIG. 4 is to the graph 300 of FIG. 3. Thus, essentially, FIG. 6 depicts a graph 600 that is a subset of the graph 500 of FIG. 5 from a different perspective. The graph 600, then, depicts all of the points from the graph 500 for which L=50. The elements that are common to both graphs are the plane 510, the circle 512, the supplemental ‘a’ axis 514, the supplemental ‘b’ axis 516, the point 518, the pixels 520 inside the circle 512, and the outlier pixels 522-524. The circle 512 is meant to be a graphical illustration of the “{a,b} variance” of the pixels from the graph 500 at the luminance level L=50. If conceptualized as a minimum bounding circle (around all pixels or around all pixels except for a few outliers), then, similar to the circle 312, various different dimensions (e.g., circumference, area, radius, diameter) of the circle 512 could be used as proxies for the degree of {a,b} variance in the plane 510.
At step 106, the computing device 200 determines the color-distribution dimensionality of the background video data to be either high-dimensional chromatic or low-dimensional chromatic. As described above, FIGS. 3-4 pertain to an example set of pixels that is considered high-dimensional chromatic in the terms of this disclosure, while FIGS. 5-6 pertain to an example set of pixels that is considered to be low-dimensional chromatic in the terms of this disclosure.
In at least one embodiment, the computing device 200 carries out step 106 at least in part by carrying out a series of sub-steps. First, the computing device 200 converts the obtained background video data, which in this example is a single frame of background-video data (though in other examples the background-video data could be from or based on multiple frames of background-video data), from the {R,G,B} color space to the {L,a,b} color space. Second, the computing device 200 calculates the {a,b} variance of the converted background pixels. Third, the computing device 200 compares the calculated {a,b} variance to an {a,b}-variance threshold. Fourth and last, the computing device 200 determines the color-distribution dimensionality of the background video data to be (i) high-dimensional chromatic if the calculated {a,b} variance exceeds the {a,b}-variance threshold or (ii) low-dimensional chromatic if the calculated {a,b} variance does not exceed the {a,b}-variance threshold. Various example ways in which these sub-steps could be carried out are described below.
The computing device 200 could calculate the {a,b} variance of the converted background pixels in a number of different ways. In general, as stated above, as used herein, the {a,b} variance of a set of pixels—in this case the converted background pixels—is essentially an expression of how spread out or concentrated the pixels in that set are with respect to the ‘a’ and ‘b’ dimensions.
In some embodiments, the computing device 200 determines an {a,b} variance at each of a plurality of different ‘L’ values (perhaps all ‘L’ values that are represented in the converted background pixels), and on a luminance-level-by-luminance-level basis decides whether the {a,b} variance at each such luminance level exceeds a single-luminance-level threshold or degree of {a,b} variance. The computing device 200 may then count up how many levels have an {a,b} variance that exceeds that single-luminance-level threshold, and then compare that count to a threshold number of levels. In such embodiments, the {a,b}-variance threshold would take the form of a certain threshold number of levels. That number could be zero or greater than zero in various different embodiments; that number could also be expressed as a fraction or percentage of the total number of luminance levels that are represented in the converted background pixels, the total number of possible luminance levels, or some other possibility.
In some embodiments, the computing device 200 similarly determines an {a,b} variance at each of a plurality of different ‘L’ values (perhaps all ‘L’ values that are represented in the converted background pixels), sums those luminance-level-specific {a,b}-variance values, and then compares that sum with an {a,b}-variance threshold, which in such embodiments would take the form of a threshold aggregate amount of {a,b} variance as opposed to a number of luminance levels as described in the previous example. And certainly numerous other manners of calculating an aggregate {a,b} variance of the converted background pixels and comparing the result of that calculation to one or more threshold values could be listed here. And in this context and in others, the use of the {L,a,b} color space is by way of example and not limitation. The {X,Y,Z} color space or any other suitable color space could be used.
With respect to the calculation of the {a,b} variance at any given luminance level, this could involve a rigorous mathematical calculation of variance (i.e., the expected value of the squared deviation from the mean in both the ‘a’ and ‘b’ axes), and could instead or in addition involve a more “rough” calculation or proxy for the {a,b} variance such as the area, circumference, radius, or diameter of a minimum-bounding circle with respect to the pixels (perhaps excluding one or more outliers) at that luminance level. And certainly the dimensions and/or area or the like of shapes other than a minimum-bounding circle could be used as well or instead.
In at least one embodiment, step 106 can be characterized as (i) determining the color-distribution dimensionality of the background video data to be low-dimensional chromatic if the background-color distribution of the background video data in the {L,a,b} color space can be expressed as:
{(L,a,b)|a=f a(L),b=f b(L)},
where fa and fb are functions and (ii) otherwise determining the color-distribution dimensionality of the background video data to be high-dimensional chromatic. This is essentially saying that, if knowing the ‘L’ value of a given pixel is sufficient to identify both the ‘a’ value and the ‘b’ value for that pixel (or at least approximate the ‘a’ and ‘b’ values closely enough because the pixels are relatively concentrated in the a-b plane for all or a sufficient number of ‘L’ values), then the background video data can be characterized as being low-dimensional chromatic; and if not, the background video data can be characterized as being high-dimensional chromatic.
Returning to FIG. 1, at step 108, the computing device 200 selects a chromatic-adjustment technique from a set of chromatic-adjustment techniques based on the determined color-distribution dimensionality of the background video data. At step 110, the computing device 200 adjusts the foreground video data using the selected chromatic-adjustment technique. The following paragraphs describe a number of different ways that the computing device 200 may carry out steps 108 and 110.
In at least one embodiment, the set of chromatic-adjustment techniques includes what is referred to herein as a white-balancing technique. In at least one embodiment, the computing device 200 selects the white-balancing technique when the color-distribution dimensionality of the background video data is determined to be high-dimensional chromatic. In at least one embodiment, the set of chromatic-adjustment techniques includes what is referred to herein as a chromatic-replacement technique. In at least one embodiment, the computing device 200 selects the chromatic-replacement technique when the color-distribution dimensionality of the background video data is determined to be low-dimensional chromatic.
This and the next few paragraphs provide an example as to how the computing device 200 could carry out the white-balancing technique in cases where that is the selected chromatic-adjustment technique. In at least one embodiment, the computing device 200 adjusts the foreground video data using the white-balancing technique at least in part by carrying out a number of sub-steps. First, the computing device 200 determines an average of the pixels of the foreground video data and an average of the pixels in the background video data. In an embodiment, the computing device 200 determines both the foreground average pixel and the background average pixel in the {R,G,B} color space. Next, the computing device 200 converts both the foreground average pixel and the background average pixel from the {R,G,B} color space to a second color space, which could be {L,a,b}, {X,Y,Z}, or some other color space. {L,a,b} is used by way of example in this part of the disclosure.
Next, the computing device 200 determines a transform matrix in {L,a,b} from the converted foreground average pixel to the converted background average pixel. In an embodiment, this transform matrix includes three separate dimension-wise ratios: the ratio of the background-average-pixel ‘L’ value to the foreground-average-pixel ‘L’ value, the ratio of the background-average-pixel ‘a’ value to the foreground-average-pixel ‘a’ value, and the ratio of the background-average-pixel ‘b’ value to the foreground-average-pixel ‘b’ value. And certainly other forms of transform matrices could be used.
The computing device 200 also converts all or substantially all of the pixels of the obtained foreground video data to the second color space, which again is {L,a,b} in this example, and transforms the converted foreground pixels in the second color space using the determined transform matrix. In the case of the transform matrix including the three ratios described in the preceding paragraph, the computing device 200 would transform the converted foreground pixels at least in part by, for each foreground pixel, multiplying the foreground pixel's ‘L’ value by the average-background-L-to-average-foreground-L ratio, (ii) multiplying the foreground pixel's ‘a’ value by the average-background-a-to-average-foreground-a ratio, and (iii) multiplying the foreground pixel's ‘b’ value by the average-background-b-to-average-foreground-b ratio. The computing device 200 may then convert the now-transformed foreground pixels back to the {R,G,B} color space.
This and the next few paragraphs provide an example as to how the computing device 200 could carry out the chromatic-replacement technique in cases where that is the selected chromatic-adjustment technique. In at least one embodiment, the computing device 200 carries out the step of converting all or substantially all of the pixels of the obtained background video data from the {R,G,B} color space to the {L,a,b} color space. It is noted, however, that this conversion step is not necessarily part of carrying out the chromatic-replacement technique, since the computing device 200 may have already converted the background pixels from {R,G,B} to {L,a,b} in order to make the determination as to whether the color-distribution dimensionality of the background video data is high-dimensional chromatic or low-dimensional chromatic. If, however, the computing device 200 had not previously converted the background pixels to {L,a,b}, the computing device 200 may do so as part of—or as a necessary precursor to—carrying out the chromatic-replacement technique. In any event, the chromatic-replacement technique is described below in a manner that presumes that the background pixels have already been converted to {L,a,b}.
As to carrying out the chromatic-replacement technique, the computing device 200 may do so at least in part by carrying out a number of sub-steps. One such sub-step involves generating an L-to-{a,b} lookup table based on the converted background pixels. An example way of doing this is now described in connection with FIGS. 5-9. As described above, FIG. 5 shows an example low-dimensional-chromatic set of pixels. In the balance of this disclosure, FIGS. 5-7 are considered to represent the set of background pixels after having been converted to {L,a,b} as described above. The specific example described herein is with respect to an ‘L’ value of 50, which as described above is selected purely by way of example.
In FIG. 5 and FIG. 6, as described above, the {a,b} variance of the pixels 520-524 in the L=50 plane 510 is represented by the size of the bounding circle 512 that surrounds the background pixels 520 in the plane 510. It is noted that this {a,b} variance could be calculated using such a circle or in any other suitable way as described above, and that the size of the circle 512 is a representation of the extent of the {a,b} variance that is present at L=50 in the background pixel set of FIG. 5. As part of generating an L-to-{a,b} lookup table based on the converted background pixels, the computing device 200 may determine a characteristic {a,b} value for L=50 in the background pixel set. One way to phrase the determination of a characteristic {a,b} value for L=50 (or any other particular ‘L’ value) in the background pixel set is idealizing or generating an idealization of the background pixels for which L=50.
FIG. 7 depicts an example idealization of the example subset of FIG. 6, in accordance with some embodiments. The graph 700 of FIG. 7 includes a number of elements of the graph 600 of FIG. 6. Among the salient differences are that the outlier points 522-524 have been removed, and the pixels 520 inside the bounding circle 512 have been reduced to a single characteristic pixel 702 having a particular ‘a’ value and a particular ‘b’ value. In this disclosure, the characteristic ‘a’ value at L=50 is referred to as a50 and the characteristic ‘b’ value at L=50 is referred to as b50. Thus, the characteristic (L=50) pixel 702 can be expressed as {a50,b50}. It is noted that the characteristic pixel 702 could be determined by averaging the {a,b} values of a plurality of pixels (such as the pixels 520 or perhaps the pixels 520-524, as examples). Another way of determining the characteristic pixel 702 could be to randomly select any one of the pixels 520 to represent the group. And certainly other approaches could be used.
FIG. 8 depicts an example idealization of the example chromatically low-dimensional background video data of FIG. 5 that incorporates the example idealization of FIG. 7, in accordance with some embodiments. In particular, FIG. 8 depicts a graph 800 that is quite similar in a number of ways to the graph 500 of FIG. 5. But instead of showing all of the background pixels 520-524, the graph 800 shows only the characteristic pixel that has been determined for each of a plurality of ‘L’ values that are present in the converted background pixels. This of course includes the characteristic pixel 702 that, as described above, corresponds to L=50. As can be seen by comparison of FIGS. 5 and 8, the column 526 of pixels in the graph 500 (which encompasses all but the outliers) has been idealized into a string (or single-pixel-wide column) of pixels by way of the idealization process described above. This information can then be populated into an L-to-{a,b} lookup table.
FIG. 9 depicts an example L-to-{a,b} look-up table that corresponds to the example idealization of FIG. 8, in accordance with some embodiments. In particular, FIG. 9 depicts an example L-to-{a,b} look-up table 900 that includes columns for each of the possible values of L (0, 1, 2 . . . 100) and shows some example rows at L=0, L=1, L=2, L=49, L=50, L=51, L=99, and L=100. Consistent with the above description, the row at L=50 corresponds to the above-described characteristic point 702 that is an idealization in the a-b plane of the background pixel set at L=50. In each of the rows, the aL and bL notation is used for each L in the table 900, and also some example values for the various characteristic ‘a’ values and characteristic ‘b’ values are included as well.
It should be noted that, while it is certainly possible that a given pixel set would have every possible value of ‘L’ represented, it is also certainly possible that there could be a number of actual implementation examples in which there are one or more empty ‘L’ rows (i.e., values of ‘L’ for which no pixels (or perhaps too few pixels) exist in the pixel set and thus for which no characteristic ‘a’ values and characteristic ‘b’ values could be determined). In some embodiments, an extrapolation process is used to fill in the entire table in case characteristic {a,b} values are needed for missing ‘L’ values. In other embodiments, extrapolation takes place on an as-needed basis for missing ‘L’ values. Such extrapolation could involve simply copying the {a,b} data from a nearest ‘L’ entry, or perhaps averaging the {a,b} values from some number of proximate ‘L’ entries that have valid {a,b) data, etc. And certainly other example approaches could be listed as well.
In at least one embodiment, as an additional sub-step in carrying out the chromatic-replacement technique, the computing device 200 converts all or substantially all of the pixels of the foreground video data from the {R,G,B} color space to the {L,a,b} color space. The computing device 200 may do this before or after the generation of the L-to-{a,b} look-up table 900, which may or may not have been done in the above-described manner.
In at least one embodiment, as a further sub-step in carrying out the chromatic-replacement technique, the computing system 200 transforms the converted foreground pixels at least in part by (a) using the respective L values of the respective converted foreground pixels to select respective replacement {a,b} values for the respective converted foreground pixels based on the L-to-{a,b} lookup table 900 and (b) replacing the respective {a,b} values of the respective converted foreground pixels with the corresponding respective selected replacement {a,b} values. In at least one embodiment, as an additional sub-step in carrying out the chromatic-replacement technique, the computing device 200 converts the transformed foreground pixels from {L,a,b} back to {R,G,B}. An example of carrying out these last two sub-steps is described below in connection with FIGS. 10-17.
FIG. 10 depicts an example table representation of example foreground video data in the {R,G,B} color space, in accordance with some embodiments. In particular, FIG. 10 depicts a foreground-{R,G,B} table 1000 having four columns: a foreground-pixel index number, an ‘R’ value, a ‘G’ value, and a ‘B’ value. The foreground-{R,G,B} table 1000 is also depicted as having a row for each of 307,200 different foreground pixels, corresponding as an example to the total number of pixels in a 640×480 image. Some example ‘R,’ ‘G,’ and ‘B’ values are shown in the foreground-{R,G,B} table 1000. For reasons that will become clearer upon reading the description of the ensuing figures, the reader's attention is called to row 1002 (foreground-pixel FG-2), row 1004 (FG-156000), and row 1006 (FG-307199). Pixel FG-2 has {R,G,B} values of {181,95,66}, corresponding to a light brown. Pixel FG-156000 has {R,G,B} values of {230,24,102}, corresponding to a pinkish red. And pixel FG-307199 has {R,G,B} values of {153,110,70}, corresponding to more of a milk-chocolate brown.
FIG. 11 depicts an example table representation of the example foreground video data of FIG. 10 after having been converted to the {L,a,b} color space, in accordance with some embodiments. In particular, FIG. 11 depicts a foreground-{L,a,b} table 1100 showing the same foreground pixels but after having been converted to {L,a,b}. The table 1100 is quite similar to the table 1000, other than that the pixels are of course expressed in the table 1100 in the {L,a,b} color space instead of in the {R,G,B} color space as is the case with table 1000. The reader will note that the same three pixels (FG-2, FG-156000, and FG-307199) that were referenced above are also called out at rows 1102, 1104, and 1106, respectively. The reader will further note that, while those three foreground pixels have three different pairs of {a,b} values, all three of those pixels have their respective ‘L’ value equal to 50.
FIG. 12 depicts an example graphical representation of the example converted foreground video data of FIG. 11 in the {L,a,b} color space, in accordance with some embodiments. In particular, FIG. 12 depicts a graph 1200 that shows a relatively scattered set of foreground pixels, which are meant to correspond to the pixels of FIGS. 10 and 11 (although of course expressed in the {L,a,b} color space in FIG. 12 as they are in FIG. 11). The graph 1200 includes an ‘L’ axis 1202, an ‘a’ axis 1204, and a ‘b’ axis 1206 that all intersect at an origin point 1208 that corresponds to the point ({L,a,b}|{0,0,0}). Each foreground pixel is represented in the graph 1200 by an empty triangle. Moreover, the relatively high breadth (i.e., variance) of the foreground pixels collectively is shown using a brace 1210.
FIG. 13 depicts an example graphical representation that shows both the example chromatically low-dimensional background video data of FIG. 5 and the example converted foreground video data of FIG. 12, both in the {L,a,b} color space, in accordance with some embodiments. In particular, FIG. 13 depicts a graph 1300 that is essentially the graphs 500 and 1200 overlaid on one another, albeit without the outlier pixels 522-524 from the graph 500. As can be seen by simple inspection of the graph 1300 of FIG. 13, it is clear that the degree of variety and scattering of the foreground pixels (as represented graphically by the brace 1200) significantly exceeds that of the background pixels (as represented graphically by the much narrower brace 526).
To wit, the graph 1300 is a depiction of the fact that, in this example, the foreground pixels as a group have significantly higher color-distribution dimensionality than do the background pixels as a group. Because the {L,a,b} color space (as does the {X,Y,Z} color space) is defined such that luminance is in a single dimension and color (or chromaticity) is in the other two dimensions, this color space provides a useful visualization tool of the color variance of a given set of pixels at various different luminance levels and also the relative color variance of two different sets of pixels.
As described above, the computing system 200 transforms the converted foreground pixels at least in part by (a) using the respective L values of the respective converted foreground pixels to select respective replacement {a,b} values for the respective converted foreground pixels based on the L-to-{a,b} lookup table 900 and (b) replacing the respective {a,b} values of the respective converted foreground pixels with the corresponding respective selected replacement {a,b} values. An example result of this sub-step of the chromatic-replacement technique is shown in FIG. 14.
FIG. 14 depicts an example table representation of the example converted foreground video data of FIGS. 11-12 after having been transformed in the {L,a,b} color space using a chromatic-replacement technique, in accordance with some embodiments. In particular, FIG. 14 depicts an example transformed-foreground-{L,a,b} table 1400 that is quite similar in layout to the foreground-{L,a,b} table 1100 of FIG. 11. What is different about the content, however, is that the actual foreground-pixel {a,b} values that are present in the foreground-{L,a,b} table 1100 of FIG. 11 have been replaced in the transformed-foreground-{L,a,b} table 1400 with {a,b} values selected by the computing device 200 from the L-to-{a,b} lookup table 900 based on the respective ‘L’ values of those foreground pixels.
As described above, in at least one embodiment, in cases where the respective ‘L’ values of the foreground pixels are present in the L-to-{a,b} lookup table 900, the computing device 200 simply replaces the corresponding {a,b} values in the table 1100 with the {a,b} values from that ‘L’ entry in the L-to-{a,b} lookup table 900; and in cases where the respective ‘L’ values of the foreground pixels are not present in the L-to-{a,b} lookup table 900, the computing device 200 replaces the corresponding {a,b} values in the table 1100 with {a,b} values derived by interpolation, perhaps in one of the manners described above, from one or more ‘L’ entries that are present in the L-to-{a,b} lookup table 900.
With respect to the foreground pixels FG-2, FG-156000, and FG-307199, it can be seen in the transformed-foreground-{L,a,b} table 1400 that, while those three pixels still have L=50, each of them now has a=48 and b=32, which are the a50 and b50 values, respectively, from the L-to-{a,b} lookup table 900. Based on the discussion above, it will be evident to one of skill in the art that, while those three pixels previously had three different colors, they have now been transformed to all having the same color due to having the same brightness value.
FIG. 15 depicts an example graphical representation of the transformed foreground video data of FIG. 14 in the {L,a,b} color space, in accordance with some embodiments. In particular, FIG. 15 depicts a graph 1500 that plots the transformed foreground pixels that are listed in tabular form in the transformed-foreground-{L,a,b} table 1400 of FIG. 14. The graph 1500 includes an ‘L’ axis 1502, an ‘a’ axis 1504, and a ‘b’ axis 1506 that all intersect at an origin point 1508 that corresponds to the point ({L,a,b}|{0,0,0}). The plot of foreground pixels in the graph 1500 appears quite similar to the idealization of the background pixels that is shown in the graph 800 of FIG. 8. Certainly there could be multiple pixels in any of the plotted locations in the graph 1500, and in that way the graph 1500 could be viewed as a plot of only the unique pixels in the transformed foreground pixels of FIG. 14. The narrow color-distribution dimensionality of the transformed foreground pixels is shown visually by the relatively narrow brace 1510.
FIG. 16 depicts an example graphical representation that shows both the example chromatically low-dimensional background video data of FIG. 5 and the example transformed foreground video data of FIG. 15, both in the {L,a,b} color space, in accordance with some embodiments. The graph 1600 of FIG. 16 is quite similar to the graph 1500 of FIG. 15, other than that the pixel column and corresponding brace 526 from FIG. 5 have been added. It is apparent from inspection of the graph 1600 that the color-distribution dimensionality of the transformed foreground pixels (as shown be the narrower brace 1510) is less than that of the original background pixels (minus the outliers) (as shown by the wider brace 526). Compared to the pre-foreground-transformation situation that is depicted in the graph 1300 of FIG. 13, the post-foreground-transformation situation that is depicted in the graph 1600 of FIG. 16 illustrates that the transformed foreground pixels are now well within the color range of the original background pixels.
FIG. 17 depicts an example table representation of the example transformed foreground video data of FIGS. 14-15 after having been converted to the {R,G,B} color space, in accordance with some embodiments. As can be seen in FIG. 17, the table 1700 has the same content as the table 1400 of FIG. 14, albeit with the {L,a,b} expressions of the foreground pixels from the table 1400 having been replaced by the {R,G,B} expressions of the foreground pixels in the table 1700. Consistent with the above description of the transformation of like-luminance foreground pixels to having the same color as one another, it can be seen that the rows 1702, 1704, and 1706 show respectively that the pixels FG-2, FG-156000, and FG-307199 each now have the {R,G,B} values of {201,79,67}, which corresponds to a dark orange.
Thus, the previous variety of light brown, pinkish red, and milk-chocolate brown among those three example pixels has been replaced by three dark-orange pixels. This would correspond to an example where the original background pixels were largely concentrated in the orange/red/brown area of the visible-light spectrum. And a foreground video that would have previously clashed with that background video (due at least to the pinkish red) has been transformed to a foreground video that appealingly sits within the relatively concentrated color range of the original background video.
Returning to FIG. 1, at step 112, the computing device 200 generates combined video data at least in part by combining the background video data with the adjusted foreground video data. This step may be performed in several different ways. In at least one embodiment, the computing device 200 combines the adjusted foreground video data with the background video data using what is known is an alpha mask, which is essentially a mapping of all of the pixel locations that will be in the combined video, where in that mapping, a ‘1’ value indicates that the corresponding pixel location should be filled with the corresponding pixel from the foreground video and a ‘0’ value indicates that the corresponding pixel location should be filled with the corresponding pixel from the background video. And certainly other techniques could be used, as known to those having skill in the art.
In at least one embodiment, generating the combined video at least in part by combining the adjusted foreground video data with the background video data includes performing a smoothing operation to a boundary between the adjusted foreground video data and the background video data. In an embodiment, the smoothing operation includes a Poisson image-blending technique. And certainly other examples could be listed.
In at least one embodiment, the method 100 also includes (i) obtaining second foreground video data and (ii) adjusting the second foreground video data using the selected chromatic-adjustment technique, and generating the combined video data includes combining the background video data with both the adjusted foreground video data and the adjusted second foreground video data. And clearly this is extendible to any number of foreground videos (e.g., personas of respective participants in a given collaborative online communication session) being obtained, chromatically adjusted based on the background video, and combined with the background video.
At step 114, the computing device 200 outputs the combined video data for display. This step may be performed in several different ways. In at least one embodiment, the computing device 200 outputs the combined video data for display via the display 220 and/or one or more other displays. In at least one embodiment, the computing device 200 outputs the combined video data via a network connection for display on one or more other devices. And certainly other examples could be listed.
Moreover, it is noted that, although the above examples and embodiments are described in the context of adjusting one or more foreground videos based on the color-distribution dimensionality of a single background video, this is not required of all embodiments. Indeed, in some embodiments, the designations of “foreground” and “background” could be reversed or not present at all (i.e., it could simply be the adjustment of a first video based on a color-distribution dimensionality of a second video). Moreover, it also need not be the case that one or more videos are chromatically adjusted based on another video and then combined with that other video; in some cases, one or more videos that are to be combined could each be chromatically adjusted based on a common reference point (e.g., a given white point, grey point, or the like). Indeed, in some embodiments, a “mood” or “style” or other setting could be available to chromatically adjust one or more videos (one or more “foreground” videos, a “background” video, and/or one or more other videos) prior to combination using chromatic profiles such as sepia, high brightness, pastels, earth tones, and/or one or more others deemed suitable by those in the art for a given implementation.

Claims (21)

What is claimed is:
1. A method comprising:
obtaining foreground video data;
obtaining background video data;
determining a color-distribution dimensionality of the background video data to be either high-dimensional chromatic or low-dimensional chromatic;
selecting a chromatic-adjustment technique from a set of chromatic-adjustment techniques based on the determined color-distribution dimensionality of the background video data;
adjusting the foreground video data using the selected chromatic-adjustment technique;
generating combined video data at least in part by combining the background video data with the adjusted foreground video data; and
outputting the combined video data for display.
2. The method of claim 1, wherein determining the color-distribution dimensionality of the background video data to be either high-dimensional chromatic or low-dimensional chromatic comprises:
converting pixels of the background video data from an {R,G,B} color space to an {L,a,b} color space;
calculating an {a,b} variance of the converted background pixels;
comparing the calculated {a,b} variance to an {a,b}-variance threshold;
determining the color-distribution dimensionality of the background video data to be high-dimensional chromatic if the calculated {a,b} variance exceeds the {a,b}-variance threshold; and
determining the color-distribution dimensionality of the background video data to be low-dimensional chromatic if the calculated {a,b} variance does not exceed the {a,b}-variance threshold.
3. The method of claim 2, wherein:
calculating the {a,b} variance of the converted background pixels comprises determining how many luminance levels in the converted background pixels have more than a luminance-level-specific degree of {a,b} variance; and
the {a,b}-variance threshold is a threshold number of luminance levels.
4. The method of claim 3, wherein the threshold number of luminance levels is zero.
5. The method of claim 3, wherein the threshold number of luminance levels is greater than zero.
6. The method of claim 2, wherein calculating the {a,b} variance of the converted background pixels comprises:
determining a respective luminance-level-specific {a,b} variance for each of a plurality of luminance levels that are represented in the converted background pixels; and
calculating the {a,b} variance of the converted background pixels to be a sum of the determined luminance-level-specific {a,b} variances.
7. The method of claim 2, wherein calculating the {a,b} variance of the converted background pixels comprises:
determining a respective luminance-level-specific {a,b} variance for each luminance level represented in the converted background pixels; and
calculating the {a,b} variance of the converted background pixels to be a sum of the determined luminance-level-specific {a,b} variances.
8. The method of claim 1, wherein determining the color-distribution dimensionality of the background video data to be either high-dimensional chromatic or low-dimensional chromatic comprises:
determining the color-distribution dimensionality of the background video data to be low-dimensional chromatic if a background-color distribution of the background video data in an {L,a,b} color space is supported by a relationship defined by:

{(L,a,b)|a=f a(L),b=f b(L)},
where fa and fb are functions; and
otherwise determining the color-distribution dimensionality of the background video data to be high-dimensional chromatic.
9. The method of claim 1, wherein:
the set of chromatic-adjustment techniques includes a white-balancing technique and a chromatic-replacement technique; and
selecting a chromatic-adjustment technique based on the determined color-distribution dimensionality comprises:
selecting the white-balancing technique when the color-distribution dimensionality of the background video data is determined to be high-dimensional chromatic; and
selecting the chromatic-replacement technique when the color-distribution dimensionality of the background video data is determined to be low-dimensional chromatic.
10. The method of claim 9, wherein adjusting the foreground video data using the white-balancing technique comprises:
determining a foreground average of pixels of the foreground video data in an {R,G,B} color space;
determining a background average of pixels of the background video data in the {R,G,B} color space;
converting the foreground average and the background average from the {R,G,B} color space to a second color space;
determining a transform matrix in the second color space from the converted foreground average to the converted background average;
converting the pixels of the foreground video data from the {R,G,B} color space to the second color space;
transforming the converted foreground pixels in the second color space using the determined transform matrix; and
converting the transformed foreground pixels from the second color space to the {R,G,B} color space.
11. The method of claim 10, wherein the determined transform matrix comprises dimension-wise ratios in the second color space of the converted background average to the converted foreground average.
12. The method of claim 10, wherein the second color space is an {L,a,b} color space.
13. The method of claim 10, wherein the second color space is an {X,Y,Z} color space.
14. The method of claim 13, wherein converting the foreground pixels from the {R,G,B} color space to the second color space comprises converting the foreground pixels from the {R,G,B} color space to an {L,a,b} color space and then from the {L,a,b} color space to the {X,Y,Z} color space.
15. The method of claim 9, further comprising converting pixels of the background video data from an {R,G,B} color space to an {L,a,b} color space, wherein adjusting the foreground video data using the chromatic-replacement technique comprises:
generating an L-to-{a,b} lookup table based on the converted background pixels;
converting pixels of the foreground video data from the {R,G,B} color space to the {L,a,b} color space;
transforming the converted foreground pixels at least in part by:
using the respective L values of the respective converted foreground pixels to select respective replacement {a,b} values for the respective converted foreground pixels based on the L-to-{a,b} lookup table; and
replacing the respective {a,b} values of the respective converted foreground pixels with the corresponding respective selected replacement {a,b} values; and
converting the transformed foreground pixels from the {L,a,b} color space to the {R,G,B} color space.
16. The method of claim 15, wherein using the respective L values of the respective converted foreground pixels to select the respective replacement {a,b} values for the respective converted foreground pixels based on the L-to-{a,b} lookup table comprises:
retrieving the respective replacement {a,b} values from the L-to-{a,b} lookup table in cases where the respective L value of the respective converted foreground pixel is listed in the L-to-{a,b} lookup table.
17. The method of claim 16, wherein using the respective L values of the respective converted foreground pixels to select the respective replacement {a,b} values for the respective converted foreground pixels based on the L-to-{a,b} lookup table further comprises:
using interpolated {a,b} values based on one or more entries in the L-to-{a,b} lookup table as the respective replacement {a,b} values in cases where the respective L value of the respective converted foreground pixel is not listed in the L-to-{a,b} lookup table.
18. The method of claim 17, wherein the interpolated {a,b} values are copied from a nearest L value that is listed in the L-to-{a,b} lookup table.
19. The method of claim 17, wherein the interpolated {a,b} values are average {a,b} values of two or more proximate entries in the L-to-{a,b} lookup table.
20. The method of claim 1, further comprising:
obtaining second foreground video data; and
adjusting the second foreground video data using the selected chromatic-adjustment technique,
wherein generating the combined video data comprises combining the background video data with both the adjusted foreground video data and the adjusted second foreground video data.
21. A system comprising:
a communication interface;
a processor; and
a non-transitory computer-readable medium storing instructions executable by the processor for causing the system to perform functions including:
obtaining foreground video data;
obtaining background video data;
determining a color-distribution dimensionality of the background video data to be either high-dimensional chromatic or low-dimensional chromatic;
selecting a chromatic-adjustment technique from a set of chromatic-adjustment techniques based on the determined color-distribution dimensionality of the background video data;
adjusting the foreground video data using the selected chromatic-adjustment technique;
generating combined video data at least in part by combining the background video data with the adjusted foreground video data; and
outputting the combined video data for display.
US15/181,653 2016-06-14 2016-06-14 Methods and systems for combining foreground video and background video using chromatic matching Active US9883155B2 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US15/181,653 US9883155B2 (en) 2016-06-14 2016-06-14 Methods and systems for combining foreground video and background video using chromatic matching

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
US15/181,653 US9883155B2 (en) 2016-06-14 2016-06-14 Methods and systems for combining foreground video and background video using chromatic matching

Publications (2)

Publication Number Publication Date
US20170359559A1 US20170359559A1 (en) 2017-12-14
US9883155B2 true US9883155B2 (en) 2018-01-30

Family

ID=60573270

Family Applications (1)

Application Number Title Priority Date Filing Date
US15/181,653 Active US9883155B2 (en) 2016-06-14 2016-06-14 Methods and systems for combining foreground video and background video using chromatic matching

Country Status (1)

Country Link
US (1) US9883155B2 (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10984589B2 (en) 2017-08-07 2021-04-20 Verizon Patent And Licensing Inc. Systems and methods for reference-model-based modification of a three-dimensional (3D) mesh data model
US11095854B2 (en) 2017-08-07 2021-08-17 Verizon Patent And Licensing Inc. Viewpoint-adaptive three-dimensional (3D) personas

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106507068B (en) 2016-11-29 2018-05-04 广东欧珀移动通信有限公司 Image processing method and device, control method and device, imaging and electronic device
CN106604001B (en) 2016-11-29 2018-06-29 广东欧珀移动通信有限公司 Image processing method, image processing apparatus, imaging device and electronic device
CN109688396B (en) * 2017-07-25 2021-02-02 Oppo广东移动通信有限公司 Image white balance processing method and device and terminal equipment
CN112866507B (en) * 2021-01-13 2023-01-10 中国传媒大学 Intelligent panoramic video synthesis method and system, electronic device and medium
WO2023107455A2 (en) * 2021-12-07 2023-06-15 The Invisible Pixel Inc. Uv system and methods for generating an alpha channel

Citations (145)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5001558A (en) 1985-06-11 1991-03-19 General Motors Corporation Night vision system with color video camera
US5022085A (en) 1990-05-29 1991-06-04 Eastman Kodak Company Neighborhood-based merging of image data
US5117283A (en) 1990-06-25 1992-05-26 Eastman Kodak Company Photobooth compositing apparatus
US5227985A (en) 1991-08-19 1993-07-13 University Of Maryland Computer vision system for position monitoring in three dimensions using non-coplanar light sources attached to a monitored object
US5343311A (en) 1992-04-14 1994-08-30 Electronics For Imaging, Inc. Indexed processing of color image data
US5506946A (en) 1991-10-01 1996-04-09 Electronics For Imaging, Inc. Selective color correction
US5534917A (en) 1991-05-09 1996-07-09 Very Vivid, Inc. Video image based control system
US5581276A (en) 1992-09-08 1996-12-03 Kabushiki Kaisha Toshiba 3D human interface apparatus using motion recognition based on dynamic image processing
US5631697A (en) 1991-11-27 1997-05-20 Hitachi, Ltd. Video camera capable of automatic target tracking
US5687306A (en) 1992-02-25 1997-11-11 Image Ware Software, Inc. Image editing system including sizing function
US5875040A (en) 1995-12-04 1999-02-23 Eastman Kodak Company Gradient based method for providing values for unknown pixels in a digital image
US6119147A (en) 1998-07-28 2000-09-12 Fuji Xerox Co., Ltd. Method and system for computer-mediated, multi-modal, asynchronous meetings in a virtual space
US6125194A (en) 1996-02-06 2000-09-26 Caelum Research Corporation Method and system for re-screening nodules in radiological images using multi-resolution processing, neural network, and image processing
US6150930A (en) 1992-08-14 2000-11-21 Texas Instruments Incorporated Video equipment and method to assist motor vehicle operators
US6288703B1 (en) * 1996-11-25 2001-09-11 Ultimatte Corporation Method for removing from an image the background surrounding a selected subject by generating candidate mattes
US20020012072A1 (en) * 2000-05-08 2002-01-31 Naoki Toyama Video mixing apparatus and method of mixing video
US20020025066A1 (en) * 1996-09-12 2002-02-28 Daniel Pettigrew Processing image data
US20020051491A1 (en) 1998-11-20 2002-05-02 Kiran Challapali Extraction of foreground information for video conference
US6411744B1 (en) 1997-10-15 2002-06-25 Electric Planet, Inc. Method and apparatus for performing a clean background subtraction
US20020158873A1 (en) 2001-01-26 2002-10-31 Todd Williamson Real-time virtual viewpoint in simulated reality environment
US6618444B1 (en) 1997-02-14 2003-09-09 At&T Corp. Scene description nodes to support improved chroma-key shape representation of coded arbitrary images and video objects
US6661918B1 (en) 1998-12-04 2003-12-09 Interval Research Corporation Background estimation and segmentation based on range and color
US6664973B1 (en) 1996-04-28 2003-12-16 Fujitsu Limited Image processing apparatus, method for processing and image and computer-readable recording medium for causing a computer to process images
US20040004626A1 (en) 2002-07-05 2004-01-08 Takashi Ida Image editing method and image editing apparatus
US6760749B1 (en) 2000-05-10 2004-07-06 Polycom, Inc. Interactive conference content distribution device and methods of use thereof
US20040153671A1 (en) 2002-07-29 2004-08-05 Schuyler Marc P. Automated physical access control systems and methods
US20040175021A1 (en) 2002-11-29 2004-09-09 Porter Robert Mark Stefan Face detection
US6798407B1 (en) 2000-11-28 2004-09-28 William J. Benman System and method for providing a functional virtual environment with real time extracted and transplanted images
US20050063565A1 (en) 2003-09-01 2005-03-24 Honda Motor Co., Ltd. Vehicle environment monitoring device
US20050094879A1 (en) 2003-10-31 2005-05-05 Michael Harville Method for visual-based recognition of an object
US6937744B1 (en) 2000-06-13 2005-08-30 Microsoft Corporation System and process for bootstrap initialization of nonparametric color models
US20050219391A1 (en) 2004-04-01 2005-10-06 Microsoft Corporation Digital cameras with luminance correction
US20050219264A1 (en) 2004-04-01 2005-10-06 Microsoft Corporation Pop-up light field
US20050262201A1 (en) 2004-04-30 2005-11-24 Microsoft Corporation Systems and methods for novel real-time audio-visual communication and data collaboration
US6973201B1 (en) 2000-11-01 2005-12-06 Koninklijke Philips Electronics N.V. Person tagging in an image processing system utilizing a statistical model based on both appearance and geometric features
US20060072022A1 (en) 2004-10-06 2006-04-06 Yoshiaki Iwai Image processing method and image processing device
US20060193509A1 (en) 2005-02-25 2006-08-31 Microsoft Corporation Stereo-based image processing
US20060221248A1 (en) 2005-03-29 2006-10-05 Mcguire Morgan System and method for image matting
US7124164B1 (en) 2001-04-17 2006-10-17 Chemtob Helen J Method and apparatus for providing group interaction via communications networks
US20060259552A1 (en) 2005-05-02 2006-11-16 Mock Wayne E Live video icons for signal selection in a videoconferencing system
US20060291697A1 (en) 2005-06-21 2006-12-28 Trw Automotive U.S. Llc Method and apparatus for detecting the presence of an occupant within a vehicle
US20070036432A1 (en) 2003-11-12 2007-02-15 Li-Qun Xu Object detection in images
US20070070200A1 (en) 2005-09-29 2007-03-29 Wojciech Matusik Video matting using camera arrays
US20070110298A1 (en) 2005-11-14 2007-05-17 Microsoft Corporation Stereo video for gaming
US20070133880A1 (en) 2005-12-09 2007-06-14 Microsoft Corporation Background Removal In A Live Video
US20070146512A1 (en) 2005-12-27 2007-06-28 Sanyo Electric Co., Ltd. Imaging apparatus provided with imaging device having sensitivity in visible and infrared regions
US20070201738A1 (en) 2005-07-21 2007-08-30 Atsushi Toda Physical information acquisition method, physical information acquisition device, and semiconductor device
US20070269108A1 (en) 2006-05-03 2007-11-22 Fotonation Vision Limited Foreground / Background Separation in Digital Images
US20080109724A1 (en) 2006-11-07 2008-05-08 Polycom, Inc. System and Method for Controlling Presentations and Videoconferences Using Hand Motions
US7386799B1 (en) 2002-11-21 2008-06-10 Forterra Systems, Inc. Cinematic techniques in avatar-centric communication during a multi-user online simulation
US20080181507A1 (en) 2007-01-29 2008-07-31 Intellivision Technologies Corp. Image manipulation for videos and still images
US7420490B2 (en) 2006-07-31 2008-09-02 Texas Instruments Incorporated Pre-charge systems and methods for ADC input sampling
US20080219554A1 (en) 2005-01-13 2008-09-11 International Business Machines Corporation System and Method for Adaptively Separating Foreground From Arbitrary Background in Presentations
US20080266380A1 (en) 2007-04-30 2008-10-30 Gorzynski Mark E Video conference system with symmetric reference
US20080273751A1 (en) 2006-10-16 2008-11-06 Chang Yuan Detection and Tracking of Moving Objects from a Moving Platform in Presence of Strong Parallax
US20090003687A1 (en) 2006-01-10 2009-01-01 Microsoft Corporation Segmenting Image Elements
US20090044113A1 (en) 2007-08-07 2009-02-12 Jones Scott T Creating a Customized Avatar that Reflects a User's Distinguishable Attributes
US7518051B2 (en) 2005-08-19 2009-04-14 William Gibbens Redmann Method and apparatus for remote real time collaborative music performance and recording thereof
US20090110299A1 (en) 2007-10-30 2009-04-30 Kddi Corporation Apparatus, method and computer program for classifying pixels in a motion picture as foreground or background
US20090144651A1 (en) 2007-08-06 2009-06-04 Apple Inc. Interactive frames for images and videos displayed in a presentation application
US20090199111A1 (en) 2008-01-31 2009-08-06 G-Mode Co., Ltd. Chat software
US7574043B2 (en) 2005-06-27 2009-08-11 Mitsubishi Electric Research Laboratories, Inc. Method for modeling cast shadows in videos
US20090245571A1 (en) 2008-03-31 2009-10-01 National Taiwan University Digital video target moving object segmentation method and system
US20090244309A1 (en) 2006-08-03 2009-10-01 Benoit Maison Method and Device for Identifying and Extracting Images of multiple Users, and for Recognizing User Gestures
US20090249863A1 (en) 2008-04-03 2009-10-08 Korea Techno Co., Ltd Vapor phase decomposition device for semiconductor wafer pollutant measurement apparatus and door opening and closing device
US7602990B2 (en) 2005-09-29 2009-10-13 Mitsubishi Electric Research Laboratories, Inc. Matting using camera arrays
US20090278859A1 (en) 2005-07-15 2009-11-12 Yissum Research Development Co. Closed form method and system for matting a foreground object in an image having a background
US20090284627A1 (en) 2008-05-16 2009-11-19 Kaibushiki Kaisha Toshiba Image processing Method
US20090290795A1 (en) 2008-05-23 2009-11-26 Microsoft Corporation Geodesic Image and Video Processing
US20090300553A1 (en) 2008-05-28 2009-12-03 Daniel Pettigrew Defining a border for an image
US7631151B2 (en) 2005-11-28 2009-12-08 Commvault Systems, Inc. Systems and methods for classifying and transferring information in a storage network
US20100027961A1 (en) 2008-07-01 2010-02-04 Yoostar Entertainment Group, Inc. Interactive systems and methods for video compositing
US20100034457A1 (en) 2006-05-11 2010-02-11 Tamir Berliner Modeling of humanoid forms from depth maps
US20100046830A1 (en) 2008-08-22 2010-02-25 Jue Wang Automatic Video Image Segmentation
US20100053212A1 (en) 2006-11-14 2010-03-04 Mi-Sun Kang Portable device having image overlay function and method of overlaying image in portable device
US7676081B2 (en) 2005-06-17 2010-03-09 Microsoft Corporation Image segmentation of foreground from background layers
US20100128927A1 (en) 2008-03-14 2010-05-27 Sony Computer Entertainment Inc. Image processing apparatus and image processing method
US7747044B2 (en) 2006-01-20 2010-06-29 The Johns Hopkins University Fusing multimodal biometrics with quality estimates via a bayesian belief network
US20100166325A1 (en) 2008-12-30 2010-07-01 Yahoo! Inc. Image segmentation
US20100171807A1 (en) 2008-10-08 2010-07-08 Tandberg Telecom As System and associated methodology for multi-layered site video conferencing
US20100195898A1 (en) 2009-01-28 2010-08-05 Electronics And Telecommunications Research Institute Method and apparatus for improving quality of depth image
US7773136B2 (en) 2006-08-28 2010-08-10 Sanyo Electric Co., Ltd. Image pickup apparatus and image pickup method for equalizing infrared components in each color component signal
US20100278384A1 (en) 2009-05-01 2010-11-04 Microsoft Corporation Human body pose estimation
US20100302395A1 (en) 2009-05-29 2010-12-02 Microsoft Corporation Environment And/Or Target Segmentation
US20100302376A1 (en) 2009-05-27 2010-12-02 Pierre Benoit Boulanger System and method for high-quality real-time foreground/background separation in tele-conferencing using self-registered color/infrared input images and closed-form natural image matting techniques
US20100329544A1 (en) 2009-06-30 2010-12-30 Sony Corporation Information processing apparatus, information processing method, and program
US20110038536A1 (en) 2009-08-14 2011-02-17 Genesis Group Inc. Real-time image and video matting
US7912246B1 (en) 2002-10-28 2011-03-22 Videomining Corporation Method and system for determining the age category of people based on facial images
US20110090311A1 (en) 2008-06-17 2011-04-21 Ping Fang Video communication method, device, and system
US20110115886A1 (en) 2009-11-18 2011-05-19 The Board Of Trustees Of The University Of Illinois System for executing 3d propagation for depth image-based rendering
US20110158529A1 (en) 2009-12-28 2011-06-30 Xerox Corporation System and method for cleanup of mrc images for improved compression and image quality
US20110193939A1 (en) 2010-02-09 2011-08-11 Microsoft Corporation Physical interaction zone for gesture-based user interfaces
US20110216965A1 (en) 2010-03-05 2011-09-08 Microsoft Corporation Image Segmentation Using Reduced Foreground Training Data
US20110216975A1 (en) 2010-03-05 2011-09-08 Microsoft Corporation Up-Sampling Binary Images for Segmentation
US20110216976A1 (en) 2010-03-05 2011-09-08 Microsoft Corporation Updating Image Segmentation Following User Input
US20110242277A1 (en) 2010-03-30 2011-10-06 Do Minh N Systems and methods for embedding a foreground video into a background feed based on a control input
US20110243430A1 (en) 2008-11-04 2011-10-06 Konica Minolta Opto, Inc. Image input apparatus
US20110249190A1 (en) 2010-04-09 2011-10-13 Nguyen Quang H Systems and methods for accurate user foreground video extraction
US20110249863A1 (en) 2010-04-09 2011-10-13 Sony Corporation Information processing device, method, and program
US20110249883A1 (en) 2010-04-09 2011-10-13 General Electric Company Methods for segmenting objects in images
US20110267348A1 (en) 2010-04-29 2011-11-03 Dennis Lin Systems and methods for generating a virtual camera viewpoint for an image
US20110293179A1 (en) 2010-05-31 2011-12-01 Mert Dikmen Systems and methods for illumination correction of an image
US20110293180A1 (en) 2010-05-28 2011-12-01 Microsoft Corporation Foreground and Background Image Segmentation
US20120051631A1 (en) 2010-08-30 2012-03-01 The Board Of Trustees Of The University Of Illinois System for background subtraction with 3d camera
US8131011B2 (en) 2006-09-25 2012-03-06 University Of Southern California Human detection and tracking system
US8175384B1 (en) 2008-03-17 2012-05-08 Adobe Systems Incorporated Method and apparatus for discriminative alpha matting
US20120127259A1 (en) 2010-11-19 2012-05-24 Cisco Technology, Inc. System and method for providing enhanced video processing in a network environment
US8264544B1 (en) 2006-11-03 2012-09-11 Keystream Corporation Automated content insertion into video scene
US8300890B1 (en) 2007-01-29 2012-10-30 Intellivision Technologies Corporation Person/object image and screening
US20120314077A1 (en) 2011-06-07 2012-12-13 Verizon Patent And Licensing Inc. Network synchronized camera settings
US8355379B2 (en) 2006-02-16 2013-01-15 Motorola Solutions, Inc. Handover between radio networks
US20130016097A1 (en) 2010-04-02 2013-01-17 Imec Virtual Camera System
US20130028476A1 (en) 2009-01-30 2013-01-31 Microsoft Corporation Pose tracking pipeline
WO2013019259A1 (en) 2011-08-01 2013-02-07 Thomson Licensing Telepresence communications system and method
US8396328B2 (en) 2001-05-04 2013-03-12 Legend3D, Inc. Minimal artifact image sequence depth enhancement system and method
US8406494B2 (en) 2008-08-08 2013-03-26 Siemens Medical Solutions Usa, Inc. Anatomical primitive detection
US20130094780A1 (en) 2010-06-01 2013-04-18 Hewlett-Packard Development Company, L.P. Replacement of a Person or Object in an Image
US20130110565A1 (en) 2011-04-25 2013-05-02 Transparency Sciences, Llc System, Method and Computer Program Product for Distributed User Activity Management
US8446488B2 (en) * 2009-03-11 2013-05-21 Samsung Electronics Co., Ltd. Method and system for focal length-specific color enhancement
US20130129205A1 (en) 2010-11-24 2013-05-23 Jue Wang Methods and Apparatus for Dynamic Color Flow Modeling
US20130142452A1 (en) 2011-12-02 2013-06-06 Sony Corporation Image processing device and image processing method
US20130147900A1 (en) 2011-12-07 2013-06-13 Reginald Weiser Systems and methods for providing video conferencing services via an ethernet adapter
US8477149B2 (en) * 2009-04-01 2013-07-02 University Of Central Florida Research Foundation, Inc. Real-time chromakey matting using image statistics
US8533594B2 (en) 2011-04-19 2013-09-10 Autodesk, Inc. Hierarchical display and navigation of document revision histories
US8533593B2 (en) 2011-04-19 2013-09-10 Autodesk, Inc Hierarchical display and navigation of document revision histories
US8533595B2 (en) 2011-04-19 2013-09-10 Autodesk, Inc Hierarchical display and navigation of document revision histories
US20130243313A1 (en) 2010-10-01 2013-09-19 Telefonica, S.A. Method and system for images foreground segmentation in real-time
US20130335506A1 (en) 2012-06-14 2013-12-19 Polycom, Inc. Managing the layout of multiple video streams displayed on a destination display screen during a videoconference
US20140003719A1 (en) 2012-06-29 2014-01-02 Xue Bai Adaptive Trimap Propagation for Video Matting
US20140029788A1 (en) 2012-07-26 2014-01-30 Jinman Kang Detecting objects with a depth sensor
US8649932B2 (en) 2006-10-27 2014-02-11 International Electronic Machines Corp. Vehicle evaluation using infrared data
US20140063177A1 (en) 2012-09-04 2014-03-06 Cisco Technology, Inc. Generating and Rendering Synthesized Views with Multiple Video Streams in Telepresence Video Conference Sessions
US20140085398A1 (en) 2012-09-21 2014-03-27 Cisco Technology, Inc. Real-time automatic scene relighting in video conference sessions
US8701002B2 (en) 2011-04-19 2014-04-15 Autodesk, Inc. Hierarchical display and navigation of document revision histories
US20140112547A1 (en) 2009-10-07 2014-04-24 Microsoft Corporation Systems and methods for removing a background of an image
US20140119642A1 (en) 2010-04-05 2014-05-01 Flashfoto, Inc. Segmenting Human Hairs and Faces
US20140153784A1 (en) 2012-10-18 2014-06-05 Thomson Licensing Spatio-temporal confidence maps
US20140229850A1 (en) 2013-02-14 2014-08-14 Disney Enterprises, Inc. Avatar personalization in a virtual environment
US20140307056A1 (en) 2013-04-15 2014-10-16 Microsoft Corporation Multimodal Foreground Background Segmentation
US8874525B2 (en) 2011-04-19 2014-10-28 Autodesk, Inc. Hierarchical display and navigation of document revision histories
US8890929B2 (en) 2011-10-18 2014-11-18 Avaya Inc. Defining active zones in a traditional multi-party video conference and associating metadata with each zone
US9065973B2 (en) 2006-12-06 2015-06-23 Cisco Technology, Inc. System and method for displaying a videoconference
US9084928B2 (en) 2013-07-31 2015-07-21 Sven Gunnar Klang Tong operating device for a pinsetter
US9542626B2 (en) 2013-09-06 2017-01-10 Toyota Jidosha Kabushiki Kaisha Augmenting layer-based object detection with deep convolutional neural networks
US20170208243A1 (en) 2014-07-18 2017-07-20 Artincam Ltd. Automatic image composition

Patent Citations (210)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5001558A (en) 1985-06-11 1991-03-19 General Motors Corporation Night vision system with color video camera
US5022085A (en) 1990-05-29 1991-06-04 Eastman Kodak Company Neighborhood-based merging of image data
US5117283A (en) 1990-06-25 1992-05-26 Eastman Kodak Company Photobooth compositing apparatus
US5534917A (en) 1991-05-09 1996-07-09 Very Vivid, Inc. Video image based control system
US5227985A (en) 1991-08-19 1993-07-13 University Of Maryland Computer vision system for position monitoring in three dimensions using non-coplanar light sources attached to a monitored object
US5506946A (en) 1991-10-01 1996-04-09 Electronics For Imaging, Inc. Selective color correction
US5631697A (en) 1991-11-27 1997-05-20 Hitachi, Ltd. Video camera capable of automatic target tracking
US5687306A (en) 1992-02-25 1997-11-11 Image Ware Software, Inc. Image editing system including sizing function
US5517334A (en) 1992-04-14 1996-05-14 Electronics For Imaging, Inc. Indexed processing of color image data
US5343311A (en) 1992-04-14 1994-08-30 Electronics For Imaging, Inc. Indexed processing of color image data
US6150930A (en) 1992-08-14 2000-11-21 Texas Instruments Incorporated Video equipment and method to assist motor vehicle operators
US5581276A (en) 1992-09-08 1996-12-03 Kabushiki Kaisha Toshiba 3D human interface apparatus using motion recognition based on dynamic image processing
US5875040A (en) 1995-12-04 1999-02-23 Eastman Kodak Company Gradient based method for providing values for unknown pixels in a digital image
US6125194A (en) 1996-02-06 2000-09-26 Caelum Research Corporation Method and system for re-screening nodules in radiological images using multi-resolution processing, neural network, and image processing
US6664973B1 (en) 1996-04-28 2003-12-16 Fujitsu Limited Image processing apparatus, method for processing and image and computer-readable recording medium for causing a computer to process images
US20020025066A1 (en) * 1996-09-12 2002-02-28 Daniel Pettigrew Processing image data
US6288703B1 (en) * 1996-11-25 2001-09-11 Ultimatte Corporation Method for removing from an image the background surrounding a selected subject by generating candidate mattes
US6618444B1 (en) 1997-02-14 2003-09-09 At&T Corp. Scene description nodes to support improved chroma-key shape representation of coded arbitrary images and video objects
US6411744B1 (en) 1997-10-15 2002-06-25 Electric Planet, Inc. Method and apparatus for performing a clean background subtraction
US6119147A (en) 1998-07-28 2000-09-12 Fuji Xerox Co., Ltd. Method and system for computer-mediated, multi-modal, asynchronous meetings in a virtual space
US20020051491A1 (en) 1998-11-20 2002-05-02 Kiran Challapali Extraction of foreground information for video conference
US7317830B1 (en) 1998-12-04 2008-01-08 Vulcan Patents Llc Background estimation and segmentation based on range and color
US6661918B1 (en) 1998-12-04 2003-12-09 Interval Research Corporation Background estimation and segmentation based on range and color
US20020012072A1 (en) * 2000-05-08 2002-01-31 Naoki Toyama Video mixing apparatus and method of mixing video
US6760749B1 (en) 2000-05-10 2004-07-06 Polycom, Inc. Interactive conference content distribution device and methods of use thereof
US6937744B1 (en) 2000-06-13 2005-08-30 Microsoft Corporation System and process for bootstrap initialization of nonparametric color models
US6973201B1 (en) 2000-11-01 2005-12-06 Koninklijke Philips Electronics N.V. Person tagging in an image processing system utilizing a statistical model based on both appearance and geometric features
US6798407B1 (en) 2000-11-28 2004-09-28 William J. Benman System and method for providing a functional virtual environment with real time extracted and transplanted images
US20020158873A1 (en) 2001-01-26 2002-10-31 Todd Williamson Real-time virtual viewpoint in simulated reality environment
US7124164B1 (en) 2001-04-17 2006-10-17 Chemtob Helen J Method and apparatus for providing group interaction via communications networks
US8396328B2 (en) 2001-05-04 2013-03-12 Legend3D, Inc. Minimal artifact image sequence depth enhancement system and method
US20040004626A1 (en) 2002-07-05 2004-01-08 Takashi Ida Image editing method and image editing apparatus
US7050070B2 (en) 2002-07-05 2006-05-23 Kabushiki Kaisha Toshiba Image editing method and image editing apparatus
US20040153671A1 (en) 2002-07-29 2004-08-05 Schuyler Marc P. Automated physical access control systems and methods
US7912246B1 (en) 2002-10-28 2011-03-22 Videomining Corporation Method and system for determining the age category of people based on facial images
US7386799B1 (en) 2002-11-21 2008-06-10 Forterra Systems, Inc. Cinematic techniques in avatar-centric communication during a multi-user online simulation
US20040175021A1 (en) 2002-11-29 2004-09-09 Porter Robert Mark Stefan Face detection
US20050063565A1 (en) 2003-09-01 2005-03-24 Honda Motor Co., Ltd. Vehicle environment monitoring device
US7831087B2 (en) 2003-10-31 2010-11-09 Hewlett-Packard Development Company, L.P. Method for visual-based recognition of an object
US20050094879A1 (en) 2003-10-31 2005-05-05 Michael Harville Method for visual-based recognition of an object
US20070036432A1 (en) 2003-11-12 2007-02-15 Li-Qun Xu Object detection in images
US7742650B2 (en) 2003-11-12 2010-06-22 British Telecommunications Plc Object detection in images
US20050219264A1 (en) 2004-04-01 2005-10-06 Microsoft Corporation Pop-up light field
US7633511B2 (en) 2004-04-01 2009-12-15 Microsoft Corporation Pop-up light field
US20050219391A1 (en) 2004-04-01 2005-10-06 Microsoft Corporation Digital cameras with luminance correction
US7463296B2 (en) 2004-04-01 2008-12-09 Microsoft Corporation Digital cameras with luminance correction
US7634533B2 (en) 2004-04-30 2009-12-15 Microsoft Corporation Systems and methods for real-time audio-visual communication and data collaboration in a network conference environment
US20050262201A1 (en) 2004-04-30 2005-11-24 Microsoft Corporation Systems and methods for novel real-time audio-visual communication and data collaboration
US7965885B2 (en) 2004-10-06 2011-06-21 Sony Corporation Image processing method and image processing device for separating the background area of an image
US20060072022A1 (en) 2004-10-06 2006-04-06 Yoshiaki Iwai Image processing method and image processing device
US20080219554A1 (en) 2005-01-13 2008-09-11 International Business Machines Corporation System and Method for Adaptively Separating Foreground From Arbitrary Background in Presentations
US7668371B2 (en) 2005-01-13 2010-02-23 International Business Machines Corporation System and method for adaptively separating foreground from arbitrary background in presentations
US7512262B2 (en) 2005-02-25 2009-03-31 Microsoft Corporation Stereo-based image processing
US20060193509A1 (en) 2005-02-25 2006-08-31 Microsoft Corporation Stereo-based image processing
US7599555B2 (en) 2005-03-29 2009-10-06 Mitsubishi Electric Research Laboratories, Inc. System and method for image matting
US20060221248A1 (en) 2005-03-29 2006-10-05 Mcguire Morgan System and method for image matting
US20060259552A1 (en) 2005-05-02 2006-11-16 Mock Wayne E Live video icons for signal selection in a videoconferencing system
US7676081B2 (en) 2005-06-17 2010-03-09 Microsoft Corporation Image segmentation of foreground from background layers
US20060291697A1 (en) 2005-06-21 2006-12-28 Trw Automotive U.S. Llc Method and apparatus for detecting the presence of an occupant within a vehicle
US7574043B2 (en) 2005-06-27 2009-08-11 Mitsubishi Electric Research Laboratories, Inc. Method for modeling cast shadows in videos
US20090278859A1 (en) 2005-07-15 2009-11-12 Yissum Research Development Co. Closed form method and system for matting a foreground object in an image having a background
US7692664B2 (en) 2005-07-15 2010-04-06 Yissum Research Development Co. Closed form method and system for matting a foreground object in an image having a background
US7755016B2 (en) 2005-07-21 2010-07-13 Sony Corporation Physical information acquisition method, physical information acquisition device, and semiconductor device
US20070201738A1 (en) 2005-07-21 2007-08-30 Atsushi Toda Physical information acquisition method, physical information acquisition device, and semiconductor device
US7518051B2 (en) 2005-08-19 2009-04-14 William Gibbens Redmann Method and apparatus for remote real time collaborative music performance and recording thereof
US7420590B2 (en) 2005-09-29 2008-09-02 Mitsubishi Electric Research Laboratories, Inc. Video matting using camera arrays
US7602990B2 (en) 2005-09-29 2009-10-13 Mitsubishi Electric Research Laboratories, Inc. Matting using camera arrays
US20070070200A1 (en) 2005-09-29 2007-03-29 Wojciech Matusik Video matting using camera arrays
US20070110298A1 (en) 2005-11-14 2007-05-17 Microsoft Corporation Stereo video for gaming
US8094928B2 (en) 2005-11-14 2012-01-10 Microsoft Corporation Stereo video for gaming
US7631151B2 (en) 2005-11-28 2009-12-08 Commvault Systems, Inc. Systems and methods for classifying and transferring information in a storage network
US7720283B2 (en) 2005-12-09 2010-05-18 Microsoft Corporation Background removal in a live video
US20070133880A1 (en) 2005-12-09 2007-06-14 Microsoft Corporation Background Removal In A Live Video
US7821552B2 (en) 2005-12-27 2010-10-26 Sanyo Electric Co., Ltd. Imaging apparatus provided with imaging device having sensitivity in visible and infrared regions
US20070146512A1 (en) 2005-12-27 2007-06-28 Sanyo Electric Co., Ltd. Imaging apparatus provided with imaging device having sensitivity in visible and infrared regions
US20090003687A1 (en) 2006-01-10 2009-01-01 Microsoft Corporation Segmenting Image Elements
US8249333B2 (en) 2006-01-10 2012-08-21 Microsoft Corporation Segmenting image elements
US7747044B2 (en) 2006-01-20 2010-06-29 The Johns Hopkins University Fusing multimodal biometrics with quality estimates via a bayesian belief network
US8355379B2 (en) 2006-02-16 2013-01-15 Motorola Solutions, Inc. Handover between radio networks
US8363908B2 (en) 2006-05-03 2013-01-29 DigitalOptics Corporation Europe Limited Foreground / background separation in digital images
US20070269108A1 (en) 2006-05-03 2007-11-22 Fotonation Vision Limited Foreground / Background Separation in Digital Images
US20100034457A1 (en) 2006-05-11 2010-02-11 Tamir Berliner Modeling of humanoid forms from depth maps
US7420490B2 (en) 2006-07-31 2008-09-02 Texas Instruments Incorporated Pre-charge systems and methods for ADC input sampling
US8411149B2 (en) 2006-08-03 2013-04-02 Alterface S.A. Method and device for identifying and extracting images of multiple users, and for recognizing user gestures
US20090244309A1 (en) 2006-08-03 2009-10-01 Benoit Maison Method and Device for Identifying and Extracting Images of multiple Users, and for Recognizing User Gestures
US7773136B2 (en) 2006-08-28 2010-08-10 Sanyo Electric Co., Ltd. Image pickup apparatus and image pickup method for equalizing infrared components in each color component signal
US8131011B2 (en) 2006-09-25 2012-03-06 University Of Southern California Human detection and tracking system
US8073196B2 (en) 2006-10-16 2011-12-06 University Of Southern California Detection and tracking of moving objects from a moving platform in presence of strong parallax
US20080273751A1 (en) 2006-10-16 2008-11-06 Chang Yuan Detection and Tracking of Moving Objects from a Moving Platform in Presence of Strong Parallax
US8649932B2 (en) 2006-10-27 2014-02-11 International Electronic Machines Corp. Vehicle evaluation using infrared data
US8264544B1 (en) 2006-11-03 2012-09-11 Keystream Corporation Automated content insertion into video scene
US20080109724A1 (en) 2006-11-07 2008-05-08 Polycom, Inc. System and Method for Controlling Presentations and Videoconferences Using Hand Motions
US20100053212A1 (en) 2006-11-14 2010-03-04 Mi-Sun Kang Portable device having image overlay function and method of overlaying image in portable device
US9065973B2 (en) 2006-12-06 2015-06-23 Cisco Technology, Inc. System and method for displaying a videoconference
US8300890B1 (en) 2007-01-29 2012-10-30 Intellivision Technologies Corporation Person/object image and screening
US20080181507A1 (en) 2007-01-29 2008-07-31 Intellivision Technologies Corp. Image manipulation for videos and still images
US20080266380A1 (en) 2007-04-30 2008-10-30 Gorzynski Mark E Video conference system with symmetric reference
US8225208B2 (en) 2007-08-06 2012-07-17 Apple Inc. Interactive frames for images and videos displayed in a presentation application
US20140300630A1 (en) 2007-08-06 2014-10-09 Apple Inc. Background Removal Tool for a Presentation Application
US20090144651A1 (en) 2007-08-06 2009-06-04 Apple Inc. Interactive frames for images and videos displayed in a presentation application
US20090044113A1 (en) 2007-08-07 2009-02-12 Jones Scott T Creating a Customized Avatar that Reflects a User's Distinguishable Attributes
US8146005B2 (en) 2007-08-07 2012-03-27 International Business Machines Corporation Creating a customized avatar that reflects a user's distinguishable attributes
US8204316B2 (en) 2007-10-30 2012-06-19 Kddi Corporation Apparatus, method and computer program for classifying pixels in a motion picture as foreground or background
US20090110299A1 (en) 2007-10-30 2009-04-30 Kddi Corporation Apparatus, method and computer program for classifying pixels in a motion picture as foreground or background
US20090199111A1 (en) 2008-01-31 2009-08-06 G-Mode Co., Ltd. Chat software
US8331619B2 (en) 2008-03-14 2012-12-11 Sony Computer Entertainment Inc. Image processing apparatus and image processing method
US20100128927A1 (en) 2008-03-14 2010-05-27 Sony Computer Entertainment Inc. Image processing apparatus and image processing method
US8175384B1 (en) 2008-03-17 2012-05-08 Adobe Systems Incorporated Method and apparatus for discriminative alpha matting
US20090245571A1 (en) 2008-03-31 2009-10-01 National Taiwan University Digital video target moving object segmentation method and system
US8238605B2 (en) 2008-03-31 2012-08-07 National Taiwan University Digital video target moving object segmentation method and system
US20090249863A1 (en) 2008-04-03 2009-10-08 Korea Techno Co., Ltd Vapor phase decomposition device for semiconductor wafer pollutant measurement apparatus and door opening and closing device
US20090284627A1 (en) 2008-05-16 2009-11-19 Kaibushiki Kaisha Toshiba Image processing Method
US8437570B2 (en) 2008-05-23 2013-05-07 Microsoft Corporation Geodesic image and video processing
US20090290795A1 (en) 2008-05-23 2009-11-26 Microsoft Corporation Geodesic Image and Video Processing
US20090300553A1 (en) 2008-05-28 2009-12-03 Daniel Pettigrew Defining a border for an image
US8331685B2 (en) 2008-05-28 2012-12-11 Apple Inc. Defining a border for an image
US20110090311A1 (en) 2008-06-17 2011-04-21 Ping Fang Video communication method, device, and system
US8446459B2 (en) 2008-06-17 2013-05-21 Huawei Device Co., Ltd. Video communication method, device, and system
US20100027961A1 (en) 2008-07-01 2010-02-04 Yoostar Entertainment Group, Inc. Interactive systems and methods for video compositing
US8406494B2 (en) 2008-08-08 2013-03-26 Siemens Medical Solutions Usa, Inc. Anatomical primitive detection
US8175379B2 (en) 2008-08-22 2012-05-08 Adobe Systems Incorporated Automatic video image segmentation
US20100046830A1 (en) 2008-08-22 2010-02-25 Jue Wang Automatic Video Image Segmentation
US8345082B2 (en) 2008-10-08 2013-01-01 Cisco Technology, Inc. System and associated methodology for multi-layered site video conferencing
US20100171807A1 (en) 2008-10-08 2010-07-08 Tandberg Telecom As System and associated methodology for multi-layered site video conferencing
US20110243430A1 (en) 2008-11-04 2011-10-06 Konica Minolta Opto, Inc. Image input apparatus
US8666153B2 (en) 2008-11-04 2014-03-04 Konica Minolta Opto, Inc. Image input apparatus
US20100166325A1 (en) 2008-12-30 2010-07-01 Yahoo! Inc. Image segmentation
US8682072B2 (en) 2008-12-30 2014-03-25 Yahoo! Inc. Image segmentation
US8588515B2 (en) 2009-01-28 2013-11-19 Electronics And Telecommunications Research Institute Method and apparatus for improving quality of depth image
US20100195898A1 (en) 2009-01-28 2010-08-05 Electronics And Telecommunications Research Institute Method and apparatus for improving quality of depth image
US20130028476A1 (en) 2009-01-30 2013-01-31 Microsoft Corporation Pose tracking pipeline
US8565485B2 (en) 2009-01-30 2013-10-22 Microsoft Corporation Pose tracking pipeline
US8446488B2 (en) * 2009-03-11 2013-05-21 Samsung Electronics Co., Ltd. Method and system for focal length-specific color enhancement
US8477149B2 (en) * 2009-04-01 2013-07-02 University Of Central Florida Research Foundation, Inc. Real-time chromakey matting using image statistics
US8503720B2 (en) 2009-05-01 2013-08-06 Microsoft Corporation Human body pose estimation
US20100278384A1 (en) 2009-05-01 2010-11-04 Microsoft Corporation Human body pose estimation
US20100302376A1 (en) 2009-05-27 2010-12-02 Pierre Benoit Boulanger System and method for high-quality real-time foreground/background separation in tele-conferencing using self-registered color/infrared input images and closed-form natural image matting techniques
US20100302395A1 (en) 2009-05-29 2010-12-02 Microsoft Corporation Environment And/Or Target Segmentation
US8379101B2 (en) 2009-05-29 2013-02-19 Microsoft Corporation Environment and/or target segmentation
US20100329544A1 (en) 2009-06-30 2010-12-30 Sony Corporation Information processing apparatus, information processing method, and program
US8320666B2 (en) 2009-08-14 2012-11-27 Genesis Group Inc. Real-time image and video matting
US20110038536A1 (en) 2009-08-14 2011-02-17 Genesis Group Inc. Real-time image and video matting
US20140112547A1 (en) 2009-10-07 2014-04-24 Microsoft Corporation Systems and methods for removing a background of an image
US20110115886A1 (en) 2009-11-18 2011-05-19 The Board Of Trustees Of The University Of Illinois System for executing 3d propagation for depth image-based rendering
US8643701B2 (en) 2009-11-18 2014-02-04 University Of Illinois At Urbana-Champaign System for executing 3D propagation for depth image-based rendering
US20110158529A1 (en) 2009-12-28 2011-06-30 Xerox Corporation System and method for cleanup of mrc images for improved compression and image quality
US8335379B2 (en) 2009-12-28 2012-12-18 Xerox Corporation System and method for cleanup of MRC images for improved compression and image quality
US8659658B2 (en) 2010-02-09 2014-02-25 Microsoft Corporation Physical interaction zone for gesture-based user interfaces
US20110193939A1 (en) 2010-02-09 2011-08-11 Microsoft Corporation Physical interaction zone for gesture-based user interfaces
US20110216965A1 (en) 2010-03-05 2011-09-08 Microsoft Corporation Image Segmentation Using Reduced Foreground Training Data
US20110216976A1 (en) 2010-03-05 2011-09-08 Microsoft Corporation Updating Image Segmentation Following User Input
US8422769B2 (en) 2010-03-05 2013-04-16 Microsoft Corporation Image segmentation using reduced foreground training data
US8411948B2 (en) 2010-03-05 2013-04-02 Microsoft Corporation Up-sampling binary images for segmentation
US8655069B2 (en) 2010-03-05 2014-02-18 Microsoft Corporation Updating image segmentation following user input
US20110216975A1 (en) 2010-03-05 2011-09-08 Microsoft Corporation Up-Sampling Binary Images for Segmentation
US20110242277A1 (en) 2010-03-30 2011-10-06 Do Minh N Systems and methods for embedding a foreground video into a background feed based on a control input
US20130016097A1 (en) 2010-04-02 2013-01-17 Imec Virtual Camera System
US9117310B2 (en) 2010-04-02 2015-08-25 Imec Virtual camera system
US20140119642A1 (en) 2010-04-05 2014-05-01 Flashfoto, Inc. Segmenting Human Hairs and Faces
US8300938B2 (en) 2010-04-09 2012-10-30 General Electric Company Methods for segmenting objects in images
US20110249883A1 (en) 2010-04-09 2011-10-13 General Electric Company Methods for segmenting objects in images
US20110249863A1 (en) 2010-04-09 2011-10-13 Sony Corporation Information processing device, method, and program
US20110249190A1 (en) 2010-04-09 2011-10-13 Nguyen Quang H Systems and methods for accurate user foreground video extraction
US8818028B2 (en) 2010-04-09 2014-08-26 Personify, Inc. Systems and methods for accurate user foreground video extraction
US9336610B2 (en) 2010-04-09 2016-05-10 Sony Corporation Information processing device, method, and program
US9053573B2 (en) 2010-04-29 2015-06-09 Personify, Inc. Systems and methods for generating a virtual camera viewpoint for an image
US20110267348A1 (en) 2010-04-29 2011-11-03 Dennis Lin Systems and methods for generating a virtual camera viewpoint for an image
US8625897B2 (en) 2010-05-28 2014-01-07 Microsoft Corporation Foreground and background image segmentation
US20110293180A1 (en) 2010-05-28 2011-12-01 Microsoft Corporation Foreground and Background Image Segmentation
US9008457B2 (en) 2010-05-31 2015-04-14 Pesonify, Inc. Systems and methods for illumination correction of an image
US20110293179A1 (en) 2010-05-31 2011-12-01 Mert Dikmen Systems and methods for illumination correction of an image
US8913847B2 (en) 2010-06-01 2014-12-16 Hewlett-Packard Development Company, L.P. Replacement of a person or object in an image
US20130094780A1 (en) 2010-06-01 2013-04-18 Hewlett-Packard Development Company, L.P. Replacement of a Person or Object in an Image
US20120051631A1 (en) 2010-08-30 2012-03-01 The Board Of Trustees Of The University Of Illinois System for background subtraction with 3d camera
US8649592B2 (en) 2010-08-30 2014-02-11 University Of Illinois At Urbana-Champaign System for background subtraction with 3D camera
US9087229B2 (en) 2010-08-30 2015-07-21 University Of Illinois System for background subtraction with 3D camera
US20130243313A1 (en) 2010-10-01 2013-09-19 Telefonica, S.A. Method and system for images foreground segmentation in real-time
US8723914B2 (en) 2010-11-19 2014-05-13 Cisco Technology, Inc. System and method for providing enhanced video processing in a network environment
US20120127259A1 (en) 2010-11-19 2012-05-24 Cisco Technology, Inc. System and method for providing enhanced video processing in a network environment
US20130129205A1 (en) 2010-11-24 2013-05-23 Jue Wang Methods and Apparatus for Dynamic Color Flow Modeling
US8533593B2 (en) 2011-04-19 2013-09-10 Autodesk, Inc Hierarchical display and navigation of document revision histories
US8874525B2 (en) 2011-04-19 2014-10-28 Autodesk, Inc. Hierarchical display and navigation of document revision histories
US8701002B2 (en) 2011-04-19 2014-04-15 Autodesk, Inc. Hierarchical display and navigation of document revision histories
US8533595B2 (en) 2011-04-19 2013-09-10 Autodesk, Inc Hierarchical display and navigation of document revision histories
US8533594B2 (en) 2011-04-19 2013-09-10 Autodesk, Inc. Hierarchical display and navigation of document revision histories
US20130110565A1 (en) 2011-04-25 2013-05-02 Transparency Sciences, Llc System, Method and Computer Program Product for Distributed User Activity Management
US20120314077A1 (en) 2011-06-07 2012-12-13 Verizon Patent And Licensing Inc. Network synchronized camera settings
WO2013019259A1 (en) 2011-08-01 2013-02-07 Thomson Licensing Telepresence communications system and method
US8890929B2 (en) 2011-10-18 2014-11-18 Avaya Inc. Defining active zones in a traditional multi-party video conference and associating metadata with each zone
US20130142452A1 (en) 2011-12-02 2013-06-06 Sony Corporation Image processing device and image processing method
US8994778B2 (en) 2011-12-07 2015-03-31 Positron Telecommunication Systems, Inc. Systems and methods for providing video conferencing services via an ethernet adapter
US20130147900A1 (en) 2011-12-07 2013-06-13 Reginald Weiser Systems and methods for providing video conferencing services via an ethernet adapter
US20130335506A1 (en) 2012-06-14 2013-12-19 Polycom, Inc. Managing the layout of multiple video streams displayed on a destination display screen during a videoconference
US9088692B2 (en) 2012-06-14 2015-07-21 Polycom, Inc. Managing the layout of multiple video streams displayed on a destination display screen during a videoconference
US20140003719A1 (en) 2012-06-29 2014-01-02 Xue Bai Adaptive Trimap Propagation for Video Matting
US8897562B2 (en) 2012-06-29 2014-11-25 Adobe Systems Incorporated Adaptive trimap propagation for video matting
US8831285B2 (en) 2012-07-26 2014-09-09 Hewlett-Packard Development Company, L.P. Detecting objects with a depth sensor
US20140029788A1 (en) 2012-07-26 2014-01-30 Jinman Kang Detecting objects with a depth sensor
US20140063177A1 (en) 2012-09-04 2014-03-06 Cisco Technology, Inc. Generating and Rendering Synthesized Views with Multiple Video Streams in Telepresence Video Conference Sessions
US8890923B2 (en) 2012-09-04 2014-11-18 Cisco Technology, Inc. Generating and rendering synthesized views with multiple video streams in telepresence video conference sessions
US20140085398A1 (en) 2012-09-21 2014-03-27 Cisco Technology, Inc. Real-time automatic scene relighting in video conference sessions
US8854412B2 (en) 2012-09-21 2014-10-07 Cisco Technology, Inc. Real-time automatic scene relighting in video conference sessions
US20140153784A1 (en) 2012-10-18 2014-06-05 Thomson Licensing Spatio-temporal confidence maps
US9269153B2 (en) 2012-10-18 2016-02-23 Thomson Licensing Spatio-temporal confidence maps
US20140229850A1 (en) 2013-02-14 2014-08-14 Disney Enterprises, Inc. Avatar personalization in a virtual environment
US9285951B2 (en) 2013-02-14 2016-03-15 Disney Enterprises, Inc. Avatar personalization in a virtual environment
US20140307056A1 (en) 2013-04-15 2014-10-16 Microsoft Corporation Multimodal Foreground Background Segmentation
US9084928B2 (en) 2013-07-31 2015-07-21 Sven Gunnar Klang Tong operating device for a pinsetter
US9542626B2 (en) 2013-09-06 2017-01-10 Toyota Jidosha Kabushiki Kaisha Augmenting layer-based object detection with deep convolutional neural networks
US20170208243A1 (en) 2014-07-18 2017-07-20 Artincam Ltd. Automatic image composition

Non-Patent Citations (53)

* Cited by examiner, † Cited by third party
Title
Akbarzadeh, A., et al., "Towards Urban 3D Reconstruction From Video," Third International Symposium on 3D Data Processing, Visualization, and Transmission, pp. 1-8 (Jun. 14-16, 2006).
Arbelaez, P., et ,al., "Contour detection and hierarchical image segmentation", Pattern Analysis and Machine Intelligence, IEEE Transactions on 33.4 (2011): 898-916.
Barnat, Jii'f, et al., "CUDA accelerated LTL Model Checking," FI MU Report Series, FIMU-RS-2009-05, 20 pages (Jun. 2009).
Benezeth et al., "Review and Evaluation of Commonly-Implemented Background Subtraction Algorithms", 2008.
Canesta™, "See How Canesta's Solution Gesture Control Will Change the Living Room," retrieved Oct. 21, 2010, from http://canesta.com, 2 pages.
Carsten, R., et al., "Grabcut: Interactive foreground extraction using iterated graph cuts", ACM Transactions on Graphics (TOG) 23.3 (2004), pp. 309-314.
Chan, S.C., et al., "Image-Based Rendering and Synthesis," IEEE Signal Processing Magazine, pp. 22-31 (Nov. 2007).
Chan, Shing-Chow, et al. "The Plenoptic Video," 15(12) IEEE Transactions on Circuits and Systems for Video Technology 1650-1659 (Dec. 2005).
Chen, Wan-Yu, et al., "Efficient Depth Image Based Rendering with Edge Dependent Depth Filter and Interpolation," IEEE International Conference on Multimedia and Expo, pp. 1314-1317 (Jul. 6, 2005).
Cheung et al., "Robust Techniques for Background Subtraction in Urban Traffice Video", 2004.
Crabb et al., "Real-Time Foreground Segmentation via Range and Color Imaging", 2008.
Debevec, Paul, et al., "Efficient View-Dependent Image-Based Rendering with Projective Texture-Mapping," In 9th Eurographics Workshop on Rendering, pp. 105-116 (Jun. 1998).
Fehn, Christoph, et al., "Interactive 3-DTV-Concepts and Key Technologies," 94(3) Proceedings of the IEEE 524-538 (Mar. 2006).
GPGPU (General-purpose computing on graphics processing units)-Wikipedia, retrieved Nov. 17, 2009, from http://en.wikipedia.org/wiki/GPGPU, 9 pages.
GPGPU (General-purpose computing on graphics processing units)—Wikipedia, retrieved Nov. 17, 2009, from http://en.wikipedia.org/wiki/GPGPU, 9 pages.
Gvili et al., "Depth Keying", 2003.
H. Y. Shum and S. B. Kang, "A Review of Image-based Rendering Techniques," Proc. IEEE/SPIE Visual Communications and Image (VCIP) 2000, pp. 2-13, Perth, Jun. 2000.
Ho, Yo-Sung, et al., "Three-dimensional Video Generation for Realistic Broadcasting Services," ITC-CSCC, pp. Tr4 through TR4 (2008).
Hradis, M., et al., "Real-time Tracking of Participants in Meeting Video", Proceedings of CESCG, Wien, 2006.
Izquierdo' M. Ebroul. "Disparity/segmentation analysis: matching with an adaptive window and depth-driven segmentation." Circuits and Systems for Video Technology, IEEE Transactions on 9.4 (1999): 589-607.
Jung, Kwang Hee, et al., "Depth Image Based Rendering for 3D Data Service Over T-DMB," IEEE, 3DTV-CON'08, Istanbul, Turkey, pp. 237-240 (May 28-30, 2008).
Kanade, Takeo, et al., "Virtualized Reality: Constructing Virtual Worlds from Real Scenes," IEEE MultiMedia, pp. 34-46 (Jan.-Mar. 1997).
Kao, Wen-Chung, et al., "Multistage Bilateral Noise Filtering and Edge Detection for Color Image Enhancement," 51 (4) IEEE Transactions on Consumer Electronics 1346-1351 (Nov. 2005).
Kipfer, Peter, "GPU Gems 3-Chapter 33. LCP Algorithms for Collision Detection Using CUDA," retrieved Nov. 17, 2009, from http://http.developer.nvidia.com/ GPUGems3/qpuqems3 ch33.html, 11 pages (2007).
Kipfer, Peter, "GPU Gems 3—Chapter 33. LCP Algorithms for Collision Detection Using CUDA," retrieved Nov. 17, 2009, from http://http.developer.nvidia.com/ GPUGems3/qpuqems3 ch33.html, 11 pages (2007).
Kitagawa et al., "Background Separation Encoding for Surveillance Purpose by using Stable Foreground Separation", APSIPA, Oct. 4-7, 2009, pp. 849-852.
Kolmogorov, et al., "Bi-Layer Segmentation of Binocular Stereo Vision", IEEE, 2005.
Kubota, Akira, et al., "Multiview Imaging and 3DTV," IEEE Signal Processing Magazine, pp. 10-21 (Nov. 2007).
Lee, D.S., "Effective Gaussian Mixture for Video Background Subtraction", IEEE, vol. 27, No. 5, dated May 2005, pp. 827-832.
Lee, Eun-Kyung, et al., "High-Resolution Depth Map Generation by Applying Stereo Matching Based on Initial Depth Information," 3DTV-CON'08, Istanbul, Turkey, pp. 201-204 (May 28-30, 2008).
Mark, William R., et al., "Post-Rendering 3D Warping," In Proceedings of 1997 Symposium on Interactive 3D Graphics, Providence, RI, pp. 7-16 (Apr. 27-30, 1997).
McMillan, Jr., Leonard, "An Image-Based Approach to Three-Dimensional Computer Graphics," University of North Carolina at Chapel Hill, Chapel Hill, NC, 206 pages (1997).
Nguyen, Ha T., et al., "Image-Based Rendering with Depth Information Using the Propagation Algorithm," Proc. of IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 4 pages (Mar. 2005).
Nguyen, Quang H., et al., "Depth image-based rendering from multiple cameras with 3D propagation algorithm," Proceedings of the 2nd International Conference on Immersive Telecommunications, 6 pages (2009).
Nguyen, Quang H., et al., "Depth Image-Based Rendering with Low Resolution Depth," 16th IEEE International Conference on Image Processing (ICIP), pp. 553-556 (2009).
Piccardi, M., "Background Subtraction Techniques: A Review", IEEE, 2004.
PrimeSense, Home Page, retrieved Oct. 21, 2010, from http://www.primesense.com, 1 page.
Saxena, Ashutosh, et al., "3-D Depth Reconstruction from a Single Still Image," 76(1) International Journal of Computer Vision 53-69 (2007).
Shade, Jonathan, et al., "Layered Depth Images," Computer Graphics Proceedings, Annual Conference Series, pp. 231-242 (Jul. 19-24, 1998).
Sheasby, G., et al., "A robust stereo prior for human segmentation", In ACCV, 2012.
Talukder, A., et al., "Real-time detection of moving objects in a dynamic scene from moving robotic vehicles," in Intelligent Robots and Systems, 2003. (IROS 2003). Proceedings. 2003 IEEE/RSJ international Conference on, vol. 2, pp. 1308-1313, vol. 2, Oct. 27-31, 2003.
Tomasi, C., et al., "Bilateral Filtering for Gray and Color Images," Sixth International Conference on Computer Vision, pp. 839-846 (1998).
Um, Gi-Mun, et al., "Three-dimensional Scene Reconstruction Using Multi-View Images and Depth Camera", pp. 271-280, SPIE-IS&t, vol. 5664, 2005.
Vazquez, C., et al., "3D-TV: Coding of Disocclusions for 2D+Depth Representation of Multi-View Images," Proceedings of the Tenth IASTED Int'l Conference: Computer Graphics and Imaging, pp. 26-33 (Feb. 13-15, 2008).
Wang, L., et al., "Tofcut: Towards robust real-time foreground extraction using a time-off camera.", Proc. of 3DPVT, 2010.
Working screenshot of Snagit manufactured by Techsmith, released Apr. 18, 2014.
Xu, F., et al., "Human detection using depth and gray images", Advanced Video and Signal Based Surveillance, 2003., Proceedings, IEEE Conference on IEEE, 2003.
Yacoob, Y., et al., "Detection, analysis and matching of hair," in Computer Vision, 2005, ICCV 2005. Tenth IEEE International Conference, vol. 1., No., pp. 741-748, vol. 1, Oct. 17-21, 2005.
Yang, Qingxiong, et al., "Spatial-Depth Super Resolution for Range Images," IEEE Conference on Computer Vision and Pattern Recognition, pp. 1-8 (2007).
Zhang, Buyue, et al., "Adaptive Bilateral Filter for Sharpness Enhancement and Noise Removal," IEEE ICIP, pp. IV-417-IV-420 (2007).
Zhang, Q., et al., "Segmentation and tracking multiple objects under occlusion from multiview video.", Image Processing, IEEE Transactions on 20.11 (2011), pp. 3308-3313.
Zitnick, C. Lawrence, et al., "High-quality video view interpolation using a layered representation," 23(3) Journal ACM Transactions on Graphics (TOG)-Proceedings of ACM SIGGRAPH 2004, pp. 600-608 (Aug. 2004).
Zitnick, C. Lawrence, et al., "High-quality video view interpolation using a layered representation," 23(3) Journal ACM Transactions on Graphics (TOG)—Proceedings of ACM SIGGRAPH 2004, pp. 600-608 (Aug. 2004).

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10984589B2 (en) 2017-08-07 2021-04-20 Verizon Patent And Licensing Inc. Systems and methods for reference-model-based modification of a three-dimensional (3D) mesh data model
US10997786B2 (en) 2017-08-07 2021-05-04 Verizon Patent And Licensing Inc. Systems and methods for reconstruction and rendering of viewpoint-adaptive three-dimensional (3D) personas
US11004264B2 (en) 2017-08-07 2021-05-11 Verizon Patent And Licensing Inc. Systems and methods for capturing, transferring, and rendering viewpoint-adaptive three-dimensional (3D) personas
US11024078B2 (en) 2017-08-07 2021-06-01 Verizon Patent And Licensing Inc. Systems and methods compression, transfer, and reconstruction of three-dimensional (3D) data meshes
US11095854B2 (en) 2017-08-07 2021-08-17 Verizon Patent And Licensing Inc. Viewpoint-adaptive three-dimensional (3D) personas
US11386618B2 (en) 2017-08-07 2022-07-12 Verizon Patent And Licensing Inc. Systems and methods for model-based modification of a three-dimensional (3D) mesh
US11461969B2 (en) 2017-08-07 2022-10-04 Verizon Patent And Licensing Inc. Systems and methods compression, transfer, and reconstruction of three-dimensional (3D) data meshes
US11580697B2 (en) 2017-08-07 2023-02-14 Verizon Patent And Licensing Inc. Systems and methods for reconstruction and rendering of viewpoint-adaptive three-dimensional (3D) personas

Also Published As

Publication number Publication date
US20170359559A1 (en) 2017-12-14

Similar Documents

Publication Publication Date Title
US9883155B2 (en) Methods and systems for combining foreground video and background video using chromatic matching
US10896358B2 (en) Image processing apparatus, display system, image processing method and storage medium for projecting a moving image onto printed matter in an overlapping manner
US8463038B2 (en) Image processing apparatus and method for correcting an image based upon features of the image
US9087385B2 (en) Method for improving images captured underwater
WO2013131311A1 (en) Method and system for carrying out vision perception high-fidelity transformation on color digital image
GB2504381A (en) Selecting a colour matching profile to increase the vividness of an image
JP5387713B2 (en) Image adjustment apparatus, image adjustment system, and program
CN111124404B (en) Custom color display method and system
US11962917B2 (en) Color adjustment method, color adjustment device, electronic device and computer-readable storage medium
Sharma Understanding RGB color spaces for monitors, projectors, and televisions
JP2016177500A (en) Image processing device, image processing system and program
WO2022120799A9 (en) Image processing method and apparatus, electronic device, and storage medium
CN113994420B (en) Metamerism stabilization via custom viewer color matching function
TWI523500B (en) Dynamic range compression method for image and image processing device
CN101304467B (en) Color processing apparatus and method
JP2007087147A (en) Image retouching method, image retouching device, and image retouching program
US8922580B2 (en) Method and system to modify a color lookup table
US11574607B2 (en) Display device and control method of display device
JP2015139131A (en) Image processing device, image processing method and program
EP3454538A1 (en) Method for determining a transfer function adapted to be used for color grading a visual content and corresponding electronic device, system, computer readable program product and computer readable storage medium
Birukov et al. Correction of color saturation for tone mapping operator
US9930349B2 (en) Image processing to retain small color/gray differences
KR102215607B1 (en) Electronic device capable of correction to improve the brightness of dark images and operating method thereof
WO2022089211A1 (en) Image quality adjustment method and device
US11869174B2 (en) Image processing apparatus, display apparatus, image processing method, and non-transitory computer readable medium

Legal Events

Date Code Title Description
AS Assignment

Owner name: PERSONIFY, INC., ILLINOIS

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:PATEL, SANJAY;YARKONY, ELAD;SIGNING DATES FROM 20160606 TO 20160613;REEL/FRAME:039166/0870

STCF Information on status: patent grant

Free format text: PATENTED CASE

AS Assignment

Owner name: HONGFUJIN PRECISION INDUSTRY (WUHAN) CO. LTD., TEX

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:PERSONIFY, INC.;REEL/FRAME:051367/0920

Effective date: 20190522

FEPP Fee payment procedure

Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

MAFP Maintenance fee payment

Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

Year of fee payment: 4

AS Assignment

Owner name: PERSONIFY, INC., CALIFORNIA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:HONGFUJIN PRECISION INDUSTRY WUHAN;REEL/FRAME:057467/0738

Effective date: 20210909