Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T11/00—2D [Two Dimensional] image generation
  • G06T11/003—Reconstruction from projections, e.g. tomography
  • G06T11/006—Inverse problem, transformation from projection-space into object-space, e.g. transform methods, back-projection, algebraic methods
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/20—Analysis of motion
  • G06T7/254—Analysis of motion involving subtraction of images
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T11/00—2D [Two Dimensional] image generation
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T11/00—2D [Two Dimensional] image generation
  • G06T11/60—Editing figures and text; Combining figures or text
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T11/00—2D [Two Dimensional] image generation
  • G06T11/80—Creating or modifying a manually drawn or painted image using a manual input device, e.g. mouse, light pen, direction keys on keyboard
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/10—Segmentation; Edge detection
  • G06T7/174—Segmentation; Edge detection involving the use of two or more images
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/10—Image acquisition modality
  • G06T2207/10016—Video; Image sequence
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/20—Special algorithmic details
  • G06T2207/20092—Interactive image processing based on input by user
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/20—Special algorithmic details
  • G06T2207/20092—Interactive image processing based on input by user
  • G06T2207/20096—Interactive definition of curve of interest

Definitions

• the present invention generally relates to digital image processing, and more particularly to matting methods and apparatus for video images.
• Image matting is the process of extracting an object from an image with some human guidance.
• Image matting may be an interactive process which relies on limited user input, usually in the form of a few scribbles, to mark foreground and background regions.
• foreground refers to the object to be extracted
• background refers to everything else in the image.
• Video matting is an extension of image matting wherein the goal is to extract a moving object from a video sequence.
• Video matting can also be used in video processing devices (including video encoders). For instance, automatic matte extraction can be used to identify a particular region in a video scene (e.g. sky area), and then apply a given processing only to that region (e.g.
• Matte extraction can also be used to guide object detection and object tracking algorithms. For instance, a matte extraction technique could be used to detect the grass area in a soccer video (i.e. the playfield) which could then use to constrain the search range in a ball tracking algorithm.
• mattes have been used to composite foreground (e.g. actors) and background (e.g. landscape) images into a final image.
• the chroma keying (blue screen) technique is a widely used method for matting actors into a novel background.
• Many of the traditional techniques rely on a controlled environment during the image capture process. With digital images, however, it becomes possible to directly manipulate pixels, and thus matte out foreground objects from existing images with some human guidance.
• Digital image matting is used in many image and video editing applications for extracting foreground objects and possibly for compositing several objects into a final image.
• image matting is usually an interactive process in which the user provides some input such as marking the foreground and possibly the background regions.
• the easier-to-use interfaces are those in which the user places a few scribbles with a digital brush marking the foreground and background regions (see FIG. 2A).
• An image matting process determines the boundary of the foreground object using the image information along with the user input.
• the user provides a rough, usually hand- drawn, segmentation called a trimap, wherein each pixel is labeled as a foreground, background, or unknown pixel.
• a trimap segmentation
• each pixel is labeled as a foreground, background, or unknown pixel.
• Other methods allow a more user-friendly scribble-based interaction in which the user places a few scribbles with a digital brush marking the foreground and background regions.
• a method for propagating user-provided foreground-background constraint information for a first video frame to subsequent frames, thereby allowing extraction of moving foreground objects in a video stream with minimal user interaction.
• Video matting is performed wherein the user input (e.g. scribbles) with respect to a first frame is propagated to subsequent frames using the estimated matte of each frame.
• the matte of a frame is processed in order to arrive at a rough foreground-background segmentation which is then used for estimating the matte of the next frame.
• the propagated input is used by an image matting method for estimating the corresponding matte which is in turn used for propagating the input to the next frame, and so on.
• FIG. I is a block diagram of an exemplary frame-wise matting process or apparatus
• FIGs. 2A-2D illustrate an interactive image matting process, in which FIG. 2A shows an image or video frame with white scribbles marking foreground and black scribbles marking background; FIG. 2B shows an extracted matte or foreground opacity for FIG. 2A; FIG. 2C shows an extracted foreground image; and FIG. 2D shows the foreground object composited with a novel background image;
• FIG. 3 is a flow diagram of a foreground-background constraint propagation method
• FIG. 5 is a block diagram of an exemplary system embodiment of the present invention.
• image matting methods process an input image / which is assumed to be a composite of a foreground image F and a background image B.
• the color of the i x pixel is assumed to be a linear combination of the corresponding foreground F 1 and background B 1 colors or intensities:
• FIG. 1 is a block diagram of an exemplary frame-wise matting process or apparatus in which the foreground is extracted on a frame-by-frame basis using image matting block 1 10. As shown in FIG.
• image matting block 1 10 for a frame at time /, image matting block 1 10 generates an associated alpha matte a' based on foreground-background (F-B) constraints c' that are based on input by a user for frame t or propagated from user input for a previous frame via constraint propagation block 120.
• c' denotes the F-B constraints of frame /. For the / th pixel, I if pixel / is marked as foreground 0 if pixel / is marked as background .
• FIG. 2B shows the matte ⁇ d) obtained for frame / based on the user-provided scribbles (c') in FIG. 2A.
• FIG. 2C shows the extracted foreground object and FIG. 2D shows an application of the matting in which the extracted foreground object is composited with a novel background image.
• matting block 1 10 can be implemented in accordance with the matting technique described in A. Levin et al., "A closed-form solution to natural image matting," IEEE Trans, on Pattern Analysis and Machine Intelligence, vol. 30, no. 2, pp. 228-242, Feb 2008.
• the matting technique of Levin et al. minimizes a quadratic cost function in the alpha matte under some constraints. Any suitable matting techniques may be used for this purpose.
• the foreground-background constraints can be derived by propagating the user-input constraints from a previous frame via constraint propagation block 120. An exemplary method for propagating the foreground-background (F-B) constraints from one frame to the next will now be described.
• FIG. 3 is a flow diagram of an exemplary method 300 of propagating F-B constraints from one frame to the next is outlined in FIG. 3.
• FIGs. 4A through 4F show illustrative images pertaining to the method 300.
• the method 300 uses the alpha matte at frame t to estimate F-B constraints at time M-I . Instead of propagating the constraints (e.g., scribbles or trimap) directly, the method uses the matte at t to generate the constraints for frame H-I . In other words, using the matte (d) from frame t, the F-B constraints (c' +l ) for the frame H-I are obtained.
• the matte (d) from frame t the F-B constraints (c' +l ) for the frame H-I are obtained.
• the alpha matte a,' for the first of a sequence of frames, i.e., for the frame of time /, is provided, such as from the matting method of FIG. 1, as an input to the method 300.
• the method 300 comprises foreground constraint propagation procedure 310 and background constraint propagation procedure 320.
• a thresholding operation is performed in which the alpha values are compared to a threshold T( g to generate a binary field ⁇ ' such that:
• the thresholding operation thus isolates pixels with a high foreground component.
• FIG. 4A illustrates the results of the thresholding step 31 1.
• FIG. 4A shows the binary field ⁇ ' obtained by thresholding the alpha matte d shown in FIG. 2B.
• values of threshold T fg are in the range of 0.5 to 0.9.
• £(5) is a disk with a radius of 5 pixels.
• the structuring element can be any suitable shape, including for example, a square, however, an isotropic shape such as a disk is preferable.
• the scale of the structuring element is preferably selected based on the desired separation between the foreground and background, and the size of the foreground and/or background, so as avoid excessive erosion of foreground and/or background pixels.
• thresholding step 31 1 yields a small foreground area that would be eliminated or reduced by morphological erosion step 312 to a foreground area smaller than a predetermined minimum size (such as a size too small to be perceived by a viewer), morphological erosion step 312 may be skipped.
• FIG. 4B illustrates the field ⁇ ' resulting from the morphological erosion of binary field ⁇ ' in step 312.
• the foreground constraints are indicated by the white pixels.
• the scale $ fg of the structuring element E ⁇ .) can be chosen based on the degree of motion of the foreground and/or background.
• c/ +l may be set as background at another point in the process, as described below, or remain undefined.
• the background constraints are determined based on the already-determined foreground constraints ⁇ / (FIG. 4B). For each pixel, a lower alpha value (a/) indicates a higher background component. By applying a threshold on (1 - a,'), the background pixels may be isolated. However, it is not desirable that the background-constrained pixels lie very far from the foreground, since this would increase the "fuzzy zone" between background-constrained and foreground-constrained pixels, possibly resulting in inaccurate foreground extraction. In order to achieve this balance, a normalized distance transform is determined at step 321 in accordance with the following expression: m ⁇ nd(i,J)
• the weight w has a value of 0.8.
• the weight w has a range of 0.5 to 0.9.
• FIG. 4C illustrates the background score field ⁇ ' determined in step 322. The brighter the pixel appears in FIG. 4C, the higher its background score.
• step 322 The background score determined in step 322 is then subjected to a thresholding operation in step 323 in which the background score field ⁇ ' is compared to a threshold to generate a binary field ⁇ ' such that:
• ⁇ ,g is a preset background score threshold.
• An exemplary range of values for threshold ⁇ , g is 0.5 to 0.9.
• ⁇ 1 F, ( ⁇ ' , E(s bs )) (8) where F E (.) denotes the morphological erosion operator and E(s) denotes a structuring element of scale s.
• F E (.) denotes the morphological erosion operator
• E(s) denotes a structuring element of scale s.
• FIG. 4D illustrates the field ⁇ ' resulting from the morphological erosion of binary field ⁇ ! in step 324.
• the white pixels in FIG. 4D indicate the background-constrained pixels.
• the erosion operation ensures that these regions lie outside the foreground in frame M-I even if the foreground (and/or background) has moved by a certain amount.
• thresholding step 323 yields a small background area that would be eliminated or reduced by morphological erosion step 324 to a background area smaller than a predetermined minimum size (such as a size too small to be perceived by a viewer)
• morphological erosion step 324 may be skipped.
• FIG. 4F illustrates the matte ⁇ +] extracted from the frame M-I using the propagated constraints.
• the matte ⁇ + ⁇ can be generated using the matting method 1 10 of FIG. 1. This matte is in turn used for deriving the constraints for frame M-2 and so on.
• the exemplary method avoids the complexity of motion estimation methods such as correlation-based template matching or optical flow and works reliably over a range of motion levels.
• prior information such as the area of the foreground object and its color distribution in the current frame is used in deriving the F-B constraints for the next frame.
• All or a subset of the parameters r fg , ⁇ ,g, ,s ⁇ g , S bg , and w can be automatically adjusted based on the prior information in order to extract an accurate matte. This process can be carried out iteratively until the matte satisfies the constraints imposed by the prior information.
• a brute force process includes trying out multiple values, preferably within predefined ranges, for each parameter and selecting the set of values that best satisfies the prior information.
• FIG. 5 is a block diagram of an exemplary system 500 in accordance with the principles of the invention.
• the system 500 can be used to generate alpha mattes, F-B constraints, and/or perform matting from a video stream.
• the system 500 comprises a frame grabber 510 and a digital video editor 520.
• Frame grabber 510 captures one or more frames of the video stream for processing by digital video editor 520 in accordance with the principles of the invention.
• Digital video editor 520 comprises processor 521 , memory 522 and I/O 523.
• digital video editor 520 may be implemented as a general purpose computer executing software loaded in memory 522 for carrying out constraint propagation and/or matting as described above.

Landscapes

• Engineering & Computer Science (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Theoretical Computer Science (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Multimedia (AREA)
• Algebra (AREA)
• Mathematical Analysis (AREA)
• Mathematical Optimization (AREA)
• Mathematical Physics (AREA)
• Pure & Applied Mathematics (AREA)
• Image Analysis (AREA)
• Image Processing (AREA)
• Studio Circuits (AREA)

Priority Applications (6)

Applications Claiming Priority (2)

Publications (1)

Family

ID=42562016

Family Applications (1)

Country Status (7)

Cited By (1)

Families Citing this family (43)

Citations (5)

Family Cites Families (1)

• 2010
  • 2010-01-05 KR KR1020117018272A patent/KR101670282B1/ko not_active Expired - Fee Related
  • 2010-01-05 CN CN201080015984.0A patent/CN102388391B/zh not_active Expired - Fee Related
  • 2010-01-05 US US13/138,384 patent/US8611728B2/en not_active Expired - Fee Related
  • 2010-01-05 BR BRPI1008024A patent/BRPI1008024A2/pt not_active Application Discontinuation
  • 2010-01-05 WO PCT/US2010/000009 patent/WO2010093408A1/en not_active Ceased
  • 2010-01-05 EP EP10741495A patent/EP2396748A4/en not_active Withdrawn
  • 2010-01-05 JP JP2011550117A patent/JP5607079B2/ja not_active Expired - Fee Related

Patent Citations (5)

Non-Patent Citations (4)

Also Published As

Similar Documents

Legal Events