US20040091047A1 - Method and apparatus for nonlinear multiple motion model and moving boundary extraction - Google Patents

Method and apparatus for nonlinear multiple motion model and moving boundary extraction Download PDF

Info

Publication number
US20040091047A1
US20040091047A1 US10/291,989 US29198902A US2004091047A1 US 20040091047 A1 US20040091047 A1 US 20040091047A1 US 29198902 A US29198902 A US 29198902A US 2004091047 A1 US2004091047 A1 US 2004091047A1
Authority
US
United States
Prior art keywords
model
motion
motions
boundary
ref
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US10/291,989
Other languages
English (en)
Inventor
Marco Paniconi
James Carrig
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sony Corp
Sony Electronics Inc
Original Assignee
Sony Corp
Sony Electronics Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sony Corp, Sony Electronics Inc filed Critical Sony Corp
Priority to US10/291,989 priority Critical patent/US20040091047A1/en
Assigned to SONY ELECTRONICS INC., SONY CORPORATION reassignment SONY ELECTRONICS INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: CARRIG, JAMES J., JR., PANICONI, MARCO
Priority to KR1020057008406A priority patent/KR101021409B1/ko
Priority to AU2003290644A priority patent/AU2003290644A1/en
Priority to JP2004551854A priority patent/JP4651385B2/ja
Priority to PCT/US2003/035512 priority patent/WO2004044842A2/en
Priority to CNA2003801030216A priority patent/CN1711776A/zh
Priority to EP03783225A priority patent/EP1561347A4/en
Priority to CN2008100883165A priority patent/CN101257632B/zh
Publication of US20040091047A1 publication Critical patent/US20040091047A1/en
Priority to US11/125,422 priority patent/US7751479B2/en
Abandoned legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/103—Selection of coding mode or of prediction mode
    • H04N19/105—Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/207—Analysis of motion for motion estimation over a hierarchy of resolutions
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/215—Motion-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/223—Analysis of motion using block-matching
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
    • H04N19/136—Incoming video signal characteristics or properties
    • H04N19/137—Motion inside a coding unit, e.g. average field, frame or block difference
    • H04N19/139—Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
    • H04N19/51—Motion estimation or motion compensation
    • H04N19/513—Processing of motion vectors
    • H04N19/521—Processing of motion vectors for estimating the reliability of the determined motion vectors or motion vector field, e.g. for smoothing the motion vector field or for correcting motion vectors
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
    • H04N19/51—Motion estimation or motion compensation
    • H04N19/537—Motion estimation other than block-based
    • H04N19/543—Motion estimation other than block-based using regions
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
    • H04N19/51—Motion estimation or motion compensation
    • H04N19/553—Motion estimation dealing with occlusions
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/12—Systems in which the television signal is transmitted via one channel or a plurality of parallel channels, the bandwidth of each channel being less than the bandwidth of the television signal
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20016—Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20021—Dividing image into blocks, subimages or windows

Definitions

  • the present invention pertains to image processing. More particularly, the present invention relates to estimation of object motion in images.
  • Standard motion modeling for video coding involves parametric models, applied to a fixed region (motion block), to estimate the motion. These approaches are limited in that the models cannot handle the existence of multiple (different) motions within the motion block. This presents a problem.
  • a basic problem in motion estimation is the ability of the model to handle multiple motion and moving object boundaries.
  • Standard motion models such as the affine or perspective models, allow for smooth deformations of a region (i.e., the motion block) to capture a coherent motion (such as translation, zoom, rotation) for all the pixels in the motion block.
  • the region or block over which the motion is estimated cannot be chosen to be to small; this is from (1) a coding point of view, since larger regions mean smaller motion overhead, and (2) from an estimation point of view, larger region allows for better estimation of motion parameters.
  • a moving object boundary within a motion region is indication of two possibly very different motions (motion of the object and motion of say the background).
  • a moving object boundary implies that some pixels will be occluded (hidden) with respect to the past or future motion estimation. This occlusion effect can bias the motion estimate, lead to higher prediction error, and make it difficult to accurately extract the object boundary.
  • FIG. 1 illustrates a network environment in which techniques of the present invention may be used
  • FIG. 2 is a block diagram of a computer system in which embodiments of the present invention may be implemented
  • FIG. 3 illustrates one embodiment of the invention in flow chart form
  • FIG. 4 illustrates in flow chart form one embodiment of video coding
  • FIG. 5 illustrates one embodiment of motion segmentation into 2 regions
  • FIG. 6 illustrates the behavior of one embodiment of a function that controls the time reference assignment of pixels
  • FIG. 7, FIG. 8, and FIG. 9 are examples illustrating how embodiments of the invention motion model, applied to a local block region, achieve separation into past and future motion references, and hence the extraction of the moving boundary is captured;
  • FIG. 10 is an example illustrating how an embodiment of the invention motion model estimated the location of a moving boundary
  • FIG. 11 is an example illustrating the comparison between a standard motion model and an embodiment of the invention motion model
  • FIG. 12 is an example illustrating 3 motions, their movement, and lowest predicted error reference frames.
  • FIG. 13 illustrates the behavior of one embodiment of an interface function which controls the time reference assignment for 3 motions.
  • the present invention involves a new motion model for estimation of object motion in video images.
  • a new motion model that involves nonlinear coupling between space and time variables is used, a type of region competition to separate multiple motions, and boundary modeling to extract an estimate of the moving object boundary.
  • the model is compact and can be used in motion segmentation and/or video coding applications.
  • the present invention is capable of handling multiple motions (two or more). However, to not unnecessarily obscure the present invention, the discussion will initially discuss two motions, with an extension to more than two motions described later in the specification.
  • the motion model is applied locally to a region/block in an image, and it may be viewed as part of a refinement stage to motion estimation or motion segmentation. That is, if after one pass of a motion estimation/segmentation algorithm of an image (say initially using a standard affine motion model) the prediction error in some region is above some quality threshold, then an embodiment of the present invention motion model may be applied to those regions.
  • FIG. 3 illustrates the process in flow chart form 300 .
  • the prediction error from a standard motion model for a region is received.
  • dashed block 314 is where some of the techniques of the invention are performed.
  • an extension in the motion model may be used for true non-rigid deformation of object boundary.
  • box 312 in FIG. 3 may also refer to a more complex model to handle true non-rigid deformation.
  • An extension such as a boundary-to-boundary matching can be used and incorporated in the structure illustrated in FIG. 3.
  • simple segmentation (for low overhead) of a motion block/region to capture multiple motions may be achieved with quadtree segmentation of blocks, where large prediction error blocks are partitioned in sub-blocks for improved motion estimation.
  • blocks with large prediction errors may be quadtree segmented with a straight line model of the boundary/partition.
  • the approach is more aligned with the motion segmentation problem itself, which involves the ability to obtain good estimates of the location and local shape of the moving object boundaries.
  • FIG. 4 illustrates in flow chart form 400 one embodiment of video coding.
  • the motion model is used to estimate motion and remove temporal redundancy, resulting in a small motion residual to code. Discussed later are additional embodiments of the invention and how the motion model may be used efficiently and effectively for coding.
  • the input images are received.
  • motion estimation is performed on a given frame and occlusion regions and moving boundaries, using the multiple motion and boundary extraction invention, are identified.
  • the remaining motion residual is coded.
  • the time variable is used for representation of 2 motions.
  • simultaneous estimation with respect to past and future is used (i.e., 2 reference frames are used), so that pixels close to the boundary that are occluded in, say the past frame, will choose estimation from the future frame (where they are not occluded), and vice-versa. It is this duality of occlusion that is exploited in the model.
  • a nonlinear aspect is used on the time variable (and hence boundary model) to control and refine the estimation of the boundary interface.
  • the extended motion model may be used locally, and as part of a successive iterative approach, as illustrated in FIG. 3. Regions that are deemed poor (because of high prediction error), say in a first pass of a segmentation process, may be re-estimated with the extended motion model to capture multiple motions and the moving boundaries.
  • the boundary is defined implicitly through the time variable in the motion model, whose functional form allows for the motion domains to be defined by regions of smooth compact support.
  • a standard motion model often used in motion estimation is the affine model, which takes the following form:
  • Nonlinear perspective models is an extension of the affine model to 8 parameters to handle projection into the image plane. The motion models are applied over some region (motion block), and estimation of the parameters can involve linear least squared projections, direct minimization of prediction error, multi-resolution minimization of prediction error, etc.
  • Embodiments of the invention include a model to account for multiple motions and estimation of moving object boundaries.
  • the model for 2 motions (more general case discussed later) takes the following form:
  • x 1 v 1 x ( x,y )+( v 2 x ( x,y ))( t 1 +1)
  • x 1 ax+by+c +( a 1 x+b 1 y+c 1 )( t 1 +1)
  • ⁇ a,b,c,d,e,f,a′,b′,c′,d′,e′,f′ ⁇ are parameters of the affine motion models.
  • the coupling to the time variable allows for the presence of 2 different motions in this embodiment (i.e., with different translation, rotation, and zooming).
  • the partition of the motion region into 2 motions is defined according to whether the region uses a past or a future frame for motion estimation. This is shown in FIG. 5.
  • motion segmentation into 2 regions is obtained by the region's frame reference for motion estimation.
  • the object moving with velocity V o is the foreground object in this example.
  • the model is determined by minimizing the prediction error (for both past and future reference frames).
  • the lowest prediction error should result in the bottom partition ( 510 ), which avoids any occlusion effect, and hence has the best potential for locating the true moving boundary.
  • 502 is the previous frame
  • 506 is the future or next frame.
  • 508 is one possible partition of the current frame into two motions.
  • 510 is another partition of the current frame into two motions and is the lower prediction error case when compared with the 508 partition.
  • the time variable in Equation (1) is a smooth function of the pixel locations, and varies from ⁇ 1 to 0.
  • a given pixel location in the motion block on the current frame defines the time variable t 1 .
  • This time variable is then used in the last 2 equations to determine the motion vectors.
  • the time variable controls the motion of the object boundary.
  • the smoothness of the interface model allows each motion region to be defined by a smooth compact support.
  • the nonlinear function F ⁇ ( s ) ( tanh ⁇ ( ( s + 0.5 ) / w ) - 1 ) 2
  • t 1 F(s) used in one embodiment of the motion model.
  • This function is characterized by a width parameter (w) and appropriately saturates at ⁇ 1 and 0.
  • a key feature in the model is the “boundary width” (w) that controls the spread of the time variable from ⁇ 1 (past frame) to 0 (future frame). Pixels near the boundary (defined by width w) are a type of mixture phase, i.e., linear combination of the 2 domains. That is, for a pixel within the boundary region, the prediction is:
  • I pred ( x,y ) (1+ t 1 ) I future ( x 1 ,y 1 ) ⁇ t 1 I past ( x 1 ,y 1 )
  • w is fixed and small during the estimation step of the motion parameters.
  • the nonlinear function F(s) in the model and the decrease of w is used to control and refine the estimate of the boundary.
  • the estimation of the motion model parameters is obtained from minimization of the prediction error.
  • e ⁇ ( x , y ) ⁇ motion_block ⁇ ( I ⁇ ( x , y ) - I pred ⁇ ( x , y ) ) 2
  • I pred ⁇ ( x , y ) ( 1 + t ⁇ ′ ) ⁇ I future ⁇ ( x ′ , y ′ ) - t ⁇ ′ ⁇ I past ⁇ ( x ′ , y ′ )
  • (x 1 ,y 1 ,t 1 ) are functions of the model parameters (see Equation (1)). Note that for each pixel, the prediction is a linear combination of past and future frames; simple bilinear temporal interpolation may be used. The estimation of the model parameters may be obtained from a steepest descent algorithm using multiple resolution layers (described below).
  • the interface parameters are chosen to be
  • i ⁇ 0.075, ⁇ 0.5, ⁇ 0.25.
  • BM block matching
  • BM is done with respect to the past; with respect to the future for set 2.
  • the set of motion vectors is then mapped onto the model parameters using Least Squares (LS). This yields an initial set of parameters (a,b,c,d,e,f) for initial condition set 1 and 2; the parameters (a 1 ,b 1 ,c 1 ,d 1 ,e 1 ,f 1 ) are initialized to 0.
  • LS Least Squares
  • the motion model was applied to a region (80 ⁇ 80 block) which contains 2 motions.
  • the original image is shown on the left, and the right image shows the segmentation of a multiple motion region into 2 regions.
  • the dark region references the past frame, and the white region references the future frame. Note that in each example the segmentation into past/future regions is consistent with the effect of occlusion being minimized, as discussed, and shown in FIG. 5.
  • Example 1 is shown in FIG. 7.
  • the fan moves to the right. Curvature of the fan object is captured, and the motion model achieves separation into past and future motion references as discussed, and shown in FIG. 5.
  • 702 is the original image, and 704 shows the segmentation of a multiple motion region into 2 regions. The dark region references the past frame, and the white region references the future frame.
  • Example 2 is shown in FIG. 8. Here, the man moves downwards. This is the same effect as in the previous example. 802 is the original image, and 804 shows the segmentation of a multiple motion region into 2 regions. The dark region references the past frame, and the white region references the future frame. The frame reference assignment is such that occlusion effect is minimized, as discussed in FIG. 5.
  • Example 3 is shown in FIG. 9.
  • the girl in the foreground moves to the left. Because the girl moves to the left, the stationary region in front of her will prefer motion estimation with respect to the past where no occlusion occurs.
  • 902 is the original image
  • 904 shows the segmentation of a multiple motion region into 2 regions. The dark region references the past frame
  • the white region references the future frame.
  • the prediction error data was calculated as the mean square error between the motion predicted region/block and the original block.
  • the standard motion model refers to a single motion affine model, often used in motion estimation.
  • the new motion model refers to an embodiment of the invention. As shown below, there is an improvement in prediction error using the new motion model.
  • Standard motion model New motion model Example 1 26.8 9.1 Example 2 22.0 8.6 Example 3 18.9 5.5
  • a large region around the objects of interest was partitioned into 80 ⁇ 80 blocks. This region was obtained from a standard type of motion segmentation (affine motion model and k-means clustering), with poorly labeled blocks (blocks with high prediction error and/or high distortion classification) identifying the regions of moving objects.
  • an embodiment of the invention new motion model was applied to a set of 80 ⁇ 80 blocks covering a large region around the moving object of interest.
  • Example 4 is shown in FIG. 10 where the thin black line 1002 is the estimation of location of the boundary using the new motion model.
  • Example 4 In Example 4 as shown in FIG. 10, the girl walks to the right, the background “moves” to the left.
  • the motion model is applied to a large region around the girl.
  • the black lines around the girl ( 1002 ) is the extracted location of the moving object.
  • the missing contour along her nose/face closely coincides with the boundary of one of the 80 ⁇ 80 blocks; thus most of the pixels in that block belong to one motion (face motion), and so the system selected one domain/state with no boundary.
  • FIG. 11 Shown in FIG. 11 is a comparison between a segmentation using an affine motion model (standard motion model) 1104 , and the improvement using the new model 1106 as disclosed in one embodiment of the invention.
  • the small picture 1102 is the original image.
  • the image 1104 is the segmentation map derived from a standard method using affine motion models. Different shades refer to different motion classes.
  • the image 1106 is the new segmentation map obtained by re-estimating the motion with the new motion model. Image 1106 shows a better outline of the girl in the image, and a smoother segmentation field than does image 1104 .
  • video coding may make use of the new motion model.
  • the model discussed above by virtue of its ability to account for 2 motions, can be applied to a large region.
  • 80 ⁇ 80 blocks were used.
  • the new motion model may be viewed as “compactly” representing different motions and boundary information.
  • the present model has 17 parameters, and if used in say 80 ⁇ 80 blocks (in a 704 ⁇ 484 image), is about 900 motion parameters; this includes all information necessary for a decoder to extract motion field and locations of some moving boundaries. Compare this to the approximately 2662 parameters needed for a very simple standard 16 ⁇ 16 Block Matching Algorithm (2 translation parameters, with no explicit moving boundary information).
  • ⁇ right arrow over (x) ⁇ refers to a pixel position on the current frame (the one whose motion is being estimated)
  • ⁇ right arrow over (x) ⁇ 1 refers to a position on the reference frame
  • ⁇ t i ref ⁇ are the M reference frames used for extraction of M motions.
  • the motion vectors ⁇ right arrow over (v) ⁇ i ⁇ are affine motion fields
  • t 1 is the continuous time variable
  • the interface equations ⁇ s j ⁇ are polynomials that model the location and shape of the boundary.
  • FIG. 12 An example for 3 motions is shown in FIG. 12.
  • the three “motions” in the image region 1300 are the middle region, which is a stationary foreground, and the other 2 regions moving as indicated by the arrows.
  • the 2 non-intersecting boundaries are shown as straight lines.
  • FIG. 13 An example of an interface function for 3 motions (2 non-intersecting boundaries) is shown in FIG. 13.
  • the width parameters ⁇ w j ⁇ may be fixed external parameters, however in general they may also be determined dynamically (which would allow the system to adjust or select the width/roughness of the boundary).
  • the invention may also be viewed as compactly representing multiple motions and boundary information.
  • FIG. 1 illustrates a network environment 100 in which the techniques described may be applied.
  • a network 102 which may be, for example, a home based network.
  • the network 102 might be or include one or more of: the Internet, a Local Area Network (LAN), Wide Area Network (WAN), satellite link, fiber network, cable network, or a combination of these and/or others.
  • the servers may represent, for example, disk storage systems alone or storage and computing resources.
  • the clients may have computing, storage, and viewing capabilities.
  • the method and apparatus described herein may be applied to essentially any type of communicating means or device whether local or remote, such as a LAN, a WAN, a system bus, etc.
  • FIG. 2 illustrates a computer system 200 in block diagram form, which may be representative of any of the clients and servers shown in FIG. 1, and which may be representative of an embodiment of the invention.
  • the block diagram is a high level conceptual representation and may be implemented in a variety of ways and by various architectures.
  • Bus system 202 interconnects a Central Processing Unit (CPU) 204 , Read Only Memory (ROM) 206 , Random Access Memory (RAM) 208 , storage 210 , display 220 , audio, 222 , keyboard 224 , pointer 226 , miscellaneous input/output (I/O) devices 228 , and communications 230 .
  • CPU Central Processing Unit
  • ROM Read Only Memory
  • RAM Random Access Memory
  • the bus system 202 may be for example, one or more of such buses as a system bus, Peripheral Component Interconnect (PCI), Advanced Graphics Port (AGP), Small Computer System Interface (SCSI), Institute of Electrical and Electronics Engineers (IEEE) standard number 1394 (FireWire), Universal Serial Bus (USB), etc.
  • the CPU 204 may be a single, multiple, or even a distributed computing resource.
  • Storage 210 may be Compact Disc (CD), Digital Versatile Disk (DVD), hard disks (HD), optical disks, tape, flash, memory sticks, video recorders, etc.
  • Display 220 might be, for example, a Cathode Ray Tube (CRT), Liquid Crystal Display (LCD), a projection system, Television (TV), etc.
  • CTR Cathode Ray Tube
  • LCD Liquid Crystal Display
  • TV Television
  • the computer system may include some, all, more, or a rearrangement of components in the block diagram.
  • a thin client might consist of a wireless hand held device that lacks, for example, a traditional keyboard.
  • the present invention can be implemented by an apparatus for performing the operations herein.
  • This apparatus may be specially constructed for the required purposes, or it may comprise a general-purpose computer, selectively activated or reconfigured by a computer program stored in the computer.
  • a computer program may be stored in a computer readable storage medium, such as, but not limited to, any type of disk including floppy disks, hard disks, optical disks, compact disk-read only memories (CD-ROMs), and magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs), electrically programmable read-only memories (EPROM)s, electrically erasable programmable read-only memories (EEPROMs), FLASH memories, magnetic or optical cards, etc., or any type of media suitable for storing electronic instructions either local to the computer or remote to the computer.
  • ROMs read-only memories
  • RAMs random access memories
  • EPROM electrically programmable read-only memories
  • EEPROMs electrically erasable programmable read-only memories
  • the methods of the invention may be implemented using computer software. If written in a programming language conforming to a recognized standard, sequences of instructions designed to implement the methods can be compiled for execution on a variety of hardware platforms and for interface to a variety of operating systems.
  • the present invention is not described with reference to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the teachings of the invention as described herein.
  • a machine-readable medium is understood to include any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer).
  • a machine-readable medium includes read only memory (ROM); random access memory (RAM); magnetic disk storage media; optical storage media; flash memory devices; electrical, optical, acoustical or other form of propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.); etc.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)
  • Image Analysis (AREA)
  • Processing Or Creating Images (AREA)
US10/291,989 2002-11-11 2002-11-11 Method and apparatus for nonlinear multiple motion model and moving boundary extraction Abandoned US20040091047A1 (en)

Priority Applications (9)

Application Number Priority Date Filing Date Title
US10/291,989 US20040091047A1 (en) 2002-11-11 2002-11-11 Method and apparatus for nonlinear multiple motion model and moving boundary extraction
CN2008100883165A CN101257632B (zh) 2002-11-11 2003-11-06 用于非线性的多运动模型和移动边界提取的方法和设备
PCT/US2003/035512 WO2004044842A2 (en) 2002-11-11 2003-11-06 Method and apparatus for nonlinear multiple motion model and moving boundary extraction
AU2003290644A AU2003290644A1 (en) 2002-11-11 2003-11-06 Method and apparatus for nonlinear multiple motion model and moving boundary extraction
JP2004551854A JP4651385B2 (ja) 2002-11-11 2003-11-06 非線形の複数の動きモデル及び移動境界を抽出する方法及び装置
KR1020057008406A KR101021409B1 (ko) 2002-11-11 2003-11-06 비선형 다중 모션 모델 및 이동 경계 추출을 위한 방법 및 장치
CNA2003801030216A CN1711776A (zh) 2002-11-11 2003-11-06 用于非线性的多运动模型和移动边界提取的方法和设备
EP03783225A EP1561347A4 (en) 2002-11-11 2003-11-06 METHOD AND APPARATUS USING A NONLINEAR MULTIPLE MOTION MODEL FOR MOBILE FRONTIER EXTRACTION
US11/125,422 US7751479B2 (en) 2002-11-11 2005-05-09 Method and apparatus for nonlinear multiple motion model and moving boundary extraction

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
US10/291,989 US20040091047A1 (en) 2002-11-11 2002-11-11 Method and apparatus for nonlinear multiple motion model and moving boundary extraction

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US11/125,422 Continuation US7751479B2 (en) 2002-11-11 2005-05-09 Method and apparatus for nonlinear multiple motion model and moving boundary extraction

Publications (1)

Publication Number Publication Date
US20040091047A1 true US20040091047A1 (en) 2004-05-13

Family

ID=32229341

Family Applications (2)

Application Number Title Priority Date Filing Date
US10/291,989 Abandoned US20040091047A1 (en) 2002-11-11 2002-11-11 Method and apparatus for nonlinear multiple motion model and moving boundary extraction
US11/125,422 Expired - Fee Related US7751479B2 (en) 2002-11-11 2005-05-09 Method and apparatus for nonlinear multiple motion model and moving boundary extraction

Family Applications After (1)

Application Number Title Priority Date Filing Date
US11/125,422 Expired - Fee Related US7751479B2 (en) 2002-11-11 2005-05-09 Method and apparatus for nonlinear multiple motion model and moving boundary extraction

Country Status (7)

Country Link
US (2) US20040091047A1 (zh)
EP (1) EP1561347A4 (zh)
JP (1) JP4651385B2 (zh)
KR (1) KR101021409B1 (zh)
CN (2) CN1711776A (zh)
AU (1) AU2003290644A1 (zh)
WO (1) WO2004044842A2 (zh)

Cited By (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7142600B1 (en) * 2003-01-11 2006-11-28 Neomagic Corp. Occlusion/disocclusion detection using K-means clustering near object boundary with comparison of average motion of clusters to object and background motions
US20070297513A1 (en) * 2006-06-27 2007-12-27 Marvell International Ltd. Systems and methods for a motion compensated picture rate converter
US20080037869A1 (en) * 2006-07-13 2008-02-14 Hui Zhou Method and Apparatus for Determining Motion in Images
EP1925159A2 (en) * 2005-09-16 2008-05-28 Sony Electronics, Inc. Adaptive area of influence filter for moving object boundaries
US20090060042A1 (en) * 2007-08-28 2009-03-05 Samsung Electronics Co., Ltd. System and method for motion vector collection based on k-means clustering for motion compensated interpolation of digital video
US20090190038A1 (en) * 2008-01-25 2009-07-30 Mediatek Inc. Method, video encoder, and integrated circuit for detecting non-rigid body motion
WO2010070128A1 (en) * 2008-12-19 2010-06-24 Thomson Licensing Method for multi-resolution motion estimation
US20110129015A1 (en) * 2007-09-04 2011-06-02 The Regents Of The University Of California Hierarchical motion vector processing method, software and devices
US8923400B1 (en) * 2007-02-16 2014-12-30 Geo Semiconductor Inc Method and/or apparatus for multiple pass digital image stabilization
US20160005184A1 (en) * 2014-07-01 2016-01-07 Irida Labs S.A. System and a method for camera motion analysis and understanding from a video sequence
CN106384361A (zh) * 2016-09-14 2017-02-08 西安理工大学 多视环境中基于水平集函数演化的非刚性目标场景流估计
US11895327B2 (en) 2006-08-02 2024-02-06 Interdigital Vc Holdings, Inc. Method and apparatus for parametric, model- based, geometric frame partitioning for video coding

Families Citing this family (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8406299B2 (en) * 2007-04-17 2013-03-26 Qualcomm Incorporated Directional transforms for intra-coding
US20090174812A1 (en) * 2007-07-06 2009-07-09 Texas Instruments Incorporated Motion-compressed temporal interpolation
US20090052532A1 (en) * 2007-08-24 2009-02-26 Simon Robinson Automatically identifying edges of moving objects
US8059909B2 (en) * 2008-04-29 2011-11-15 Sony Corporation Adaptive generation of irregular spatial sub-sampling for images
US8059908B2 (en) * 2008-04-29 2011-11-15 Sony Corporation Adaptive area of influence filter for irregular spatial sub-sampled images
US8055087B2 (en) * 2008-04-29 2011-11-08 Sony Corporation Sample level variation for spatial sub-sampled images
KR101502362B1 (ko) 2008-10-10 2015-03-13 삼성전자주식회사 영상처리 장치 및 방법
WO2012012582A1 (en) * 2010-07-21 2012-01-26 Dolby Laboratories Licensing Corporation Reference processing using advanced motion models for video coding
JP2013048717A (ja) 2011-08-31 2013-03-14 Sony Corp 画像処理装置及び方法、記録媒体、並びにプログラム
EP2805306B1 (en) * 2012-01-19 2016-01-06 Thomson Licensing Method and device for generating a motion field for a video sequence
WO2013137890A1 (en) * 2012-03-15 2013-09-19 Intel Corporation Hierarchical motion estimation employing nonlinear scaling and adaptive source block size
CN104253650B (zh) * 2013-06-27 2016-12-28 富士通株式会社 信道内非线性损伤的估计装置及方法
US10306229B2 (en) 2015-01-26 2019-05-28 Qualcomm Incorporated Enhanced multiple transforms for prediction residual
CN104661031B (zh) * 2015-02-16 2017-12-22 华为技术有限公司 用于视频图像编码和解码的方法、编码设备和解码设备
US10623774B2 (en) 2016-03-22 2020-04-14 Qualcomm Incorporated Constrained block-level optimization and signaling for video coding tools
US11323748B2 (en) 2018-12-19 2022-05-03 Qualcomm Incorporated Tree-based transform unit (TU) partition for video coding
CN110139100B (zh) * 2019-05-16 2021-07-06 东莞职业技术学院 多图像运动估计的扩展块匹配及运动矢量估计算法

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6400831B2 (en) * 1998-04-02 2002-06-04 Microsoft Corporation Semantic video object segmentation and tracking
US6658059B1 (en) * 1999-01-15 2003-12-02 Digital Video Express, L.P. Motion field modeling and estimation using motion transform
US6665423B1 (en) * 2000-01-27 2003-12-16 Eastman Kodak Company Method and system for object-oriented motion-based video description

Family Cites Families (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
ATE109604T1 (de) 1987-12-22 1994-08-15 Philips Nv Videosignalkodierung und -dekodierung mit einem adaptiven filter.
JP3531186B2 (ja) * 1992-09-18 2004-05-24 ソニー株式会社 ビデオ信号符号化方法及び装置
JP3679426B2 (ja) 1993-03-15 2005-08-03 マサチューセッツ・インスティチュート・オブ・テクノロジー 画像データを符号化して夫々がコヒーレントな動きの領域を表わす複数の層とそれら層に付随する動きパラメータとにするシステム
JPH08223577A (ja) * 1994-12-12 1996-08-30 Sony Corp 動画像符号化方法及び装置、並びに動画像復号方法及び装置
JP3432937B2 (ja) * 1995-03-15 2003-08-04 株式会社東芝 移動物体検知装置及び移動物体検知方法
ES2153597T3 (es) 1995-10-25 2001-03-01 Koninkl Philips Electronics Nv Sistema y metodo de codificacion de imagen segmentada, y sistema y metodo de decodificacion correspondiente.
KR970025184A (ko) 1995-10-26 1997-05-30 김광호 예측매크로블럭 변환을 이용한 동화상 복호기
US5778097A (en) 1996-03-07 1998-07-07 Intel Corporation Table-driven bi-directional motion estimation using scratch area and offset valves
EP0894403B1 (en) 1996-04-19 2002-08-28 Nokia Corporation Video encoder and decoder using motion-based segmentation and merging
JP3876392B2 (ja) 1996-04-26 2007-01-31 富士通株式会社 動きベクトル探索方法
US5991447A (en) 1997-03-07 1999-11-23 General Instrument Corporation Prediction and coding of bi-directionally predicted video object planes for interlaced digital video
US6249548B1 (en) 1998-07-10 2001-06-19 U.S. Phillips Corporation Motion vector processing
KR100415266B1 (ko) * 2000-05-11 2004-01-16 가부시끼가이샤 도시바 물체영역정보 기술방법과 물체영역정보 생성장치 및 기록매체
US6731799B1 (en) * 2000-06-01 2004-05-04 University Of Washington Object segmentation with background extraction and moving boundary techniques
US7092550B2 (en) * 2001-12-12 2006-08-15 Sony Corporation Implementation of hough transform and its application in video motion analysis
BR0304545A (pt) 2002-01-14 2004-11-03 Nokia Corp Método de codificação das imagens em uma sequência de vìdeo digital para fornecer os dados de vìdeo codificados, codificador de vìdeo, método de decodificação dos dados indicativos de uma sequência de vìdeo digital, decodificador de vìdeo, e, sistema de decodificação de vìdeo
US6646578B1 (en) 2002-11-22 2003-11-11 Ub Video Inc. Context adaptive variable length decoding system and method

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6400831B2 (en) * 1998-04-02 2002-06-04 Microsoft Corporation Semantic video object segmentation and tracking
US6658059B1 (en) * 1999-01-15 2003-12-02 Digital Video Express, L.P. Motion field modeling and estimation using motion transform
US6665423B1 (en) * 2000-01-27 2003-12-16 Eastman Kodak Company Method and system for object-oriented motion-based video description

Cited By (23)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7142600B1 (en) * 2003-01-11 2006-11-28 Neomagic Corp. Occlusion/disocclusion detection using K-means clustering near object boundary with comparison of average motion of clusters to object and background motions
USRE42790E1 (en) 2003-01-11 2011-10-04 Neomagic Corporation Occlusion/disocclusion detection using K-means clustering near object boundary with comparison of average motion of clusters to object and background motions
EP1925159A2 (en) * 2005-09-16 2008-05-28 Sony Electronics, Inc. Adaptive area of influence filter for moving object boundaries
EP1925159A4 (en) * 2005-09-16 2009-09-23 Sony Electronics Inc ADAPTIVE INFLUENCE FILTER ZONE FOR MOTION OBJECT LIMITS
US8340185B2 (en) * 2006-06-27 2012-12-25 Marvell World Trade Ltd. Systems and methods for a motion compensated picture rate converter
US20070297513A1 (en) * 2006-06-27 2007-12-27 Marvell International Ltd. Systems and methods for a motion compensated picture rate converter
US20080037869A1 (en) * 2006-07-13 2008-02-14 Hui Zhou Method and Apparatus for Determining Motion in Images
US7783118B2 (en) 2006-07-13 2010-08-24 Seiko Epson Corporation Method and apparatus for determining motion in images
US11895327B2 (en) 2006-08-02 2024-02-06 Interdigital Vc Holdings, Inc. Method and apparatus for parametric, model- based, geometric frame partitioning for video coding
US8923400B1 (en) * 2007-02-16 2014-12-30 Geo Semiconductor Inc Method and/or apparatus for multiple pass digital image stabilization
US20090060042A1 (en) * 2007-08-28 2009-03-05 Samsung Electronics Co., Ltd. System and method for motion vector collection based on k-means clustering for motion compensated interpolation of digital video
US8861603B2 (en) * 2007-08-28 2014-10-14 Samsung Electronics Co., Ltd. System and method for motion vector collection based on K-means clustering for motion compensated interpolation of digital video
US20110129015A1 (en) * 2007-09-04 2011-06-02 The Regents Of The University Of California Hierarchical motion vector processing method, software and devices
US8605786B2 (en) * 2007-09-04 2013-12-10 The Regents Of The University Of California Hierarchical motion vector processing method, software and devices
US8208552B2 (en) * 2008-01-25 2012-06-26 Mediatek Inc. Method, video encoder, and integrated circuit for detecting non-rigid body motion
US20090190038A1 (en) * 2008-01-25 2009-07-30 Mediatek Inc. Method, video encoder, and integrated circuit for detecting non-rigid body motion
FR2940492A1 (fr) * 2008-12-19 2010-06-25 Thomson Licensing Procede d'estimation de mouvement multi-resolutions
WO2010070128A1 (en) * 2008-12-19 2010-06-24 Thomson Licensing Method for multi-resolution motion estimation
US20160005184A1 (en) * 2014-07-01 2016-01-07 Irida Labs S.A. System and a method for camera motion analysis and understanding from a video sequence
US9508026B2 (en) * 2014-07-01 2016-11-29 Irida Labs S.A. System and a method for camera motion analysis and understanding from a video sequence
US20170076466A1 (en) * 2014-07-01 2017-03-16 Irida Labs S.A. System and a method for camera motion analysis and understanding from a video sequence
US10185897B2 (en) * 2014-07-01 2019-01-22 Irida Labs S.A. System and a method for camera motion analysis and understanding from a video sequence
CN106384361A (zh) * 2016-09-14 2017-02-08 西安理工大学 多视环境中基于水平集函数演化的非刚性目标场景流估计

Also Published As

Publication number Publication date
US7751479B2 (en) 2010-07-06
EP1561347A4 (en) 2009-03-18
KR101021409B1 (ko) 2011-03-14
AU2003290644A1 (en) 2004-06-03
WO2004044842A3 (en) 2005-02-03
CN101257632A (zh) 2008-09-03
CN101257632B (zh) 2011-12-14
KR20050072139A (ko) 2005-07-08
WO2004044842A2 (en) 2004-05-27
US20050213660A1 (en) 2005-09-29
JP2006505870A (ja) 2006-02-16
EP1561347A2 (en) 2005-08-10
CN1711776A (zh) 2005-12-21
JP4651385B2 (ja) 2011-03-16
AU2003290644A8 (en) 2004-06-03

Similar Documents

Publication Publication Date Title
US7751479B2 (en) Method and apparatus for nonlinear multiple motion model and moving boundary extraction
US8682063B2 (en) Methods and apparatus for dynamic color flow modeling
US8923638B2 (en) Algorithm selection for structure from motion
US9087390B2 (en) High-quality upscaling of an image sequence
US7194110B2 (en) Method and apparatus for tracking features in a video sequence
EP1612733B1 (en) Color segmentation-based stereo 3D reconstruction system and process
US6266443B1 (en) Object boundary detection using a constrained viterbi search
US7957610B2 (en) Image processing method and image processing device for enhancing the resolution of a picture by using multiple input low-resolution pictures
US20140126818A1 (en) Method of occlusion-based background motion estimation
Bei et al. Learning semantic-aware dynamics for video prediction
US8711938B2 (en) Methods and systems for motion estimation with nonlinear motion-field smoothing
US7916791B2 (en) Method and system for non-linear motion estimation
US8000527B2 (en) Interactive image segmentation by precomputation
US20180005039A1 (en) Method and apparatus for generating an initial superpixel label map for an image
US7110453B1 (en) Motion or depth estimation by prioritizing candidate motion vectors according to more reliable texture information
US9317928B2 (en) Detecting and tracking point features with primary colors
KR101438451B1 (ko) 비고정 카메라 영상에 대한 이중모드 ｓｇｍ 기반의 이동체 고속 검출 방법 및 이를 위한 컴퓨터로 판독가능한 기록매체
JP2980810B2 (ja) 動きベクトル探索方法と装置
David et al. Foreground/background segmentation with learned dictionary
US20240029281A1 (en) Method for reducing error of depth estimation model, electronic device, and non-transitory storage medium
JPH0837664A (ja) 動画像符号化／復号化装置
JP2934151B2 (ja) 動きベクトル探索方法と装置
CN114257748A (zh) 视频防抖方法及装置、计算机可读介质和电子设备
AU5426801A (en) Estimating a property of one or more regions of a two dimensional set of data

Legal Events

Date Code Title Description
AS Assignment

Owner name: SONY ELECTRONICS INC., NEW JERSEY

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:PANICONI, MARCO;CARRIG, JAMES J., JR.;REEL/FRAME:013958/0469

Effective date: 20021111

Owner name: SONY CORPORATION, JAPAN

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:PANICONI, MARCO;CARRIG, JAMES J., JR.;REEL/FRAME:013958/0469

Effective date: 20021111

STCB Information on status: application discontinuation

Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION