KR20080055965A - Encoder assisted frame rate up conversion using various motion models - Google Patents

Abstract

An Encoder Assisted Frame Rate Up Conversion (EA-FRUC) system that utilizes various motion models, such as affine models, in addition to video coding and pre-processing operations at the video encoder to exploit the FRUC processing that will occur in the decoder in order to improve the modeling of moving objects, compression efficiency and reconstructed video quality. Furthermore, objects are identified in a way that reduces the amount of information necessary for encoding to render the objects on the decoder device.

Description

ENCODER ASSISTED FRAME RATE UP CONVERSION USING VARIOUS MOTION MODELS}

This patent application contains the provisional application number 60 / 721,375 entitled (a) Method of Encoder Auxiliary Frame Rate Upconversion Using Different Motion Models, filed September 27, 2005 and (b) September 27, 2005. Priority is filed in Provisional Application No. 60 / 721,376, entitled "Methods and Apparatus for Encoder Auxiliary Frame Rate Upconversion," filed on May, incorporated herein by reference.

This disclosure relates to a method and apparatus for encoding video data.

Today there are video formats that support various frame rates. The following formats are currently most widely used, listed in order of supported frames per second (fps): 24 (film negative), 25 (PAL), 30 (generally interlaced video), And 60 (high resolution (HD), such as 720p). The frames are suitable for most applications, but the frame rates are sometimes reduced to rates as low as 15, 10, 7.5 or 3 fps to achieve the low bandwidth required for mobile handset video communications. While the low rates allow low end devices with lower computational capabilities to display some video, the resulting video quality may not be “jerkiness” (ie, with a slide show effect) rather than smooth in motion. Experience. Also, reduced frames often do not accurately track the amount of motion in the video. For example, fewer frames should be reduced during "high motion" video content portions such as those occurring at sporting events, while more frames may be reduced during "low motion" video content portions such as those occurring at talk shows. have. Video compression is content dependent and can be integrated by analyzing motion and texture features in the sequence to be coded to improve video compression efficiency.

Frame Rate Up Conversion (FRUC) is a process that uses video interpolation in a video decoder to increase the frame rate of reconstructed video. In FRUC, interpolated frames are generated using the frames received as references. Currently, systems implementing FRUC frame interpolation include approaches based on motion compensation interpolation and processing of transmitted motion vectors. FRUC is also used for conversion between various video formats. For example, in telecine and reverse telecine applications, which are film-to-video tape delivery techniques that adjust respective color frame rate differences between film and video, progressive video (24 frames / second) is NTSC interlaced. Is converted into video (29.97 frames / second).

Another FRUC approach uses weighted-adaptive motion compensation interpolation (WAMCI) to reduce block compression artifacts caused by defects in motion estimation and block-based processing. This approach is based on interpolation by additive polymerization of multiple motion compensated interpolation (MCI) images. Block compression artifacts at the block boundaries are also reduced in the proposed method by applying a technique similar to overlapped block motion compensation (OBMC). Specifically, to reduce blurring while processing overlapping areas, the method uses motion analysis to adaptively determine the block motion type and adaptively applies OBMC. Experimental results suggest that the proposed approach improves the results by significantly reducing block compression artifacts.

Another FRUC approach uses vector reliability analysis to reduce compression artifacts caused by the use of any motion vectors incorrectly transmitted from the encoder. In this approach, motion estimation is used to construct motion vectors that are compared to the transmitted motion vectors to determine the most desirable approach for frame interpretation. In existing upconversion algorithms using motion estimation, the estimation process is performed using two adjacent decoded frames to construct motion vectors that allow the frame to be interpolated. However, these algorithms attempt to improve the utilization of transmission bandwidth without considering the amount of computation required for motion estimation operations. In comparison, in up-conversion algorithms using transmitted motion vectors, the quality of the interpolated frames is highly dependent on the motion vectors obtained by the encoder. Using a combination of these two approaches, the transmitted motion vectors are first analyzed to determine whether they can be used to construct interpolated frames. The method used for interpolation is then adaptively selected from three methods: local motion-compensation interpolation, global motion-compensation interpolation and frame-repeat interpolation.

Although FRUC techniques are generally implemented as post-processing functions of a video decoder, a video encoder is therefore typically not included in this operation. However, in an approach referred to as encoder-assisted FRUC (EA-FRUC), the encoder is still capable of generating motion vectors or motion vectors while still allowing the decoder to regenerate significant portions of frames independently without the vector or residual data being removed. It may be determined whether the transmission of specific information related to the reference frames (eg, residual data) is to be removed. For example, the bidirectional predictive video coding method has been introduced as a method for improving B-frame coding in MPEG-2. In this method, the use of an error criterion is proposed to enable the application of real motion vectors in motion-compensated predictive coding. Distortion measurements are based on absolute difference sum (SAD), but such distortion measurements are known to be insufficient in providing actual distortion measurements, especially where the amount of motion between two frames in a sequence needs to be limited. Also, since classifications for threshold variations are preferably content dependent, changes in thresholds are sorted using fixed thresholds when those thresholds should be variable.

FRUC video compression techniques include using encoder enhancement information, and use block-based motion prediction with moving motion models to model the motion of objects within video frames. Block-based motion prediction uses a time correction structure inherent to video signals. Moving motion modeling, such as used by block-based motion prediction, allows video to move through a motion that moves in a plane that is more parallel or less parallel to the lenses of the video capture device while maintaining a fixed shape. Time redundancy within the signals can be reduced or eliminated. The moving motion model uses two parameters per encoded block.

In motion-compensated prediction and transform coding based on hybrid video compression, video frames are partitioned by conventional encoders according to the use of a moving motion model, in which case the partitions remain fixed while experiencing the moving motion. To determine the location of the object bodies. For example, a video sequence of a person calling a camera while a car is passing can include a still image showing the fixed background of the sequence, a video object showing the head of the caller, an audio object showing the voice associated with the person, And another video object representing a moving car on a sprite having a rectangular support area. In still images, the position of the page can be moved temporarily.

Unfortunately, moving model motion prediction cannot accurately predict or account for motion for objects in a motion that requires two or more parameters per block. Independently moving objects in combination with camera motion and focal length changes generate complex motion vectors that must be efficiently approximated for motion prediction. Thus, the rest of the signal (known as a prediction error) has power to be considered, so video frames containing the movement are inefficient for compression. When video frames containing the objects are interpolated using block-based motion prediction, both the subjective or objective quality of the interpolated frame is due to the limitations of the motion model infrastructure that move to account for block motion dynamics. low. In addition, when video sequences are partitioned according to moving model motion prediction, the efficiency of the algorithm for processing interpolations of an object that experiences arbitrary movement and deformations is limited.

Provide high quality interpolated frames in a decoder device that properly models moving objects while reducing the amount of bandwidth required to transmit information to perform interpolation, and provide the frames for multimedia mobile devices that rely on low power processing. An approach that reduces the amount of computation needed to produce is desirable.

Certain aspects disclosed herein provide a variety of motion in addition to video coding and pre-processing operations in a video encoder to utilize FRUC processing that will occur within a decoder to improve modeling, compression efficiency, and reconstructed video quality of moving objects. Provides encoder assisted frame rate up-conversion (EA-FRUC) using models.

In one aspect, a method of processing multimedia data is disclosed. The method comprises partitioning at least one of the first and second video frames into a plurality of partitions, modeling information for at least one object in at least one of the partitions, wherein the modeling information is the first one. And associated with second video frames, generating an interpolation frame based on the modeling information, and encoding information based on the interpolation frame, wherein the encoding information is temporarily located with the interpolation frame. Used to generate a co-located video frame.

In another aspect, an apparatus for processing multimedia data is disclosed. The apparatus comprises means for dividing at least one of the first and second video frames into a plurality of partitions, modeling information for at least one object in at least one of the plurality of partitions, the modeling information being the Means for determining an associated with first and second video frames, means for generating an interpolation frame based on the modeling information, and encoding information based on the interpolation frame, the encoding information being temporarily associated with the interpolation frame. Means for generating a co-located video frame.

In a further aspect, an apparatus for processing multimedia data is disclosed. The apparatus comprises a partitioning module configured to partition at least one of the first and second video frames into a plurality of partitions, modeling information for at least one object in at least one of the plurality of partitions-the modeling information A modeling module configured to determine an associated with the first and second video frames, a frame generation module configured to generate an interpolated frame based on the modeling information, and an encoding configured to generate encoding information based on the interpolated frame. A module, and a sending module, configured to send the encoding information to a decoder.

In another aspect, a machine readable medium is disclosed that includes instructions for processing multimedia data. The instructions may cause the machine to divide at least one of the first and second video frames into a plurality of partitions, the modeling information for at least one object in at least one of the partitions, the modeling information being Determine an associated with the first and second video frames, generate an interpolated frame based on the modeling information, and encode information based on the interpolated frame—the encoding information is temporarily associated with the interpolated frame. Used to generate a co-located video frame.

In another aspect, a processor for processing multimedia data is disclosed. The processor divides at least one of the first and second video frames into a plurality of partitions, and modeling information for at least one object in at least one of the partitions, wherein the modeling information is determined by the first and second images. Is associated with two video frames, generates an interpolated frame based on the modeling information, and encodes information based on the interpolated frame, the encoding information being temporarily co-located with the interpolated frame. located) is used to generate a video frame.

Other objects, features and advantages will be apparent to those skilled in the art from the following detailed description. However, it is to be understood that the detailed description and specific examples are provided by way of illustration and not limitation, displaying illustrative aspects. Various changes and modifications can be made in the following description without departing from the spirit of the invention.

1A is an illustration of an example of a communication system implementing an encoder assisted frame rate up-conversion (EA-FRUC) system using various motion models in accordance with an aspect for delivery of streaming video.

1B is an illustration of an example of an EA-FRUC device configured to use various motion models in accordance with an aspect for delivery of streaming video.

2 is a flow diagram illustrating the operation of the EA-FRUC system of FIG. 1A configured to use various models.

3 is a flowchart illustrating encoded video data for upsampling using object-based modeling information and decoder information.

4 is a flow diagram illustrating determining modeling information for objects in a video frame in accordance with an aspect of the present invention.

5 is a flow diagram illustrating determining motion vector erosion information for objects in a video frame using affine models.

6 illustrates an upsampled encoded video data bitstream using object-based modeling information and decoder information using a decoding device configured to decode motion models within a moving motion model infrastructure in accordance with certain aspects of the present invention. This is a flowchart illustrating decoding.

In one aspect of an encoder assisted-FRUC (EA-FRUC) system as disclosed herein, the encoder knows in advance the FRUC algorithm used in the decoder as well as accesses the source frames. The encoder is further configured to use various motion models, including moving motion models, to accurately model moving objects within the source frames. The encoder using the interpolated frame generated therewith sends additional information to assist the decoder when performing FRUC and to improve the decisions made during interpolation. Using the knowledge that FRUC will be performed within the decoder, the EA-FRUC system improves compression efficiency (and thus improves the use of transmission bandwidth), and reconstructed video quality (including the representation of reconstructed moving objects). We use various motion models, namely video coding and pre-processing operations in the video encoder to improve the performance. In particular, various motion model information from the encoder, such as affine motion modeling, is generally transmitted by the encoder to supplement or replace the information provided to the decoder so that the motion modeling information can be used within the encoder assisted FRUC.

In one aspect, the information provided by the encoder is not only the spatial (eg, refinements, model decisions, neighbor features) and time (eg, motion vector (s) determinations) features of the image to be interpolated at the decoder. Parameters such as normal information (B or P) frame coding generated by the FRUC process and different information with respect to interpolated frames. The information provided by the encoder further includes various motion models selected to accurately and efficiently represent the moving objects from the original video stream.

Some motion prediction techniques can be used for video compression in addition to moving motion. Additional motion types include rotational motion; Zoom-in and zoom-out motion; Deformations where changes in structures and shape violations of scene objects occur under the assumption of a rigid body; Affine motion; Global motion; And object-based motion. Affine motion models support a number of motion types including moving motion, rotational motion, cropping, translation, transformations and object scaling for zoom-in and zoom-out scenarios. The affine motion model is more versatile than the moving model because it incorporates different motion times. The affine motion model uses six parameters per encoded block, taking into account rotation, scaling and truncation. Thus allowing higher adaptability to the actual dynamic motion of the objects in the scene.

Object-based motion prediction techniques are used for video frames for a scene that includes multiple objects that experience different motion types. In the above reimbursements, no single motion model can capture different dynamics, but instead the size of the models can be used, in which case separate models are explicitly made for each object in the scene.

Certain aspects of the encoder device discussed herein evaluate properties of a decoder device to be used to decode data encoded by the encoding device, and to improve compression efficiency, performance, object rendering at the decoder device when interpolating frames. Optimize the encoding of the data. For example, the decoder device may improve FRUC or error concealment. In one aspect, video frames are divided into a set of regions of non-uniform size and non-uniform shape based on operations, time varying dynamics or uniquely identifiable objects. According to certain aspects, the encoder device analyzes the video data (in segments of varying duration) to determine the location of global motion. Once the position of the global motion is determined, the relevant model parameters and signals are estimated using various motion models, such as affine motion models. An affine motion model is then created that describes the moving, rotating, scaling, and morphologically changing transforms for each object or partitions. The partitioning information can be used with the associated models to generate a predictive signal that can reduce the power of the residual signal. The partition map along with the associated model contains type and parameter information and is sent to the decoder device. The remaining signals can be individually compressed and sent to the decoder device to enable higher quality reconstruction. In certain aspects, the decoder device may analyze the encoded data using the information in the encoded motion model within the modified moving motion model framework.

Certain aspects describe a process for identifying objects in which encoding significantly reduces the amount of information needed to render the objects at the decoder device. In some of the above aspects, one background object and any number of foreground objects are identified using image segmentation, graph based techniques or scene configuration information. The background object is then classified. If object-based scene analysis, including the two steps described above, is performed on a detailed or complete video sequence of the video sequence, then the evolution of each object and its dynamic behavior are accurately determined by the appropriate motion-deformation model. Can be explained. For example, for an object that experiences uniform moving motion, the overall trajectory can simply be described by a motion vector (normalized with respect to the nominal interframe spacing). The information can be used in conjunction with the visual data of a single snapshot of the object to accurately render the object at the decoder device until the object moves out of the scene or some of its motion or visual characteristics change. Changes in one of the motion or visual characteristics of the object may be used to identify a minimal non-uniform visual sampling pattern for the object. In a similar manner, potentially complex motion trajectories and occlusion properties can be determined for previously identified objects in the scene.

In the following description, specific details are given to provide a thorough understanding of aspects of the present invention. However, it will be appreciated by those skilled in the art that the features may be practiced without the specific details. For example, electronic components may be shown in block diagrams in order not to obscure aspects in unnecessary description. In other instances, the devices, other structures and techniques may be shown in detail to further illustrate the aspects.

Aspects of the invention may be described as a process depicted in a flow diagram, flow diagram, structure diagram or block diagram. Although the flowchart describes the operations as a sequential process, many of the operations may be performed simultaneously or sequentially, and the process may be repeated. In addition, the order of the operations may be rearranged. The process ends when the operations end. Processes may correspond to methods, functions, procedures, subroutines, subprograms, and so forth. When a process corresponds to one function, the end corresponds to the return of the function or the main function of the function.

1A is an illustration of an example of a communication system implementing an encoder assisted frame rate up-conversion (EA-FRUC) system using various motion models in accordance with an aspect for delivery of streaming video. System 100 includes encoder device 105 and decoder device 110.

Encoder device 105 includes frame generator 115, modeler 120, divider 160, multimedia encoder 125, memory component 130, processor 135, and receiver / transmitter 140. The processor 135 generally controls the overall operation of the example encoder device 105.

The divider component 160 divides the video frames into different blocks to allow motion models to be associated with subset regions of the video frame. Analysis of motion-deformation information can be used to segment the initial scene / frame successfully, and the minimum time sampling of the frames that need to be compressed and transmitted in contrast to the frames that can be successfully interpolated based on the data of the transmitted frames. Can be used to determine In certain aspects, the (minimum) number of sampling instances is based on when the motion-deformation dynamics experience changes. Appropriate frame interpolation can thus be performed based on the proper partitioning of motion-modified dynamics.

The modeler component 120 is configured to determine motion models and associate them with objects found within video frames containing the scene.

Frame generator component 115 generates interpolated frames that use data from the original video stream as well as information at the decoder that will be used to decode the data sent by encoder device 105. Systems and methods for generating interpolated frames are disclosed in US Patent Publication 2006/0165176 entitled "Methods and Apparatus for Encoder Coordinated-Frame Rate Upconversion (EA-FRUC) for Video Compression". Incorporated by reference.

Multimedia encoder 125 includes a converter / quantizer component that transforms and / or quantizes video (or audio or private caption text) data from the spatial domain to another domain, such as the frequency domain in the case of discrete cosine transform (DCT). It may include subcomponents. The multimedia encoder can include an entropy encoder component. The entropy encoder component can use context-adaptive variable length coding (CAVLC). The encoded data may use quantized data, transformed data, compressed data, or any combination thereof. The memory component 130 is used to store information such as raw video data to be encoded, encoded video data to be transmitted, header information, header directory, or intermediate data operated by various encoder components.

In the above example, receiver / transmitter component 140 includes circuitry and / or logic used to receive data to be encoded from an external source 145. External source 145 can be, for example, external memory, the Internet, live video and / or audio feeds, and receiving data can include wired and / or wireless communications. Transmitter 140 also includes circuitry and / or logic, such as a transmitter for transmitting (Tx) the encoded data over network 150. Network 150 may be part of a wired system such as a telephone, cable and fiber optic or wireless system. In the case of wireless communication systems, network 150 may comprise part of a code division multiple access (CDMA or CDMA 2000) communication system, or optionally the system may be a frequency division multiple access (FDMA) system, an orthogonal frequency division multiple access Time division multiple access (TDMA) systems, such as (OFDMA) systems, GSM / GPRS (Universal Packet Radio Service) / EDGE (Enhanced Data GSM Environment), or TETRA (Terrestrial Relay Lines) mobile phone technology for the service industry, Wideband code division multiple access (WCDMA), high data rate (1xEV-DO or 1xEV-DO Gold multicast) systems, or any wireless communication system using a combination of the above techniques. The transmitted data may comprise a number of bit streams, such as video, audio and / or private captions.

One or more elements of the encoder device 105 shown in FIG. 1 may be omitted, rearranged, and / or combined. For example, processor component 135 may be external to encoder device 105.

Decoder device 110 includes components similar to encoder device 105, including multimedia decoder 165, memory component 170, receiver 175, and processor 180. Decoder device 110 receives encoded multimedia data transmitted via network 150 or from external storage 185. Receiver 175 includes logic for receiving encoded data from external storage 185 as well as circuitry and / or logic used to receive (Rx) encoded data in conjunction with network 150. External storage 185 may be, for example, external RAM or ROM or a remote server.

The multimedia decoder 165 includes circuitry and / or logic used to decode the received encoded multimedia bitstreams. Subcomponents of the multimedia decoder 165 may include an inverse quantization component, an inverse transform component, and various error recovery components. Error recovery components replace and / or replace data that cannot be corrected by lower level error detection and correction components (Reed-Solomon coding and / or turbo-coding) as well as lower layer methods. Or higher layer error recovery and / or error hiding used for hiding.

Decoded multimedia data may be displayed using display component 190, stored in external storage 185, or stored in internal memory component 170. Display component 190 may be an integrated part of decoder device 110. Display component 190 includes portions such as video and / or audio display hardware and logic, including a display screen and / or speakers. Display component 190 may also be external peripheral devices. In this example, receiver 175 includes logic used to communicate decoded multimedia data to external storage component 185 or display component 190.

It should be noted that one or more elements of decoder device 110 shown in FIG. 1 may be omitted, rearranged, and / or combined. For example, the processor 180 may be an external device of the decoder device 110.

1B is an illustration of an example of an EA-FRUC device 155 configured to use various motion models in accordance with an aspect for delivery of streaming video. The EA-FRUC device 100 configured to use various motion models includes a module 161 for dividing the first and second video streams, a module 121 for determining modeling information, a module for generating an interpolation frame ( 116 and a module 126 for generating encoding information.

In one aspect, means for dividing at least one of the first and second video frames into a plurality of partitions includes a module 161 for dividing the first and second video frames. In one aspect, the means for determining modeling information for at least one object in at least one of the plurality of partitions includes a module 121 for determining modeling information. In one aspect, means for generating an interpolation frame based on the modeling information includes a module 116 for generating the interpolation frame. In one aspect, means for generating encoding information based on the boy frame includes a module 126 for generating encoding information.

2 is a flow diagram illustrating operation of the EA-FRUC system of FIG. 1A configured to use various motion models. First, in step 201, video data is encoded for upsampling using object based modeling information and information on decoder device 110 as described in detail with reference to FIG. Next, in step 202, the encoded information is sent to the decoder device 110. In certain aspects, the encoded information is sent from the transmitter module 140 of the encoder device 105 to the receiver 175 of the decoder device 110. Upon receiving the encoded information, in step 203 the process ends when the decoder device 110 regenerates the compressed version of the original video data using the encoded object-based modeling information to decode the encoded information. . Step 203 will be further described with reference to FIG. 6.

3 is a flow diagram illustrating encoding video data for upsampling using object based modeling information and decoder information. First, in step 301, modeling information is determined for the objects in the video frame as further described with reference to FIG. Next, at step 302, the information in the decoding system to be used to decode the encoded video data is further used to upsample the encoded video. Finally, in step 303, the encoded video bitstream is generated as discussed in US Patent Publication 2006/0002465 entitled "Methods and Apparatus for Using Frame Rate Upconversion Techniques in Scalable Video Coding", The patent is incorporated herein by reference.

4 is a flow diagram illustrating determining modeling information for objects in a video frame in accordance with an aspect of the present invention. In the aspect described, moving objects are identified using specific techniques herein related to recognizing objects experiencing certain motions and deformations. In other aspects, objects may be identified by uniformly applying motion-compensated prediction and transform coding based on a hybrid video compression scheme to each video frame as is known in the art. Also in the aspects discussed, affine models commonly referred to as object-based affine models or local GMC are used to cover a portion of the video frame. In this case, the encoder device 105 performs object segmentation to determine the position of the objects in motion, and then updates the affine model estimate using the affine model itself and the object descriptor. For example, the binary bitmap may indicate the boundaries of the described object within the video frame. In aspects in which the affine model covers the entire video frame, global motion compensation (GMC) is used. For GMC cases, the six parameters used in the affine model motion are used to describe the motion of the frame and are sent to the decoder device 110 without any other motion information included in the bit stream. In still other aspects, affine models and other motion models may be used.

First, in step 401, the video frame is divided into blocks. In certain aspects, the blocks have a fixed size and shape. In other aspects, the frame is non-uniform in size and / or non-uniform based on one or a combination of prominent motion-deformation behaviors, time varying dynamics in regions, and factors including uniquely identifiable objects. It can be divided into blocks of the form.

Next, in step 402, one background object is identified and zero or more foreground objects are identified. In certain aspects, identification may be performed using image segmentation. Image segmentation may be combined with thresholding in addition to pixel domain characteristics such as brightness and color values, as well as with region-based methods to determine specific statistics of the characteristics such as mean, variance, standard deviation, minimum-maximum, median and Analyzing other statistics. In other aspects, the identification can be performed using Markov random fields or Fractals. In other aspects, the identification is performed using edge / contour detection including Watershed transform into gradient images and shape models. In further aspects, identification may be performed using connection-maintaining mitigation-based segmentation methods generally referred to as an active contour model. In other aspects, the identification may be performed using time information such as motion fields. In certain aspects, image segmentation may occur using a combination of some or all of the image segmentation approaches described above within a single basic structure.

In certain aspects, objects may be identified using graph-based techniques such as local and global, semantic and statistical (strength / texture) group rules (CUES). In further aspects, the identification of the above-described objects may be performed using scene composition information available from the creation-tool. In certain aspects, the background object and any foreground object may be identified using a combination of some or all of the aforementioned identification approaches within a single infrastructure.

Then, in step 403, the background object is classified. In certain aspects, the background object may be classified as a still image where one transmission of the background object satisfies future frame interpolation and / or decoding / reconstruction operations at decoder device 110. In other aspects, the background object is classified as a still (almost stationary) image that experiences stagnant motion such as pan, scroll, rotate, zoom in or zoom out motion. In such a case, the encoder device 105 properly selects to send specific sample states of the background image in combination with the description of the global motion model. The transmission may be satisfied for frame interpolation and / or decoding / reconstruction tasks at the decoder device 110. In further aspects, the classification of the background object may not belong to one of the two classes described above, in which case the potentially high density time sampling of the states of the background image is a successful frame interpolation at the decoder device 110. And / or by encoder device 105 to support decoding / reconstruction.

Next, in step 404, motion vector information for the objects identified from the video data is processed. Mossen vector information may be processed using the systems and methods disclosed in US Patent Publication 2006/0018382 entitled “Methods and Apparatus for Motion Vector Processing” and incorporated herein by reference. In step 405, the estimated affine models are associated with the moving objects. The affine model can be estimated based on the minimal reduction in performing the discrete planar motion vector field approximation. Each affine model associated with each identified moving object is specified in step 406 using motion vector erosion information, as described below with reference to FIG. 5, and then using motion based object segmentation. It is further specified at 407. The further specifications are used to update each individual affine model in step 408, and the process eventually ends when an object descriptor is created for the affine models in step 409.

FIG. 5 is a flow diagram illustrating determining motion vector erosion information for objects in a video frame using api models. First, at step 501, encoder device 105 determines an affine model for associating with a moving object. The encoder device 105 then proceeds to step 502 with the first macro block of the object map for the video frame, and at step 503, for each macro block of the object map, the encoder device 105 determines the macro in decision step 504. The block determines if the affine model determined from step 501 matches. If the macro block does not match the affine model, the affine model based object map is updated using the matching macro block in step 505. The encoder device 105 advances to the next macro block in step 506 by returning to step 503. However, if the macro block does not match the affine model, then the decoder device immediately proceeds to the next macro block in step 506 by returning to step 503. Otherwise, the process terminates.

While block-based motion compensation using a moving model is widely deployed in decoder devices (software or hardware aspects of devices), for EA-FRUC to use different motion models to be implemented within decoder devices, Motion information from the encoder device 105 is described within a moving block based motion vector basic structure. In certain aspects, the process of describing different motion models in the moving block-based motion framework of decoder device 110 may generate block motion vectors of smaller block sizes in order to generate motion vectors for larger block sizes. Can be run recursively.

When using the information in the motion model encoded in the video bit stream, decoder device 110 uses the portion of the plurality of pixels used to display the object in the original video to select motion vectors for the selected moving objects. Create In certain aspects, the selected pixels may be evenly distributed within the block. In other aspects, the pixels may be randomly selected from the block.

In certain aspects, multiple motion vectors of the blocks are then integrated to produce a single motion vector representing the block, which motion vector may further be subjected to subsequent processing such as vector planarization as described above. In other aspects, the selected pixel or object motion vector can be used as a seed motion vector for the motion estimation module to generate a motion vector indicative of the block of interest.

6 illustrates an upsampled encoded video data bitstream using object-based modeling information and decoder information using a decoder device configured to decode motion models within a moving motion model framework in accordance with certain aspects of the present invention. This is a flowchart illustrating decoding.

In step 601, the decoder device 110 receives encoded information for a video bitstream that includes two reference frames. Next, at decision step 602, the decoder device 110 determines whether the bit stream includes an encoder enhanced interpolation frame. If an encoder enhanced interpolation frame is included, in step 603 the decoder device adds an encoder enhancement associated with various motion models in addition to the reference frame to generate a video frame that is temporarily co-terminal with the interpolated frame. Use an interpolation frame containing the compiled information. In other words, the decoder device uses its associated reference frame together with the encoder enhanced interpolation frame to produce a video frame that replaces the interpolation frame. However, if at step 602 the decoder device 110 determines that the encoder enhanced interpolated frame information is not included in the bit stream, then at step 604 the decoder device 110 generates a bidirectional frame (B-frame). We will use the frame of reference to create it.

Those skilled in the art will appreciate that information and signals may be represented using any of a number of different technologies and techniques. For example, data, instructions, instructions, information, signals, bits, symbols, and chips that may be referenced throughout the description may include voltages, currents, electromagnetic waves, electromagnetic fields, or electromagnetic particles, By optical systems or optical particles, or any combination thereof.

Those skilled in the art will also recognize that the logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or a combination thereof. To clearly illustrate the interchangeability of the hardware and software, various elements, blocks, modules, circuits, and steps have been described above with regard to their functionality. Whether the functionality is implemented in hardware or software is determined by the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.

The various logical blocks, modules, and circuits described in connection with the embodiments disclosed herein may be general purpose processors, digital signal processors (DSPs), application integrated circuits (ASICs), field programmable gate arrays (FPGAs), or It may be executed or performed using other programmable logic devices, discrete gate or transistor logic, discrete hardware elements, or any combination thereof designed to perform the functions disclosed herein. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, eg, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other configuration.

The steps of a method or algorithm described in connection with the embodiments disclosed herein may be immediately implemented in hardware, in a software module executed by a processor, or in a combination thereof. Software modules are known to those skilled in the art in the form of RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM or any other storage medium. Exemplary storage media are connected to a processor capable of reading information from and recording information from the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium may reside within an ASIC. The ASIC can reside in a user terminal. In the alternative, the processor and the storage medium may reside as discrete components in a user device.

The foregoing description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without the use of the present invention. Accordingly, the invention is not limited to the described embodiments but is to be accorded the widest scope indicated in the principles and novel features disclosed herein.

Claims (20)

As a multimedia data processing method, Dividing at least one of the first and second video frames into a plurality of partitions; Determining modeling information for at least one object in at least one of the partitions, the modeling information associated with the first and second video frames; Generating an interpolation frame based on the modeling information; And Generating encoding information based on the interpolated frame, wherein the encoding information is used to generate a video frame temporarily co-located with the interpolated frame. The method of claim 1, wherein determining model information for at least one object in one of the partitions comprises: Determining block based motion field estimation; Identifying at least one object based on the block based motion field estimation; And Determining an affine model for the at least one object. The method of claim 1, Using color features to identify boundaries of the at least one object. The method of claim 1, Using texture features to identify boundaries of the at least one object. The method of claim 1, Using pixel region features to identify boundaries of the at least one object. The method of claim 1, Determining motion vector erosion information associated with one of the partitions, wherein the transmitted encoding information includes the motion vector erosion information. The method of claim 1, Wherein said modeling information comprises an affine model. The method of claim 7, wherein Wherein said affine model comprises at least one of translation, rotation, shearing, and scaling motion. The method of claim 1, The modeling information comprising a global motion model. A multimedia data processing device, Means for partitioning at least one of the first and second video frames into a plurality of partitions; Means for determining modeling information for at least one object in at least one of the plurality of partitions, the modeling information associated with the first and second video frames; Means for generating an interpolation frame based on the modeling information; And Means for generating encoding information based on the interpolated frame, wherein the encoding information is used to generate a video frame temporarily co-located with the interpolated frame. The method of claim 10, wherein the determining means, Means for determining block based motion field estimation; Means for identifying at least one object based on the block based motion field estimation; And Means for determining an affine model for the at least one object. The method of claim 10, And means for using color features to identify boundaries of the at least one object. The method of claim 10, And means for using texture features to identify boundaries of the at least one object. The method of claim 10, And means for using pixel region features to identify boundaries of the at least one object. The method of claim 10, Means for determining motion vector erosion information associated with one of the partitions, wherein the transmitted encoding information includes the motion vector erosion information. The method of claim 10, And the modeling information comprises an affine model. The method of claim 16, Wherein said affine model comprises at least one of translation, rotation, shearing, and scaling motion. A multimedia data processing device, A partitioning module configured to partition at least one of the first and second video frames into a plurality of partitions; A modeling module configured to determine modeling information for at least one object in at least one of the plurality of partitions, the modeling information associated with the first and second video frames; A frame generation module configured to generate an interpolation frame based on the modeling information; An encoding module configured to generate encoding information based on the interpolated frame; And And a transmitting module configured to transmit the encoding information to a decoder. A machine readable medium comprising instructions for processing multimedia data, the instructions causing a machine to perform the following operations when executed: Split at least one of the first and second video frames into a plurality of partitions; Determine modeling information for at least one object in at least one of the partitions, the modeling information associated with the first and second video frames; Generate an interpolation frame based on the modeling information; And And generate encoding information based on the interpolated frame, wherein the encoding information is used to generate a video frame temporarily co-located with the interpolated frame. A processor for processing multimedia data, Split at least one of the first and second video frames into a plurality of partitions; Determine modeling information for at least one object in at least one of the partitions, the modeling information associated with the first and second video frames; Generate an interpolation frame based on the modeling information; And And generate encoding information based on the interpolated frame, wherein the encoding information is used to generate a video frame temporarily co-located with the interpolated frame.

Images