JP2005504490A - Video coding based on visemes - Google Patents

Abstract

A video processing system for processing a stream of frames of video data. A system for determining whether a frame of input video data corresponds to at least one predetermined viseme; a viseme library storing a frame corresponding to at least one predetermined viseme; A packaging system including an encoder for encoding each frame corresponding to at least one predetermined viseme, the encoder pre-stored in the viseme library to encode the current frame Frame is used. Also included is a receiver system that includes a decoder that decodes the encoded frame of video data and a frame reference library that stores the decoded frame, the decoder being current encoded. Provided by the receiver system using a previously decoded frame from the frame reference library to decode the previously decoded frame, and the previously decoded frame belongs to the same viseme as the current encoded frame Is done.

Description

[0001]
The present invention relates to video encoding and decoding, and more particularly to systems and methods based on visemes that encode video frames.
[0002]
As the need for remote video processing applications (e.g., video conferencing, video telephony) is increasing, it has become important to provide a system that can efficiently transmit video data over a limited bandwidth. One solution for reducing bandwidth consumption is to use a video processing system that can encode and decode the compressed video signal.
[0003]
There are currently two techniques for performing video compression: waveform-based compression and model-based compression. Waveform-based compression uses, for example, relatively mature techniques provided by MPEG and ITU standards (eg, MPEG-2, MPEG-4, H.263, etc.). Alternatively, model-based compression is a relatively unmature technology. A typical approach used for model-based compression involves generating a three-dimensional model of a human face and then deriving a two-dimensional image that underlies a new frame of video data. If most of the transmitted video image data is repetitive, as in the case of head and shoulder images, model-based coding can achieve a much higher degree of compression.
[0004]
Thus, compression techniques based on current models are useful for applications such as videoconferencing and videophones, and the computational complexity of generating and processing three-dimensional images makes such systems difficult and prohibitive. Tends to be expensive. Therefore, there is a need for an encoding system that can achieve the compression level of a system based on a model without requiring computational overhead for processing 3D images.
[0005]
The present invention addresses not only the above-mentioned problems but also other problems by providing a novel model-based coding system. In particular, the input video frame is thinned out so that only a part of the entire frame is actually encoded. The frames to be encoded are encoded using predictions from previously encoded frames and / or from frames generated from a dynamically generated viseme library.
[0006]
In a first aspect, the present invention is a video processing system for processing a stream of frames of video data, the viseme determining whether an input frame of video data corresponds to at least one predetermined viseme. A packaging system including a weaving system, a viseme library that stores frames corresponding to at least one predetermined viseme, and an encoder that encodes each frame corresponding to at least one predetermined viseme. The encoder provides a video processing system that uses pre-stored frames in the viseme library to encode the current frame.
[0007]
In a second aspect, the invention is a method of processing a stream of frames of video data, determining whether each frame of input video data corresponds to at least one predetermined viseme; Storing a frame corresponding to at least one predetermined viseme in the viseme library; and encoding each frame corresponding to the at least one predetermined viseme; A method is provided for using a frame previously stored in a viseme library to encode a current frame.
[0008]
In a third aspect, the present invention is a program product stored on a recordable medium that, when executed, processes a stream of frames of video data, wherein the input video data frames are at least A system for determining whether or not one predetermined viseme corresponds, a viseme library storing frames corresponding to at least one predetermined viseme, and encoding each frame corresponding to at least one predetermined viseme The encoding system provides a program product that uses previously stored frames in the viseme library to encode the current frame.
[0009]
In a fourth aspect, the present invention is a decoder for decoding an encoded frame of video data encoded using a frame associated with at least one predetermined viseme, Has a frame reference library for storing the frames, and the decoder uses previously stored frames in the frame reference library to decode the current encoded frame, and the previously stored frames are A decoder is provided having a morphing system that reconstructs frames of video data that belong to the same viseme as the current encoded frame and that were removed during the encoding process.
[0010]
Hereinafter, exemplary embodiments of the present invention will be described with reference to the accompanying drawings. In the drawings, like reference numerals indicate like elements. Referring to the drawings, FIGS. 1 and 2 show a video processing system for encoding video images. While the embodiments described herein focus primarily on applications relating to facial image processing, it should be understood that the present invention is not limited to facial image coding. FIG. 1 includes an encoder 14 that generates encoded video data 50 from input frames of video data 32 and audio data 33. FIG. 2 illustrates a video receiver system 40 that includes a decoder 42 that decodes the video data 50 encoded by the video packaging system 10 of FIG. 1 and generates decoded video data 52.
[0011]
The video packaging system 10 of FIG. 1 processes an input frame of video data 32 using a viseme identification system 12, an encoder 14, and a viseme library 16. In a typical application, an input frame of video data 32 may include multiple images of a human face, for example, typically processed by a video conference system. The input face 32 is examined by the viseme identification system 12 to determine which frames correspond to one or more predetermined visemes. A viseme can be defined as a generic facial image that can be used to describe a particular sound (eg, to create the shape of the mouth needed to speak “sh”). A viseme is visually equivalent to a phoneme or a unit of sound in a spoken word.
[0012]
The process of determining which image corresponds to a viseme is performed by the audio segmentation unit 18 that identifies a phoneme in the audio data 33. Each time a phoneme is identified, the corresponding video image can be tagged as belonging to the corresponding viseme. For example, each time a phoneme “sh” is detected in the audio data, the corresponding video frame can be identified as belonging to the “sh” viseme. The process of tagging video frames is handled by the mapping system 20 that maps identified phonemes to visemes. Note that explicit identification of a given posture or facial expression is not necessary. Rather, video frames belonging to known visemes are implicitly identified and classified using phonemes. It should be understood that any number or type of visemes can be generated, including silent visemes, that have no speech corresponding to a certain period of time (eg, 1 second).
[0013]
If the frame is identified as belonging to a viseme, the frame is stored in the viseme library 16. The viseme library 16 is based on visemes so that frames tagged as belonging to a common viseme are stored together in one of a plurality of model sets (eg, V1, V2, V3, V4). It can be physically or logically arranged. Initially, each model set has a null set of frames. Each model set grows as more frames are processed. A threshold may be set for a given model set size to prevent model sets that are too large. A first-in first-out system that discards frames can be used to remove excess frames after threshold criteria are met.
[0014]
If the input frame does not correspond to a viseme, the frame thinning system 22 thins or erases the frame, that is, transmits it to the trash can 34. In this case, the frame is not stored in the viseme library 16 and is not encoded by the encoder 14. However, information regarding the position of any decimation frame may be explicitly or implicitly incorporated into the encoded video data 50. This information can be used by the receiver to determine where to reconstruct the decimated frame, as described below.
[0015]
Assuming that the input frame corresponds to a viseme, the encoder 14 encodes the frame using, for example, a block-by-block prediction method, and this is output as encoded video data 50. The encoder 14 includes an error prediction system 24, detailed motion information 25, and a frame prediction system 26. The error prediction system 24 encodes the prediction error in any known manner, for example as given in the MPEG-2 standard. Detailed motion information 25 may be generated as side information that may be used by morphing system 48 at receiver 40 (FIG. 2). The frame prediction system consists of two images: (1) a motion compensated previously encoded frame generated by the encoder 14 and (2) an image retrieved from the viseme library 16 by the retrieval system 28. And predict the frame. In particular, an image retrieved from the viseme library 16 is retrieved from a model set that includes the same viseme as the frame being encoded. For example, if the frame includes an image when a human face utters the voice “sh”, the previous image from the same viseme is selected and searched. The search system 28 searches for an image that is closest to the mean square. Thus, rather than relying on temporal proximity (ie, neighboring frames), the present invention can select the closest match for any previous frame, regardless of temporal proximity. Good for finding very similar previous frames, the prediction error is small and a very high degree of compression can easily be achieved.
[0016]
Referring now to FIG. 2, a video receiver system 40 that includes a decoder 42, a reference frame library 44, a buffer 46, and a morphing system 48 is shown. Decoder 42 decodes incoming frames of encoded video data 50 that use the same parallel technology as video packaging system 10. In particular, the encoded frame is decoded using (1) the immediately preceding decoded frame and (2) an image from the reference frame library 44. The images from the reference frame library are the same as those used to encode the frames and can be easily identified with reference data stored in the encoded frames. After being decoded, the frame is stored in reference frame library 44 (to decode future frames) and transferred to buffer 46.
[0017]
If one or more frames were originally decimated (eg, indicated as “??” in buffer 46), morphing system 48 may interpolate between encoded frames 53 and 55, for example. Can be used to reconstruct the decimation frame. Such interpolation techniques are described, for example, by Ezzat and Poggio, “Miketalk; A tapping facial display based on morphing views,” Proc. Computer Animation Conference, pages 96-102, Philadelphia, PA, USA, 1998. Morphing system 48 may use detailed motion information provided by encoder 14 (FIG. 1). After the frame is reconstructed, it can be output with the decoded frame as a complete set of encoded video data 52.
[0018]
The systems, functions, methods, and modules described herein may be implemented as hardware, software, or a combination of hardware and software. These can be realized by any kind of computer system or other device adapted to carry out the method described above. A typical combination of hardware and software may be a general purpose computer system having a computer program that, when loaded and executed, controls the computer system to perform the methods described herein. Alternatively, special purpose computers may be used that include dedicated hardware that performs one or more functional tasks of the present invention. The present invention also has all the features that enable the implementation of the methods and functions described herein and is embedded in a computer program product capable of performing these methods and functions when loaded into a computer system. Can be. In this specification, a computer program, a software program, a program, a program product, or software is specified directly or after (a) or (b) in a system having information processing capability. Means any representation in any language, code, or notation of a set of instructions intended to perform the functions.
[0019]
The following description of the preferred embodiment of the present invention has been presented for purposes of illustration and description. They are not intended to be exhaustive or to limit the invention to the precise forms disclosed, and obviously many modifications and variations are possible from the above teachings. Such modifications and variations that may be apparent to a person skilled in the art are intended to be included within the scope of this invention as defined by the accompanying claims.
[Brief description of the drawings]
[0020]
FIG. 1 illustrates a video packaging system having an encoder according to a preferred embodiment of the present invention.
FIG. 2 is a video receiver system having a decoder according to a preferred embodiment of the present invention.

Claims (14)

A video processing system for processing a stream of frames of video data,
A vise-mesh weaving system that determines whether a frame of input video data corresponds to at least one predetermined viseme;
A viseme library storing frames corresponding to the at least one predetermined viseme;
A packaging system comprising: an encoder that encodes each frame corresponding to the at least one predetermined viseme;
The encoder uses pre-stored frames in the viseme library to encode a current frame;
Video processing system. The video processing system of claim 1, wherein the viseme identification system includes an audio segmentation unit that identifies phonemes associated with frames of the video data in an audio data stream. The video processing system of claim 2, wherein the viseme identification system maps identified phonemes to the at least one predetermined viseme. The video processing system of claim 2, wherein the viseme identification system tags frames with associated phonemes. The video processing system of claim 1, further comprising a frame skipping system that removes frames that do not correspond to the at least one viseme. A decoder for decoding encoded frames of video data;
A receiver system including a frame reference library for storing decoded frames;
The decoder uses a previously decoded frame from the frame reference library to decode a current encoded frame, and the previously decoded frame is the current encoded frame. 6. A video processing system according to claim 5, which belongs to the same viseme as the frame. The video processing system of claim 6, wherein the receiver system further comprises a morphing system that reconstructs frames removed by the decimation system. 8. The video processing system of claim 7, wherein the encoder generates detailed motion information used by the morphing system to reconstruct a frame. A method of processing a stream of frames of video data,
Determining whether each frame of input video data corresponds to at least one predetermined viseme;
Storing a frame corresponding to the at least one predetermined viseme in the viseme library;
Encoding each frame corresponding to the at least one predetermined viseme,
The method of encoding, wherein a frame previously stored in the viseme library is used to encode a current frame. Decoding encoded frames of video data;
Providing a frame reference library for storing the decoded frames;
The decoding step uses a previously decoded frame from the frame reference library to decode a current encoded frame, and the previously decoded frame is the current encoding frame. The method according to claim 9, wherein the method belongs to the same viseme as the generated frame. A program product stored on a recordable medium that, when executed, processes a stream of frames of video data,
A system for determining whether a frame of input video data corresponds to at least one predetermined viseme;
A viseme library storing frames corresponding to the at least one predetermined viseme;
A system for encoding each frame corresponding to the at least one predetermined viseme,
A program product, wherein the encoding system uses previously stored frames in the viseme library to encode a current frame. The program product of claim 11, wherein the determining system includes an audio segmentation unit that identifies phonemes in an audio data stream associated with the frame of video data. The program product of claim 11, wherein the determining system maps the identified phonemes to the at least one predetermined viseme. A decoder for decoding an encoded frame of video data encoded using a frame associated with at least one predetermined viseme,
A frame reference library for storing decoded frames, wherein the decoder uses previously stored frames in the frame reference library to decode a current encoded frame, and the previous The frame stored in belongs to the same viseme as the current encoded frame,
A decoder having a morphing system for reconstructing frames of video data removed during the encoding process.

Images