CA1323093C - Method and apparatus employing adaptive filtering for efficiently communicating image sequences - Google Patents

Abstract

ABSTRACT OF THE DISCLOSURE

A method and apparatus for encoding and transmitting and/or decoding and receiving a sequence of image frames employ selectively a measure of motion displacement for generating a motion compensated image and filter selection data. The motion compensated image, together with an error signal, provides the receiver with full detail regarding the next frame of a sequence. In some instances, however, the motion displacement data is inaccurate for portions of the next image frame. In those instances, the method and apparatus of the transmitter determine whether filtering of a portion of the motion compensated image provides a lower error energy for the resulting prediction of the image. If a lower error energy can be achieved by low-pass filtering, the data regarding the additional processing of the motion compensated reconstructed image is transmitted to the receiver. In those instances, at the receiver, the method and apparatus determine from the filter selection data that filtering of a portion of a previously received image provides a lower error energy for the resulting prediction of the image. If a lower error energy is not achieved by low-pass filtering, the data from the communications path regarding the motion displacement enables the receiver to reconstruct at least a block portion of the image frame.

Description

A MET~OD AND APPARATUS EMPLOYING ADAPTIVE FILTERING
FOR EFFICIENTLY COMMUNICATING IMAGE SEQUENCES

Backqround of the Invention The invention relates generally to data communication and signal processing method~ and apparatus, and in particular to a method and appara~us for reliably and efficiently communicating sequences of image data over a communications channel, for example, a 56 kilobit per second telephone communications channel.
The transmission of sequences of images, and in particular ~ 10 se~uences of naturally occurring images such as those represented ; by a television signal, has been the subject of a significant amount of investigation. Typically, investigators have relied upon the highly redundant nature of successive images in the sequence and have often modeled the image data as a Markov process with a correlation coefficient close to unity. The three-dimensional Markov model provides a motivation for utilizing l differential pulse-code-modulation (DPCM) and transform coding `.t, techniques to take account of the interframe redundancy.
8y analyzinq the nature of typical moving video, it is easy to become convinced that the principal change occurring between successive frames is the inhomogeneous motion of the objects within the frame. It has also been recognized that an accurate `' apparatus and method of estimating and compensating for this spatially dependent motion enables the construction of an interframe data compression method and apparatus which can have , . ~ .
:. ' ~ ',: . ' .

1 3~3093 substantially better performance than can be achieved by sending a signal representative merely of the difference between successive frames.
As a result, various motion compensating coding methods and apparatus have been developed. These systems typically are either receiver-based motion compensation systems or transmitter-based motion compensation systems. In the receiver-based motion compensation system, the receiver makes a prediction as to the motion and compensates the previous frame for the expected motion.
The transmitter, operating in the same manner, then sends only an error signal describing what must be done at the receiver in order to correct the receiver predicted frame. The error signal is typically coded in order to reduce its bandwidth.
For a transmitter-based motion compensation system, the motion estimation process occurs only at the transmitter.
Displacement vectors are generally determined over various re~ions of the image and this data is then transmitted to the receiver along with an error information data signal. At the receiver the compensation process i~ performed on the previously coded image first using the motion information provided by the transmitter.
The error signal data provided by the transmitter is then added to the thus compensated receiver image in order to maintain picture quality.

: .

.. -,, . ~ . .. .

1 3230~3 l A particularly advantageous and efficient transmitter-based motion compensation system is described in U.S. patent numbers 4,727,422 issued February 23, 1988, 4,703,350 issued October 27, 1987 and 4,661,849 issued April 28, 1987, all in the name of srian L. Hinman, assigned to the assignee of this application. The transmitter-based motion compensation system described there, while performing exceedingly well, encodes the motion compensation signals using a lossy compression encoder and hence can introduce artifacts which are not in the original image. Accordingly, the resulting receiver reconstructed image can be degraded somewhat.
The motion compensation process can provide a bad prediction of the new image for several other reasons. For example, the new image frame may not be able to be described by data which details displacements of the previous image;
or there may be newly exposed areas of the image, a deformation of the image objects of the old image, illumination changes, or reflections in the image objects.
In addition, the error estimation procedure may fail to track the motion in the old image; the motion estimate may be applied to a region with non-uniform motion, that is, rotation; and coarse quantization can introduce noise into the previous frame that has been used for the prediction.
Thus, if the motion compensated data predictions are used in areas of the image where it performs poorly, artifacts will appear in the receiver reconstructed image, , . ` - , -. .

.- . . ~. - . . .

. ,, . ~ ~ .

-1 especially at high compression rates. Image detail from the previous image will generate noise in the new image. It is to this proble~, wherein a prediction performs poorly, that the present invention is directed.
S It is therefore an object of the present invention to transmit sequences of images over a communications channel using relatively low bandwidth, and providing high reliability and fidelity. Other objects of the invention are a motion compensation method and apparatus which reliably provide either an accurate estimate of the displacement of the pixels of a scanned image in a sequence or an alternative compensation function, and an improved motion estimation method and apparatus which enable real-time, reliable, and accurate determination of each new image in an image transmission device.
Summary of the Invention The invention relates to a method and apparatus for encoding and transmitting a sequence of image frames.
The method features estimating, for successive images of the sequence, and for each of a plurality of multi-pixel blocks of the image, a measure of image difference for each block between successive images; generating, using the measures of image difference, a difference compensated image; selecting, for each block of an image frame representation, one block chosen from between corresponding blocks of, respectively, the difference compensated image and a low-pass filter representation of the difference compensated image, and 1 323~93 transmitting filter selection data over a communications channel to enable a receiver to determine for a block whether to employ low-pass filtering in a block in generating a prediction image from a receiver difference compensated image, the filter selection data being dependent upon the selecting step at the transmitter.
In another aspect, the invention relates to a method and apparatus for encoding and transmitting a sequence of image frames. The method features the steps of dividing the image into a plurality of block regions, generating a data coding of the high spatial frequency data content of the image in each block region;
and determining, for each block region an error value for each o a low-pass filtered image prediction of the block represented by the coded data and an image prediction from the coded data without the filtering. The method further features transmitting the coded data and a filter selection data over a communications channel to a receiver to enable the receiver to determine for each block region of a predicted receiver image whether to employ low-pass filtering in generating a particular block.
In yet another aspect, the method relates to estimating, for successive images of the sequence, and for each of a plurality of multi-pixel, non-overlapping blocks of the image, a measure of the motion displacement of each block from one image frame to a next image frame. The method further features generating, using the measures of motion displacement, a motion compensated image;
selecting, for each block of an image frame representation, one - . . . . . . . :

- - ,. ... : - ..

. ~ . . - . - , : . . .:

1 32~0~3 block chosen from between corresponding blocks of, respectively, the motion compensated image and a low-pass filter representation of the motion compensated image; and transmitting filter selection data over a communications channel to enable a receiver to determine whether to employ low-pass filtering in ~enerating a prediction image from a motion compensated receiver image, the filter selection data being dependent upon the selecting step at the transmitter.
The apparatus of the inventlon features elements for estimating, for successive images of the sequence, and for each of a plurality of multi-pixel blocks of the image, a measure of im~ge difference for each block between successive images; circuitry for generating, u~lng the measures of image difference, a difference compensated image; circuitry for selecting for each block of an image frame representation, one block chosen from between corresponding blocks of, respectively, the difference compensated image and a low-pass filtered representation of the difference compensated image; and elements for transmitting filter selection data over a communications channel to enable a receiver to determine for a block whether to employ low-pass filtering in a block in generating a prediction image from a receiver difference compensated image, the filter selection data being dependent upon the selecting circuitry at the transmitter.

~ . ,, ~. ,, ` - ~

In another aspect, the apparatus of the invention relates to circuitry for dividing the input image into a plurality of block regions, and features elements for generating a data coding of the high spatial frequency information content of the image, elements for determining, for each block region, an error value for each of a low-pass filtered image prediction of the block represented by the coded data and of an image prediction using the coded data without filtering, and circuitry for transmitting the coded data and filter selection data over a communications channel to enable a receiver to determine, for each block region, whether to employ low-pass filtering in generating a receiver prediction image frd~
the received previous recon~tructed image for that block.
In a particular aspect of the invention, the apparatus relates to elements for estimating, ~or successive images of the sequence, and for each of a plurality of multi-pixel blocks of the image, a measure of motion displacement of each block from one image frame to a next image frame; and features elements for generating, using the measures of motion displacement, a motion compensated image; circuitry for selecting, for each block of an image frame, one block chosen from between corresponding blocks of, respectively, the motion compensated image and a low-pass filter representation of the motion compensated image; and elements for transmitting filter selection data over a communications channel to enable a receiver to determine whether to employ low-pass filtering in a block in generating a received .

1 3230q3 l prediction image from the receiver motion compensated image, the filter selection data being dependent upon the selecting means at the transmitter.
The invention also, in another aspect, relates to a method and apparatus for decoding and receiving an encoded sequence of image frames, preferably in connection with a motion compensation image system. The method features receiving filter selection data from a communications path for determining the data source for reconstructing a next ]0 block of an image frame at the receiver. In one uncoded mode of operation, a low-pass image is derived from a previously received image frame, while in a second coded mode of operation, image data from the communications path, typically representing displacement data, is used for reconstructing the image frame. Typically, the method employ3 in connection with the low-pass ima~e, applying a block averaging filter process at each picture element of the block of the image frame to be generated.
The receiver apparatus of the invention features circuitry for receiving the filter selection data from a communications channel for determining, for a block of the image frame being received, whether to employ a low-pass filter in connection with reconstructing the block of the image or whether to employ circuitry which derives the block being constructed in accordance with motion displacement information from the data channel.

: ,~
, ~

. .
. .

~ 32~093 1 Brief deseription o~ the Drawings Other objeets, features and advantages of the invention will appear from the following deseription of particular preferred embodiments taken together with the drawings in whlch:
Figure 1 is an eleetrical block diagram of a typieal image communications system in accordance with ~he claimed invention;
Figure 2 is an electrieal bloek diagram of the transmitter of a motion-compensated image eneoding apparatus employing adaptive filtering in aeeordanee with the invention Figure 3 is an eleetrieal bloek diagram of the receiver of a motion-eompensated image eoding system employi.ng adaptive filtering for reeeiving the ehannel slgnals from the transmitter of Figure 2;
Figure 4 is an electrieal block diagram of an alternate transmitter ultilizing adaptive filtering in a closed loop motion-compensated image coding system in aeeordance with the invention;
Figure 5 is an eleetrical block diagram of the transmitter of a motion-compensated image coding system employing adaptive filtering whieh utilizes temporal DPCM
coding of the motion data in accordance with the invention;

.

- " -1 32309~

Figure 6 is an electrical block diagram of the receiver of a motion-compensated coding system employin~ adaptive coding and utilizing temporal DPCM coding of the motion data in accordance with the invention;
Figure 7 is an electrical block diagram of a motion-compensation apparatus for a field of undersampled motion vectors;
Figure 8 is a diagrammatic representation of the spatial interpolation process; ~ :
Figure 9 is an electrical block diaqram of an iterative spatial-domain motion estimation apparatus utilizing adaptive, steepest descent error minimization;
Figure 10 is an electrical block diagram of an iterative spatial-domain motion estimation apparatus utilizing adaptive, steepest descent error minimization with an improved data processing structure:
Figure 11 is a diagrammatic presentation of the relative locations of adjacent blocks used in predicting an initial displacement motion vector value: and Figure 12 is a block diagram of an adaptive filter according to the illustrated embodiment of the invention.

Descri~tion of Particular Preferred Embodiments Referring to Figure 1, a communications system 6 has a transmitter 8 which, in accordance with a preferred embodiment of the invention, has a camera 10 for providing a video signal to an _g_ ",,,,,", " :., , ",.", "

.. i ~ ., : , :

1 3230q3 analog-to-digital converter and frame buffer 12. The frame buffer portion of the analog-to-digital converter and frame buffer 12 is capable of storing a full frame of the video, sampled to, for example, eight bits across a 256x240 pixel raster.
The entire coding and motion compensation process takes place in the digital domain. The transmitter has an error signal circuitry 14 and a motion estimation and coding circuitry 16. A
channel encoder 18 codes the outputs of the error circuitry 14 and motion estimation and coding circuitry 16 and passes the thus encoded data onto a channel 20 for transmission to a receiver 21.
The illustrated motion estimation and coding circuitry 16 o ~igure 1, in accordance with a preferred embodiment of the invention, and referring to Figure 2, compares a present oriqinal input frame image, available over line~ 22, with the previous original input image, available in this illustrated embodiment from a frame buffer 24. A motion estimator circuitry 26 generates a measure of the motion displacement between the frames input thereto, and can be any of a number of motion estimation devices as are well known in the art. In a preferred embodiment to be described hereinafter, the motion estimation device uses an adaptive steepest descent error minimization method to generate the motion displacement measures.
The output of the motion estimator 26 is a field of motion vectors which, as noted above, provide a measure of the motion displacement between input frames. This vector field provides a description of how to map a previous input frame or input image from buffer 24 into the best approximation of the present input frame or image over lines 22. By "best" is meant an error metric such as, for example, a mean-squared-error error measure.
Typically, the motion estimator uses a region matching techni~ue between non-overlapping blocks of the previous and present input images. Should motion occur for a region in the previous image, the estimator will determine which block in the present image is the best match for the block in the previous image, and the value of the displacement is the difference between a new coordinate pair for the block in the present image and the original coordinate pair for the block in the earlier image. That determination defines the motion vector to be associated with the block in the previous image.
Since scenes are generally composed of several large objects moving uniformly over time, there is a high degree of correlation in the motion vector field. To avoid transmitting redundant information, and to reduce the data bit requirements, the preferred embodiment of the invention modifies the motion vector field, thereby losing some information, to facilitate the compression of the motion representing data. In the illustrated embodiment, this operation is represented by a "lossy compressor"
28 which reduces the amount of data, and hence the bandwidth, required to represent the motion vector field. Noting the similarity between motion vector field and natural images, --11-- , :
.

.. , .. ; .. ~ ,, , . , . : . . ~ . . . . .

.

1 3230q3 1 predictive, transform, or interpolative coding of the two independent components of the vector field can be employed by the lossy compressor 28.
Thus the lossy compressor circuitry 28 is employed for coding the motion vector field available over lines 32, and provides, over lines 30, a coded motion signal representative of the motion vectors. This output of the lossy compressor, as noted above, will not, upon decodins, exactly reproduce the signals over lines 32 (which provide the measure of motion displacement) and, therefore, have some error signal associated with them. Nevertheless, the reduction in the data requirements of a lossy compressor, when compared to, for example, a PCM exact coding method, are so substantial, that the use of a lossy compressor, is a significant advance in the art. One preferred ]ossy compression circuitry employs a discrete cosine transform and the circuitry incorporates a processing method described in the applicant's co-pending Canadian application Serial No. 510,748, entitled "Method and System for Adapting a D~igitized Signal Processing System for Block Processing with Minimal Blocking Artifacts" filed on June 3, lg86. The inventor is Henrique Malvar.
The output of the lossy compressor circuitry over lines 30, as noted above, is passed to the encoder 18. In addition, those signals are employed by the error circuitry 14 for de~ermining , ' .

1 3230q3 what the receiver would have seen, absent any errors in the channel, and thereby providing the mechanism for determining the prediction error signal, that is, the signal representing the difference between what the receiver would have predicted based upon the coded motion signal representation over lines 30, and the true image input.
The output of the lossy compressor over lines 30 is used by a reconstructor circuitry 34 for producing, at its output, a signal representative of the measure of motion displacement, the motion vectors, on lines 32. The difference between the signals over lines 36, the output of the reconstruction circuitry, and the signals over lines 32, represents the coding error introduced by the lossy compression apparatus 23. The output of the reconstruction apparatus 34, over lines 36, is directed to a motion field interpolation circuitry 38 which operates in the spatial domain to associate with each picture element a motion displacement vector. Thus, while the input signals over lines 36 represent motion displacements for groups or regions of elements, for example, the picture elements of a 4x4 block, the motion field interpolator, as described in more detail below, resolves that data so that there is associated with each picture element, a motion displacement vector. The resulting output of the motion field interpolator, over lines 40, is designated the motion reconstruction signal.

- .. ..

1 3230~3 The motion reconstruction signal is applied to a motion compensation apparatus 42 which forms part of an error reconstruction loop 43. The error reconstruction loop includes a frame buffer 44, a lossy compression circuitry 46, a reconstruction circuitry 48, and a selectively operable adaptive filter 49. The input to the lossy compression circuitry 46, over lines 50, is the er~or signal which is to be transmitted to the receiver. That error signal is coded to reduce its bandwidth and the resulting signal, the error reconstruction signal over lines 52, is delivered to the channel encoder 18. The lossy compressor 46 can be any of the two-dimensional block encoders such aq a t transform or DPCM encoder, and is preferably the encoder described 5 1 O, '7 )Jr ~
in Malvar's co-pending application Serial No. ~40,~06, referred to above. The error reconstruction signal is also sent to the reconstruction apparatus 48 which provides the inverse operation of the lossy compressor 46. There results, therefore, at the output of the reconstruction apparatus 48, an error reconstruction image over lines 54. The error reconstruction image is added to the expected output of the motion compensator, (which is the estimated receiver image over lines 61) and the resulting signal, an estimated previous receiver image (the predicted receiver image for the previous frame), is stored in the frame buffer 44.
The estimated receiver image is also applied to a differencing apparatus 60 which takes the difference between what the receiver is expected to predict, the signal over lines 61, and .
, .

:, . . . :
- . . . .

-the actual image available from A-D converter and frame buffer 12.
The output of the differencing apparatus 60 is the error signal input to the lossy compressor 46 over lines 50. If desired, a "leak" constant, "a", can be used to multiply the output of the S adaptive filter 49 over lines 61a by a factor less tha~ or equal to 1. This factor, "a," is indicated at 62 and provides for gradually eliminating the effect of errors in the channel.
The output of motion compensator 42 represents the predicted receiver image as a result of reconstructing the output data from lossy compressor 28. For those blocks wherein the motion compensation data is accurate, the signal over lines 56 is applied directly, by the adaptive filter 49, to its output over lines 61a.
~or those blocks for which motion compensation is not adequate, as described in more detail below, the adaptive filter can, in effect, low-pass filter the output data over lines ~6 and provides over lines 61a, for those regions wherein filtering is advantageous, a low-pass version of the motion compensation reconstructed receiver image. In addition, the adaptive filter provides filter data, over lines 63, to the channel encoder for transmission to the receiver.
As noted above, the input to the frame buffer 44 is the estimated previous receiver image. This receiver image, which takes into account all data received by the receiver, corresponds to the reconstructed receiver image for a frame. The image output from the frame buffer over lines 64 is the image which the motion :. ~ ,, . , : . .. .
.

compensation circuitry 42 modifies in accordance with the output of the motion field interpolator 38 over lines 40.
At the receiver 21, referring to Figure 3, the data from the channel is decoded by a channel decoder circuitry 70 and the resulting receiver error reconstruction signal over lines 72 and filter data signal over lines 73, and receiver coded motion signal representation over lines 74 are delivered to reconstruction circuitry 76, motion compensator 99, and reconstruction circuitry 78 respectively. The output of the error reconstruction circuitry 76 is delivered to a recovery loop 80 in which motion compensating signals over lines 82 are added to the error image representatian over lines 84 to produce a reconstructed receiver signal over lines 86. That signal is delivered to a temporal frame interpolator 88, which can add one or more frames between the successive received frames over lines 86, for delivery to a digital-to-analog circuitry 90 and from there to a monitor 92 for viewing.
The frame interpolator 88 interpolates in the temporal domain in accordance with motion reconstruction signals received over lines 94. Those signals are generated by a motion field interpolator 96 corresponding to the motion field interpolator 38 of the Figure 2. That motion field interpolator, as noted above, provides a motion vector for each picture element of the image and hence allows the frame interpolator to accurately predict what the image would have been at any selected time between received . ~
.
;.
.

frames. The reconstructed receiver images over lines ~6 are successively stored in a frame ~uffer 98 and are delivered to a motion compensator 99 which also receives signals from the motion field interpolator 96. The motion compensator 99 includes S circuitry for adaptively filtering the input from buffer 98. The output of the motion compensator, representing the expected receiver image in the absence of an error correction, corresponds to the signal over lines 61a in the transmitter, and is delivered to the adder 100 for combination with the output of the error reconstruction circuitry over lines 84. As before, the output of the motion compensator circuitry can be modified by a "leak"
parameter, "a," whereby the effects of errors in the channel can be slowly eliminated. (The "leak" parameter, "a," for both the transmitter and receiver must be identical in value.) Referring to Figure 4, in an alternative embodiment of the invention, the motion estimation circuitry 26 can receive an input image corresponding to the previous frame, not from the frame buffer 24 which provides an open loop circuitry, but from the frame buffer 44 to provide closed loop operation. In certain aspects of operation, such closed loop operation is effective for reducing overall error. This, however, is not always true. The alternate input to the motion estimation circuitry 26 is, in the Figure 4 embodiment the predicted reconstructed, previous frame output at the receiver.

., ,~
- , . , i : .

, Referring now to Figure 5, the transmitter ean employ a temporal differential pulse code modulation of the output of the motion estimator circuitry 26. In this illustrated embodiment, the lossy compressor input is no longer received directly from the motion estimator circuitry 26 as in the Figure 2 embodiment, but instead is a differential signal corresponding to the difference between the estimated reconstructed signal for the previous frame, available over lines 110, and the present motion displacement vector signal over lines 32. These two signals are differenced at an adder 112 and that difference is passed to the lossy compressor circuitry 28, operating as described in connection with Figure ~.
The output of the reconstruction circuitry 34 is modified, in this illustrated embodiment, by the previous output of the reconstruction circuitry as is available from a buffer circuitry 114. These two signals are added at 116. The output from adder circuitry 116 is a coded motion signal representation corresponding to that available in the illustrated Figure 2 embodiment over lines 36. The loop can compensate for channel error using the multiplication circuitry 118 having a "leak"
parameter, a, which can range between 0 and 1.
Referring to Figure 6, the receiver structure for the transmitter of Figure 5 employs a differential loop 120 having a frame buffer 122 for storage of the reconstructed motion displacement vector signals from reconstruction apparatus 78. The receiver, thus, adds, using an adder 124, to the output of the . - . .. . .. . . .

. - ~ - . : :: .

,. . :: .. - ..

-reconstruction circuitry 78, the previous motion displacement vector which is stored in buffer 122. The result, over lines 126, is directed to the motion field interpolator 96. As before, a "leak" parameter, "a," can be employed to remove, over time, channel errors in the reconstruction signal.
The Motion Field Interpolator (38, 96) ~ eferring to Pigure 7, the motion field interpolator 38 receives from the reconstruction circuitry 34, a m~tion displacement vector over lines 36 for each block region of the image. For example, for a color television video signal, a typical luminance imaqe can have a block size of 8x8 pixels whi~e a typical chrominance image can have a block size of sxs pixels.
The motion field interpolator,-employing a selected interpolation method, then associates with each pixel of the frame, an interpolated motion vector displacement value.
In accordance with the preferred aspect of the invention, a raised cosine interpolation function i5 employed for associating with each pixel a displacement value. ~he interpolation function is stored at 130. The input vector field is stored in a buffer 132 and has a low resolution corresponding to the block size.
Thus, in accordance with this aspect of the invention, each displacement vector from reconstruction circuitry 34 is associated with the center of a multi-pixel region. Thus, for a 4x4 block region, referring to Figure 8, the interpolation vectors are associated with a center position, that is, positions 400, 402, --19~ :.

, .- , . . .
.. . . . - .. ~.... . ~ ~ - .
~, , , : ,, .
. ~

404 which are not, for a 4x4 block, associated with any one picture element. The interpolation process, performed by a vector interpolator 124 operates in both the X and Y directions. ~hus, the displacement vector associated with the 4x4 pixel block region centered at 400 and the corresponding displacement vector associated with the region centered at 402, for example, can be interpolated with respect to the X direction, while the displacement vectors at regions centered at 400 and at 404 can be employed with respect to a Y interpolation. In general, the interpolation process employs a plurality of displacement vectors surrounding a point of interest for deriving the value of the displacement vector at the point. Thus, an X and Y interpolated displacement vector value is assigned by the vector interpolator 134 in accordance with the interpolation function being used for lS each pixel of the image. In other embodiments of the invention, interpolation functions other than the raised cosine can be employed. For example, a linear interpolation function, or a trapezoidal shaped interpolation function can be employed, the latter for reducing the computational load.
The output of the transmitter motion field interpolator 38, or the receiver motion field interpolator 96 ~which operates in a manner identical to that of interpolator 38,) is directed to the full motion compensation circuitry 42 in thè transmitter and to the full motion compensation circuitry 80 and frame interpolation circuitry 88 in the receiver. The full motion compensation '' 1 3230q3 circuits q2 and 80, using incoming frame image data from frame buffers 44 and 98 respectively and the motion field interpolator output data, produce, in the transmitter, the estimated receiver image over lines 56, and in the receiver, the received estimated image over lines 82. The motion compensation circuitry maps each output pixel location to a location in the previous frame as indicated by the displacement vector value associated with that output pixel location. The displacement vectors are specified by the motion field interpolation circuitry associated therewith. In particular, referring to Figure 7, this is the output of vector interpolator 134.
There results, however, for some coordinate locations of the new image field, a mapping f rom pixel coordinates in the previous frame which do not fall upon a grid location. That is, the interpolated motion displacement vector may call for a movement of, for example, 1 1/4 picture elements. In these instances, the motion compensation apparatus employs a spatial (or pixel) interpolation, for example a linear spatial interpolation of the 2X2 block of pixels surrounding the noninteger location, for determining a pixel value from the previous frame. Other interpolation functions could, of course, be employed, for example, the value selected for the noninteger coordinate of the previous f rame can be the value of the pixel closest thereto.
Alternatsly, a raised cosine interpolation function can be employed, .
, - . ~ . . ~

`3 ~ 3230q3 In the illustrated embodiment of the invention, the receiver also employs the output of mo~ion field interpolator to create one or more frames between those which are actually transmitted. In accordance with this aspect of the invention, the temporal frame 5 interpolator 88, in the illustrated embodiment, receives the -values of the motion field interpolation circuitry to determine the image values for a frame positioned in time, in the particular illustrated embodiment, one-half the distance between the transmitted and received frames. In the illustrated embodiment, this function is performed by halving the output displacement vectors from the motion field interpolator 96. Thus, if a pict~e element, from one transmitted frame to the next, were displaced two pixel positions in the X direction and 4 pixel positions in the Y direction, the temporal frame interpolator would provide an intermediate frame wherein that picture element was displaced one position in the X direction and two positions in the Y direction.
In this manner, a frame half way between two received frames can be added to the picture image sequence to provide a better visual rendition.
The motion field interpolator 38 and motion compensator 42 can be implemented in hardware, in software, or in a combination of hardware and software. Attached hereto as Appendix 8 is a software implementation of a color video signal processing apparatus wherein the luminance (8x8 block regions) and chrominance (4x4 block regions) signals are processed.

The Lossv Compressor In accordance with a preferred embodiment of the invention, the lossy compression circuitry 28 can be any of a plurality of circuitries. For example, the lossy compression circuitry can use transform coding, differential pulse code modulation, or any other lossy coding method. The output of the motion estimation circuitry 26 has the appearance, in essence, of a highly correlated image wherein each picture element represents not a portion of a visual image but, rather, a displacement value.
According to the preferred embodiment of the invention, a block transform encoding method is employed for coding the motion estimation output si~nal.
In the illustrated embodiment of the invention, the lossy compression circuitry can be a discrete cosine transform operating on the displacement vector output of the motion estimation circuitry 26 to provide coefficient data which is then adaptively subjected to a threshold and uniform quantization process. In this circumstance, the reconstruction circuitry 34 then becomes the inverse discrete cosine transform as is well known in the art.
In other embodiments of the invention, the lossy compression circuitry 28 can be embedded in a differential pulse code modulation circuitry, for example that described in connection ;
with Figure 4. In this instance, the temporal differential pulse code system employs a pulse code modulation coder of limited bit .. . . ..
, j ,, , .............................. :

' ' ' . ' ''~ ' ' ` ' ' ~

l structure, and it ls this lossy coder which provides the errors which are compensated for by the system as described hereinabove.
In yet another aspect of the invention, and as described the above-noted co-pending Canadian applicatlon 570,748, filed on June 3, 1986, and assigned to the assignee of this invention, preprocesslng techniques can be employed for improving even more upon the transform method described hereinabove, to substantially eliminate the block coding artifacts. In still another embodiment of the invention, the lossy compressor can be implemented using a short-space Fourier transform, such as that described in U.S. patent number 4,703,349, issued October 27, 1987 for a "Method and Apparatus for Multi-Dimensional Signal Processing Using a Short-Space Fourier Transform, and assigned to the assignee of the present invention.
The Motion Estimator Referring now to Figure 9, in accordance with the illustrated embodiment, the motion estimation circuitry 26 employs an iterative spatial domain approach which quickly converges through an error minimization process for determining the displacement vectors of an input signal. In the preferred embodiment, an adaptive, steepest descent approach method is employed. The method has good convergence behavior.

~. .

, In Hinman, B., "Theory and Application of Image Motion Estimation," Master's Thesis, M.I.T. (1984~, equation 3.12 describes the iterations for determining a displacement vector v for a pixel having a location ~ in an image f. The new image is represented by g. That equation is reproduced herein as Equation 1 where the "i" represents an earlier value of the displacement vector and "i ~ 1" represents a next value o~ the displacement vector.

vi+l = Vi 2e [9(~) - f(~-Vi)] ~7(.) f(p _vi) ~ (Equation 1) Equation 1 can be implemented in accordance with Figure 9 where the input g and f frames are stored in buffers 200 and 202 respectively. The gradient value of the f rame i5 determined by a gradient computer 204 and spatial interpolators 206, 208 provide the values of ~ required by the summation in Equation 1. A
summing element 210 and a multiplication element 212 yield the error gradient value which is accumulated in an accumulator 214.
The output of the accumulator 214 is multiplied by a step size adaptor 215 at a multiplier 216; and the result is used to update the vector displacement value in an accumulator 218. The accumulator receives the initial vector prediction value from a prediction hardware 220.
The upda~ed vectors are employed by interpolators 206 and 208 in determining the values of "f" and the vector output field is - ~
~ ~ .

.; - -1 32~093 stored in a buffer 222. The entire apparatus operates under the control of a system controller 224 which monitors the value of the error gradient accumulator output and in response thereto adaptively chanqes the step size.
Referring now to Figure 10, an improved method for implementing the adaptive steepest descent error minimization iterative spatial-domain motion estimator 26, defined by equation 1, provides for use of a single interpolator 240 operating upon the f frame and a gradient computation circuitry then operating upon the interpolated value output of interpolator 240. There results from this circuitry the elimination of one of the Figur~ 9 interpolators thus providing a savings of hardware and/or software.
In employing this method, the three important parameters are lS the initial step size, e, the stopping or cutoff threshold, T, and the predictor set, ak. The initial step size plays an important role in determining the rate at which the method converges upon the motion vector displacement. When e is chosen to have a very small value, the method requires an excessive number of iterations before reaching a sufficiently small neighborhood of the minimum.
As the value of e increases, the rate of convergence also increases but there comes a value when e is so large that the search procedure will oscillate about the minimum without converging. The value of e must be determined by empirical tests.

:' ' . :
. ;, : "`

For a block size of 4x4, a value of 3xlO-5 has been found to provide the best convergence behavior.
The cut-off threshold is compared to the magnitude of the error gradient times the initial step size, e, for each iteration.
This threshold must be chosen with two constraints in mind.
First, it should be sufficiently small so that a displacement close to the actual displacement is reached. The second and opposing constraint is that as the threshold decreases, the number of required iterations increases dramatically. Thus, as with the step size, the appropriate cut-off threshold must be found empirically. (When the step size varies, as described below, t~e threshold, T, as defined above continues to be compared against the multiplicative product of the initial step size, e, and the error gradient.) In determining the cutoff threshold empirically, two indicators are of interest. They are the average number of iterations per block and the motion compensated frame difference energy. In order to find the best value of T, the estimation and compensation process is run several times. Starting with a small cut-off threshold, a minimal value of the motion compensated frame difference energy is determined. As the threshold is increased, the average number of iterations steadily drops while the motion compensated frame difference energy remains essentially constant.
Eventually, however, a value of the threshold is reached at which the estimated displacements become inaccurate and the motion . ~

compensated frame difference energy begins to rise. In seeking the point of inflection in the motion compensated frame difference energy as a function of T, a value of 7x10-3 was found to be most appropriate. This threshold corresponds to a minimum vector update of 7x10-3 pixels.
The selection of the predictor coefficients affects both the average number of iterations and the motion compensation frame difference energy.
The selection of the initial value of the displacement vector for each block is, in the illustrated embodiment, chosen as a linear combination of displacement vectors adjacent thereto. t Thus, referring to Figure 11, the steepest descent method provides access to the displacement vectors above and to the left of a present displacément vector VO. Thus, in accordance with the illustrated embodiment, the present displacement vector is defined by equation 2 as:

VO = ~ ajVj (Equation 2) =

The aj are the predictor coefficients while the Vj are the previously determined displacement vectors.
~he preferred vector predictors, aj, are al = 0.3, a2 = ;
a3 = 0.4 and a4 = 0.3. For these values of the vector predictors, in combination with the step value and threshold values no~ed above, for one set of data, the average number of iterations was 6.71, and the motion compensation frame difference energy was 14.1 d~'s lower than the non-motion compensated frame difference energy.
In accordance with a preferred embodiment of the invention, the selected val~e of e, the initial step size can be adapted to change as a function of the error gradient. ~hus, according to thi~ illustrated embodiment of the invention, and under the control of the system controller 224, the step size value is adapted to change, depending upon the present gradient error value, or the present and one or more previous error gradient values, so that the step size for the vector displacement does ~ot become unreasonably small. A description of two methods for selecting step size is provided in the Hinman thesis beginning at page 33.
lS According to the preferred embodiment of the invention, the second method, defined by Equations 3.18, 3.19, and 3.20 of ~he Hinman thesis is modified to reduce the processing load of the system. In this e~bodiment, the equations 3.18, 3.19, and 3.20 are modified so that:

(new step size) = (old step size) CX (Equation 3) ' .~

-where r 1.4 if R ~0 ~ 0.3 if R< 0 (Equation 4) and R = ~Gradient E(Vl)] [Sign [Gradient E(V~ t [Gradient E(V )] [Sign [Gradient E(V 1 )]]

and "Gradient E" represents the gradient of the present x or y error function, and "Sign [~]" e~uals +l depending upon the sig~
of the argument. Equation 4, thus defined, provides a third method for an adaptive steepest descent approach. As noted ahove, however, the cutoff threshold value, T, is measured against the initial step size. ~hus, the illustrated T can be defined as the product of a constant times the error gradient.
Further, in accordance with the preferred embodiment of the invention, the system controller 224 also prevents the displacement increment step size from being greater than a selected threshold, for example, greater than one pixel, and further prevents the maximum value of the displacement vector from being greater than a second selected threshold, for example greater than 7 1/2 pixels. In this manner, more control of the steepest descent process is available and singularities which prevent convergence can be avoided.

- , - ,.~. : , . . . .

The illustrated motion estimator can be implemented in hardware, in software, or in a combination thereof. In one particular embodiment, attached hereto as Appendix C, a software implementation of an adaptive steepest descent method and apparatus is provided.
Adaptive Filter (49) As noted above, the output of the motion compensator 42 represents the receiver motion compensated image as a result of applying the coded motion signal representation from the lossy compressor 28 to the previous frame which is stored in frame buffer 44. The receiver motion compensated ima~e, under certai~
circumstances, can be improved in various spatial regions (blocks) by selectively applying a spatial low-pass filter. To decide whether the filtering process should be performed on a block of the receiver motion compensated image, a comparison of the squared prediction error between the original image and the unfiltered and filtered prediction is performed on a block-by-block basis.
A block, in this illustrated embodiment, i5 designated as an 8x8 grouping of picture elements. For each block, the prediction having the least error energy ~weighted, for example, by a constant to provide a bias in favor of unfiltered prediction) is selected, and that decision information, the filter data available over line 63, is made available to the encoder 18. The filter data is sent to the receiver to enable it to perform the same operation as the transmitter on each corresponding block during :, , -. .

the receiver image reconstruction process. After the adaptive low-pass filtering decision has been made for a block, the resulting block image, whether or not filtered, is subtracted as noted above from the new uncoded image to generate an error image over lines 50.
~ eferring to Figure ~2, the adaptive filter has ~our stages of operation. Pirst, the motion compensated image block on line 32 is low-pass filtered by a filter element 400. Second, an energy measurement is made at enerqy measurement circuitries 402 and 404 for, respectively, both the filtered and unfiltered versions of the block in comparison to the original image. Thi~d, a decision is made at a comparator 406 to determine if the low-pass filtered block should be inserted in place of the unfiltered receiver motion compensated image. Finally, the information describing the filter usage for each block is encoded by an encoder 408 to provide the filter data on lines 63 which is sent to the encoder 18. The output of the comparator 406 controls a multiplexor 410 to provide the prediction image output over lines 61a.
The filtering operation is performed, in accordance with the present invention, over composite blocks of the receiver motion compensated image having a size of sixteen picture elements by sixteen picture elements. The low-pass filter, in accordance with the illustrated embodiment of the invention, provides for a simple 5x5 block averaging. The averaging process is performed, in ., . :: . ~ :
., ,. . . . . ~-, .

- : ~ - ' , , ~ : . ~

1 3230q3 accordance with this embodiment of the invention, as a length five moving average on all of the rows followed by a length five moving average of all of the columns.
Each 16x16 composite block at an edge of the image is enlarged by adding two row~ above or below and/or two columns to the right or to the left of the existing block in order to provide the data needed for the 5xS average. The result is thus generated for each of the sixteen pixels in the row by averaqing the current pixel value with the value of the two pixels to the left and the two pixels to th~ right. For the first pixel in each row, the averaging is done explicitly in accordance with the illustrated ~
embodiment o~ the invention; however, for the remaining pixels of the row, the average can be determined by successively adding the next pixel to the right and subtracting the next pixel to the left of the length five region. The resulting values are stored in a separate 16x16 array. After filtering of all rows, the same averaging procedùre is performed on all of the sixteen columns and the results are stored in a separate 16x16 array. For those composite blocks at the edge of a frame, the two extra rows or columns are obtained by repeating the row or column at the edge of the composite block.
once the filtered version of the reference block has been generated, the error energy is determined for both the predicted receiver motion compensated block and the filtered motion compensated block. The energies are determined for each of the ~'` ` :

8x8 blocks in the overall 16x16 composite block. The energy values, in accordance with the illustrated embodiment of the invention, are determined by a sum of squares operation for every picture element of the block. The comparison is made to the S original image which is available to the adaptive filter from the analog-to-digital conversion and frame buffer 12, over lines 22.
The two energy measurements that result for each of the 8x8 blocks are compared to determine whether the filtered block should be used as a replacement for the unfiltered block in the motion compensated frame. The energies are compared to each other using a constant weighting multiplier. After the weighting, the bloc~
having the smaller energy value is selected. If the unfiltered error energy is lower, the 8x8 block from the motion compensator 42 remains unchanged and is directed to the leak multiplication circuitry and thence to adder 60. If the filter error energy is lower, then the original 8x8 block is replaced by the corresponding 8x8 filtered block from the adaptive filtering process.
For each 16x16 composite block, the result of the comparisons are stored in a list and a four bit code is generated which indicates which of the four blocks is to be filtered and which is not.
In accordance with the preferred embodiment of the invention, the four bit codes that describe the filter data for each 16x16 ~5 composite block are coded using a variable length code. The . ' ~, :, ' '` ' ~

1 323Qq3 variable length coding procedure takes advantage of the fact that filtered blocks tend to appear in clusters.
At the receiver, the variable length filter data is decoded and the data, referring to Figure 3, is directed to the motion compensator 99 In this embodiment, the motion compensator includes circuitry for determining when filtering is to be applied to its input data. If filtering is employed, indicated by a non-zero decoded code word, then the 16x16 composite block is low-pass filtered in accordance with the code word and the appropriate 8x8 blocks available at the input of the motion compensator 99 are replaced by their corresponding filtered block versions.
In accordance with the illustrated embodiment o~ the invention, the preferred weighting constant is equal to one.
Thus, the error associated with the filtered output must be smaller than the error associated with the unfiltered output before a filtered block will be substituted for that unfiltered block.
The adaptive filter method and apparatus can be implemented in hardware, in software, or in a combination of the two.
Attached hereto as Appendix A is a software implementation of the method.
Additions, subtractions, deletions, and other modifications of the preferred particular embodiments of the invention will be apparent to those skilled in the art and are within the scope of the following claims.

i, : :.

Claims (20)

The embodiments of the invention in which an exclusive property or privilege is claimed are defined as follows:

1. A method for encoding and transmitting a sequence of image frames comprising the steps of estimating, for successive image of the sequence, and for each of a plurality of multi-pixel blocks of the image, a measure of motion displacement of each block between successive images, generating, using said measures of motion displacement, a motion compensated image, selecting, for each block of an image frame representation, one block chosen from between corresponding blocks of, respectively, said motion compensated image and a low-pass filter representation of said motion compensated image, and transmitting filter selection data over a communications channel to enable a receiver to determine for a block of the image being transmitting whether to employ low-pass filtering in said block of the image in generating a prediction image from a motion compensated receiver image, said filter selection data being dependent upon the selecting step at the transmitter.

2. The method of claim 1, wherein said selecting step comprises the steps of generating a low-pass image representation of each block of the motion compensated image, determining a value of a motion compensated image error associated with each block of the motion compensated image, determining a value of a low-pass image error associated with each block of the generated low-pass image representation, and setting, in response to said motion compensated and low-pass image error values, for each block, a data bit identifying whether to subject the block of the motion compensated image to low-pass filtering.

3. The method of claim 2, wherein said representation generating step comprises the step of applying a block averaging filter process at each picture element of a block.

4. The method of claim 2, further comprising the step of compressing said data bits using a variable length coding procedure.

5. Apparatus for encoding and transmitting a sequence of image frames comprising means for estimating, for successive images of the sequence, and for each of a plurality of multi-pixel blocks .

of the image, a measure of motion displacement of each block between successive images, means for generating, using said measures of motion displacement, a motion compensated image, means for selecting for each block of an image frame representation, one block chosen from between corresponding blocks of, respectively, said motion compensated image and a low-pass filtered representation of said motion compensated image, and means for transmitting filter selection data over a communications channel to enable a receiver to determine for a block of the image being transmitted whether to employ low-pass filtering in said block of the image in generating a prediction image from a motion compensated receiver image, said filter selection data being dependent upon the selecting means at the transmitter.

6. The apparatus of claim 5 wherein said selecting means comprises means for generating a low-pass filtered image representation of each block of the reconstructed motion image, means for determining a value of a reconstructed image error associated with each block of the reconstructed motion image, means for determining a value of a low-pass image error associated with each block of the generated low-pass image representation, and means for setting, in response to said reconstructed and low-pass image error values, for each block, a data bit identifying whether to subject the block of the reconstructed motion image to low-pass filtering.

7. The apparatus of claim 6 wherein said representation generating means comprises means for applying a block averaging filter process at each picture element of a block of the image.

8. A method for encoding and transmitting a sequence of image frames comprising the steps of dividing the image into a plurality of block regions, generating a data coding of the high spatial frequency information content of said image, determining for each block region an error value for each of a filtered image prediction of the block represented by said data coding and of an image prediction using said data coding without filtering, and transmitting said data coding and a filter selection data over a communications channel to a receiver to enable said receiver to determine, for each block region of an image, whether to employ low-pass filtering in generating a receiver prediction image from a received data coding.

9. The method of claim 8 further wherein said generating step comprises the steps of estimating, for successive images of the sequence, and for each block, a measure of motion displacement of each block from one image frame to a next image frame, and encoding, using a lossy compression device, said motion displacement measures.

10. An apparatus for encoding and transmitting a sequence of image frames comprising means for dividing the image into a plurality of block regions, means for generating a data coding of the high spatial frequency information content of said image, means for determining for each block region an error value for each of a filtered image prediction of the block represented by said data coding and of an image prediction using said data coding without filtering, and means for transmitting said data coding and a filter selection data over a communications channel to a receiver to enable said receiver to determine, for each block region of the image, whether to employ low-pass filtering in generating a receiver image from a received data coding.

11. The apparatus of claim 10 further wherein said generating means comprises means for estimating, for successive images of the sequence, and for each block, a measure of motion displacement of each block from one image frame to a next image frame, and means for encoding, using a lossy compression device, said motion displacement measures.

12. A method for encoding and transmitting a sequence of image frames comprising the steps of selecting, for successive images of the sequence, and for each of a plurality of multi-pixel blocks of the image, one block chosen from between corresponding blocks of, respectively, a previous image and a low-pass filter representation of said previous image, and transmitting filter selection data over a communications channel to enable a receiver to determine for a block of the image being transmitted whether to employ low-pass filtering in said block of the image in generating a prediction image from a previous receiver image, said filter selection data being dependent upon the selecting step at the transmitter.

13. Apparatus for encoding and transmitting a sequence of image frames comprising means for selecting, for successive images of the sequence, and for each of a plurality of multi-pixel blocks of the image, one block chosen from between corresponding blocks of, respectively, a previous image and a low-pass filtered representation of said previous image and means for transmitting filter selection data over a communications channel to enable a receiver to determine for a block of the image being transmitted whether to employ low-pass filtering in said block of the image in generating a prediction image from a previous receiver image, said filter selection data being dependent upon the selecting means at the transmitter.

14. A method for decoding and receiving an encoded sequence of image frames comprising the steps of receiving, from a communications path, filter selection data for determining a data source for reconstructing a next block of an image frame at the receiver, and selecting, in an uncoded mode of operation, a low-pass image derived from a previously received image frame, and in a second coded mode of operation, image data from said communications path for reconstructing, from the image data, a block of the image frame.

15. A method for decoding and receiving an encoded sequence of image frames comprising the steps of receiving, from a communications path, filter selection data for determining a data source for reconstructing a next block of an image frame at the receiver, selecting, in an uncoded mode of operation, a low-pass image derived from a previously received image frame, and in a second coded mode of operation, image data from said communications path for reconstructing, from the image data, a block of the image frame, and applying a block averaging filter process at each picture element of the block of the image frame to be generated in said first uncoded mode of operation.

16. A method for decoding and receiving an encoded sequence of image frames comprising the steps of receiving filter selection data over a communications channel for determining for a block of the image frame being received whether to employ low-pass filtering in said block of the image in generating a prediction image from a previous receiver image, said filter selection data being generated by an image encoding transmitter, and generating each block of the image frame using the block, chosen in accordance with the said filter selection data, selected from a block of a previous receiver image, and a low-pass filter representation of a block of said previous image.

17. An apparatus for decoding and receiving a sequence of image frames comprising means for receiving filter selection data over a communications channel for determining, for a block of the image frame being received, whether to employ low-pass filtering in said block of the image in generating a prediction image in a motion compensated receiver image, and means for generating said motion compensated image using, in a first mode, said low-pass filtering and, in a second mode, a block of a previous receiver image frame using data measuring a motion displacement.

18. The apparatus of claim 17 wherein said image generating means comprises means for applying a block averaging filter process at each selected picture element of a block of the image frame.

19. In an image transmission and receiving process for communicating a sequence of image frames, wherein the image is divided into a plurality of block regions and the data transmission generates, selectively, high spatial frequency block displacement information relating to the image and filter selection data, a decoding method for comprising the step of determining from filter prediction data for each block of an image frame being received from a communications channel whether in a first operating mode, a low pass filter image frame of a block of a previously received image is to be employed for generating the predicted image representation for a block region of the image being received, or, in a second operating mode, further following high spatial frequency block displacement information content of the image shall be employed for generating the predicted image representation for the block region.

20. In an apparatus for decoding and receiving a sequence of image frames having means for dividing the image into a plurality of block regions, means for generating a data coding of the high spatial frequency information content of the image, and means for determining for each block region an error value for each of a filtered image prediction of the block represented by the data coding and an image prediction using the data coding without filtering, a receiving apparatus comprising means for receiving a filter selection data over a communications channel for operating, in a first mode, for reconstructing said block region of the image employing a low-pass filtered version of a previously received image for said block and, in a second mode, employing the high spatial frequency information content of the image data being received from the communications path.