AU622422B2 - Video coder - Google Patents

Description

I

OPI DATE 23/05/89 APPLN. ID 25585 88

PCT

AOJP DATE 29/06/89 PCT NUMBER PCT/GB88/00871 INTERNATIONAL APPLICATIA P ISF D J E R Et E 4ENT COOPERATION TREATY (PCT) (51) International Patent Classification 1 (l1) I lk atid f Publication Number: WO 89/ 04101 H04N 7/137 A l (43) International Publication Date: 5 May 1989 (05.05.89) (21) International Application Number: PCT/GB88/00871 (74) Agent: LLOYD, Barry, William; British Telecommunications Public Limited Company Intellectual Pro- (22) International Filing Date: 17 October 1988 (17.10.88) perty Unit, 151 Gower Street, London WCIE 6BA

(GB),

(31) Priority Application Number: 8724789 (81) Designated States: AU, DK, FI, jP, NO, US.

(32) Priority Date: 19 October 1987 (19.10.87) (33) Priority Country: GB Published With international search report.

(71) Applicant (for all designated States except US): BRIT- ISH TELECOMMUNICATIONS PUBLIC LIMIT- ED COMPANY [GB/GB]; 81 Newgate Street, London ECIA 7AJ (GB).

(72) Inventors; and Inventors/Applicants (for US only) CARR, Michael, Douglas [GB/GB] "St Hilda", 137 Kirton Road, Trimley St Martin, Ipswich, Suffolk IPIO OQL (GB).

LEANING, Anthony, Richard [GB/GB]; 43 Elmhurst Drive, Ipswich, Suffolk (GB).

(54) Title: SIGNAL CODING IN PIXEL

I

(57) Abstract SELECT I AND DIFF.

The current frame of the picture is com- 1 pared block-by-block with the previous frame BUFFER to identify changed picture elements (pixels). The 2 4 OUT resulting matrix is matched ("vector quantised") HANGE to one of a set of standard matrices ("VQ DETECT 34 shapes"). Only those pixels flagged by the selected VQ shape are transmitted plus a "VQ shape code". This technique is modified in that the current frame is also compared (30) with a "background" frame stored (31) at the transmitter and 33 at the receiver. Where, for any block, all the pix- DECODE els flagged by the VQ shape are indicated by the second comparison as being the same as the background, the pixels are not sent instead a shape code plus a "copy bacLground" instruction CHANGE is transmitted. DETECT

BACKGROUND

FRAME

S STORE LOAD d-36 32 r -i L-.J CHANGE I_ I MONITOR L S- ia SIGNAL CODING The present invention relates to coding of video signals, especially using conditional replenishment coding, where information is generally transmitted only in respect of elements of a frame of the picture which have changed relative to a previous frame; the transmitted data being used at a receiver to update a stored version of the picture.

Thus picture elements of each block of an image to be coded are compared with those of the corresponding i. 10 block of a previously coded image to determine whether 9. **the block has changed between the two images; if so, j: picture element data are generated for output.

Such a system is described in international patent So application published under no. W086/03922, which also proposes that the block be compared with the corresponding block of an earlier (reference) image. If S"they are deemed to be the same, no picture data are generated instead, a codeword is produced to indicate that the receiver is to obtain its data from a locally stored replica of the reference image.

According to one aspect of the present invention, there is provided a method of coding an image comprising, for each of a plurality of blocks of a frame in a series of frames of an image: comparing picture elements of the block with those of the corresponding block of a previously coded frame of the image to produce a matrix of values each indicating 2 whether the corresponding element is, in accordance with a predetermined criterion, deemed to have changed between the two images; (ii) matching the matrix to one of a predetermined set of such matrices each of which identifies a region of the block as being deemed to have changed, and generating a codeword identifying that one matrix of the set; (iii) comparing picture elements of the block with those of the corresponding block of a reference image composed of blocks from at least one earlier frame to determine i. whether the block is, in accordance with a predetermined criterion, deemed to have changed between the two images; (iv) coding for output those elements within the identified region, unless all the elements within the region are identified by comparison step (iii) as being unchanged relative to the reference image, whereupon a codeword indicating this is generated.

In another aspect, the invention provides an apparatus for coding an image comprising: means for comparing, for each of a plurality of 9 blocks of a frame in a series of frames of an image, picture elements of the block with those of the corresponding block of a previously coded frame of the image to produce a matrix of values each indicating whether the corresponding element is, in accordance with a predetermined criterion, deemed to have changed between the two images; t i rr ~1 ~i i ~L~ia~cl3-sr* I-- 3 i

I

ii i; V4" (ii) means for matching the matrix to one of a predetermined set of such matrices each of which identifies a region of the block as being deemed to have changed, and generate a codeword identifying that one matrix of the set; (iii) a store arranged to store a reference image composed of blocks from at least one earlier frame and means for comparing picture elements of the block with those of the corresponding block of the reference image 10 to determine whether the block is, in accordance with a predetermined criterion, deemed to have changed between the two images; (iv) means for coding for output those elements within the identified region, unless all the elements within the 15 region are identified by comparison step (iii) as being unchanged relative to the reference image, whereupon a codeword indicating this is generated.

The matching step often termed vector quantisation of two-dimensional maps has been proposed previously for picture coding, for example in European patent application serial no. 0239076A, where blocks of transform coefficient blocks are classified.

The present invention, however, further exploits the vector quantisation, in that not only can the classification be used (if desired) to reduce the amount of information that has to be transmitted to indicate which elements have been coded for output, but also the number of occasions on which a "reference" codeword is instead generated are increased.

:::ai

;;I

i 4 3a Some embodiments of the invention will now be described, by way of example, with reference to the accompanying drawings, in which: Figure 1 is a block diagram of a coder according to one embodiment of the invention; Figure 2 shows a typical bit map produced by the change detector of the coder of Figure 1; Figure 3 illustrates a few standard bit patterns used by the vector quantiser of the coder of Figure 1; and Figure 4 is a block diagram of a decoder according to another embodiment of the present invention.

Figure 1 shows a conditional replenishment video coder, where video signals (assumed to be in digital form) a a.

0

S

^in^ LL__ C 44 -1-4-

F

are supplied to an input 1. The current frame of a picture is compared in a change or movement detector 2 with the output of a local decoder 3. The local decoder produces a "previous frame" output which is the same as S that produced by a remote decoder; the object of the comparison is to identify those parts of the picture which have changed so that only information concerning those parts needs to be sent to the decoder, to update a stored representation of the frame.

The signals are processed on a block-by-block basis an 8 x 8 block is assumed and the change detector 2 therefore Droduces an 8 x 8 bit map. A typical mao Is shown in fioure 2, where the elements of the block corresponding to Dicture elements (pixels) which have chanced are shown shaded.

oIlyn practice, transr.ission of information concerning o nly the changed elements involves a significant addressing overhead and therefore it is preferred to match the bit maD to one of a limited number typically forty of standard shapes (a few are illustrated in figure 3).

Since it is preferable to transmit information ar.

L .i u n nch a nge A. ixel ;6ha fi to r-ansr -i't 4nforma" -ri changed pixel, the saoe chosen is the smallest i.e.

with the least numer of shaded eleent.s) which has a shaded area covering the shaded elements of the bit map.

This process is termed vector quantisation (VQ) and is indicated as vector quantizer 4 in figure 1. One transmits, for the block, a VQ number idetifin the chosen shaPe, along with information concernino oixels deemed in accordance with the chosen shaoe to have changed. The outut data are cobine and buffered in a buffer 5 Drior to :assace to an outut 6. s is conventional in su:h sste.Ms, the buffer is used to s.cot: varato-: ns in the rate at which data are oenerated (due t -Y m Ir 4 AlV ,1 the picture content dependent coding) and interface to a regular'transmitted rate and the buffer fullness state used to control the rate of generation (eg by varying the change detector thresholds (control line The pixels deemed to have changed are coded in an encoder 8 (for example in the manner described in our International patent application PCT/GB88/00709 (publication no. W089/02206), and European patent application no. 88307981.6 (publication no. EP306250)).

Many televised scenes especially in a videoconference or videotelephone environment contain moving persons or objects set against a fixed background. Pixels observed as changed by the change detector 2 will relate either to objects which have changed their position (or entered the scene) or to parts of the background uncovered by the object. In the present coder, a second change detector 30 is also shown, which compares the current frame with a reference or background frame stored in a frame store 31. Acquisition of the stored background frame will be discussed further below, but in figure 1 is assumed to have been acquired from the local decoder 3 in response to manual operation of a switch 32 at the commencement of a transmission; a code being transmitted to the receiver to initiate similar action at the remote decoder.

The change detector 30 produces a bit map identifying those pixels of the current frame which are i the same as the background.

'A IL yV V Y7 K yi ;I i i i' f.

5a If this shows that the new image is different from the background for any of the pixels declared as changed by the VQ shape the background information cannot be used and information concerning those pixels is transmitted along with the VQ number.

U

*ft ft ft.

ft ft ft ft ft ft. ft ft.

ft *ft ft ft SO ft.

ft ft ft ft ft aft ft ft ft ft Oft ft ft 3 /4" I I

I>

2:

<I

6- *6

C.

C C

C

9* C C C C

C

C.

C

C CC C

C

CC

C

If, however the changed area as given by the VQ shape covers only pixels which are identified by the edetecotor as being the same as the corresponding pixels of the stored background frame, then the VQ number is accompanied by a reserved codeword indicating "background"l and no further information needs to be transmitted for the block in question.

These functions are accomplished in figure 2 by a comparator 33. whose output 34 overrides the operation of the encoder 8.

Figure 4 shows a decoder. An- inDt Drocessor receives the c-oded i4npult signals. T-he bac'7gour] frare is stored in a background store 41. in normal cPeration, the processor 40 uses the pixel information received to uodate v ia line 42 the contents of a f;rame store 43, using the received VQ numibers via line 44 to control the frare store adressinc. When, however, itreeves the reserved "lbackground"1 codeword (v ia line 45) It recovers the relevant. pixels (identified by th e V2I -ximber) from the lbackcrounc4 store and enters then :ntO th-e frame store 43. T hs S. illu-strat-ed scher-aticaly by a changeover switch 4 AE The frame sto-re 43 :is reac6 t~ (b utu control" maeans 47) to pro duce the received4 video at an, cu t zut 4E. As in t-he coder, the bjackcrc'Id store 41 is loaded frmthe decoded image in stor-:e 43 when a 'load background' instruction is received (line 49).

Note that the local decoder 3 of Ficure 1 can be of th- e same construction as the decodAer .fFigure 4, althougmh, of course, in Dra-ti-e the*cc=' decoder would use the background store 32 rather than i one of its own.

The chnedet ectors 2,30 an n rciebe an-, conen~oaly used, or may both b sdsrbdi u abov e-m e n :fo n e, paten-. T'-~a~os Te vector -2 LL~ WO 89/04101 PCT/GB88/00871 -7quantiser 4 can again be a known device. One possibility is that described in our International patent application no. PCT/GB87/00816 (publication no. W088/04084) and European patent application no. 8627787 (publication no.

0272794).

Simpler but cruder, another option is to convert the 8x8 matrix to a 4x4 matrix by creating each 'new' element as an OR function of four 'old' elements; this reduces the number of elements in the matrix to a size (16) which can be used to address a look-up table in the form of a 64 Kbyte read only memory in which the appropriate VQ numbers are stored.

The background scene may be 'frozen' manually at the commencement of a transmission. It may, but does not have to, consist solely of a fixed background. For example it may include seated figures (thereby covering the situation where a person momentarily passes his hand over his face if the face forms part of the "background", the face does not have to be retransmitted).

It may be desirable to include Drovision for updating the background. For example, the incoming video could be monitored at the coder (by a unit 35 shown dotted in Figure 1) and parts of the picture which differ frcm the original background but have remained unchanged for a i 25 predetermined period of time inserted into the frame store, a signal being sent (via line 36) to the decoder to instruct it to do likewise.

An alternative method of updating the background store abandons any attempt at identifying genuine background, but instead forms a reference image which is a composite of blocks taken from preceding images over a period. Each frame period, data for a few eight) selected blocks scattered over the imace area is entered into the background stores, the block selection being such that 3 suc t WO 89/04101 PCT/GB88/00871 8 different blocks are selected from each frame until the whole image area has been covered. Assuming 1024 blocks per frame at 25 frames per second, this represents a period of approximately 40 seconds. The blocks could be loaded from the frame store 43 of the decoder, and a corresponding store in the local decoder 3, the change monitor then being replaced by a simple address generator to select the appropriate blocks. Clearly, this is less effective, in that some blocks of the reference image will not represent background material, but a significant coding advantage is still obtained, and it has the merit of simplicity.

A variation of this approach provides that data for selected blocks, instead of being drawn from the decoded image, are actually transmitted i.e. transmission of the whole block is forced even if only a part, or none, of it is indicated by the detector 2 and quantiser 4 as moving.

In this case, the frame stores 31, 41 take their input from the input 1 and processor 40 respectively. This also has the benefit of ensuring that transmission errors on blocks which rarely change do not persist in the decoded image.

1

Claims (4)

1. 2. A method according to claim 1 in which, in the coding of the elements within the identified region, at least some of the elements are coded as the difference between that element and a predicted value for that element derived from one or more previously coded X~n I 10 elements of the block, the sequence of coding of the elements being dependent on the orientation of the identified region.
2. 3. An apparatus for coding an image comprising: means for comparing, for each of a plurality of blocks of a frame in a series of frames of an image, picture elements of the block with those of the corresponding block of a previously coded frame of the image to produce a matrix of values each indicating whether the corresponding element is, in accordance with a predetermined criterion, deemed to have changed between the two images; (ii) means for matching the matrix to one of a predetermined set of such matrices each of which identifies a region of the block as being deemed to have changed, and generate a codeword identifying that one matrix of the set; e (iii) a store arranged to store a reference image composed of blocks from at least one earlier frame and means for comparing picture elements of the block with 'those of the corresponding block of the reference image j i to determine whether the block is, in accordance with a predetermined criterion, deemed to have changed between the two images; (iv) means for coding for output those elements within the identified region, unless all the elements within the region are identified by comparison step (iii) as being unchanged relative to the reference image, whereupon a codeword indicating this is generated.
3. 4. An apparatus according to claim 3, including means A Z I I L_.J CHANGE I MONITORL,3 4a LI 11 U. 4 :4 I*: I': I1 4 .4 liti for comparing successive images to identify parts of the image which have remained unchanged for a predetermined period of time, and in the event of such parts being identified, to: update the stored reference image; and code for output data indicating which parts of the reference image have been thus updated. An apparatus according to claim 3, including means arranged in operation to effect periodic replacement of a minority of the blocks of the reference image stored in the store by the corresponding blocks of a recent image, the blocks being differently selected for successive replace.ient- such that the store always contains a composite image made up of blocks from a plurality of frames of the image being coded.
4. 6. A decoder for use with the coder of claim 3 or comprising: a frame store for storing a received image; a second frame store; control means responsive to received data to update the contents of the second frame store using the received data and responsive to a received codeword to update the second frame store with information from the first frame store; means for repetitively reading the contents of the second frame store to produce a video output signal. DATED this 28th day of January 1992 BRITISH TELECOMMUNICATIONS public limited company Attorney: PETER HEATHCOTE Fellow Institute of Patent Attorneys of Australia of SHELSTON WATERS i