WO2001043448A1

WO2001043448A1 - Apparatus and method for image data coding with additional data embedding

Info

Publication number: WO2001043448A1
Application number: PCT/GB2000/004671
Authority: WO
Inventors: Arthur John William Mitchell
Original assignee: Snell & Wilcox Limited
Priority date: 1999-12-06
Filing date: 2000-12-06
Publication date: 2001-06-14
Also published as: AU2189301A; GB2357210A; GB9928836D0

Abstract

By taking advantage of the time delay arising from the buffer in an MPEG encoder, information concerning future bit rate is inserted at regular locations in the bit-stream. In consequence, a multiplexer or data inserter receives advance notice of opportunities to insert data into the bit-stream. Advance notice enables data to be drawn from secondary storage and to be formatted optimally. In both cases, the effective data-carrying capacity of the stream is increased.

Description

APPARATUS AND METHOD FOR IMAGE DATA CODING WITH ADDITIONAL DATA EMBEDDING

This invention relates to compressed data streams and in one important example to the filling of unused data-carrying capacity in digital media transmission, and other systems, with extra data as and when capacity allows.

This practice is referred to as opportunistic data insertion.

The international standard ISO/IEC 13818 (generic coding of moving pictures and associated audio information), known commonly as MPEG2, has been globally adopted as the method of encoding media and binding them together in a multiplex for transmission as digital services. MPEG2 is an important example of an application for the present invention.

In this system, media are compressed by removing temporally and spatially redundant information. Psycho-visual and psycho-acoustic models also reduce information content. The resulting digital data is packetised and bound together to form a collection of services called a multiplex.

In many systems the rate of coded data from the video encoding process varies depending on the difficulty encountered when trying to maintain a constant image quality. This results in a general short fall of data from the encoders to ensure a working overhead of capacity. The spare capacity of the multiplex is then filled with ancillary data in a process known as opportunistic data insertion.

The management of opportunistic data insertion is complex and troublesome. If the available data-carrying capacity is to be fully utilised, the data handling system is required to react extremely quickly and flexibly when an opportunity for data insertion arises. The data handling system is, however, required also to communicate with the various sources of data and must perform

"housekeeping" tasks such as moving data between tiers of storage, ensuring redundancy in the event of equipment failure, regular archiving, and so on. Many of these demands conflict with the requirement for fast and flexible reaction to data insertion opportunities. It is thus the case that the design of data handling systems is a severe obstacle to the goal of maximising the carriage of data in digital media streams.

It is an object of the present invention to address this problem. Accordingly, the present invention consists, in one aspect, in compression apparatus comprising an input; a processing unit; and an output, the processing unit being arranged to provide at the output a compressed stream having a variable bit rate, there being a time delay between the input and the output, characterized in that the processing unit is adapted to provide in the time frame of the output, advance information concerning the bit rate of the output. Advantageously, the advance information is inserted into the output stream.

In a further aspect, the present invention consists in a compressed bit stream having a variable bit rate, wherein there is inserted in the bit-stream at each of a number of repeated locations, advance information concerning a parameter of the stream at a future location. Preferably, the parameter is the bit rate or one a more parameters useful in estimating or deducing bit rate.

Advantageously, said repeated locations are repeated at least once per frame in the compressed bit-stream.

In yet a further aspect, the present invention consists in opportunistic data insertion apparatus adapted to read from the bit-stream, advance information concerning the bit-rate at future locations in the bit-stream.

In still a further aspect, the present invention consists in a process of multiplexing variable bit rate streams with dynamic allocation of an available multiplex bit rate capacity among the respective streams, wherein there is read from at least one of the streams advance information concerning the bit rate of the stream at a future location.

The invention will now be described by way of example with reference to the accompanying drawings, in which:

Figure 1 is a block diagram of an exemplary data server; Figure 2 is a block diagram of an encoder according to the prior art; Figure 3 is block diagram of an encoder according to the present invention; and

Figure 4 is a block diagram of a multiplexer arrangement according to the present invention.

Briefly, the invention relates in this example to the optimisation of the system that inserts data into the MPEG2 stream and enables greater assurance that all spare capacity is used. To explain the invention it is necessary to understand a typical data insertion system. Reference is directed to Figure 1 for a block diagram of an example data server. There are several concurrent operations a data server may be required to perform, some of which are listed below:

1. Data must be added to the server from input streams such as a network or backup device.

2. Data must archived for safety and record keeping.

3. Data must be sent to the insertion point, the multiplexer.

4. Data may need to be duplicated within a disk sub-system for fault tolerance.

5. Data may also be replicated between servers so that on a failure of one the second steps in to maintain the service.

6. The system must respond to control and information requests from users or automatic systems.

Figure 2 shows a typical example of an encoding and transmission system. The source video is analysed and then encoded according to the MPEG2 standard. In many systems, the encoded MPEG2 bit-stream will be transcoded to a different bit-rate. The encoded data is fed into the encoder buffer ready to be included in the output data stream. The encoder buffer is used to govern the feedback algorithm that sets the level of compression and hence controls the output bit rate.

Between the analysis of the incoming video frame and the appearance of that frame as coded information in the output stream, there will be a significant delay. The length of this delay will of course vary in time and will also vary between encoding systems. An average delay of 0.5 seconds would, however be typical. It will be recognised that in the management of data handling systems, 0.5 seconds is a considerable period of time.

The delay is inherent in the MPEG2 compression system and can only be reduced, not eliminated. It arises in various ways; the most obvious of which is the encoder buffer. There are also delays arising from the need for picture reordering and from processing in the compression engine itself.

Figure 3 demonstrates the same encoding and transmission system as Figure 2, but with the inclusion of the present invention proposed. Here information from the coding process is carried forward through the encoder, avoiding the delay of the buffer and is passed out to the following data server. The data may be sent on a separate interface or may be included in suitable points in the MPEG multiplex.

The information sent from the encoder may include, but is not limited to, the number of bits currently being used to code the image and an indication of the time of encoding. The time might be the time of day, a relative time (i.e. in 300 ms time) an independent count or the program clock reference (PCR). From this and any other information it receives, the data server may calculate the capacity of data available in the future and by comparing the time code to the PCR, or other reference, know how soon this capacity will be available. From this information the server may modify is behaviour in many ways, some of which are:

1. Change the balance of time expended on its processes, e.g. if little bandwidth will be available for insertion it may boost its backup or update rate. If more bandwidth will be available, the priority of other processes might be reduced to guarantee that the demand is met. 2. Schedule data requests more efficiently to reduce the peak demands on the disk and network channels. 3. Choose the most efficient order to send blocks of data to minimise the period that each virtual file is "open" i.e. present in the multiplex. 4. Increase advanced data requests to cope with peaks in demand without abandoning sending opportunities.

By way of further explanation under point No. 3, a convenient mechanism for inserted data is in Programme Specific Information (PSI) packets of MPEG2. Every decoder will have the capacity to read and interpret PSI packets and there is therefore considerable advantage for certain types of inserted data of utilising the PSI stream. Indeed, there are at least two recommended protocols for inserting data in the PSI stream. A potential difficulty, however, is that many encoders are capable of reading only one type of PSI packet at any one time. Thus, if an encoder is reading the Programme Allocation Table (PAT) from the PSI stream, all other packets are effectively locked out until such time as reading of the table is complete. More importantly, if a large file is being carried as opportunistic data within the PSI stream, all other packets in the PSI stream may be locked out from the time the first packet containing data from the file is opened, until reading is completed of the last packet containing file data. In the case of a large file, with a relatively small bit rate available for opportunistic data, this 'lock out" time may be unacceptably large. In accordance with the present invention, the advance bit rate information would enable the server to select a location for inserting the file data at which enough contiguous or closely spaced packets are available to carry the file with minimum lock out of other PSI packets. It will be seen from Figure 3 that information can be made available to the data server from the compression engine or from the mode decision system. The information from the compression engine may simply take the form of the number of bits being fed into the buffer for a particular picture. This is precise information which the data server receives "in advance" by an amount of time determined by buffer throughput. Still "earlier" information is made available from the mode decision system. This will not represent a precise number of bits but will provide a reasonable estimate of the number of bits that will be used in the compression engine to encode the particular picture being considered in the mode decision system.

It has been explained that the information can - in one example of the invention - be presented to the data server as part of the MPEG2 transport stream. The MPEG2 standard provides ample opportunity for the insertion of such private data. This approach is particularly valuable where the data server is remote from the encoder. It also allows the benefit of the invention to be achieved where an MPEG2 transport stream, generated by an encoder in accordance with this invention, is recorded on conventional recording equipment before passing to a data server in accordance with this invention.

The stream itself may then be regarded as embodying the present invention, in the sense that - at intervals - it includes information concerning the bit rates utilised subsequently in the stream.

The information may in the case of a transport stream take the form of the available bit capacity for data insertion, being the maximum capacity of the transport stream less the aggregate instantaneous bit rates of the services it contains. An alternative is for each service to carry advance information of its own bit rate utilisation.

It has been clearly shown that there is advantage in a data server reading advance bit rate information from an incoming transport stream, with there being no physical link between the server and the encoder. The question may then be asked of how that data server reacts when it receives a transport stream from a conventional encoder and not from an encoder according to the present invention. In one approach, the data server will in such circumstances operate conventionally. An alternative, however, is for the conventional transport stream to be pre-processed by the insertion at appropriate intervals of advance bit rate information. The data server will then operate in the same optimised manner whether it is receiving a stream from an encoder according to this invention or a stream from a conventional encoder that has been processed in accordance with this invention.

Thus, in a further embodiment of the present invention, a transport stream of conventional form is analysed and, with appropriate delay, information concerning bit rate utilisation at one point in the transport stream is inserted at a point earlier in the stream. Whilst this of course introduces further processing and further delay, it may be useful in certain non-real time applications, or applications in which delay of that sense is not critical. In this embodiment, the amount of advance notice that can be given is, of course, not limited to the delay inherent in a compression unit. It is accordingly possible to select a desired look ahead period for the advance information that is being inserted into the stream. The advance bit rate information can also be inserted in an off-line software process.

Whilst the description of this invention has focused on data insertion, there will be other uses for the advance bit rate information that the invention provides. One such example is multiplexing. In a typical situation, a so-called statistical multiplexer will monitor the output of a bank of encoders, each providing a service, and will exercise control over the instantaneous bit rates of the encoders to allocate the available bit rate dynamically between the services in accordance with variations in picture material. It is not always convenient to provide the necessary control paths from the multiplexer to the encoders; some or all of the encoders may be in physically remote locations and some material may be available only in encoded form. In these multiplexing arrangements, it will be recognised that advance warning in the incoming transport streams of subsequent bit rate utilisation, (or of parameters from which subsequent bit rate utilisation may be estimated or deduced) will greatly assist the taking of decisions on bit rate allocation. This advance information is preferably sent continuously or at least at the picture rate or at least once per Group of Pictures (GOP). Decisions can be taken on the basis of more complete information and provisional decisions taken early can be used in electing an optimum allocation strategy. There is shown in Figure 4 a multiplexer receiving a variety of inputs that in different ways take advantage of the present invention.

Thus, an encoder 400 according to the present invention provides an MPEG output with inserted look-ahead information. A transcoder 402 is controlled by the multiplexer 404 to provide the required bit rate. The look-ahead information stripped out in transcoder 402 is fed forward to the multiplexer to inform the bit rate allocation process with advance bit rate information.

A file server 406 holds pre-recorded bit-streams which have look-ahead information inserted (in real time at the point of encoding or in an offline process, as examples). This advance information is made use of by the multiplexer 404 in its control of the bit rate allocated to other services, which may include the output of a conventional encoder 408 having its bit rate controlled in conventional fashion by the multiplexer, or data provided from data source 412.

If an MPEG bit-stream has been encoded conventionally, it can if appropriate have look-ahead information inserted as at box 410. It will be understood that this invention has been described by way of examples only and that a wide variety of further modifications are possible without departing from the scope of the present invention. Thus the advance information which is inserted into the bit-stream (or otherwise conveyed to a multiplexer or data inserter) may represent the bit rate itself or information which alone or with other information enables the bit rate to be deduced or estimated.

Such information might include, for example, the location of intra coded frames, edit points or frames which have proved particularly difficult to code.

Claims

1. Compression apparatus comprising an input; a processing unit; and an output, the processing unit being arranged to provide at the output a compressed stream having a variable bit rate, there being a time delay between the input and the output, characterised in that the processing unit is adapted to provide in the time frame of the output, advance information concerning the bit rate of the output.

2. Apparatus according to Claim 1 , wherein said advance information is inserted into the output stream.

3. Apparatus according to any one of the preceding claims, wherein the processing unit comprises a compression encoder adapted to receive an uncompressed input.

4. Apparatus according to Claim 3, wherein the compression encoder comprises a compression engine and a buffer, the compression engine being adapted to deliver a compressed bit-stream to said output through the buffer and wherein information concerning bit rate is derived from the compression engine and inserted in the bit-stream between the buffer and the output.

5. Apparatus according to Claim 4, wherein the compression encoder further comprises a mode decision system adapted to receive from the input an uncompressed signal and to deliver mode decisions to the compression engine, wherein information concerning bit rate is derived from the mode decision system and inserted in the bit-stream between the buffer and the output.

6. Apparatus according to Claim 1 , wherein the processing unit comprises a compression transcoder adapted to receive a compressed input.

7. A compressed bit-stream having a variable bit rate, wherein there is inserted in the bit-stream at each of a number of repeated locations, advance information concerning a parameter of the bit-stream at a future location.

8. A bit-stream according to Claim 7, wherein said advance information concerns the bit rate of the bit-stream at a future location.

9. A bit-stream according to Claim 7, wherein said locations comprise a regular continuous structure.

10. An MPEG bit-stream according to Claim 7, wherein said locations appear at least once per Group of Pictures and preferably once per picture.

11. Bit-stream apparatus comprising an analysis unit for extracting information from a bit-stream, a bit-stream delay and an information inserter for inserting extracted information into the bit-stream after the delay, thereby to provide an output bit-stream having at each of a number of repeated locations, advance information concerning the stream at a future location.

12. Apparatus according to Claim 11 , wherein the advance information concerns bit rate.

13. Apparatus according to Claim 11 , wherein said locations comprise a regular continuous structure.

14. MPEG apparatus according to Claim 11 , wherein said locations appear at least once per Group of Pictures and preferably once per picture.

15. Opportunistic data insertion apparatus adapted to read from the bit- stream, advance information concerning the bit-rate at future locations in the bit-stream.

16. A process of multiplexing variable bit rate streams with dynamic allocation of an available multiplex bit rate capacity among the respective streams, wherein there is read from at least one of the streams advance information concerning the bit rate of the stream at a future location.

17. A bit-stream mastering process, comprising the steps of generating information about the bit-stream at a first location and inserting a representation of that information into the bit-stream at a second location which is in advance of the first location.