GB2322509A - Communication system - Google Patents

Abstract

A method of transmitting information includes the steps of segmenting a block of information into a plurality of sub-blocks (102), determining the respective requirements for transmission of respective ones of the plurality of sub-blocks (104) and assigning a priority to each one of the plurality of sub-blocks. A resource of a communication channel is dynamically assigned (106) according to the respective requirements and prioritisation and respective sub-blocks on the respective dynamically assigned resource transmitted (108). In this manner, objects from a video scene for example are divided, prioritised and allocated resource according to their particular needs and/or available resources.

Description

COMMUNICATION SYSTEM AND METHOD FOR TRANSMITTING INFORMATION Field of the Invention This invention relates to communication systems. The invention is applicable to, but not limited to, video image transmission in such communications systems, and in particular in the allocation of available resource.

Background of the Invention In data transmission systems, bandwidth is generally a scarce resource. This is especially true for video/image communications since images have large bit requirements (typically one standard television resolution still image is over 0.5 Mbytes in size) compared to other forms of data e.g. speech, text etc.

In such image/ video transmission systems, a number of strategies exist to allocate the available data capacity in a predetermined manner.

For example, if multiple sources are sharing one data channel, capacity is allocated according to known rules e.g. a priority order is assigned to the sources, or some form of statistical multiplex is used where sources are allocated and de-allocated capacity depending on their activity. In such rule-based systems, predetermined priority ordering can lead to sources receiving additional data bits at a time when the user is no longer interested in them. This leads to problems where non-critical objects are assigned a disproportionate amount of the bit budget. An example of this might be in a surveillance application. MPEG4 has been defined such that a target number of bits are allocated to each of the "video object planes" (VOPs) in the source scene. VOPs could be a set of pixels with some semantic meaning such as the groups of pixels representing, for example, the entry gate, the security guard, vehicles approaching from the road etc.

When a new object enters the scene, and a new VOP is thereby defined, the target bit rate available for each of the VOPs will fall, as the total available bit rate must be re-allocated amongst all the VOPs including the new one. If, for example, the new object had been a vehicle approaching from another direction which could pose a threat, then the viewer would much rather view this object with high fidelity. If however it were just a stray dog passing by, the viewer would want only just enough quality to identify it as a dog.

In MPEG4, there is no explicit strategy for dealing with the above case. Without particular user intervention, the dog and the threatening vehicle would have exactly the same impact on the scene. Potentially all objects could suffer reduced quality for the sake of a stray dog, or an important vehicle might not be noticed.

A further technology available for video/ data transmissions is statistical multiplexing. Statistical multiplexing assigns data capacity to sources based on their activity, but this does not necessarily result in an improvement in picture quality as perceived by the viewer. For example, an apparently higher activity could be the result of a noisy source. In addition, extra bits allocated to a data source stream may be used to enhance the quality in an area of the image/video which is of low interest to the user.

A yet further technique, sometimes used to provide an enhancement over such allocation strategies, is to provide for the compression of the data to be transmitted. In video communication, a typical requirement for a compression scheme would be more than 200:1. Such high compression ratios lead to degradation of the video signal, for example loss of spatial or temporal resolution (e.g. blurring, judder etc.). Video compression systems are typically based on a frame by frame compression, where the number of resultant bits per frame (and therefore per second) varies with the scene content.

Typical video compression schemes result in a variable amount of data being produced per input video frame. These schemes will use a "rate buffer" to produce a fixed rate output for transmission on fixed rate channels. A simple method is typically used whereby the buffer is filled at a variable rate, but emptied at a fixed rate.

With the practical limitation that such a buffer cannot be arbitrarily large, there are a number of strategies employed to ensure that buffer over/under flow does not occur. To avoid overflow, the buffer is monitored.

When it reaches a certain level, a signal is fed back to the video compression unit to decrease the average of the variable bit rate it is producing. This can result in significantly decreased quality across the whole output picture, and may occur at a time where the viewer is most interested in the scene.

To avoid underflow, a similar feedback system can be used to increase the number of bits produced by the video compression unit, but additionally, one has the option to include null bits to pad out the bitstream.

Such "stuffing" is inefficient as non-useful bits are transmitted.

Thus' there exists a need for an improved method of transmitting video/ data that is known to the receiver to allow adequate decoding of the transmitted video/ data.

Summary of the Invention In a first aspect of the present invention a method of transmitting information is provided. The method includes the steps of segmenting a block of information into a plurality of sub-blocks, determining the respective requirements for transmission of respective ones of the plurality of sub-blocks and negotiating between the plurality of sub-blocks for communication resource. A communication channel resource is dynamically assigned according to the respective requirements and respective sub-blocks are transmitted on the respective dynamically assigned resource.

In this manner, objects, from for example a video image, are segmented and optionally allocated a priority. Available resource is then dynamically allocated for transmission of the video images based on at least, priorities assigned and/or resource available and/or processing requirements and/or variable data rates etc.

In the preferred embodiment of the invention, a priority is optionally assigned to at least one of the plurality of sub-blocks. Furthermore, the information to be transmitted includes video images and the step of dynamically assigning a communication channel resource includes assigning bandwidth on demand, in a real-time manner, for the respective video images. The information also includes speech associated with a particular speaker of a respective video image and text associated with one area of the respective video image.

Preferably, the step of segmenting includes segmenting an image/video stream into a plurality of sub-blocks based on at least one of the following characteristics: contrast, texture, edge, motion, rate of change, shape, colour, opacity, composition order, size and spatial or temporal location. The step of dynamically assigning resource preferably includes at least one of the following: user-defined parameters, all resource being allocated to the highest priority sub-block, all sub-blocks for transmission receive at least a portion of the available resource, resources are allocated by a negotiation/ bargaining process according to the respective dretermination and prioritisation of the sub-blocks, sub-blocks having a lower processing requirement are allocated resource first, and un-transmitted sub-blocks remaining from a previous unfulfilled transmission request are allocated resource first. In the preferred embodiment a block or sub-block of information includes clusters of video objects or scenes, regions of video objects or scenes, sets of video objects or scenes and/or sections of video objects or scenes. The step of determining is preferably based upon at least one of the following: rule-based, fuzzy logic, neural networks, pattern classifiers and includes analysing an available processing resource in accordance with the information to be transmitted.

In the preferred embodiment of the invention, the method further includes the step of processing the sub-block information based on a characteristic of the information, wherein the characteristic includes at least one of the following: additional spatial resolution, additional temporal resolution of an object, additional detail of one part of the object, sizel shape transformation, opacity information, compositing information, luminance and shading updates, colour changes, encoding procedure and parameters to be encoded.

In a second aspect of the present invention, a communication system for transmission of video images is provided. The communication system includes segmenting means for segmenting video images, a processor operably coupled to the segmenting means for determining the respective transmission requirements of segmented video images, and negotiation means for negotiating for a communication resource for at least one video image segment based on its respective requirements. A communication resource allocator dynamically assigns resource of a communication channel according to the respective transmission requirements and optionally user-defined and/or system defined prioritisation. Preferably the communication system further includes input means for receiving a user-defined priority of each segmented video image.

In this manner, available resource is then dynamically allocated for transmission of the video images based on at least, priorities assigned and/or resource available and/or processing requirements and/or variable data rates etc.

A preferred embodiment of the invention will now be described, by way of example only, with reference to the drawings.

Brief Description of the Drawings FIG. la shows a video scene for transmission.

FIG. ib and FIG. ic show the video scene of FIG. la segmented into data objects, according to alternative embodiments of the invention.

FIG. 2 is a block diagram of the segmentation, decision and object processing of a typical video scene, according to the preferred embodiment of the invention.

FIG. 3 is a flowchart of a method of transmitting images, according to the preferred embodiment of the invention.

Detailed Description of the Drawings Refening first to FIG. la, a video scene 10 for transmission is shown. The video scene includes motional and static objects.

FIG. ib and FIG. 1c show the video scene 10 of FIG. la, segmented into data objects, according to the preferred embodiment of the invention.

One form of segmentation, segmentation by shape, is shown in FIG. ib.

FIG. 1b shows the video scene 10 segmented by shape into three video scenes, a first video scene 20 containing the train 12, a second video scene 22 containing the bus 14 and a third video scene 24 containing the static objects of the tree 16 and the house 18. An alternative form of segmentation, segmentation by speed of movement, is shown in FIG. ic.

An image/ video source comprises a series of scenes which themselves are made up of objects. An object is preferably defined by its uses, behaviour, semantics etc. and is treated as a block of information. A block or sub-block of information includes clusters of video objects or scenes, regions of video objects or scenes, sets of video objects or scenes and/or sections of video objects or scenes. As an example, in a videotelephony scene, only a single object might be defined e.g. the speaker, or alternatively, a number of objects might be defined e.g. the speaker's eyes, mouth, hands, and background objects such as potted plants or books etc. In operation, segmentation of a scene to recover the objects is performed, for example, based on contrast, texture, edge and/or motion information, user-defined etc. In the preferred embodiment of the invention, the segmentation and/or any priorities assigned to the segments is preferably controlled and defined by user-inputs.

In operation, Object A, for example the train 12, calculates that it requires 512 bits of data in the next twenty msec interval in order to update itself as the user requires. Object B, for example the bus 14, calculates that it also requires 512 bits of data in that twenty msec interval to provide additional spatial detail required by the user. Other usage on the channel has left only 512 bits available in that twenty msec interval for both objects.

It is within the contemplation of the invention that the objects themselves define both how much capacity they require and what that capacity will be used for. Rather than a pre-defined rule-based system dynamically assigning a higher priority to one source over the other, which may no longer be relevant to the user, Objects A and B negotiate with each other over the available capacity. The outcome, i.e. what proportion is allocated to each object, will depend on the amount of bargaining resources each has at that point in time. However, in the preferred embodiment of the invention, the user is able to override such decisions, and change amounts of bargaining resources at any time. In addition, the allocation of priorities to particular objects to assist in the negotiation /arbitration process is system and/or user defined. When object priorities are system defined, the user is able to make priority inputs off-line (in a non-real-time manner), which then subsequently override the system priorities.

Preferably, if insufficient capacity is available to meet the requirements of the object(s) which win(s) the negotiation, those objects, or the user, will be required to also dynamically assign a priority order to the data they wish to transmit. If, however, an object at a future point in time has more bargaining resources than it decides it requires at that moment, it can choose to lend such resources to other objects until such time as it needs them again e.g. when the object undergoes a transformation in the scene.

The object(s) makes its own decisions about how it wants to use the capacity that it won in negotiations. These might include, inter alia, transmission of: additional spatial resolution, additional temporal resolution i.e. object updates, additional detail in one area of the object, motion information indicating a change in position of the object, size and shape transform information indicating changes in the appearance of the object, opacity information e.g. to make the object invisible at the display, compositing information to show how the object is located in the scene with respect to other objects, luminance and shading updates, colour changes, method of encoding (e.g. transform based, wavelet, fractal etc.), encoding parameters (e.g. quantisation rules, error protection etc.).

The decision methods for obtaining bargaining resources and use by objects of the bandwidth acquired may be rule based, or perhaps use some form of fuzzy logic, or incorporates decision(s) making based on neural networks or pattern classifiers etc. A rule based approach includes objects receiving bargaining resources on a "merit basis" where the user defines the importance of each object, assigns a priority and selects certain desired parameters of each object to focus on, for example spatial and temporal resolution, colour rendition etc. Objects themselves follow certain rules to determine e.g. their temporal and spatial resolution requirements, updating needs etc., and gain bargaining power from these. Capacity negotiations might then be based on "highest bidder", with the possibility of allowing all objects a minimnm share in the channel to guarantee some transmission of all objects if the user requires this.

In the preferred embodiment of the invention, the system delivers the picture quality users require for objects they are interested in. Instead of the channel allocating bandwidth to the various sources, imagel video objects negotiate with each other for the capacity that is available. Each imagel video object obtains some bargaining power based on some simple rules which include, for example, priority of the source data stream, user requirements for image/video object quality (e.g. resolution, refresh rate, definition etc.), etc.

It is within the contemplation of the invention, that the proposed technique is suitable for any data transmission application where capacity is limited, and/or fixed, and/or shared. Such capacity limits occur in, for example, ISDN lines, shared satellite channels, TETRA slots, packet based data networks etc. A typical application would be where more than one image/ video source is being multiplexed into one fixed rate data channel, where imagel video sources are sharing a data channel with other services e.g. audiol voice, other data, emergency data, error protection codes etc., or where a single image/ video source contains objects of different levels of interest to the user.

The technique is applicable to other multimedia data which are defined as objects. For example, an audio object which is linked to a video object or a text object which is to be overlaid on the displayed image. The technique is also applied to other limited resources, for example, processor power, time available on a multiplex, etc. It is equally applicable to other parameters such as user attention, memory usage.

Referring now to FIG. 2, a block diagram of the segmentation, decision and object processing, according to the preferred embodiment of the invention, is shown. A video scene 32 is input to the segmentation element 30 where individual elements of the video scene 32 are extracted into a number of segmentation objects (Oi) 34, as shown in FIG. 2. Each object has knowledge about itself which determines the bandwidth which it needs in order to be transmitted, for example such knowledge includes: size, spatial location, texture (e.g. pattern complexity and resolution), motion (e.g. direction and speed), rate of change and/or update, shape, composition order (e.g. from a chromakey), opacity (e.g. from mixing/editing functions) and/or colour and colour variations. From their self-knowledge, the objects determine their individual demands 42 (di) on thechannel. The arbitration forum 40 allows the objects to negotiate to gain their demands 42 according to the amount of bargaining power 44 each has, and the availability of scarce resources 46 such as: channel capacity at that time, processing power, memory, time sharing on a multiplex etc. Demands 42 (di) will change in time as objects change in the source scene.

Examples of bandwidth demands from an individual video object include: (i) a requirement for P bits in a t msec interval (for example a car moving slowly across a scene requires shape and texture information to be transmitted first so that it can be reconstructed at the receiver), (ii) no new bit allocation (the car is now stationary and nothing has moved in front of it to obscure it), (iii) pv bits to send a motion vector (the car starts moving again with a simple translational motion), (iv) Ps bits to signal a shape change (the car moves away from the camera so appears to be getting smaller), (v) Q bits (Q P) as a new object has appeared (someone previously obscured has got out of the car) and/or (vi) no further bit allocations (the car has driven out of the scene).

The set of bargaining powers 44 (Ri) are acquired by each object in a number of ways, which include: (a) user intervention e.g. user clicks on objects and assigns a priority list, (b) system defined e.g. by object size (for example a vehicle has priority over a dog), speed of motion (for example a fast moving person could be a threat in a personal safety situation), colour (for example cars other than blue ones should not enter the car park), (c) feature defined e.g. by object resolution (for example lettering on a vehicle in a surveillance operation must be transmitted with a lot of bandwidth for the user to derive evidence), variance of colour (for example small colour dynamic ranges require less bits per colour word), spatial frequency content etc., and/or (d) class defined e.g. motion vectors defined as a class of higher importance (and so receive more bargaining power) than a class defined of high frequency discrete cosine transform (DCT) coefficients.

The assignment of the bargaining power set 44 (Ri) could also take into account spatial resolution required by the user, temporal resolution required by the user, shape and/or texture and/or colour definition required by the user, amount and significance of detail in the object compared to other objects in the scene, amount and significance of temporal change in the object etc. and/or a default value(corresponding to a predetermined pattern). In addition, the bargaining power 44 (Ri) can be varied at any time depending on the user's preference, changes to the object, scene changes etc.

In this manner, all objects determine their demands for data allocation for transmission, and required processing power. At any time instant, the amount of available capacity in the channel is known, as well as processing power available and other critical information such as timing (not all objects can use their allocation in the same time interval).

The output from the arbitration forum 40 is a series of decisions 48 (Di).

FIG. 2 highlights some of the format opportunities 50 available for the decisions of each object. These include compress by lossy DCT encoding 58, compress by lossy wavelet encoding 62, application of a high level error protection on the video data 60, provision of a shape code 66 and/or provide sub-band decomposition 64. The object bit stream 56 (Bi) is then transmitted in the chosen format.

In this manner, each object, from say a video scene, is allocated a priority and uses any available resource negotiated for within the arbitration forum for transmission purposes.

Each object requires a certain amount of processing (measured in time, instructions or million instructions per second (MIPS)) prior to transmission. This again is governed by factors such as object size, coding scheme and user preferences . Although the desired amount of processing is a function of the result of the bandwidth negotiation, it also has implications on the bandwidth negotiation itself. It may not be possible to split the available bandwidth up into the proportions desired by the bandwidth negotiation if the processing power cannot achieve this in an acceptable time. Hence the DSP should be considered both as a criterion for the objects and a resource to be bargained for, and should ideally be included in the resource bargaining part of the algorithm combined with the bandwidth negotiation. However it is occasionally more practical to carry out the DSP bargaining after the bandwidth negotiation stage to limit the processing overheads incurred in the resource allocation parts of the algorithm.

It is within the contemplation of the invention that it is not limited to video objects, although these have been used for illustration and explanation of the concept. The invention is also applied wherever data is grouped into sets with a certain meaning that allows them to be defined as objects. An example of a time varying audio object would be speech associated with a particular speaker in a multimedia transmission. A text object would include overlay characters associated with one area of an image.

Hence, application of the invention is beneficial wherever scarce resources need to be secured. In multiplex systems where for example, several data streams are being multiplexed into one channel, a second layer of negotiation (arbitration) can be added. In this, each user's arbitration forum assesses total demands made from all objects, and requests this allocation from the shared network (d). It has bargaining power (R) which could be dependent upon, for example, the importance the user has given to that particular video stream, the type of data carried on that stream (e.g. audio having priority over video), etc. The network arbitration process allocates decisions (D) to each user, which is then shared among the individual objects as described above.

Referring now to FIG. 3, a flowchart showing a method for transmitting images, according to the preferred embodiment of the invention, is provided. The method include segmenting a block of information, for example video images, into a plurality of sub-blocks, as shown in step 102, based on at least one of the following characteristics: contrast, texture, edge, motion, rate of change, shape, colour, opacity, composition order, size and spatial location. The respective requirements for transmission of respective ones of the plurality of sub-blocks are determined, as in step 104 and a priority assigned to each one of the plurality of sub-blocks, as shown in step 106. A resource of a communication channel is dynamically assigned as in step 108 according to the respective requirements and prioritisation and respective sub-blocks on the respective dynamically assigned resource transmitted, as shown in step 110.

Preferably a block or sub-block of information includes clusters of video objects or scenes, regions of video objects or scenes, sets of video objects or scenes and/or sections of video objects or scenes. Furthermore, the-resource preferably includes bandwidth on demand for the allocation of the respective video images.

In the preferred embodiment of the invention, important variables such as, for example, encoding method and bit usage are dynamically determined by the objects themselves. The preferred embodiment of the invention proposes an entirely new model of data bandwidth sharing that ensures the system delivers the picture quality users require for objects they are interested in. Instead of the channel allocating bandwidth to the various sources, image/video objects negotiate with each other for the capacity that is available. Each image/video object obtains some bargaining resources based on some simple rules which include, for example, priority of the source data stream, user requirements for image/video object quality (e.g. resolution, refresh rate, definition etc.), etc.

The method is suitable for any data transmission application where capacity is limited and/or share, such as ISDN lines, shared satellite channels, TETRA slots, packet based data networks etc. where more than one image/video source is being multiplexed into one fxed rate data channel, where image/video sources are sharing a data channel with other services e.g. voice, other data, emergency data, error protection codes etc., and/or where a single image/video source contains objects of different levels of interest to the user.

Thus, a communication system and method of transmitting information are provided giving an improvement over current technologies.

Claims (14)

Claims

1. A method of transmitting information comprising the steps of: segmenting a block of information into a plurality of sub-blocks; determining respective requirements for transmission of respective ones of the plurality of sub-blocks; negotiating between the plurality of sub-blocks for communication resource; assigning dynamically a communication resource according to the respective negotiation process; and transmitting respective sub-blocks on the respective dynamically assigned resource.

2. A method of transmitting information according to claim 1 wherein the information to be transmitted includes video images and the step of dynamically assigning a communication channel resource includes assigning bandwidth on demand for the respective video images.

3. A method of transmitting information according to claims 1 or 2, wherein the information further includes at least one of: speech associated with a particular speaker of a respective video image, text associated with one area of the respective video image.

4. A method of transmitting information according to any one of the preceding claims, further comprising the step of assigning optionally a priority to each one of the plurality of sub-blocks.

5. - A method of transmitting information according to any one of the preceding claims wherein the step of segmenting includes segmenting a picture into a plurality of sub-blocks based on at least one of the following characteristics: contrast, texture, edge, motion, rate of change, shape, colour, opacity, composition order, size and spatial location.

6. A method of transmitting information according to any one of the preceding claims wherein the step of dynamically assigning resource includes at least one of the following parameters: all resource being allocated to the highest priority sub-block, all sub-blocks for transmission receive at least a portion of the available resource, subject to the outcome of a negotiation/bargaining process, resources are allocated according to the respective determination and prioritisation of the sub-blocks, sub-blocks having a lower processing requirement are allocated resource first, and sub-blocks remaining from a previous unfulfilled transmission request are allocated resource first.

7. A method of transmitting information according to claim 6 wherein the step of dynamically assigning resource includes allocating a processing resource and the parameter is user defined.

8. A method of transmitting information according to any one of the preceding claims wherein the step of determining is based upon at least one of the following: rule-based, fuzzy logic, neural networks, pattern classifiers.

9. A method of transmitting information according to any one of the preceding claims wherein the step of dynamically assigning resource is performed in a real-time manner.

10. A method of transmitting information according to any one of the preceding claims further comprising the step of: processing the sub-block information based on a characteristic of the information, and wherein the characteristic includes at least one of the following: additional spatial resolution, additional temporal resolution of an object, additional detail of one part of the object, size/ shape transformation, opacity information, compositing information, luminance and shading updates, colour changes, encoding procedure and parameters to be encoded, amplitude and dynamic range of the object.

11. A communication system for transmission of video images comprising: segmenting means for segmenting video images; a processor operably coupled to the segmenting means for determining the respective transmission requirements of video image segments; negotiation means for negotiating for a communication resource for at least one video image segment based on its respective requirements; and communication resource allocator for dynamically assigning resource of a communication channel according to the negotiation process.

12. A communication system for according to claim 11, further comprising: input means for receiving a priority of at least one video image segment, wherein negotiating for a communication resource for video image segments includes the priority of the at least one video image segment.

13. A communication system substantially as described herein with respect to FIG. 1 or FIG 2 of the drawings.

14. A method of transmitting information substantially as described herein with respect to FIG. 3 of the drawings.