GB2435141A

GB2435141A - Adaptive re-framing of images

Info

Publication number: GB2435141A
Application number: GB0602870A
Authority: GB
Inventors: Michael James Knee
Original assignee: Snell and Wilcox Ltd
Current assignee: Snell Advanced Media Ltd
Priority date: 2006-02-13
Filing date: 2006-02-13
Publication date: 2007-08-15
Also published as: GB0602870D0

Abstract

The invention relates to the adaptive re-framing of pictures in which a region of interest in a picture is determined and the selected region of interest is re-sized. The region of interest may be determined based on a motion analysis or segmentation of the picture, or by prediction from a determined region of interest in another picture of a sequence of pictures. The region of interest may also be defined based on camera metadata or a measure of confidence or probability that a picture element or region belongs to the region of interest. The shape and/or size of the region of interest may be dependent on the shape and/or size of a target identified within the image. The size of the region of interest may be selected such that the size of the target remains constant. The pictures may be images in a video sequence and the re-framed pictures may be compression coded and transmitted for display on a mobile phone or PDA.

Description

1 2435141

ADAPTIVE RE-FRAMING

The present invention relates to the adaptive re-framing and/or re-sizing of pictures.

BACKGROUND OF THE INVENTION

Increasingly, pictures are being captured in wide-screen or High Definition (HDTV) formats in order to provide a higher quality picture for viewers. In the future it is envisaged that it will be desirable to transmit High definition or standard definition (SD) pictures to small screen devices such as mobile telephones and Personal Digital Assistants (PDAs), for example over a wireless communication channel. A picture framed for display on a full-size screen may not be optimal for display on a small-screen device. In addition only a restricted bandwidth may be available for the transmission of the picture. In such situations it has been recognised to be desirable to select only portion of an original picture for presentation to the viewer.

SUMMARY OF THE INVENTION

Accordingly the present invention seeks to provide methods and apparatus enabling improved picture quality adaptive re-framing of pictures.

According to one aspect of the present invention, there is provided a method of re-framing a picture comprising the steps of: determining a region of interest in a picture; and re-sizing the selected region of interest.

BRIEF DESCRIPTION OF THE DRAWINGS

For a better understanding of the present invention, and to show how it may be brought into effect, reference will now be made, by way of example, to the accompanying drawings in which: Figure 1 shows a block diagram illustrating adaptive re-framing in accordance with a first embodiment; Figure 2 shows more general representation of one aspect of the invention; and Figure 3 shows a block diagram illustrating adaptive re-framing in accordance with a second embodiment.

DETAILED DESCRIPTION OF THE INVENTION

Generally, it is envisaged that the present invention will be implemented to facilitate the transmission of content, such as live sports programs, to small-screen devices such as mobile phones and PDAs, and the invention will now be described in this context. However, it will be apparent that the invention can also be applied to other situations, and so the invention is not intended to be limited to this application.

An exemplary embodiment of the invention will now be described with reference to Figure 1, which shows a block diagram illustrating adaptive re-framing in accordance with an embodiment.

The input picture sequence 2 is applied to a motion analysis block 4, which measures and analyses the motion present in the picture.

As will be known to a skilled person there are many known techniques, such as block-matching or phase correlation, for measuring motion in areas of a picture that is part of a moving picture sequence. Typically a number of motion vectors forming a motion vector field will be established, where the motion vectors represent the direction and magnitude of motion in areas of the picture. Motion vectors may be calculated for each pixel of the picture, but more commonly will be calculated for a group of pixels, such as a block of pixels.

Once motion in the picture has been established, the motion is analysed in order to determine a region of interest. Many picture sequences and in particular pictures of sporting events will contain a single object of interest in the foreground moving across a background. Examples of such a picture sequence would be a picture sequence tracking a show-jumping horse, or a single or group of racing horses or cars or people, for example.

Analysis of the measured motion pattern of a picture in such a picture sequence would be expected to show stationary or substantially uniform motion (if the camera is panning) of the background and a different generally uniform motion of the foreground (for example for racing cars) or a different more complicated motion pattern of the foreground (for example for racing horses or people). Thus an analysis of the motion pattern within a picture can be used to identify a region of interest including the foreground and to generate region of interest information 6.

Region of interest information may in principle be represented in any fashion and may describe a region of interest of any shape. However, in general it is expected that the region of interest information will describe a rectangular region of interest.

In some embodiments the region of interest information 6 may be temporally smoothed and advanced in time relative to the picture. Picture-by-picture variations in the shape or size of the region of interest can thus be smoothed, enabling an effective simulation of a camera that pans and zooms smoothly. If the region of information is temporally advanced slightly the region of interest will track slightly in advance of the action, simulating the action of a skilled camera operator.

Region of interest information 6 generated by the motion analysis block 4 is passed from the motion analysis block 4 to a window calculation block 8, which can then calculate co-ordinates 10 of the desired window to be selected from the input picture. The window calculation block 8 may also be supplied with picture limit information 12 to ensure that the re-framed window remains within the original picture and/or conforms to the size or aspect ratio of the required output picture.

The window co-ordinates 10 calculated by the window calculation block 8 are supplied to the re-scale window block 14. The re-scale window block 14 is also supplied with the input picture 2. The re-scale window block 14 selects the portion of the picture identified by the window co-ordinates 10, re-sizes as necessary, and supplies an output picture 16.

Typically the output picture 16 may be compression coded, for example MPEG2 compression coding and then formatted for transmission to a small screen device such as a mobile phone or pda.

Alternative embodiments may also include additional information to define a region of interest and/or to calculate the window co- ordinates as shown in dotted lines in Figure 1. In particular, in some embodiments, camera metadata, such as information relating to the pan and/or tilt and/or zoom and/or movement of the camera, for example, which provide a priori motion knowledge, may be used by the motion analysis block 4 in determining region of interest information. Additionally or alternatively camera metadata may be supplied to a camera metadata processor 18 to generate region of interest information 20 to be input to the window calculation block 8. Additionally or alternatively in some embodiments foreground/background segmentation information may be used in segmentation block 22 to generate additional region of interest information 24 for supply to the window calculation block 8.

If the window calculation block 8 receives a region of interest information from more than one source (for example region of interest information 6, 20, 24) the window calculation block 8 may based the determined window co-ordinates 10 based on any combination of the received region of interest information.

It should be noted that the region of interest information, whether derived from the motion analysis block, segmentation block 22 or camera metadata processor 18, may in some embodiments be information that provides a measure of the confidence or a probability measure for the inclusion of each pixel or region of pixels in the region of interest.

A more general representation of one aspect of the present invention is shown in Figure 2. Figure 2 is very similar to Figure 1 and as the same reference numerals refer to the same or similar components and further description of these same or similar components will be omitted for brevity.

In Figure 2 the motion analysis block 4 has been replaced with a general region of interest determination block 26. The motion analysis carried out by motion analysis block 4 of Figure 1 as described above in connection with Figure 1 is one method of determining a region of interest.

In one alternative embodiment the region of interest determination is based on establishing a definition of the region of interest for at least one picture in a sequence of pictures. The region of interest information for each subsequent picture in the sequence can be derived by spatial and/or temporal interpolation of the region of interest defined for a preceding picture. If interpolation is used to define the region of interest, preferably information from pictures both before and after the current picture in the sequence may be used. Although the use of information from pictures after the current picture in the sequence requires pictures to be re-ordered and introduces delay, the resulting interpolation is likely to be more reliable.

The definition of the region of interest in a picture, which then serves as the basis for interpolation of the region of interest in subsequent pictures, may be established in a number of ways. In one embodiment the region of interest may be defined by human intervention. In other embodiments automatic or semi-automatic techniques may be used. In particular, segmentation techniques such as foreground/background segmentation techniques may be used to define a region of interest.

The definition of the region of interest in a picture may be established twice in each picture sequence, for example before and after every shot change, or may be established periodically, for example every n frames.

Alternatively segmentation techniques may be used in every picture to define a region of interest therein.

Figure 3 shows a further aspect of the present invention. Again, Figure 3 is similar to Figure 1 and as the same reference numerals refer to the same or similar components, further description of these same or similar components will be omitted for brevity.

Figure 3 shows an arrangement that is particularly adapted to re-framing of pictures where the region of interest is likely to be centred on a particular object, which may move within the picture. This is particularly applicable to ball-based sports such as football or golf.

In this embodiment a ball tracker block 28 is provided, which analyses the input pictures and provides ball centre information 30 and ball size information 32 to the window calculation block 8. Various techniques for providing this information are known to a skilled person and so these techniques will not be described in any more detail.

The window calculation block 8 determines the window co-ordinates io based on the ball centre information 30 and ball size information 32, possibly in conjunction with region of interest information 20, 24 from the camera meta-data processor 18 and segmentation block 22 respectively.

In embodiments of the invention the window calculation block 8 determines the window co-ordinates io to ensure, so far as is possible, that the size of the ball in the picture is constant, and the ball is located within a predefined area of the output picture. Thus for example the output picture may generally be centred on the ball.

The remaining operation of this embodiment is the same as described above with reference to Figure 1.

Additionally or alternatively additional processing of the picture may be carried out so as to alter the size and/or colour of part of the picture. In particular, in ball-based sports the ball may be made bigger and/or coloured a different colour or otherwise enhanced to increase its visibility. This technique enhances the contrast between the ball and the background action, enabling action to be followed more easily on a small screen. This technique also reduces the possibility that restrictions in the bit rate available to send the picture to the viewer, for example over a wireless link to a mobile phone, will result in the ball disappearing in the received picture.

The present invention has been described with reference to an uncompressed video signal. However, the invention may also be applied to a compressed or partially uncompressed video signal.

The present invention thus provides improved techniques for re-framing pictures.

The invention has been described herein by way of example and variations may be made to the described arrangements without departing from the scope of the invention. The invention encompasses methods and apparatus embodying the techniques described herein and may be implemented in hardware and/or software as will be apparent to a skilled person.

Claims

CLAIMS

1. A method of re-framing a picture comprising the steps of: determining a region of interest in a picture; and re-sizing the selected region of interest.

2. The method of re-framing a picture as claimed in claim 1 wherein the region of interest is determined based on a motion analysis of the picture.

3. The method of re-framing a picture as claimed in claim 2 wherein the region of interest is determined based on the steps of measuring motion; establishing a motion pattern of the measured motion; determining the region of interest dependent on the established motion pattern.

4. The method of re-framing a picture as claimed in any preceding claim wherein the region of interest is determined based on prediction from a determined region of interest in at least one other picture of a sequence of pictures.

5. The method as claimed in claim 4 wherein the prediction is based on temporal and/or spatial interpolation.

6. The method of re-framing a picture as claimed in any preceding claim wherein the region of interest is determined based on a segmentation of the picture or at least one other picture in the sequence of pictures.

7. The method of re-framing a picture as claimed in any preceding claim wherein the region of interest is defined dependent on camera metadata.

8. The method of re-framing a picture as claimed in any preceding claim wherein the step of determining a region of interest in a picture comprises the steps of:identifying a target; identifying at least one of the size and/or shape of the target; and determining the size and/or shape of the region of interest dependent on the size and/or shape of the target.

9. The method as claimed in claim 8 wherein the region of interest is substantially centred on the target.

10. The method as claimed in claim 8 or 9 wherein the size of the region of interest is selected such that the size of the target remains generally constant.

11. The method as claimed in any preceding claim wherein the region of interest is determined based on a measure of confidence or probability that a picture element or region belongs to the region of interest.

12.A method of re-purposing a video signal comprising the steps of: re-framing the pictures of the video signal as claimed in any preceding claim; and compression coding the resulting picture.

13.A program carrier storing processor-implementable instructions for carrying out the method of one of claims 1-11.

14. Apparatus for image analysis adapted to carry out the method of one of claims 1-11.