GB2553125A

GB2553125A - Comparing video sequences using fingerprints

Info

Publication number: GB2553125A
Application number: GB1614435.4A
Authority: GB
Inventors: Diggins Jonathan
Original assignee: Snell Advanced Media Ltd
Current assignee: Snell Advanced Media Ltd
Priority date: 2016-08-24
Filing date: 2016-08-24
Publication date: 2018-02-28
Anticipated expiration: 2036-08-24
Also published as: GB2599877B; GB2599877A; EP3287946A1; US10395121B2; US20190377954A1; US20180060667A1; GB201614435D0; GB2553125B; US11222209B2

Abstract

To compare the fingerprints of two video streams of differing image repetition rates (101, 103, Fig. 1) a converted sequence of fingerprints is generated from the first or second sequence of fingerprints by combining the values of neighbouring fingerprints (111, Fig. 1). A correlation is then performed between the converted sequence of fingerprints and the non-converted sequence of fingerprints 815. Spatial fingerprints, temporal fingerprints or a combination of the two are compared. Temporal fingerprints may be generated by dividing the image into blocks and comparing luminescence (Fig. 6). Spatial fingerprints may be generated by discarding the edge of the image; dividing the remaining image into blocks; and averaging the luminescence of the blocks (Fig. 7). The combination of two fingerprints may be a weighted average based on the temporal location of the neighbouring fingerprints. Shot changes may be detected by threshold comparison and fingerprints of the shot change ignored; given less weighting when being combined with other fingerprints; or replaced with a fingerprint taken before the shot change. Cadence may also be detected by detecting temporal fingerprints with a zero change and replacing the fingerprint with an estimated value or not using the fingerprint in the combining operation.

Description

(71) Applicant(s):

Snell Advanced Media Limited

Turnpike Road, NEWBURY, Berkshire, RG14 2NX,

United Kingdom (72) Inventor(s):

Jonathan Diggins (56) Documents Cited:

EP 2077663 A1 WO 2007/148290 A2 (58) Field of Search:

INT CL G06K, G06T, H04N Other: WPI, EPODOC

WO 2010/046990 A1 US 20080007614 A1 (74) Agent and/or Address for Service:

Mathys & Squire LLP

The Shard, 32 London Bridge Street, LONDON, SE1 9SG, United Kingdom (54) Title of the Invention: Comparing video sequences using fingerprints

Abstract Title: Comparing video sequences of differing image rates using fingerprints (57) To compare the fingerprints of two video streams of differing image repetition rates (101, 103, Fig. 1) a converted sequence of fingerprints is generated from the first or second sequence of fingerprints by combining the values of neighbouring fingerprints (111, Fig. 1). A correlation is then performed between the converted sequence of fingerprints and the non-converted sequence of fingerprints 815. Spatial fingerprints, temporal fingerprints or a combination of the two are compared. Temporal fingerprints may be generated by dividing the image into blocks and comparing luminescence (Fig. 6). Spatial fingerprints may be generated by discarding the edge of the image; dividing the remaining image into blocks; and averaging the luminescence of the blocks (Fig. 7). The combination of two fingerprints may be a weighted average based on the temporal location of the neighbouring fingerprints. Shot changes may be detected by threshold comparison and fingerprints of the shot change ignored; given less weighting when being combined with other fingerprints; or replaced with a fingerprint taken before the shot change. Cadence may also be detected by detecting temporal fingerprints with a zero change and replacing the fingerprint with an estimated value or not using the fingerprint in the combining operation.

Figure 8

Low(er) image rate ’ video source

Shol Change Cadence

Detection and Detection and

Adding Scaiing

Cadence -> Fingerprint Compensation Compensation Detection Fiag Values

Forming Converted Sequence o; Fingerprints iemporal Identification

Filtering of Optimum

High(er) Shot Change . Cadence image rate ——-¼ Detection and —> Detection and video source Compensation Compensation

803

807

817

819

Figure

109 i·^ o

o

1—f co o

2/9

Figure

111 “-i o v \

o

CD

Figure 3

-£° jap

CD

Φ u 4-J O« ω is o ω */>

SP o £ <u

E o >

OJ ao ro di o

Oi

T3 >

a>

+-i ro a>

Ox

S»

Σ3

O l/i

4/9

411

401

5/9

501

505

CO ο

νθ

609

Ο

5.0

7/9

Γχ <U

V ho

705 co o

o r^·

8/9 fC <u □f ho g §

CO σι o

LD

O o

k A

GJ u

fO £ „ 0 co

CD v>

£ Ό >

J0 &p ±

GJ +u

Π3

Sw_

GJ ao rc £

OJ

U)

Ξ

ΙΛ

GJ

Έ ’>

Γ'*'· o

CO o

Media match verification Lipsync measurment

COMPARING VIDEO SEQUENCES USING FINGERPRINTS

This invention relates to the control and monitoring of media workflows and in one example to the comparison of video for purposes such as monitoring delay and confirming content.

Many international video standards are in everyday use around the world, which have different image rates. In an increasingly automated world, intelligent monitoring of ‘live’ system workflows in which media may originate from different video standards, without the intervention of human operators, is a highly desirable capability and a commercial driving force. The problem is that, in order to do this, media identification, system delay and lip-sync monitoring applications need to be capable of accommodating comparisons between different video image rates, across multiple test points at physically separated positions along the broadcast chain, and on a large number of channels. There is currently no industry monitoring solution available to do this.

The idea of using video fingerprints (which are also sometimes referred to as ‘signatures’, or ‘hash’ values) in order to characterise the video content dynamically with a low-bandwidth representation, is not new. Fingerprints from different test points can usefully be transmitted to a central application for correlation. But existing systems, including such fingerprinting approaches, deal only with comparisons between video standards with the same image rate, or involve invasive processes, such as the insertion of appropriate test stimuli. These kinds of system address the problem to a degree, but are only useable in offline, set-up scenarios. They are not suitable for dynamic, noninvasive monitoring.

In one embodiment a method of comparing a first sequence of fingerprints representing a first sequence of video images at a first image repetition rate with a second sequence of fingerprints representing a second sequence of video images at a second image repetition rate, different from said first image repetition rate is devised. The fingerprints may be spatial fingerprints derived each from the content of a single video image, temporal fingerprints derived each from a difference in content between two successive video images or a combination of spatial and temporal fingerprints. Embodiments of such a method may comprise the steps of generating from the first sequence of fingerprints a converted sequence of fingerprints at said second image repetition rate and performing a correlation between said converted sequence of fingerprints and said second sequence of fingerprints. In some embodiments generating the converted sequence of fingerprints comprises, for a fingerprint in the converted sequence, forming a combination of the values of a number of neighbouring fingerprints in the first sequence of fingerprints. In other embodiments apparatus, systems and computer products are envisaged that may perform such methods.

The invention will now be described by way of example with reference to the accompanying drawings, in which:

Figure 1 is an illustration of a high frequency sequence of fingerprints and a low frequency sequence of fingerprints.

Figure 2 is an illustration of the conversion of a low frequency sequence of fingerprints to a converted sequence of fingerprints with a higher frequency.

Figure 3 is an illustration of a low frequency sequence of temporal fingerprints and a high frequency sequence of temporal fingerprints and the difference in the average values of said fingerprints.

Figure 4 is an illustration of the conversion of a sequence of images from a first frequency to a second higher frequency through the introduction of cadence by repeating images a set number of times in the higher frequency sequence.

Figure 5 is an illustration of the detection of a shot change in a low frequency sequence of fingerprints and a high frequency sequence of fingerprints.

Figure 6 is an illustration of the generation of a temporal fingerprint for an image from the image and the image immediately preceding it in a video sequence.

Figure 7 is an illustration of the generation of a spatial fingerprint from an image.

Figure 8 is a flow diagram showing the method of comparing a high frequency sequence of fingerprints and a low frequency sequence of fingerprints to determine if they correlate with one another.

Figure 9 is a flow diagram showing the use of the correlation of a sequence of high frequency fingerprints and a sequence of low frequency fingerprints in determining if the content of European 50Hz content is the same as the content of US 59.94Hz content.

Figure 1 shows a first sequence of fingerprints 103 with a low frequency representing a first sequence of video images at a first image repetition rate. Figure 1 also shows a second sequence of fingerprints 101 with a high frequency representing a second sequence of video images at a second image repetition rate. The second image repetition rate is different from the first image repetition rate and in this example the second image repetition rate is higher than the first image repetition rate.

The fingerprints each represent an image from an image sequence. The fingerprints may be either temporal, or spatial, or a combination of the two. A temporal fingerprint is based on a measure of the difference between an image and the image that immediately precedes it (or a selected other image such as the image that immediately precedes it with the same parity). The generation of the fingerprints from their corresponding image is described with reference to Figures 6 and 7.

Arrow 105 is the axis of time. Figure 1 therefore shows that the low frequency sequence of fingerprints 103 is periodic and repeats with an interval shown by arrow 109. The high frequency sequence of fingerprints 101 is also periodic and has a period shown by the second arrow 107.

Comparing the two sequences of fingerprints directly will not give an accurate reflection of the correlation between the two sequences. In any comparison at least some of the fingerprints would have an offset in position in the sequence from one another, and the average values of the fingerprints in the two sequences may be different. Moreover shot changes and cadence in one or both of the sequences may mean that any attempt to correlate the two sequences would lead to an inaccurate result.

Figure 2 shows the low frequency sequence of fingerprints 103 representing a low frequency sequence of video images at a first image repetition rate. Figure 2 also shows a converted sequence of fingerprints 111. This sequence 111 has a higher frequency and image repetition rate than the low frequency sequence of fingerprints 103. The converted sequence of fingerprints 111 can be generated from the low frequency sequence of fingerprints 103. The generation may comprise forming a combination of the a number of neighbouring low frequency fingerprints 103 to form a converted fingerprint 111. Arrows 113 show which of the low frequency fingerprints 103 neighbour each converted fingerprint 111. Each converted fingerprint 111 is a combination of the low frequency fingerprints 103 that are connected to it via an arrow 113.

One way of generating the converted sequence of fingerprints is to form a combination of the values of the two values of the neighbouring fingerprints in the first sequence of fingerprints 101.

In one embodiment this generation of the converted sequence of fingerprints 111 may be achieved by performing an average of two neighbouring fingerprints. In one embodiment this may be a weighted average, and preferably the weighting of this average may be dependent upon how close each of the neighbouring first fingerprints is to the position of the converted fingerprint. Other methods of generating the converted fingerprints may be used.

Figure 2 shows an arrow 115 that links a low frequency fingerprint 103 with a converted fingerprint 111. These fingerprints are at the same position as one another on the time axis. In some embodiments the value of this converted fingerprint is taken as being the value of the low frequency fingerprint at this same time without performing an average. This is shown in Figure 2 as this converted fingerprint only has one arrow linking it to a single neighbouring fingerprint from the low frequency fingerprint sequence 103.

Each fingerprint from the low frequency sequence may be the neighbour to a plurality of converted fingerprints. This is shown by arrows 117 that show a low frequency fingerprint being a nearest neighbour to three separate converted fingerprints. Two converted fingerprints in the sequence have the same two neighbouring low frequency fingerprints as one another. In some embodiments these will still have different values as the weighting of each neighbour will vary dependent upon the position of each converted fingerprint in the sequence.

In one embodiment low frequency sequence 103 is the same low frequency sequence 103 that is shown in Figure 1. In some embodiments the converted sequence of fingerprints 111 has the same repetition rate and period as high frequency sequence of fingerprints 101 shown in Figure 1. Preferably the converted sequence of fingerprints 111 is also in phase with the second sequence of fingerprints 101. This may not always be the case, however it may be advantageous. The converted sequence of fingerprints may be generated in a generation module.

In one embodiment in which the fingerprints are temporal fingerprints the values of the first sequence of fingerprints may be scaled. Scaling is discussed below with reference to Figure 3. This scaling may be done before the converted sequence of fingerprints is generated. In another embodiment the first sequence of fingerprints is not scaled, and instead the converted sequence of fingerprints is scaled. In this embodiment the same scaling factor is used as in the embodiment above. In another further embodiment no scaling is performed on either first sequence of fingerprints or the converted sequence of fingerprints. In some cases this may be due to the detection of cadence as discussed below in relation to Figure 4.

Figure 3 illustrates the low frequency sequence of fingerprints 103, and the high frequency sequence of fingerprints 101. In the embodiment illustrated in Figure 3 these fingerprints are temporal fingerprints. The generation of a temporal fingerprint is illustrated in Figure 6. These fingerprints are each associated with an image and are derived from the temporal difference between the image and the one preceding it. The temporal difference between the images in the high frequency sequence 101 is depicted by 4_low305. The temporal difference between the images in the low frequency sequence is depicted by4_HiGH307.

As the low frequency sequence of fingerprints 103 and the high sequence of fingerprints 103 have different repetition rates the difference between successive images in the image sequence the fingerprints represent will be different. A larger amount of time between images in an image sequence will make it more likely that, on average, the difference between successive images is greater. Therefore4_LOW305 will on average be greater than 4_High 307.

So on average the values of the temporal fingerprints making up low frequency sequence of fingerprints 103 will be larger than the values of the fingerprints making up high frequency sequence of fingerprints 101.

When comparing video image sequences with different image rates using temporal fingerprints it is thus advantageous to scale the fingerprints appropriately so that they have a similar average magnitude before performing any further processing. In one embodiment this can be done by noting that the equation 1 below holds:

^high ~ 4_lowP^er'^OdHIGH ~ P^er'^OdLOW

This can be manipulated to form equation 2:

4HIGH ~ 4_LoW(7 ) /HIGH

This equation can be used to scale the temporal fingerprints of the video image sequences with the lower image repetition rate by multiply it by the ratio of the two image rates.

Figure 4 shows one example of the introduction of cadence into a sequence of video images. If a video is shot at a certain image rate (for example 24Hz) but is shown to audiences at another image rate then cadence may be introduced. This is the introduction of duplicate images into the video sequence so that some of the images repeat themselves. Common forms of cadence include 3:2 cadence (shown in Figure 4) where images are alternately repeated three times and then two times. This may raise the image rate of a video sequence form 24 Hz to 60Hz. Another common form of cadence is 2:2 In this instance video recorded at 24Hz is “played fast” at 25Hz and then each image is repeated. This creates a video with an image rate of 50Hz.

Figure 4 shows a sequence of images 401. This has a period shown by arrow 413. The images are replicated (as shown by arrows 405 and 407). This is done in a periodic fashion. In this case a first image is repeated twice, and then a next image is repeated once, and then the next twice and so on. This produces a new sequence of images 403 in which cadence has been introduced. This new sequence of images has a different period 411.

Temporal fingerprints measure the difference between an image in a video sequence and the image that immediately precedes it (or the image immediately preceding it with the same parity) in the same video sequence. If cadence has been introduced to a video sequence, as shown in Figure 4, then in some cases an image will be immediately preceded by an identical image. This means that the temporal fingerprint of the second image will have a value of zero or close to zero.

Some embodiments, described below, are envisaged in which cadence is detected in either of the first sequence of fingerprints 103, or the second sequence of fingerprints 101, or both.

For example, a method is contemplated in which cadence is detected of any 3:2 or 2:2 repetition of video images resulting from a previous film to video conversion process. The temporal fingerprints derived from these repeating video images are then replaced by a neighbouring non-zero temporal fingerprint, or another estimated value. This method may be used in conjunction with the method outlined for comparing a first sequence of fingerprints and a second sequence of fingerprints outlined with reference to Figure 2.

The cadence detection may comprise identifying temporal fingerprints with values of zero, or close to zero. Alternatively, cadence detection may comprise identifying temporal fingerprint values of a video sequence that come in identical, or near identical sequential pairs.

In the event cadence is detected, in one embodiment, replacing temporal fingerprints derived from repeating video images by a neighbouring non-zero fingerprint or other estimated value comprises re-constituting the original temporal differences between the images. This may be done by deleting the fingerprints with a negligible value and then adjusting the positions of the remaining fingerprints in the sequence. In one embodiment this can be done by adjusting the positions of the remaining fingerprints in the sequence so that they occur at regular intervals, and so that they have an image rate of 24Hz.

As discussed above with reference to Figure 2 it is envisaged that temporal fingerprints of sequences with different repetition rates do not always have to be scaled. For instance if cadence is detected scaling is not always necessary. A video may originally have a frequency of 24Hz. Several versions of the video may be produced including a 3:2 cadence version shown at a frequency of 59.94Hz and a 2:2 cadence version shown at 50Hz. For both of these examples the temporal non-zero differences between consecutive images are the same as the original 24Hz so scaling is unnecessary. Figure 5 illustrates a first sequence of fingerprints 503, and a second sequence of fingerprints 501. Arrows 511 and 509 illustrate where a shot change takes place in the video sequences associated with these fingerprint sequences. The dashed sequences 505 and 507 are fingerprints after the shot change.

In some embodiments a method is envisaged in which shot changes are detected in the first and/or second sequence of video images and wherein temporal fingerprints derived from video images across a shot change are ignored or given reduced prominence in the correlation step. This method may be used in conjunction with the method of comparing a first sequence of fingerprints with a second sequence of fingerprints set out in relation to Figure 8.

The shot changes may be detected by identifying temporal fingerprints that are above a pre-set threshold. As the image before a shot change likely bears little or no relation to the image immediately following a shot change there is likely to be a large temporal difference between them. This leads to a large temporal fingerprint. Therefore by finding large temporal fingerprint changes the shot changes can be identified.

In some embodiments ignoring or giving reduced prominence to temporal fingerprints derived across a shot change comprises replacing the temporal fingerprints derived across a shot change with either the temporal fingerprint immediately preceding or succeeding the shot change, or a combination of the two. In some embodiments this combination may be an average of the two fingerprints.

Figure 6 illustrates a first image 601 and a second image 603. The first and second images are from the same sequence of video images. These sequences may be the sequences shown in any preceding Figure. The second image is a single video image. The first image is a previous single video image from the same video sequence. It occurs immediately before the single video image in the video sequence.

A temporal fingerprint may be formed by dividing the single video image into a number of blocks 607 and then dividing the previous single video image into the same blocks 605 as the single video image. The average luminescence of each block may then be found. This allows the corresponding blocks of the images to be compared with one another.

The differences between each of the blocks that correspond to one another are then found, and an overall difference between the images can be determined by calculating the sum of these absolute differences. This difference is the temporal fingerprint of the single video image. The fingerprints shown in Figures 1 and 2 may have been calculated this way. A fingerprint generation module may generate the fingerprints associated with an image sequence.

Arrow 609 shows that the two blocks of the images 605 and 607 correspond with one another. Arrow 611 shows that image 603 immediately follows image 601 in the video image sequence.

Figure 7 illustrates one embodiment of calculating a spatial fingerprint of an image 701. The spatial fingerprint of image 701 may be formed by discarding the edge of the image 705. This leaves only the central portion 703 of the image 701. The centre of the image can then be divided into a number of blocks. The average luminescence of these sections can then be found. In some embodiments these values may be summed or averaged to calculate a single value for the spatial fingerprint of image 701. A fingerprint generation module may generate the fingerprints associated with an image sequence.

Figure 8 shows a low frequency input of a first sequence of fingerprints 103 representing a first sequence of video images at a first image repetition rate. Figure 8 also shows a high frequency input of a second sequence of fingerprints 101 representing a second sequence of video images at a second image repetition rate. The second image repetition rate is different from the first image repetition rate. A first sequence of converted fingerprints may be generated from the first sequence of fingerprints that correspond with the first image sequence. The converted sequence of fingerprints has an image repetition rate that is the same as the second image repetition rate. A correlation may then be performed between the converted sequence of fingerprints and the second sequence of fingerprints.

Figure 8 illustrates a method of comparing a first sequence of fingerprints with a second sequence of fingerprints. In this example the first sequence of fingerprints has a low input rate (and so a low image repetition rate) and the second sequence of fingerprints has a high rate input (and so a high image repetition rate). This method may be performed on a single apparatus, or the steps may be performed on separate apparatus. Each apparatus may comprise a processor and a data storage. The processor(s) is configured to perform the steps shown in Figure 8. The data storage may record the results, or provide input data to the processor(s).

In some embodiments, and as illustrated in Figure 8, both the low rate input and the high rate input may first have shot changes detected and then compensated 801. This may done as described above with reference to Figure 5.

Optionally cadence may then be detected and compensated for 805. This may done as described above with reference to Figure 4. In the event cadence is present in the low rate input a cadence flag 809 may be used to indicate that the temporal fingerprints in the low rate input do not need to be scaled.

Optionally the low rate input may then be compensated 811. This may be done as described with reference to Figure 3.

A converted sequence of fingerprints is then formed 813. In some embodiments this may be as described with reference to Figure 2.

The converted low rate input is then correlated 815 with the high rate input. This is done to identify the similarity between the two inputs. The correlation between the converted sequence of fingerprints and the second sequence of fingerprints may comprise performing one of: a discrete correlation function, a phase correlation process, a simple sum of absolute minimum differences as a function of offset, or an alternative matching function which compares the two inputs and returns a function which has an optimum point at an offset.

The correlation 815 may be configured to measure the correlation between the inputs to sub-sample precision (i.e. a fraction of an image period). The correlation may be performed by a correlator.

Temporal filtering 817 may be used in some embodiments to ensure that over a given temporal window the correlation is accurate.

Identification of optimum 819 is used to identify the optimum point in the temporally filtered correlation function, which establishes both how strong the correlation between the two inputs is.

Steps 801, 805, 809, 811, 817 and 819 are optional steps. These may be performed as part of the method. Some of these steps may be performed and others may not. For example shot change compensation may not be performed but cadence compensation may be. Additionally in another alternative embodiment, the sample scaling and forming of the converted sequence of fingerprints may be performed on the high rate input, rather than on the low rate input. In this embodiment all of the features bar the fingerprint conversion and scaling are once more optional and any combination of them may be used.

In another embodiment the scaling and fingerprint conversion steps can be applied to both inputs. This would allow the fingerprint conversion to convert the first and second fingerprints to two sequences of fingerprints both with the same pre-set repetition rate. This may be advantageous in video libraries so that a standardised version of all the content can be produced. The scaling would then also have to be done to scale the temporal fingerprints to the pre-set standardised image repetition rate.

Figure 9 shows one implementation of the method set out with reference to Figure 8. In this example a live media, such as a sporting event is being shown around the world. Different countries use different broadcast standards therefore the broadcast must be converted to each of these different standards.

In the embodiment shown in Figure 9, the live media is first edited 901 before being delivered around the world. This is an optional step.

In this Example the live media is being sent to the USA. Therefore the image rate is converted 903 from 50Hz to 59.94Hz. The broadcast is then encoded for sending 905. This broadcast is then received in the USA and is decoded 907. The received broadcast is fingerprinted 909 to form a sequence of fingerprints corresponding to the US broadcast. The original live media is also fingerprinted 911. This allows the two fingerprint sequences to be correlated with each other 911. This is done using the method described above.

This allows quick verification of whether the content that has been sent to the US is the same content as the original live stream. This media match verification is useful for ensuring that the correct content is always being sent.

This method also allows for lip synchronisation measurements to be made to ensure that both broadcasts have an accurate lipsync. Because the images are now being broadcast with a different repetition rate the original audio may not completely match up with the new rate. This measurement aids in the determination of any problems so that a better lipsync can be added to the broadcast.

In another embodiment the method set out above may be used to compare content from a film archive. In this context cadence is more likely to be present because cadence is applied to many films. In this example, the fingerprinting of both films may be performed at the same location, and in some embodiments, by the same electronic device. In some embodiments this same electronic device may be used to perform the correlation.

It will be appreciated from the discussion above that the embodiments shown in the Figures are merely exemplary, and include features which may be generalised, removed or replaced as described herein and as set out in the claims. With reference to the drawings in general, it will be appreciated that schematic functional block diagrams are used to indicate functionality of systems and apparatus described herein. For example the steps shown in Figures 8 and 9 may be combined into single steps. These steps may also be performed on a single apparatus, or each step may be performed at a separate apparatus. The apparatus performing the method steps may include a data storage and a processor. Alternatively the functionality provided by the data storage may in whole or in part be provided by the processor. In addition the processing functionality may also be provided by devices which are supported by an electronic device. It will be appreciated however that the functionality need not be divided in this way, and should not be taken to imply any particular structure of hardware other than that described and claimed below. The function of one or more of the elements shown in the drawings may be further subdivided, and/or distributed throughout apparatus of the disclosure. In some embodiments the function of one or more elements shown in the drawings may be integrated into a single functional unit.

The above embodiments are to be understood as illustrative examples. Further embodiments are envisaged. It is to be understood that any feature described in relation to any one embodiment may be used alone, or in combination with other features described, and may also be used in combination with one or more features of any other of the embodiments, or any combination of any other of the embodiments. Furthermore, equivalents and modifications not described above may also be employed without departing from the scope of the invention, which is defined in the accompanying claims.

In some examples, one or more memory elements can store data and/or program instructions used to implement the operations described herein. Embodiments of the disclosure provide tangible, non-transitory storage media comprising program instructions operable to program a processor to perform any one or more of the methods described and/or claimed herein and/or to provide data processing apparatus as described and/or claimed herein.

The processor of any apparatus used to perform the method steps (and any of the activities and apparatus outlined herein) may be implemented with fixed logic such as assemblies of logic gates or programmable logic such as software and/or computer program instructions executed by a processor. Other kinds of programmable logic include programmable processors, programmable digital logic (e.g., a field programmable gate array (FPGA), an erasable programmable read only memory (EPROM), an electrically erasable programmable read only memory (EEPROM)), an application specific integrated circuit, ASIC, or any other kind of digital logic, software, code, electronic instructions, flash memory, optical disks, CD-ROMs, DVD ROMs, magnetic or optical cards, other types of machine-readable mediums suitable for storing electronic instructions, or any suitable combination thereof.

Claims

1. A method of comparing a first sequence of fingerprints representing a first sequence of video images at a first image repetition rate with a second sequence of fingerprints representing a second sequence of video images at a second image repetition rate, different from said first image repetition rate, in which the fingerprints are spatial fingerprints derived each from the content of a single video image, temporal fingerprints derived each from a difference in content between two video images or a combination of spatial and temporal fingerprints, comprising the steps of generating from the first sequence of fingerprints a converted sequence of fingerprints at said second image repetition rate or a common repetition rate and performing a correlation between said converted sequence of fingerprints and said second sequence of fingerprints or a second converted sequence of fingerprints generated from the second sequence of fingerprints at said common image repetition rate, where generating a converted sequence of fingerprints comprises, for a fingerprint in the converted sequence, forming a combination of the values of a number of neighbouring fingerprints in the first or second sequence of fingerprints.

2. A method according to claim 1, wherein the combination is a weighted average based on the temporal location of the respective neighbouring fingerprints in the first sequence of fingerprints with respect to the fingerprint in the converted sequence.

3. A method according to claims 1 and 2, wherein two neighbouring fingerprints are used to generate a converted fingerprint.

4. A method according to any one of the preceding claims in which a temporal fingerprint in the first sequence of fingerprints is scaled in accordance with a ratio of the first and second image repetition rates.

5. A method according to any one of the preceding claims in which shot changes are detected in the first and/or second sequence of video images and wherein temporal fingerprints derived from video images across a shot change are ignored or given reduced prominence in the correlation step.

6. A method according to claim 5, wherein the shot changes are detected by identifying temporal fingerprints that are above a set threshold.

7. A method according to claim 5 or claim 6, wherein ignoring or giving reduced prominence to temporal fingerprints derived across a shot change comprises replacing the temporal fingerprints derived across a shot change with either the temporal fingerprint immediately preceding it, the temporal fingerprint immediately succeeding it, or a combination of the two.

8. A method according to any one of the preceding claims in which the cadence is detected of any 3:2 or 2:2 repetition of video images resulting from a previous film to video conversion process and wherein temporal fingerprints derived from repeating video images are replaced by a neighbouring non-zero temporal fingerprint or other estimated value.

9. A method according to claim 8, wherein cadence detection comprises identifying temporal fingerprints with values of zero, or close to zero.

10. A method according to claim 8, wherein cadence detection comprises identifying temporal fingerprint values of a video sequence that come in identical, or near identical sequential pairs.

11. A method according to one of claims 8 to 10, wherein replacing temporal fingerprints derived from repeating video images by a neighbouring non-zero fingerprint or other estimated value comprises re-constituting the original temporal differences between the images.

12. A method according to one of claims 8 to 11, wherein in the event cadence is detected, not performing a scaling of the first temporal fingerprints in accordance with a ratio of the first and second image repetition rates.

13. A method according to any preceding claim, wherein performing a correlation between said converted sequence of fingerprints and said second sequence of fingerprints comprises performing one of: a discrete correlation function, a phase correlation process, a simple sum of absolute minimum differences as a function of offset, or an alternative matching function which compares the two inputs and returns a function which has an optimum point at an offset position.

14. A method according to claim 13, wherein the correlation ofthe converted sequence of fingerprints and said second sequence of fingerprints is configured to measure the correlation or matching function to sub-sample precision.

15. A method according to any preceding claim, wherein a temporal fingerprint is formed by dividing the single video image into a number of blocks;

dividing the previous single video image ofthe same video image sequence into the same blocks as the single video image;

determining the average luminescence of each block of both images; comparing the average luminescence of each block ofthe single video image with the corresponding blocks ofthe previous single video image.

16. A method according to any preceding claim, wherein a spatial fingerprint is formed by discarding the edge ofthe single video image, leaving only the centre ofthe single video image;

dividing the centre of the single video image into one or more blocks; and averaging the luminescence of each block.

17. A system configured to compare a first sequence of fingerprints representing a first sequence of video images at a first image repetition rate with a second sequence of fingerprints representing a second sequence of video images at a second image repetition rate, different from said first image repetition rate, in which the fingerprints are spatial fingerprints derived each from the content of a single video image, temporal fingerprints derived each from a difference in content between two video images or a combination of spatial and temporal fingerprints, comprising the steps of generating from the first sequence of fingerprints a converted sequence of fingerprints at said second image repetition rate or a common repetition rate and performing a correlation between said converted sequence of fingerprints and said second sequence of fingerprints or a second converted sequence of fingerprints generated from the second sequence of fingerprints at said common image repetition rate, where generating a converted sequence of fingerprints comprises, for a fingerprint in the converted sequence, forming a combination of the values of a number of neighbouring fingerprints in the first or second sequence of fingerprints.

18. A system according to claim 17, wherein the system comprises at least one fingerprint creation module, wherein the fingerprint creation module is configured to form the fingerprints.

19. A system according to one of claims 17 or 18, wherein the system comprises a correlator configured to perform the correlation between said converted sequence of fingerprints and said second sequence of fingerprints.

20. A system according to any one of claims 17 to 19, comprising a generation module configured to generate the converted sequence of fingerprints.

21. A system according to any one of claims 17 to 20, wherein the combination is a weighted average based on the temporal location of the respective neighbouring fingerprints in the first sequence of fingerprints with respect to the fingerprint in the converted sequence.

22. A system according to any one of claims 17 to 21, wherein two neighbouring fingerprints are used to generate a converted fingerprint.

23. A system according to any one of claims 17 to 22 in which a temporal fingerprint in the first sequence of fingerprints is scaled in accordance with a ratio of the first and second image repetition rates.

24. A system according to any one of claims 17 to 23 in which shot changes are detected in the first and/or second sequence of video images and wherein temporal fingerprints derived from video images across a shot change are ignored or given reduced prominence in the correlation step.

25. A system according to claim 24, wherein the shot changes are detected by identifying temporal fingerprints that are above a set threshold.

26. A system according to claim 24 or claim 25, wherein ignoring or giving reduced prominence to temporal fingerprints derived across a shot change comprises replacing the temporal fingerprints derived across a shot change with either the temporal fingerprint immediately preceding it, the temporal fingerprint immediately succeeding it, or a combination of the two.

27. A system according to any one of claims 17 to 26 in which the cadence is detected of any 3:2 or 2:2 repetition of video images resulting from a previous film to video conversion process and wherein temporal fingerprints derived from repeating video images are replaced by a neighbouring non-zero temporal fingerprint or other estimated value.

28. A system according to claim 27, wherein cadence detection comprises identifying temporal fingerprints with values of zero, or close to zero.

29. A system according to claim 27, wherein cadence detection comprises identifying temporal fingerprint values of a video sequence that come in identical, or near identical sequential pairs.

30. A system according to one of claims 27 to 29, wherein replacing temporal fingerprints derived from repeating video images by a neighbouring non-zero fingerprint or other estimated value comprises re-constituting the original temporal differences between the images.

31. A system according to one of claims 27 to 30, wherein in the event cadence is detected, not performing a scaling of the first temporal fingerprints in accordance with a ratio of the first and second image repetition rates.

32. A system according to any one of claims 17 to 31, wherein performing a correlation between said converted sequence of fingerprints and said second sequence of fingerprints comprises performing one of: a discrete correlation function, a phase correlation process, a simple sum of absolute minimum differences as a function of offset, or an alternative matching function which compares the two inputs and returns a function which has an optimum point at an offset position.

33. A system according to claim 32, wherein the correlation of the converted sequence of fingerprints and said second sequence of fingerprints is configured to measure the correlation or matching function to sub-sample precision.

34. A system according to any one of claims 17 to 23, wherein a temporal fingerprint is formed by dividing the single video image into a number of blocks;

dividing the previous single video image of the same video image sequence into the same blocks as the single video image;

determining the average luminescence of each block of both images; comparing the average luminescence of each block of the single video image with the corresponding blocks of the previous single video image.

35. A system according to any one of claims 17 to 34, wherein a spatial fingerprint is formed by discarding the edge of the single video image, leaving only the centre of the single video image;

36. An apparatus configured to perform the method of any of claims 1 to 16.

37. A computer program product comprising program instructions configured to program a processor to perform the method of any of claims 1 to 16.

Intellectual

Property

Office

Application No: GB1614435.4 Examiner: Mr Chris Davidson