EP2078277A2

EP2078277A2 - Method for comparing groups of images

Info

Publication number: EP2078277A2
Application number: EP07821748A
Authority: EP
Inventors: Frédéric Lefebvre; Ayoub Massoudi
Original assignee: Thomson Licensing SAS
Current assignee: Thomson Licensing SAS
Priority date: 2006-10-24
Filing date: 2007-10-24
Publication date: 2009-07-15
Also published as: WO2008049851A2; WO2008049851A3

Abstract

The invention relates to a method for comparing a group of images, called candidate group, with reference groups of images, said reference groups being stored in a database with their representing key frames to which are associated local features and global features. The method comprises the following steps: - detecting an image within said candidate group, called key frame; - computing a global feature of said key frame; - comparing said computed global feature with the global feature of each key frame of the reference group; - if the differences between said computed global feature and each stored global feature are above a predefined threshold then: computing a local feature of said key frame; comparing said computed local feature with the local feature of each key frame of the reference group.

Description

METHOD FOR COMPARING GROUPS OF IMAGES

The invention concerns a method for comparing groups of images.

FIELD OF THE INVENTION

The invention concerns mainly the field of content identification and authentication.

BACKGROUND OF THE INVENTION

New multimedia database search tools require rapidity, compression resistance and copyright. Fingerprinting is largely used to search a content in large multimedia databases. A fingerprinting design extracts discriminating features, called fingerprints, typical for each image/video and thus specific to each picture/video. The fingerprint can also be called image DNA or video DNA. Database management is the first application. Fingerprinting is also a technical issue for the detection of illegal copy and streaming monitoring. Security and output bit length are the fingerprinting weaknesses.

The visual hash is a fingerprinting technique with crypto-system constraints. A visual hash function computes a unique constant condensed version of the content, called visual digest. A small change of the picture/video leads to a small change of the visual digest. Visual hash is used in multimedia content authentication We call visual hash function, a hash function designed for images and video contents. Visual hash function uses fingerprinting techniques with crypto-system constraints. A visual hash function verifies:

• Easy to compute

A one-way function or cryptographic hash function f has the property "ease of computation": for every input x (from domain of f) f(x) is 'easy' to compute.

• Fixed output bit length A hash function f maps an input x of arbitrary bit length to an output f(x) of fixed bit length.

• Pre-image resistant

Given any image y, for which there exists an x with f(x)=y, it is computationally infeasible to compute any pre-image x' with f(x')=y.

• Weak collision resistance

Given any pre-image x it is computationally infeasible to find a 2nd pre- image x' ≠ x with f(x)=f(x'). Two pre-images x, x' are different if and only if their contents are different. • A soft (perceptive) digest

The image f(x) must be resistant and robust, i.e. shall remain the same before and after attacks, if these attacks do not alter the perceptive contents. f(x) ~ f(x') if χ = χ' . χ = χ' means that x' is a version of x (same visual content). For video/picture application, the image f(x) is called visual digest. A small change of a content leads to a small change of the visual digest. A high change of the content leads to a high change of the visual digest. An image visual digest can be:

• A global digest of the image. • A set of local digests computed on regions of an image.

A video visual digest can be:

• a global digest of the video.

• a set of image visual digest of all video frames, a set of image visual digests of all video key frames.

The video fingerprinting consists in describing the video content as set of temporal features, local image features or global image features. Temporal features, local and global image features do not consume the same time. Global description is very fast but it is not very robust. Local description is time demanding but it is very robust against attack.

The invention proposes an innovative video content identification process which combines a visual hash function and a local fingerprinting. Thanks to a visual hash function, we observe the video content variation and we detect key frames. A local image fingerprint technique characterizes the detected key frames. The set of local fingerprints and global fingerprints for the whole video summarizes the video or fragments of the video.

SUMMARY OF THE INVENTION

To this end, the invention relates to a method for comparing a group of images, called candidate group, with reference groups of images, said reference groups being stored in a database with their representing key frames to which are associated local features and global features. According to the invention, the method comprises the following steps:

- detecting an image within the candidate group, called key frame;

- computing a global feature of the key frame;

- comparing said computed global feature with the global feature of each key frame of the reference group;

- if the differences between said computed global feature and each stored global feature are above a predefined threshold then:

^■ computing a local feature of the key frame;

^■ comparing the computed local feature with the local feature of each key frame of the reference group.

According to a preferred embodiment, the step of computing a global feature computes a visual digest of each key frame.

According to a preferred embodiment, the method comprises

- a step of detecting shot boundaries,

- a step of detecting among the key frames of each shot, a key frame having the smallest content variation of the candidate group, called stable frame, said method being performed only on the stable frames detected among the key frames.

Preferentially, the step of computing the local features of each key frame comprises the steps of:

- detecting interest points, - computing a local descriptor for said interest points,

According to a preferred embodiment, when said global feature of a key frame of the reference group match the global feature of a key frame of the candidate group, said key frame is detected as a copy.

According to a preferred embodiment, when said local feature of a key frame of the reference group match the local feature of a key frame of the candidate group, said key frame is detected as a copy.

According to a preferred embodiment, the method further comprises the step of

- deciding that the candidate group is a copy of the reference group when a predetermined number (p) of key frames of the candidate groups are identified as a copy of said predetermined number of key frames of the reference group.

BRIEF DESCRIPTION OF THE DRAWINGS Other characteristics and advantages of the invention will appear through the description of a non-limiting embodiment of the invention, which will be illustrated, with the help of the enclosed drawing.

- Figure 1 represents an image in the P-Q space, - Figure 2 represents a selection of points in an image used for the computation of the visual digest,

- Figure 3 represents a selection of preferred points in an image for the computation of the visual digest,

- Figure 4 represents a key point orientation used for fingerprinting computing,

- Figure 5 represents a nine region decomposition used for the computation of the descriptor.

- Figure 6 represents a flowchart of a video fingerprinting computation and of an indexing/identification process according to the invention, DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

Embodiments of the present invention may be implemented in software, firmware, hardware or by any combination of various techniques. For example, in some embodiments, the present invention may be provided as a computer program product or software which may include a machine or computer-readable medium having stored thereon instructions which may be used to program a computer (or other electronic devices) to perform a process according to the present invention. In other embodiments, steps of the present invention might be performed by specific hardware component that contain hardwired logic for performing the steps, or by any combination of programmed computer components and custom hardware components.

Thus, a machine-readable medium may include any mechanism for storing or transmitting information in a form readable by a machine (for instance a computer). These mechanisms include, but are not limited to, floppy diskettes, optical disks, hard disk drives, holographic disks, compact disks read-only memory (CD-ROMs), magneto-optical disks, read-only memory (ROMs), random access memory (RAM), Erasable Programmable Read-only memory (EEPROM), magnetic or optical cards, flash memory, a transmission over the Internet, electrical, optical, acoustical or other forms of propagated signals (for instance carrier waves, infrared signals, digital signals, etc), or the like.

Unless specifically stated otherwise as apparent from the following discussion, it is appreciated that discussions utilizing terms such as "processing" or "computing" or "calculating" or "determining" or the like, may refer to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices.

In the following detailed description of the embodiments, reference is made to the accompanying drawings that show, by way of illustration, specific embodiments in which the invention may be practices. In the drawings, like numeral describe substantially similar components throughout the several views. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention. Other embodiments may be utilized and structural, logical, and electrical changes may be made without departing from the scope of the present invention. Moreover, it is to be understood that the various embodiments of the invention, although different, are not necessarily mutually exclusive. For example, a particular feature, structure, or characteristic described in one embodiment may be included within other embodiments.

In Figure 6, the modules represented are functional units which may or may not correspond to physically distinguishable units. For example, these modules or some of them may be grouped together into a single component, or constitute functionalities of one and the same piece of software. A contraήo, certain modules may possibly be composed of separate physical entities.

We are now going to describe how the visual hash function used for obtaining the visual digest, said visual digest enabling to detect the shot boundaries. The visual digest is based on pseudo variance of the luminance of the points selected in a P-Q image representation. On figure 1 , to select a point in P-Q image representation, we consider points along a line passing through the image center, and characterized by its angular orientation θ. For an angular orientation θ, the coordinates of a point (x,y) according to (X₁Y) axis (original domain) become (p, θ) according to the (P₁Q) axis :

We select the set of points on a line passing through the image center, with the orientation θ. The discrete version of this line is a strip depicted in figure 2. θ is equal to 0.179 degree. We decide to use only the p-axis to characterize the pixels of each angular orientation θ. For an angular orientation θ, a point (x,y) is characterized by the couple (p,θ).

For a strip width of 1 , a point (x,y) is a selected point if its coordinate p satisfies:

-0.5 ≤p-p'≤ 0.5 (1)

With ( (p',θ) the coordinates of the middle point (x',y') for a same given θ.

So the equation (1) becomes :

-0.5<(x-x')cosθ + (j-/)sinθ < 0.5 (2)

For a strip width of η, we extend (2) to :

-η/2≤(x-x')cosθ + (j-/)sinθ <η/2 (3)

The preferred embodiment takes care of the importance of a point in the image. The importance of a point (x,y) in an image can be weighted by the relative position of this point (x,y) to the center (x',y'). The distance rof a point (x,y) from the ellipse (figure 3) image center (x',y') verifies:

r(x,y)<dφ) (4)

Where: d_{2(Q) =} {height I if .{width I if

^~ {height 12) ². cos² (θ ) + {width 12) ². sin ² (θ )

For the given orientation θ, only light grey pixels are selected. For other orientations, only dark grey pixels can be selected, white pixels are rejected. If a discretization of 1 ° angle is selected, the visual digest is composed of 180 elements. Each element of the visual digest is thus computed by:

Elt(θ) = (I(p,θ) - Mean(θ)f

• l(p,θ) is the value of the selected point (p,θ) (preferably the luminance of the pixel (p,θ))

• Mean(θ) is the classical mean value of l(p,θ) i Α — .V /(p,θ) .

^{J V}Θ P =1

• N_θ is card({p,θ }) (strip width of η )

The image visual digest of a frame / is:

With card(VD(i)) = nθ

We are now going to describe how the key frames are detected.

The evolution of the visual digest distance over a group of frames allows the detection of key frames. A shot boundary is a brutal variation of the visual digest inside a group of frames. The set of frames in between two shot boundaries is called a shot. The stable frame presents the smallest distance variation of visual digest inside a shot. The goal of this step is to extract stable frames. When a stable frame is detected, it can be characterized by using local fingerprinting.

First a shot boundary detection is performed.

An automatic threshold process determines brutal transitions along the video and detects shot boundaries. The global automatic threshold process is based on two thresholds:

• Pseudo-global threshold, denoted T_gι_obaι • Adaptive threshold, denoted T_/oca/ Two frames with the same content have the same (or closed to) visual digest. Instead of the histogram, the visual hash function described previously is chosen to study the video content variation. Such an approach presents the additional advantage to be very fast and not CPU/memory demanding. Thus, the visual digest distance measures the similarity between frames. Inside a window S, the shot boundary corresponds to the frame where a pick of the visual digest distance is detected. This frame is localized in the center of the window S. The shot boundary on a window S, denoted SB, is calculated as below:

SB = i\dist (VD(i), VD(i + 1)) > m_Αχ(T_global (i, L₁ ), T_local (i, L₂ ))

And the set of shot boundaries along a video is denoted ShotBound

ShotBound = {SB}

The cardinality of ShotBound depends on video activity. A pseudo-global threshold, denoted T_gι_obaι(iLi), is defined on a large sliding window S? of size 2Li+1, centered on frame /. Letting μ(i) and σ(i) denote the mean and the variance of dist(VD(k),VD(k + 1)) measured for all k in S₁=Ii-L₁^L₁]. We define our proposed pseudo-global threshold as: T_globαl(i,L_ι) = μ(i) + α_ι.σ (i)

A local and adaptive threshold, denoted Tι_oca{i, L₂), is computed on a small sliding window S₂ of size 2L₂+1, with L₂ « L₁, centered on frame i. Specifically, we have:

T_locαl (i, L₂) = α₂.d_msκ (i)

where d_max(i) is the second maximum value of dist(VD(k),VD(k + X)) measured for k in S₂=[i-L₂;i+L₂]. Empirical values: ^=20, L₂=I 2, αi=3 and α₂=2 .

Stable frame detection A stable frame is the frame with the smallest content variation along a shot. For such a frame, the content distance between this frame to the other neighbor frames will be very low. Two frames with the same content have the same (or closed to) visual hash (global fingerprint in our case).

Inside a shot, for each group of 2L₃+1 frames, an average of the content image distance (at the position )) is given by: With j = L₃

The stable frame is the frame which provides the smallest average of content image distance. This stable frame must have a well distributed content information.

Dist(l) = Min({Dist(j)}) StαbleFrαme = I .

\Entropy(VD(l)\ ≥ Threshold

We obtain one stable frame per shot.

The preferred value for L₃ is 5.

In the previous step, a stable frame is detected. Now, we can characterize this frame by using local fingerprinting. The local fingerprint process is divided in two steps:

• Interest points detection

• Local description of interest points We are now going to describe the interest points detection

The interest points detection is based on a Difference Of Gaussian. It consists in detecting repeatable interest points. An interest point (also called key point) is repeatable if its pixel location is stable. It must be resistant to scale change, rotation, filtering... We use a cascade of filtered image. The Gaussian kernel is a scale-space kernel candidate. The theoretical interest of such an approach is that the difference of two Gaussians with respective variances k.σ and σ is a very good approximation of the normalized Gaussian Laplacian:

G(jc, y, kσ ) - G(x,y,σ ) ∞ (k - I)O ²V²G

The convolution of this difference with the image will lead to the Difference of Gaussians function (DOG image):

D(x,y,c) = (G(x,y,kc) - G(x,y,c)) * I(x,y) (6)

Therefore, for an input image, cascades of filtered images are built and called "octaves", and then the difference of Gaussians is computed for each octave. The extrema of the DOG Gaussians represent potential locations of interest points. Not all of these locations contain relevant information. Thus, a further threshold is performed on these points. Only the points with good contrast and precise space localization are kept. The locations of the detected key points in the scale space are then stored.

We are now going to describe the local description of the key points.

In the previous step, we performed key points detection and we stored their localization in the scale-space. In this step, we locally characterize each key point by computing a local descriptor. The descriptor must be both discriminant and invariant to a certain number of transformations. A discriminant descriptor is a descriptor which provides representative and different values for each different content. The description of a key point must be different from the description of other key points. An efficient descriptor allows a correct matching in a large database of descriptors with high probability.

To compute a descriptor, we consider a circular neighborhood of radius β (e.g 20) (Figure 6) in order to be invariant to rotation. We compute the key point orientation, Keyθr(x,y), by summing the gradient vectors in a small disc Disc(x,y) around the key point.

Keyθr(x, y) = V orientation(x" , y" )

(x",y")eDisc(x,y)

The gradient of a pixel (x,y) in a disc of radius 1 is given by:

magnitude(x,y) = ^(d _xL(x,y)f + (d _yL(x,y)f orientation{x,y) = Arc tan(θ _yL(x, y)ld _xL{x,y))

Where L(x,y) is the Gaussian image:

L(x,y,o) = G(x,y,o) * I(x,y)

The orientation of the resulting vector gives the key point orientation (figure 4).

According to this orientation, the neighboring disc is then divided into nine regions (figure 5). Each pixel within the disc belongs to one unique region depending on its polar position.

With :

• a the key point orientation

• θ the polar position of the pixel (x,y) within the disc

• rthe distance from the key point to the pixel (x,y)

For each of the nine regions, we compute a local histogram of sixteen bins. Histo(i,θ,k)=#Pixel(x",y") \ orientation(x",y")= θ, (x",y") C R(i,k) With

• k the selected key point (x,y) • θ the pixel orientation (from 0° to 360° with a step angle of 22.5°, so there are 16 bins)

• / the index of the selected region (from 1 to 9)

• R(i,k) the region / of the disc centered in the selected key point k

The final descriptor of a key point k (KD) is the concatenation of the nine histograms.

KD(k) = [Histo(i,Q ,k)]

The Local Fingerprint of a stable frame /, called also shot fingerprint (SF) , is the set of all KD:

SF(i) = {KD(k)}

The cardinality Of LF(T) depends on image activity.

The Video Fingerprint VF is the set of shot fingerprints SF: VF = [SF(J)]

We are now going to describe, with the help of figure 6, an application of video matching where the local and global features are used for detecting if two different video documents are similar.

Video movies are recorded in a database, each video movie is called a reference group of images. In a step E1 , global image features (visual digest) for each frame of each movie are calculated by applying a visual hash function as described earlier in this document. The global image features are typical and unique for each of the different image contents. The cardinality of each global image features is constant for each frame of the video.

For each group of n frames, the distance of the successive global image features is computed. The evolution of this distance detects shot boundaries and key frames. A shot boundary is a brutal variation of the representative features inside a group of n frames. The set of frames surrounded by two shot boundaries is called a shot. In a preferred embodiment, stable frames are detected among the key frames as presented earlier but the invention can also be applied to key frames without detection of stable frames. The stable frame presents the smallest distance variation of image features inside a shot. For a stable frame, the frame content information is correctly distributed (above an entropic threshold).

In a step E3, a local fingerprint is calculated on each of the detected keyframes. The local fingerprint represents as mentioned earlier, the local fingerprint of a key frame is SF(i), which calculation is described earlier.

In a step E4, the global fingerprint of only the key frames is calculated. A global fingerprint of a key frame i is the visual hash VD(i) as defined earlier.

The local fingerprints and the global fingerprints are stored in a step E5 in a database, for instance on a hard disk drive, a DVD, an holographic storage or any other memory or recording medium.

When a candidate movie is submitted to the database, a visual hash function is performed during a step E6. This candidate movie is submitted to the database in order to check if it is a copy of one document of the database for instance.

A visual digest is obtained for the candidate movie. In a step E7, the key frames are detected as explained earlier. In a step E8, a global fingerprint is calculated for each of the key frames. In a step E9, a local fingerprint is calculated on each of the key frames.

A comparison is performed in a step E10 between the global fingerprint of the key frame of the candidate movie and the global fingerprint of the key frames stored in the database. The comparison is done using the Euclidian distance.

The global description VD_c(i) is compared to all VD(i) of the database. The comparison, called first pass, is done using Euclidean Distance:

distEuclid(VD, VD _c ) = - VD _c (/)|

A candidate VD₀ matches with a reference VD if

distEuclid(VD,VD_c) ≤ Te

The empirical value Te depends on the application and attacks applied on the reference movie. This first pass is very fast and brings excellent results for most of attacks.

When there is a match, in a step E11 , the keyframe is detected as being a copy and a decision step E14 is performed (detailed later).

When there is no match in step E11 , the local fingerprints of the key frame of the candidate movie and of the database are compared in a step E12. The comparison is done using the Euclidian distance. If there is a match, in step E13, the current key frame is detected as a copy and the decision step E14 is performed. When there is no match, in step E13, the next key frame is selected and the comparison of step E10 is performed again on this new key frame.

During the decision step E14, when p key frames are detected as a copy, the decision step detects a section. When q sections are detected, the candidate movie is detected as being a copy of the movie stored in the database for which key frames local/global fingerprints have matched.

The invention does not need to make the comparison of all the key frames of the movie. When a predetermined number of key frames (or stable frames) have been detected as being a copy, the process can be stopped.

This predetermined number is based on several parameters related to the content of the movie, to the reliability of the fingerprint calculation. Therefore the invention allows a fast process for matching videos.

This invention can be used for many purposes.

A first application is to retrieve a video or a fragment of video in a database using video fingerprint as a database index.

A second application is to identify, in real time (or closed to), a video stream by matching the current fingerprints of the current video with all fingerprints of a video database even if strong distortions are applied to an original content. It is a copy detection application.

A third application is to find a model of distortion applied to an original movie. By matching the local fingerprints of an original copy and a pirate copy, a model of distortion is computed. It is a co-registration application. This co- registration technique can be used in an informed watermarking algorithm.

A fourth application can be the recovery of a particular release of a movie. For instance, when a plurality of shots is taken, it is sometimes, later on, needed to recover which shot was used for which copy. The current invention enables to recover the right shot.

The invention is not limited to the cited applications given here above.

Claims

1. Method for comparing a group of images, called candidate group, with reference groups of images, said reference groups being stored in a database with their representing key frames to which are associated local features and global features, characterized in that the method comprises the following steps:

- detecting an image within said candidate group, called key frame; - computing a global feature of said key frame;

- if the differences between said computed global feature and each stored global feature are above a predefined threshold then: ■ computing a local feature of said key frame;

^■ comparing said computed local feature with the local feature of each key frame of the reference group.

2. Method according to claim 1 characterized in that the step of computing a global feature computes a visual digest of each key frame.

3. Method according to claim 1 or 2 characterized in that it comprises

- a step of detecting shot boundaries,

4. Method according to claim 1 characterized in that the step of computing the local features of each key frame comprises the steps of:

- detecting interest points,

- computing a local descriptor for said interest points,

5. Method according to one of the preceding claims characterized in that when said global feature of a key frame of the reference group match the global feature of a key frame of the candidate group, said key frame is detected as a copy.

6. Method according to one of claims 1 to 4 characterized in that when said local feature of a key frame of the reference group match the local feature of a key frame of the candidate group, said key frame is detected as a copy.

7. Method according to one of the previous claims characterized in that it further comprises the step of