EP1979876A1

EP1979876A1 - Method of verifying the saliency areas of a multimedia document, method of creating an advertising document, method of automatically verifying the saliency areas of an internet site, and associated computer program product

Info

Publication number: EP1979876A1
Application number: EP07726278A
Authority: EP
Inventors: Olivier Le Meur; Xavier Castellan; Michel Kerdranvat
Original assignee: Thomson Licensing SAS
Current assignee: Thomson Licensing SAS
Priority date: 2006-02-03
Filing date: 2007-02-01
Publication date: 2008-10-15
Also published as: CN101379529A; FR2897183A1; JP2009525525A; CN101379529B; WO2007088193A1; JP4896160B2

Abstract

The invention relates to a method of verifying the saliency areas of a multimedia document. According to the invention, the method comprises the steps of: determination (E1) of the areas of the said document that are considered to be key areas on the basis of a priori data extracted from the said document, application (E2) to the said document of a model based on the compilation of saliency maps so as to obtain a saliency map of the said document, comparison (E3) between the areas of the said document that are considered to be key and the said saliency map.

Description

Method of verifying the saliency areas of a multimedia document, method of creating an advertising document, method of automatically verifying the saliency areas of an Internet site, and associated computer program product.

The invention relates to a device and a procedure for verifying the saliency areas of a multimedia document.

The invention applies more particularly to the evaluation of content and more particularly of documents such as advertising documents. Today, when one wants to bring a content to the fore, various techniques for creating contents are known: quite simply, while writing a text document, one overscores, one writes in bold, one frames, etc. More sophisticated techniques are used by advertisers which may, for video or audio, combine image processing techniques and sound processing techniques, more generally signal processing techniques.

It is difficult to evaluate the effectiveness of these techniques. The advertisements are very often attractive but the various effects implemented are not always all effective and could certainly be improved.

Currently, oculomethc techniques can aid in the determination of saliency areas. Such techniques however are manual and unwieldy to implement since they do not allow automation of the procedure on a plurality of documents, Web pages, video documents.

For this purpose, the invention proposes a method of verifying the saliency areas of a multimedia document. According to the invention, the method comprises the steps of:

- determination of the areas of the said document that are considered to be key areas on the basis of a priori data extracted from the said document,

- application to the said document of a model based on the compilation of saliency maps so as to obtain a saliency map of the said document, - comparison between the areas of the said document that are considered to be key and the said saliency map.

According to a preferred embodiment, the method comprises a step of calculating a ratio of the saliency contained in the key areas of the said document to the total saliency of the said document.

According to a preferred embodiment, the method comprises a step of calculating the visual path indicating the course of the eye during the viewing of the said document, the said visual path being calculated on the basis of the said saliency map and composed of the pixels of the image whose saliency is greater than a threshold.

According to a preferred embodiment, during the search for the said visual path,

- the pixel having the maximum saliency in the said document is searched for, an area of the said document around the said maximum is disabled and - the next pixel having the maximum saliency is searched for, until all the pixels having a saliency greater than the said threshold have been obtained.

Advantageously, an analysis report comprising the saliency areas, the said visual path, and a set of sub-images representative of the saliency areas of the said document is compiled.

According to a preferred embodiment, the said report comprises the key areas of the said document.

The invention also relates to a method of creating an advertising document. According to the invention, the method alternately comprises a step of creating the content followed by a method of verifying the saliency areas of the said document according to the invention, the said steps being repeated so long as the saliency areas do not correspond to the key areas of the said document.

The invention also relates to a method of automatically verifying the saliency areas of an Internet site. According to the invention, the method comprises for each page of the site the steps of

- determination of the areas of the said page that are considered to be key areas on the basis of a priori data extracted from the said page,

- application to the said page of a model based on the compilation of saliency maps so as to obtain a saliency map of the said page,

- comparison between the areas of the said page that are considered to be key and the said saliency map.

The invention also relates to a computer program product comprising program code instructions for the execution of the steps of the methods according to any one of the embodiments of the invention when the said program is executed on a computer.

The invention will be better understood and illustrated by means of wholly nonlimiting advantageous exemplary embodiments and modes of implementation, with reference to the appended Figure 1 representing the verification procedure according to a preferred mode of the invention.

The invention is particularly advantageous in documents of an advertising nature where the components of the document are particularly important by dint of their economic impact. Specifically, advertising films, be they a video, simply audio or of still image type in a magazine, are particularly cultivated for placing messages that are attractive to the consumer. However sometimes, these messages are not necessarily the areas which will attract the human eye the most and in this case, the advertiser will not have created a document having the hoped for effect.

The present invention is thus aimed at affording means of analysing the documents, whether or not they are advertising documents, so as to verify that these documents have the desired impact on their recipients. Thus, procedures exist for calculating saliency maps for a document.

These procedures make it possible to determine the areas which will attract the eye. They are based on calculations taking into account various parameters of the image and on spatial and/or temporal analyses of the image and/or of the video.

A saliency map is a topographic representation of the perception of an image. The visual relevance of each pixel of the image is determined. The map is normalized for example between 0 and 255. The higher the value, the more the pixel attracts the eye.

Such procedures are for example described in the European patent application in the name of the company Thomson Licensing SA, published on 22 June 2005 under the number 1544792.

Figure 1 represents the steps of the method according to a preferred mode of the invention.

A paper document D of commercial document type is available. This document has been created on the basis of a piece of publishing software, for example acrobat writer (trademark registered by the company adobe), word (trademark registered by the company microsoft), of HTML type, the electronic version of which is available. It is possible therefore to extract from this document a priori information which reflects what the author of the document desired to bring to the fore. For example, in a document of word type, it is possible to judge the importance of the information as a function of the size of the characters used, of the character font used, of the overscohng, of the colour used. It is also possible to use the location on the sheet, in general the important information being placed at the centre of the image.

In an html type document, the "tags" are also an aid to the determination of the points that the author wants to bring to the fore.

Thus, in a manual manner by observation of the document D, or in an automatic manner by automatic extraction of the characteristics mentioned above, it is possible to compile (step E1 ) the a priori information representative of the key points of the document D according to the author. It is thus possible to obtain the geographical location of the areas of interest of the document.

Moreover, a model making it possible to build the saliency map of a document is available. During a step E2, this model is applied to the document D. The model is for example a piece of software receiving the document D as input. The model analyses the characteristics of the document that are necessary for the extraction of saliency maps so as to obtain the saliency map of the document D.

Such procedures are for example described in the European patent application in the name of the company Thomson Licensing SA, published on

22 June 2005 under the number 1544792. They are based on calculations taking into account various parameters of the image and based on spatial and/or temporal analyses of the image and/or of the video.

Information of text type as well as information of face type are also extracted since usually they are also representative of information considered to be important, when the text is included in a video document.

The consideration of the information of face or text type during the compilation of saliency maps is in particular described in the document "attention model based progressive image transmission " by Hu, Xie, Chen, Ma as well as in the document "a visual attention model for adapting images on small displays" by Chen et Al and published in the journal "multimedia systems" in 2003.

During a step E3, on the basis of the saliency map, an analysis report is obtained containing the following information:

- A heat map indicating the attraction areas,

- A map of the visual path indicating the visual course of the average observer.

This visual course is obtained by searching through the saliency map for the local maxima having a saliency value greater than a threshold, as a function of the number of maxima that one desires to obtain or as a function of the global saliency of the image. The search for the local maxima can for example be done by disabling a circular area (representing the fovea of radius equal to 0.5 of a visual degree) about the (n-1 ) th maximum and by seeking the global maximum of this new map and so on and so forth for each maximum. For each maximum an area is therefore obtained and thereafter the centres of the areas n-1 ,n,... are joined so as to obtain the visual course of an average observer. - A perceptual summary containing n sub-images of the image to be analysed. These sub-images are areas fixed by the observer.

Such a procedure for determining the sub-images is described in the

European patent application filed on 19 September 2005 under the number

05291938.8 in the name of the company Thomson Licensing. This procedure is also described in the publication by O. Le Meur, X. Castellan, P. Le Callet and D. Barba, Efficient saliency-based repurposing method, ICIP 2006.

The sub-images are composed of the pixels about the local maxima. The number of sub-images is determined by fixing a threshold and sub- images are defined only for the maxima having a saliency value greater than this threshold.

This information makes it possible to detect the areas of the image that are actually perceived by the human eye.

Thereafter, the areas indicated by the author of the document or extracted automatically during step E1 and the areas indicated in the analysis report are compared. It is thus possible to compare what the author desired to bring to the fore in the document with the information that actually catches the human eye, by placing opposite one another the a priori information and the information of the analysis report.

The analysis report can also provide this analysis directly by implementing a step of direct comparison between the saliency map and the a priori data. Specifically, it is possible to compare the geographical location of the saliency areas and of the areas considered to be important by the author.

It is possible for example to calculate the ratio of the saliency contained in the areas delimited by the author of the document and the global saliency. This ratio can be an indicator of the relevance of the design of the document. It is thus possible for the creator to evaluate the value of his document in terms of attractiveness to the eye.

The invention therefore advantageously proposes a procedure for evaluating documents of a commercial or advertising nature.

The invention can also be seen as being a remote method for the evaluation of advertising documents. It is actually possible to envisage the model being available on an Internet site and the client submitting his document in electronic form so as to ascertain the points/areas which will in reality catch the human eye. The user of the service, as a function of the result of the comparison can then modify his document and resubmit it in an iterative manner in such a way that the information that he desires to bring to the fore is indeed the information on which the human eye will focus.

The method can also have as application the evaluation of several documents submitted by clients following a call for tenders. It is thus possible to analyse in an automatic and objective manner which document is the one that will have the most impact on the consumers.

In another embodiment, the document D is not a paper document or a still image, of the advertising insert type but a video document like the advertisements transmitted on televisual channels. Such a document can also be analysed according to the method implemented in the invention. Actually, it is also possible to compile a saliency map on the basis of the temporal information. Such a method is described in the document by O. Le Meur, P. Le Callet, D. Barba and D. Thoreau, "A spatio-temporal model of the selective human visual attention", Proc. ICIP-05 (IEEE International Conference on Image Processing), Genoa, Italy, September 2005. It is therefore possible to thus also obtain the saliency map of the video document.

To obtain the a priori information on the video data, it is possible to obtain it by way of the author of the video. In another embodiment, it is also possible to employ this procedure by integrating it directly into tools for aiding the creation of documents. Documents are then actually obtained for which the eye is focused on the areas that are actually considered to be key in the document. It is thus possible to alternately create content, apply the verification method according to the invention and modify it so long as the saliency areas and the key areas do not coincide.

In yet another embodiment, it is also possible to envisage the document to be analysed being an Internet site and each page of the site being analysed regularly, in an automatic or manual manner. Thus, it is possible to envisage an automatic procedure for running the method according to the invention on all the Web pages of the site and displaying of the report, comparative result between the data considered to be key in the page by the author and the data of the page that are actually viewed by the eye if they are available, or blindly if the key data are not available.

Claims

1. Method of verifying the saliency areas of a multimedia document characterized in that it comprises the steps of:

- determination (E1 ) of the areas of the said document that are considered to be key areas on the basis of a priori data extracted from the said document,

- application (E2) to the said document of a model based on the compilation of saliency maps so as to obtain a saliency map of the said document,

- comparison (E3) between the areas of the said document that are considered to be key and the said saliency map.

2. Method according to Claim 1 characterized in that it comprises a step of calculating a ratio of the saliency contained in the key areas of the said document to the total saliency of the said document.

3. Method according to Claims 1 or 2 characterized in that it comprises a step of calculating the visual path indicating the course of the eye during the viewing of the said document, the said visual path being calculated on the basis of the said saliency map and composed of the pixels of the image whose saliency is greater than a threshold.

4. Method according to Claim 3 characterized in that during the search for the said visual path,

- the pixel having the maximum saliency in the said document is searched for,

- an area of the said document around the said maximum is disabled and

- the next pixel having the maximum saliency is searched for, until all the pixels having a saliency greater than the said threshold have been obtained.

5. Method according to one of the preceding claims characterized in that an analysis report comprising the saliency areas, the said visual path, and a set of sub-images representative of the saliency areas of the said document is compiled.

6. Method according to Claim 5 characterized in that the said report comprises the key areas of the said document.

7. Method of creating an advertising document characterized in that it alternately comprises a step of creating the content followed by a method of verifying the saliency areas of the said document according to one of Claims 1 to 6, the said steps being repeated so long as the saliency areas do not correspond to the key areas of the said document.

8. Method of automatically verifying the saliency areas of an Internet site characterized in that it comprises for each page of the site the steps of

- determination of the areas of the said page that are considered to be key areas on the basis of a priori data extracted from the said page, - application to the said page of a model based on the compilation of saliency maps so as to obtain a saliency map of the said page,

9. Computer program product characterized in that it comprises program code instructions for the execution of the steps of the methods according to any one of Claims 1 to 8, when the said program is executed on a computer.