WO2014012662A1

WO2014012662A1 - Selecting a set of representative images

Info

Publication number: WO2014012662A1
Application number: PCT/EP2013/002117
Authority: WO
Inventors: Fabian Nater; Helmut Grabner; Luc VAN GOOL
Original assignee: Eth Zurich
Priority date: 2012-07-20
Filing date: 2013-07-17
Publication date: 2014-01-23

Abstract

The invention relates to an image selection device (100) and to a method for selecting a set of representative images IE={IE₁, IE₂, IE₃, IE_N} from a set of images l={l₁, I₂, I₃,..., I_M} recorded and transmitted by an image source (1 ). An image processor (101 ) is configured to determine a set of representations R={R₁, R₂, R₃, · · ·, R_M} of the set of images l={l₁, I₂, I₃,..., I_M} A representative set builder (103) determines different test sets (RE^(q)= {RE₁ ^(q), RE₂ ^(q), RE₃ ^(q),..., RE_N ^(q)}) each test set (RE^(q)) containing N image representations (RE_i ^(q)) taken from the set of image representations (R). The representative set builder (103) further assigns a score (RS^(q)) to each test set (RE^(q)), the assigned score (RS^(q)) being determined dependent on at least two of the following variables: local density, dissimilarity, and compliance with a predefined attribute. A test set (RE = {RE₁, RE₂, RE₃, RE_N}) is selected out of the different test sets (RE^(q)) dependent on the assigned scores (RS^(q)). An image selector (104) determines the set of representative images (IE) by selecting input images (h) from the set of input images (I_i) corresponding to the image representations (RE_i) in the selected test set (RE).

Description

SELECTING A SET OF REPRESENTATIVE IMAGES

FIELD OF THE INVENTION

The present invention relates to an image selection device and a method for selecting a set of representative images from a set of images from at least one image source.

BACKGROUND OF THE INVENTION

A large number of image sources is constantly connected to the Internet, thereby providing a large amount of images to users. For example, by using easily deployable webcams, touristic or public places are monitored, hotels show their environment, ski resorts show the current weather, etc. Due to the large amount of image sources and images provided, it is difficult for users to find and access images, which are interesting for him or her. Image sources may be listed in directories, such as webcam directories, and a user may find image sources according to a directory label, such as, for example, mountain views, landscapes, public places, streets, etc. However, images of such image sources may not be of much interest, in particular when images with essentially the same image content are provided and/or displayed, such as the same mountain view or landscape, which only slowly changes due to different light conditions during a day. For example, a user with an interest in specific objects or attributes, such as mountain birds in a mountain view or a heavy storm in landscapes, wishes to reliably find image sources displaying such images of interest. The desire for spotting events of current interest was already addressed in US6757682 and US 2008/0320159.

The same concept is also valid for any other application, where an uninteresting, static or monotonic image stream is desired to be condensed or compiled to a number of essential, representative images. This is for example the case for many surveillance cameras. Most of the time, the image content in the images provided by such surveillance camera does not change, or shows repeated activity, which is not interesting or useful for any viewer. In rare moments, special events happen which are desired to be detected and reported. Or in a retrospective analysis, representative event images give an ideal overview of what happened in the past.

SUMMARY OF THE INVENTION

It is an object of this invention to provide an image selection device and a method for automatically selecting a set of representative images from a set of images from an image source, thereby reliably selecting the images of interest. According to the present invention, the object is achieved by the features of each independent claim. In addition, further advantageous embodiments are provided in the dependent claims and the description.

According to a first aspect of the present invention, an image selection device is provided for selecting a set of N representative images from a set of M input images di- rectly or indirectly received from, or recorded and/or transmitted by an image source. The image selection device comprises an image processor configured to determine a set of image representations from the set of input images. Preferably, each input image may be encoded into a corresponding image representation. In another embodiment, multiple input images may be encoded into a corresponding image representation. A representative set builder is configured to select a set of representative image representations, which selected set of representative image representations also is denoted as selected test set given that it is selected from a choice of multiple test sets. An image selector is configured to determine the set of representative images by selecting the images corresponding to the image representations present in the selected test set. The set of representative images is a condensed and compiled version of the image set and therefore N M.

According to a second aspect of the present invention, a method is provided for selecting a set of N representative images from a set of M input images directly or indirectly received from, or recorded and/or transmitted by an image source. Input images of the set are encoded into corresponding image representations collectively forming a set of image representations. Preferably, each input image may be encoded into a corresponding image representation. In another embodiment, multiple input images may be encoded into a corresponding image representation. A set of representative image rep- resentations is selected, which selected set of representative image representations also is denoted as selected test set given that it is selected from a choice of multiple test sets. The set of representative images is determined by selecting the images corresponding to the image representations present in the selected test set. The set of representative images hence is a condensed and compiled version of the image set and therefore N M.

According to the present invention, in the representative set builder different test sets are assembled from the image representations contained in the set of image representations wherein each test set contains N image representations given that the finally to be determined set of representative images is desired to contain N images. A score is attributed to each test set and a test set is selected out of the different test sets dependent on the assigned scores. For example, the test set with the highest score amongst the different test sets is selected. In another embodiment, one of the test sets with a score above a threshold is selected. In a further embodiment, the test set with a score closest to predefined target score is selected. A test set is assigned a score dependent on at least two out of the following three variables: • a frequency at which image content contained in one or more, and preferably all of the image representations of the test set is contained in the set of image representations,

• a measure for a dissimilarity between image content con- tained in at least two, and preferably all of the image representations of the test set,

• a measure for a similarity between image content contained in one or more, and preferably all of the image representations of the test set and a predefined image content and/or a measure for one or more of the image representations of the test set fulfilling a predefined attribute such as, for example, recording date, time, etc.

It is emphasized that in case of determining the score only dependent on two of the three variables, the third variable does not need to be determined or otherwise addressed given that is not used.

In a preferred embodiment, the score is determined based on all three variables. In that sense, a set of representative image representations should well represent image content that often occurs in the input image set and should well represent diverse image content contained in the input image set that may appear rarely and is different from other image content, and especially different from the high frequent occurring image content, and should include image content that is similar to predefined image content, and/or image representations that fulfill predefined attributes. Values of each of the three variables may preferably be mapped into individual scores, and the score of a test set may in a preferred embodiment involve a combination of the three individual scores. According to embodiments of the present invention, the set of input images can be obtained from any imaging source and/or can be embodied as distinct images separated in time and/or space, and/or can originate from a capturing device in the form of a video at any, possibly variable frame rate, and/or can also be obtained from a fixed or steerabie camera installed in a surveillance setting or from a webcam accessible through the internet.

In an embodiment, the image processor is configured to deduce the set of image representations from the set of input images. An image representation can correspond to one image and can be obtained by using any type of local and/or global image descrip- tion including information about color, intensity, edges, texture, image segments, or any combination of those. Alternatively or in addition, an image representation can be deduced from more than one image and can be obtained by using any type of description including motion, flow, depth or 3D information. Any of the above listed characteristics of an image or an image sequence which support describing an image or an image sequence are also denoted as image features. Additionally each image representation can be transformed to a binary image representation by means of comparing feature values of different elements in an image representation. For example, in a grey tone image, elements of the image may be defined as pixels or regions of pixels. In an image representation of a region of such image, the grey tone of this region as the feature of interest may be compared to the grey tone of, for example, an adjacent region, such that the result may be of binary nature, i.e. "more dark" or "less dark" than the grey tone in the adjacent region.

In order to compare image representations amongst each other, or to compare an image representation with a representation of a predefined image content, any distance measure can be used, including Euclidean distance or the Hamming distance in the case of binary features. In particular, the use of binary features and the Hamming dis- tance is of great interest if efficiency needs to be guaranteed, as for real-time processing.

In one embodiment, the set of representative images is determined once the set of input images is available entirely, for example at the end of a recorded video. In another embodiment, the set of representative images is determined on-the-fly during image acquisition. By doing this, a set of representative images is available at any time during image acquisition, which is particularly suitable for real-time live processing.

In the on-the-fly case, it is preferred that an alert is triggered each time a new input image representation is included in the selected test set which then constitutes a new selected test set. The corresponding new input image may be of particular interest, as its image representation has increased the score of the selected test set of image representations. In this case, the new input image is directly routed to the output in order to have it displayed and possibly have a viewer notified in real time.

In one embodiment, the set of representative images can be displayed to a user or experienced by a user in any form on any display device, such as in a web-browser, on a computer monitor, a tv screen, in a photo-frame, on a mobile phone or a tablet pc. In a different embodiment, the set of representative images can be used for further analysis in the same device or in a different processing device connected thereto.

In one embodiment, if multiple image sources are available simultaneously, these mul- tiple sources are processed in parallel and a test set and the corresponding set of representative images is selected for each source. These sets are sent to the output in parallel, or an alert is triggered each time any selected test set is updated, and the according new input image is sent to the output. Additionally, the selected test sets can be ranked, e.g. according to the respective scores, which results in a ranking of the image sources.

According to a further aspect of the present invention, a computer program element is provided for automatically performing a method for selecting a set of N representative images from a set of M > N input images according to any of the preceding embodiments when executed on a processor.

BRIEF DESCRIPTION OF THE DRAWINGS

The herein described invention will be more fully understood from the detailed description of preferred embodiments given herein below and the accompanying drawings, which should not be considered as limiting the invention described in the appended claims. The drawings are showing:

Fig. 1 in a block diagram an overview of a system including an image selection device according to an embodiment of the present invention, for processing images of an image source in order to select representative images and display them;

Fig. 2 in a flow chart a sequence of steps involved for selecting a set of representative images according to an embodiment of the present invention;

Fig. 3 in a diagram an image selection device for selecting a set of representative images according to an embodiment of the present invention, wherein the processing for selecting the set of representative images is performed after all images to be evaluated are available; Fig. 4 in a diagram an image selection device for selecting a set of most representative images according to an embodiment of the present invention, wherein the processing for selecting the set of representative images is performed during a recording of the images whereby the set of representative image representations is updated on-the-fly;

Fig. 5 in a diagram an image selection device for selecting a set of most representative images according to an embodiment of the present invention, wherein a new image representation replacing an image representation in the selected test set triggers a display of the corresponding input image; and

Fig. 6 in a diagram a plurality of image sources and an image selection device according to an embodiment of the present invention, configured for selecting sets of representative images.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Figure 1 schematically shows an image selection device 100 configured to receive a set of images l={li, .U I_M} recorded and transmitted by an image source 1 , and further configured to select a set of representative images IE^IE_L IE₂, IE₃, IE_N}, and further configured to display the set of representative images IE on a user device 2.

The image selection device 100 includes various functional modules, which are preferably implemented as programmed software modules comprising computer program code for directing one or more processors of a computer to perform functions as described in the following. The computer program code is stored on a tangible computer- readable medium, which is connected fixed or removable to the respective computer. One skilled in the art will understand, however, that in alternative embodiments the functional modules may be implemented fully or partly by way of hardware. In a variant, the image selection device 100 is further connected to or includes a memory, which is configured to store images as well as variables and parameters.

The image source 1 is configured to provide still or moving images I, as for example a video stream, a webcam stream, a surveillance video, a photo collection, digital pictures or other images. In an embodiment, the images I, originate from a video stream, wherein the images I, are recorded at a relatively high frame rate of several pictures per second, for example at 25 fps (fps: frames per second), or at any other frame rate. In another embodiment, the images I, are provided by a webcam at a relatively low frame rate, for example one image I, every ten minutes, or an image per any other time duration. In a variant, the images I, are provided according to an image or video stream standard, for example according to a JPEG standard, according to an MP4 standard, or according to another standard. For example, the image source 1 is connected to a data network, such as the Internet, and is configured such that the images I, provided by the image source 1 are accessible by users connected to the data network. For example, an IP data channel (IP: Internet Protocol) is established between the image source 1 and the image selection device 100. In another embodiment, the image source 1 such as a camera is only connected to a personal computer or a server computer having the image selection device 100 embodied therewith, such that all images I, - also denoted as image data in the following - provided by the image source 1 are kept privately and are processed on this computer. In a variant, the image source 1 is a surveillance camera in a CCTV network (CCTV: Closed Circuit Television) and the image data is not transmitted to any remote user device 2. In another embodiment, the image selection device 100 is embedded in the image source 1 and selects images IE, to be trans- mitted to a monitoring device.

On the remote user device 2, one or more representative still or video image compilations IE - also referred to as sets of representative images - are displayed for example on a screen. In a variant, any device connectable to a data network such as the Internet or a telecommunication network and capable of receiving still and/or video image data may be deployed. For example, an IP data channel may be established between the user device 2 and the image selection device 100 or a mobile terminal may be con- nectable to a mobile network according to the GSM or UMTS standard (GSM: Global System for Mobile communications; UMTS: Universal Mobile Telecommunications System). The user device 2 may be embodied as one of a web-browser, a TV screen, a computer monitor, a photo-frame, a mobile phone or a tablet PC. The manner in which the compilation of representative images IE can be displayed on the remote user de- vice 2 includes the representative images IE, themselves, a montage of the representative images IE,, animated or faded representative images IE,, cropped or scaled representative images IE,. Alternatively, any number of specific images or specific ordering can be selected for displaying, according to different criteria, including date/time, image similarity, image quality or score. Altematively, a user interface is provided on the user device 2 in order to interactively browse the set of representative images IE.

In a different embodiment, the image selection device 100 is an intermediate processing device and its output is linked to a different computer program, which uses the generated compilation of representative images IE, as input for further processing and/or information retrieval purposes. For example, the compilations IE can be used to efficiently compare the content of images broadcast by different image sources or may serve as input for efficiently indexing large amounts of image data.

The compilation that is formed by the set of representative images IE^IE_L IE_2i IE₃

IE_N} gives the user an overview about the visual content - also referred to as image content - which was included in the set of images

. ,- ··, IM} recorded and trans- mitted by the image source 1. The compilation IE of the visual content preferably is complete in the sense that it covers situations that may occur regularly but also situations that may occur rarely in the observed scene. This means that the compilation IE includes typical image content e.g. a scene or a situation that is observed frequently, but also abnormal or extraordinary image content, e.g. events that are statistically abnormal and/or rare and/or of a predefined particular interest. In order to obtain a compact compilation, it is preferred to include diversity of image content in the compilation and it is preferred not to include similar image content for multiple times. Such a compilation permits a human observer to efficiently grasp the important and interesting aspects of a scene, as it covers maximally diverse image content. And preferably, frequently appearing images permit to capture and understand the abnormal situations more easily. Furthermore, depending on the setting, some particular aspects are more relevant than others and are prioritized. According to an embodiment of the present invention, three characteristics, i.e. frequency, diversity which is also denoted as dissimilarity, and relevance of the image content are included simultaneously in the processing of the compilation IE. However, in other embodiments, any two of the three characteristics which are represented by corresponding variables may be combined without the need of including the third one. In a very preferred embodiment, the compilation IE contains high frequent and diverse image content. The resulting compilation IE may in one embodiment be displayed in real-time, for example, and is ideal to monitor video streams in live mode, and provide real-time information on the images recorded by the image source 1. As shown schematically in Figure 1 , the image selection device 100 is configured to receive images I, recorded and transmitted by the image source 1 and contains three main building blocks: An image processor 101 , a representative set builder 103 and an image selector 104. The three main building blocks are interconnected in order to select a representative and meaningful compilation of images IE among the images I, obtained from the image source 1. This compilation IE is made available for display in any user device 2 as described above. Figure 2 shows a flow chart of steps executed by the image selection device 100 of Figure 1 , for example, in order to select the set of representative images

IE₂,

IE₃ IE_N} from the set of images

l₂, l₃ I_M} recorded and transmitted by the image source 1 . In step S1 , the image processor 101 determines a set of image repre- sentations

R₂, R3 R_M} from the set of images i. in step S2, the representative set builder 103 determines different test sets of image representations RE^(q) = {RE_! ^(q), RE₂ ^(q), RE₃ ^(q), RE_N ^(q)} and selects one test set of image representations

RE₂, RE₃ RE_N}. In step S3, the image selector 104 selects a set of representative images IE corresponding to the previously selected test set RE. As schematically shown in Figure 2, the step S1 may include two steps S1 1 and S12 for encoding one or multiple input images I, into a feature vector R, [1...v] of length v, i.e. the image representation R,. These steps S1 1 and S12 are performed in the image processor 101 wherein the set of image representations R is deduced from the set of input images I. Features of an image I, are determined in a feature representation step S1 1 wherein preferably a single image I, in the image set I is mapped into a corresponding image representation R, by using any type of description of features of the image including global and/or local image features based on information about one or more of color, intensity, edges, texture or image segments. Alternatively or in addition, an image representation R, can also be deduced from more than one image and can be obtained using any other feature describing e.g. motion, speed or direction. In an embodiment, an image representation R, includes an entire frame of the image data, or a sub-part of the frame, possibly specified by an algorithm or by user selection. Any combination of the mentioned feature types can also be used. Alternatively or in addition, the image representation R, may be augmented by features such as 3D or depth information captured by a depth camera, or in the case of a video, by using audio data coming along with the images. In the second, optional step S12, the image representation R, obtained in step S1 1 can additionally be transformed into a binary image representation BR, by any type of binarization process. In one embodiment, this binary im- age representation BR, is obtained from comparing selected individual feature values in each image representation Ri. The indices, also referred to as elements of the compared features can be selected randomly but are kept fix for the entire set of images I. The binary image representation BR, mainly has advantages in terms of efficiency, as the comparison between binary features based on the Hamming distance measure can be efficiently implemented on common computer hardware using XOR operations (XOR: Exclusive Or).

As indicated in Figure 2, in step S2 different test sets of image representations RE^(q) are assembled and one test set of image representations RE is selected out of the test sets RE^(q) , which at the same time represents a selection of image representations R, out of the initial set of image representations R, which steps are preferably implemented in the representative set builder 103. The selected test set RE is a condensed and compiled version of the set of representations R and therefore N M. The value of N can be fixed or be selected by the user according to his or her actual needs. In this condensation process, the different test sets RE^(q) each containing N image representations are searched for a maximum score RS^(q) assigned. In a variant, a score RS^(q) is assigned to each test set RE^(q) possible to be built from the set of N image representations R and the test set RE with the maximum score RS^(q) assigned is selected.

In one embodiment, the score RS^(q) to be assigned to a test set RE^(q) is calculated in step S21 and is composed by at least two out of three individual scores which are calculated respectively in steps S211 , S21 1 and S213. In step S211 a first individual score Sden is determined by estimating a local density of the image representations REi^(q) contained in the present test set RE^(q). In step S212, a second individual score S_di_S is determined by estimating a dissimilarity of each of the image representations REj^(q) in the test set RE^(q) with respect to the other image representations RE ^q) in the test set RE^(q). And in step S213, a third individual score S_pre is determined dependent on a compliance of image content or other image characteristics with a predefined image content or predefined attributes respectively. The score RS preferably is a combination of the at least two out of the three individual scores S_den, Sdis. S_pre calculated in the according steps, or in a very preferred embodiment a combination of all the three individual scores S_den, S_di_S, S_pre. The combination of the individual scores S_den, S_di_S, S_pre as appli- cable may include linear combination, weighted average, product, majority, max and min combination methods as well as scaling of the individual scores S_den S_di_S S_pre as applicable.

The step S21 1 is owed to the desire, that in a preferred embodiment a representative compilation of image representations IE preferably shall contain one or more image representations R, the image content of which appear with a high frequency in the set of image representations R, such that these one or more image representations reflect image content, which occurs often in a similar manner and is common or usual in the input image stream. While it is preferred to include only one image representation of the same image, there may be several different high frequent image contents each of which is preferably represented in the compilation IE by means of a singe image representation. Hence, the frequency of appearance of such representations R, in the set R may qualify for detecting such frequently appearing image content. For determining a frequency of image content, in a preferred embodiment, each image representation R| in the set of image representations R is attributed a local density value DEN, calculated as the number of other image representations R, in the set R that lie within a fixed or predefined distance p from R, or alternatively as the inverse of the average distance between representation R, and the k representations R_j with minimal distance Dij (i,j e [1 , ... M] and j≠ i). A total density score - which is also denoted as first individual score S_den - for a specific test set of image representations RE^(q) is then the sum of local density values DEN, for all image representations R, in the test set RE^(q).

The step S212 is provided for including images in the compilation IE, which represent image content that is rare and/or unique and/or diverse from high frequent image con- tent, meaning that the compilation IE preferably captures images IE, representing diverse image content. Mostly, an image representation R, that is diverse in its image content from the image content of other image representations R_j at the same time occurs at a low frequency. In a preferred embodiment, a test set of image representations RE^(q) is attributed a higher score if it contains image representations REj ^(q) that are highly dissimilar.

More precisely, for each image representation RE,^(q) (i e [1 , ... N]) contained in the test set of image representations RE^(q), a distance measure to any other image representation RE_j ^(q) contained in the test set RE^(q) (j e [1 , ... N] and j ≠ i) is calculated and the minimal distance measure is taken as a measure for the dissimilarity of the subject image representation REi^(q). A second individual score S_di_S - which is also denoted as dissimilarity score - for the test set RE^(q) is then the sum of the minimal distances of all image representations REj^(q) in the test set RE^(q). Formally this can be written as S_di_S = sumi(min_j(Di _j)) with i,j e [1 , ... N] and j≠ i. The employed distance measure D_hl to de- termine the dissimilarity between two image representations REi^(q) and RE ^q) can be any type of distance measure, including Euclidean distance or the Hamming distance in the case of binary features. In consequence, a test set RE^(q) has a higher score when it scores a higher dissimilarity score S_di_S i.e. if it contains diversely scattered and/or distinct and/or dissimilar image representations REi^(q). The step S213 may be included when a set of representative images IE is desired to include specific predefined image content which may be image content of particular interest for the user. This image content preferably is specified prior to starting the processing of the representative image set by a user. In one embodiment, such image content may contain, for example, human faces or persons. Therefore, the representa- tive image set IE is designed to contain many images with humans or faces. In another embodiment, for example in a webcam scene, the user might want to know what happens on the street, but ignores weather changes. Hence, a predefined image content referring to street views may be considered in the scoring of the test sets RE^(q). In another embodiment, a very basic concept, such as motion might matter, therefore preferably image sequences including motion of objects or persons should be considered for the representative set IE. In a different approach, other attributes of user interest may be applied to the selection of the representative image set IE. Such attribute may, e.g. be linked to the elapsed time since an image l,was recorded, such that newer images I, or images I, recorded at a particular time of day of week/month are attributed higher individual scores, i.e. specifically a higher individual third score. In another embodiment, a measure of image quality may contribute in the third individual score such that colorful or aesthetic images are favored and blurred or out-of-focus images l, are penalized. Also a combination of predefined attributes can be included. The searched attribute does not necessarily need to be described in the space of image representations Ri as defined previously, but an image representation R, is attributed a higher score, if and possibly the more the underlying image characteristics comply with the predefined attribute. The third individual score S_pre of compliance with the predefined attribute may in the event of multiple attributes be a combination of the measure of fulfillment of all the various attributes by the image representations REi^(q) in the test set RE^(q).

As schematically shown in Figure 2, in step S3 a set of representative images

IE₂, IE₃ IE_N} is determined by selecting the images I, in the set of input imag

I2. I3 IM}. which correspond to the image representations RE, contained in the selected test set RE={RE₁, RE₂, RE₃, RE_N} as obtained in step S2. The step S3 is performed within the image selector 104 and preferably supplies the set of representative images IE to the user device 2 for displaying or for subsequent processing. Figure 3 schematically shows one embodiment of an image selection device 100, connected to an image source 1 and a user device 2. In this embodiment, the image selection device 100 selects the set of representative images IE starting from an entire batch of images that was provided by the image source 1. Hence, the entire batch of images represents the set of images

IM} to be evaluated. The evaluation in this scenario is conducted off-line, i.e. after the complete set of input images I is received at the image selection device 100. A structure 102 schematically outlines the set of input images i₂,i3 ΪΜ}, each of which images \, is represented by a corresponding image representation R, collectively forming a set of image representations

R₂,

R₃ R_M}, as explained previously. In this alternative, the representative set builder

103 executes the selection procedure based on the entire set of image representations

R and selects the N image representations RE^RE^ RE₂, RE₃ RE_N} that best serve the desire for a diverse image content in the compilation IE.

In a first approach, all possible combinations of N image representations Ri are deter- mined / assembled into a number of test sets RE^(q) and are scored before the test set RE with the highest score RS is selected for compilation purposes

However, in a different embodiment, a less time and power consuming processing ap- proach is introduced: In a first step, the representative set builder 103 is configured to select one image representation R, randomly, or alternatively the image representation R, with the highest density score S_den in the set of image representations R, or alternatively the image representation R, with the highest score S_pre measuring the compliance with a predefined image content or attribute. This image representation R, initially may define an interim test set IRE¹ with only one member R,, i.e. N=1. Next, interim test sets IRE^2(q) with two members N=2 are formed by adding iteratively the other image representations R_k≠ R, to the initial test set. A score IRS^2(q) is determined for each of these interim test sets IRE^2(q) of degree N=2, and the interim test set IRE² with the highest score IRS², e.g. may be selected. In another round, interim test sets IRE^3(q) with N=3 members are formed by adding iteratively remaining image representations R with Ri ≠Ri and R|≠R_k not yet included into the selected interim test set IRE² of degree N=2. Again, the score IRS^3(q) is determined for each of these interim test sets IRE^3(q) of de- gree N=3, and the interim test set IRE³ with the highest score IRS³, e.g. may be selected. This is continued up to the point where an interim test set IRE^{N 1} of degree N-1 is selected. Now, test sets RE^(q)of degree N are formed on basis of the selected interim test set IRE^N"1 of degree N-1 by iteratively adding all remaining image representations in the set R that are not yet member of the selected interim test set IRE^N_1 of degree N- 1. For each test set RE^(q) the score RS^(q) is determined, and the test set RE with the highest score RS is selected.

Mathematically, an initial interim test set IRE¹ with a single member N=1 is incrementally completed to a selected test set RE with N representations by incrementally adding representations R_j not yet being member of the so far selected interim test set IRE^X. In each of these iterations, the image representation Rj is added to the so far selected interim set IRE^X, which maximizes the representativeness score IRS^X of IRE^X namely IRE^X+1 = {IRE^X u R_j}, where j is selected according to j = argmaxj IRS({IRE Rj}) and j e [1 , ... M], excluding the representations R, already contained in IRE^X . This iterative procedure is continued until in a last round a test set RE is selected out of the multiple test sets RE^(q) of degree N determined by way of evaluating the assigned scores RS^(q) such that the predefined number N of representations is attained. Alternatively, a different stopping criterion can be introduced, for example such that the selected test set RE is complete if the increase in terms of score RS of { RE^{N 1} Rj} is smaller than a predefined or data-dependent threshold. Once the test set RE containing the image representations RE, considered as most representative with respect to the compilation of image content is complete, the image selector 104 selects the images IE, corresponding to the image representations RE, in the selected test set RE, and delivers them as a set of representative images IE to an output interface of the image selection device 100. When determining the set of representative images IE in this way, all images I, yet available are taken into account in the set of input images I, or alternatively the set of images I is restricted to images I, occurring in a selected temporal window out of the overall available images. Figure 4 schematically shows a different embodiment of an image selection device 100, connected to an image source 1 and a user interaction device 2. In this embodiment, the image selection device 100 selects the set of representative images IE during runtime, when the image source 1 continuously delivers new input images I, from the image stream, such that the entire set of images I is not available at the beginning of the processing but rather will be completed during the processing. In this alternative, the selection is done on-the-fly and represents a compilation IE out of the images I, available at a time t of compilation. A new image l_M+i received after this time t is first transformed to its corresponding new image representation R_M+1 in the image proces- sor 101 and subsequently the representative set builder 103 determines whether the new representation R_M+1 is to be included in the selected test set of representations RE which determines the set of representative images IE. During the first N images received, the corresponding representations R, are simply added to the yet present image representations given that the desired length N of the finally to be selected test set is not reached yet. Hence, after having received the first N images the then current test set of image representations RE contains all the N image representations

R₂, R₃, R_N} available at this point in time t, which test set at the same time represents the selected test set RE due to lack of alternative test sets containing a different combination of image representations. Subsequently, for each newly incoming image rep- resentation R_M+i, the representative set builder 103 preferably checks if the score RS^(q) of a new test set RE^(q) is increased if the new image representation R_M+i would replace any one of the present image representations RE, in the selected test set RE. In more detail, the representative set builder determines N new test sets RE^(q) by replacing each image representation RE, in the selected test set RE by the new image represen- tation R_M+i and calculates the score RS for each new test set RE^(q), namely for the new test sets {Ri, RE₂, RE₃ RE_N}, {RE_{1 f} R,, RE₃ RE_N} {REi, RE₂, RE₃ R,}. If any of the calculated scores RS^(q) of the new test sets RE^(q) exceeds the score RS of the selected test set RE, the selected test set RE is replaced with this new selected test set RE^(q) and preferably with the test set showing the highest score out of the new test sets RE^(q) and the selected test set RE, with the new image representation R_M+ now being included and the corresponding new image I +_I being added by the image selector 104 to the new set of representative images IE. Otherwise, the selected test set RE does not change. One advantage of this embodiment is that at any time, a set of representative images IE is available at the output interface of the image selection device 100, which represents the compilation of representative images in the image stream until that moment. Hence, also if the supply of input images I, from the image source 1 stops for any reason, a selection of representative images IE up to this time is instantaneously available. In this embodiment, the device for selecting a set of repre- sentative images IE is well adapted to a processing environment with limited memory space, compared to the embodiment described in Figure 3.

Figure 5 schematically shows an image selection device 100 in a variant of the previous embodiment shown in Figure 4, which is configured to provide an alert each time an unexpected event happens. As described previously, the selected test set RE can be updated by the representative set builder 103 during runtime, by building new test sets RE^(q) and replacing one of the image representations RE, of the selected test set RE by the new image representation R_M+i and determining the scores RS^(q) for the new test sets RE^(q). If the inclusion of a new image representation R_M+i in the selected test set RE produces a higher score RS the corresponding test set RE^(q) is selected and thus represents a new selected test set RE which contains the image representation R_M+_I describing a new, unexpected image content present in the corresponding new image l_M+i , that obviously particularly characterizes the image set I, and therefore is of special concern. The representative set builder 103 preferably flags the event of inclusion of a new image representation R_M+i in the selected test set RE to an image router 105, which transmits the new image I_M+_I directly to the output of the image selection device 100. This mechanism is useful in the case where the image selection device 100 serves as real-time alert device, which can provide suspicious, unexpected or in- teresting events in real-time to a user device 2 or for displaying or further processing by the user device 2 or a different device.

Figure 6 shows schematically a plurality of image sources 1 1 , 12, 1 n, which are each configured to record and transmit images. The images are received by the image selection device 100, which is configured to select for each image source 11 , 12, 1 n a set of representative images 41 , 42, 4n. In a variant, these representative images can be transmitted to a user device 2 separately for each image source 1 1 , 12, 1n. In another variant, the image selection device 100 further comprises an output selector 106, which selects one representative set of images 4x among all the sets of repre- sentative images 41 , 42 4n according to the respective scores RS. In another variant, the output selector 106 is configured to select the set of representative images 4x or a combination of sets of representative images to be displayed by further using one or more user preferences 5, 51 , 52, 5m. These user preferences 5, 51 , 52, 5m can be available as reference images or reference image representations, to which the images in the sets of representative images 41 , 42, 4n or the representations in the corresponding selected test sets can be compared to, or alternatively as any other selection criteria. The output selector 106 is furthermore configured to combine any of the variants described here.

In a different embodiment, the image selection device is configured to receive data from a plurality of image sources and to determine representative image sets and corresponding selected test sets for each of the image sources. In case any of these selected test sets is updated and a new image representation is added to any of these selected test sets as described previously for one image source, the image selection device is furthermore configured to provide an alert at the output and send the accord- ing new image to the user device. This embodiment is especially suitable, if many imaging devices may be monitored simultaneously and an alert can indicate which of the image sources might be of particular interest for a viewer or a user. In a variant, the assignment of a score to a test set and hence the selection of a test set may depend on the reflection of predefined image content in the subject test set. For example, predefined image content may include image content in form of a description of the environment such as "indoor", "outdoor", "mountain view", "landscape view", "sea view", etc., but also "clouds", "rain", "sunny weather", "sunrise", "sunset", etc. Other predefined image content may include image content in form of object and animal types, such as "spider", "bird", etc. or "car", "bus", "truck" etc.

In another variant, predefined attributes may be investigated as to match with one or more of the image representations in the test set and be scored according to a meas- ure of fulfilment: For example, such attribute may be embodied as the detection of different transmission errors, e.g. indicative of a broken camera or a failure in the data transmission. Predefined image content and/or predefined attributes may be assigned by looking up respective data in an expert database, including, detecting a known attribute in the query image. The expert database may be generated by analysis of im- ages by an expert user in order to determine a attribute C_s, which is linked to the respective representation R_s of the image l_s. and stored in the expert database. Any other technique to generate an expert database or to assign an attribute to a representation may be used. This may make use of meta-information, ie. information obtained from a camera location or website content available on the same website as the webcam image, or user view statistics and/or semantic user votes.

In a variant, the image selection device is configured to generate a map based on the physical locations of the image sources responsible for the sets of representative images for example. In another variant, the sets of representative images based on the evaluation of image data from multiple image sources are randomly arranged, are sort- ed according to further user preferences or detector scores, such as a score reflecting a level of user interest, or a score reflecting one or more weather attributes or a brightness of the sun in an image, or grouped according to subgroups as, for example, vil- lage webcams, mountain webcams, etc. The image selection device may transmit a notification to a user for further interaction after the set of representative images 41 , has been selected. Moreover, summaries may be generated from each set of representative images, for example during a time interval, for example during the last day, week, month, etc. The image selection device may be configured to generate images of partial or scaled down images, for example, which may be displayed in a pop-up window to a user, as a screen saver, in a digital photo frame or on wall screens.

Claims

An image selection device (100) for automatically selecting a set of N representative images (IE={IE_{1 t} IE₂, IE₃ IE_N}) from a set of M≥N input images

I2, l3, - ■, l }) from at least one imaging source (1 ), comprising

a. an image processor (101 ) configured to encode input images (I,) of the set (I) into corresponding image representations (R,) collectively forming a set of image representations

, R2, 3 RM});

b. a representative set builder (103) configured to determine different test sets (RE^(q) = {REi^(q), RE₂ ^(q), RE₃ ^(q) RE_N ^(q)}) each test set (RE^(q)) containing N image representations (RE,^(q)) taken from the set of image representations (R); and configured to assign a score (RS^(Q)) to each test set (RE^(q)), the assigned score (RS^(q)) being determined dependent on at least two of the following variables:

• a frequency at which image content contained in one or more of the image representations (REi^(q)) of the test set (RE^(q)) is contained in the set of image representations (R).

• a measure for a dissimilarity between image content contained in at least two of the image representations (REi^(q)) of the test set (RE^(q)),

• a measure for a similarity between image content contained in one or more of the image representations (REi^(q)) of the test set (RE^(q)) and a predefined image content and/or a measure for one or more of the image representations (RE|^(q)) of the test set (RE^(q)) fulfilling a predefined attribute; and configured to select a test set (RE = {REi , RE₂, RE₃ RE_N}) out of the different test sets (RE^(q)) dependent on the assigned scores (RS^(q)); c. an image selector (104) configured to determine the set of representative images (IE) by selecting input images (I,) from the set of input images (!) corresponding to the image representations (REi) in the selected test set (RE).

The image selection device (100) according to claim 1 , wherein the image processor (101 ) is configured to determine the image representation (Rj) of one or more input images (li) by determining one or more of:

one or more of global and/or local image features including one or more of an intensity of the image (I,), a color of the image ( ), edges of the image (I,), a texture of the im

one or more of motion or other image features including one or more of 3D information, depth information or audio data coming along with the images (li),

and is configured to arrange image feature values resulting therefrom in a vector (Ri [1 ...v]) of length v as corresponding image representation (R,).

The image selection device (100) according to claim 2, wherein the image processor (101 ) is furthermore configured to determine a binary image representation (BR,) for each image representation (R) in the set of image representations (R) according to the result of a comparison of image feature values (R,[a]) and (Ri[b]) for elements (a) and (b) in the input image. (I,)

The image selection device (100) according to one of claims 1 to 3, wherein the representative set builder (103) is configured to determine the frequency of the image content of an image representation (R,) in the set of image representations (R) by determining a density value (DEN,) representing a number of other image representations (R_j) located within a predefined distance measure (D ) from the image representation (R,). 5. The image selection device (100) according to claim 4, wherein the representative set builder (103) is configured to determine the frequency of image content for each image representation (REj^(q)) in the test set (RE^(q)) by determining the corresponding density value (DEN,); and is configured to determine a first individual score (S_den) by adding the determined density values (DEN,).

6. The image selection device (100) according to one of claims 1 to 5, wherein the representative set builder (103) is configured to determine for each image representation (REi^(q)) in the test set (RE^(q)) the measure of dissimilarity by determining distance measures (Dy) to each other image representation (RE ^q)) of the test set (RS^(q)), and by selecting the smallest distance measure (Dy) as the measure of dissimilarity.

7. The image selection device (100) according to claim 6, wherein the representative set builder (103) is configured to determine a second individual score (S_di_S) by adding the selected smallest distance measures (Dy).

8. The image selection device (100) according to one of claims 1 to 7, wherein the representative set builder (103) is configured to determine the measure of similarity between an image representation (REi^(q)) of the test set (RE^(q)) and a prede- fined image representation representing the predefined image content by determining a distance measure (Dy) there between.

9. The image selection device (100) according to claim 8, wherein the representative set builder (103) is configured to determine a distance measure (Dy) between each image representation (REi^(q)) in the test set (RE^(q)) and the predefined image representation; and is configured to determine a third individual score (S_pre) by adding all the determined distance measures (Dy).

10. The image selection device (100) according to claim 2 in combination with one of claims 4 to 9, wherein the distance measure (D^) is a distance between vectors of the respective image and/or predefined image representations, and in particular is one of the Euclidean distance and the Hamming distance.

11. The image selection device (100) according to a combination of two of claims 5, 7 or 9, wherein the representative set builder (103) is configured to calculate the score (RS^(q)) for a test set (RE^(q)) by a combination of the first, second or third individual score (S_den, Sdis, Spre) as applicable, which combination includes one or more of determining a weighted average, an addition, a product, a majority, a max value and/or a min value of the applicable individual scores (S_den, Sdis. S_pre)-

12. The image selection device (100) according to one of claims 1 to 11 , wherein the representative set builder (103) is configured to select the test set (RE) out of the different test sets (RE^(q)) according to one of: the highest score (RS^(q)), a score (RS^(q)) higher than a threshold, a score (RS^(q)) closest to a predefined target score.

13. The image selection device (100) according to one of claims 1 to 12, wherein the representative set builder (103) is configured to determine the test sets (RE^(q)) of image representations

• by selecting an interim test set with N-1 image representations

• by adding image representations (R_k) to the selected interim test set taken from the set of image representations (R) excluding image representations (R,) already included in the selected interim test set.

The image selection device (100) according to claim 13, wherein the representative set builder (103) is configured to determine the test sets (RE^(Q)) by selecting interim test sets starting with selecting one image representation (R,) as a selected interim test set and by repeating

• iteratively adding to the selected interim test set each image representation (R_K) taken from the set of representations (R) excluding image representations (R,) already included in the selected interim test set for determining different interim test sets,

• determining the scores for the different interim test sets, and

• selecting an interim test set out of the different interim test sets dependent on the determined scores.

The image selection device (100) according to one of claims 1 to 14, wherein for taking into account a new image representation (R ₊I) of a new input image (I +I ) from the at least one image source (1 ), the representative set builder (103) is configured to determine N new test sets (RE^(Q)) by iteratively replacing each image representation (RE,) in the selected test set (RE) with the new image representation (RM₊I); and is configured to assign a score (RS^(Q)) to each new test set (RE^(Q)) and is configured to select a new test set (RE) out of the N new test sets (RE^(Q)) and the selected test set (RE) dependent on the assigned scores (RS^(Q)).

16. The image selection device (100) according to one of claims 1 to 15, further comprising an alert mechanism (105) which is configured to output a new input image (IM₊I) during run-time if its corresponding image representation (RM+I) is in- eluded in the selected test set (RE) or in the selected new test set where applicable.

17. The image selection device (100) according to one of claims 1 to 15, wherein the image selector (104) or the alert mechanism (105) where applicable is configured to initiate displaying the set of the representative images (IE) in the form of one or more of the following: displaying, browsing or interacting with a human viewer in a web browser, in an application, as a video, as a slide-show, as a screen-saver, on a mobile device, or in a photo frame.

18. The image selection device (100) according to one of claims 1 to 17, wherein the image processor (101 ) is configured to receive images from a plurality of image sources (11 , 12, 1 n); wherein the image selector (104) is configured to select a set of representative images (41 , 42, 4n) for each of the image sources (11 ,

12 1n), and wherein the image selection device (100) is configured to initiate displaying all sets of representative images (41 , 42, 4n) or a new input image (l_M+i) where applicable on a user device (2).

19. The image selection device (100) according to claim 18, further comprising a selector (106) configured to rank the plurality of image sources (1 1 , 12, 1n) according to the scores (RS) of the corresponding selected test sets (RE).

20. A method for automatically selecting a set of N representative images (IE={IE , IE₂, IE₃, IE_N}) from a set of M≥N input images

, --, I }) from at least one imaging source, comprising

a. encoding input images (b) of the set (l={li, I2J3 I_M}) into corresponding image representations (Ri) collectively forming a set of image representations (R={R!, R₂, R₃ RM}), b. determining different test sets (RE^(q) = {REi^(q), RE₂ ^(q), RE₃ ^(q) RE_N ^(q)}) each test set (RE^(q) ) containing N image representations (REi) taken from the set of image representations (R), and assigning a score (RS^(q)) to each test set (RE^(q)), the assigned score (RS^(q)) being determined dependent on at least two of the following variables:

• a frequency at which image content contained in one or more of the image representations (REi^(q)) of the test set (RE^(q)) is contained in the set of image representations

(R).

• a measure for a similarity between image content contained in one or more of the image representations (REi^(q)) of the test set (RE^(q)) and a predefined image content and/or a measure of one or more of the image representations (REi^(q)) of the test set (RE^(q)) fulfilling a predefined attribute, and selecting a test set (RE = {RE^ RE₂, RE₃ RE_N}) out of the different test sets (REi^(q)) dependent on the assigned scores (RS^(q)), c. determining the set of representative images (IE) by selecting input images (li) from the set of input images (I) corresponding to the image representations (RE,) in the selected test set (RE).

A computer program product comprising program code means for automatically performing a method for selecting a set of N representative images

IE₂,

IE₃ IE_N}) from a set of > N input images

l₂, I3 I_M}) according to claim 20 when executed on a processor. An image selection device (100) for automatically selecting a set of most representative images (IE={IE , IE₂, IE₃ IE_N}) from a set of input images (Ι={Ί, l₂, l₃ I_M}) with M > N from at least one imaging source recorded at unknown, possibly variable frame-rate, comprising

a. an image processor (101 ) encoding the input images

, I2J3 IM}) in image representations

, R₂, R3 R_M});

b. a representative set builder (103) scoring a possible subset (RE^(q) =

{RE^ RE₂ ^(q), RE₃ ^(q) RE_N ^(q)}) with N representations (REi) taken from the set of representations (R), such that a higher score is assigned to a subset (RE^(q)) according to one or a combination of the following criteria:

• (RE^(q)) contains common, usual and frequently appearing image content with respect to the initial set (R),

• (RE^(q)) contains rare, unique, or maximally dissimilar representations from the initial set (R),

• (RE^(q)) contains images exhibiting particular, predefined concepts,

and selecting one of the higher, preferably the highest scoring set (RE^(q)) as the set of most representative representations

, RE₂,

c. an image selector (104) selecting the images in the set of input images I, which correspond to the representative representations in (RE) as the set of most representative images

, IE₂, IE₃ IE_N}).