WO2014009751A1

WO2014009751A1 - Method and apparatus for image searching

Info

Publication number: WO2014009751A1
Application number: PCT/GB2013/051873
Authority: WO
Inventors: Benjamin Kenneth Coppin; Mustafa SULEYMAN
Original assignee: DeepMind Technologies Ltd
Current assignee: DeepMind Technologies Ltd
Priority date: 2012-07-13
Filing date: 2013-07-12
Publication date: 2014-01-16
Anticipated expiration: 2015-01-13
Also published as: US20140019431A1

Description

METHOD AND APPARATUS FOR IMAGE SEARCHING

FIELD OF INVENTION

This invention relates to a method and apparatus for searching for images, in particular the field known as CBIR (Content based image retrieval) or reverse image searching.

BACKGROUND TO THE INVENTION

The weakness with many traditional image search technologies is that they rely on a user describing the search in keywords. This works extremely well in some cases (e.g., "photo of Barack Obama") but does not work for images that have no keyword descriptions, or for concepts that can only be described by reference to an object or another photo, or an abstract idea. CBIR to allows a user to search for images based on content rather than by entering keyword queries. Two well known examples of TinEye: http://www.tineye.com/ and CBIR are Google™ image search

http://www.google.com/insidesearch/features/images/searchbyimage.html. Search is, fundamentally, a way to explore a space. In the case of a text-based search engine, the space that is being searched is typically web pages, and the space is organised around the textual content of those pages, and relationships between those pages. This space can be visualised as a large graph or tree, that is, as a series of nodes connected by arbitrary connections. Navigating the space is only really possible by using keywords to identify potentially relevant pages or following links from one site to another.

The problem with this model is that it means that a user can become disoriented and lose any ability to perceive the current position relative to the search space. This means that a user is often unable to follow a single strand through the space. After entering a query in a search-engine and clicking on a promising-seeming link, the user may be taken to a page, which may or may not contain the desired results. If the linked page does not contain the desired result, the user's only option is to follow links from that page or go back to the search results and try again. Perhaps the user has found an interesting page, containing resulting images similar to the desired result. Often, the only way to improve the search is to go back to the search engine and enter a modified query, using keywords learned from the previously visited interesting page. This means that the user needs to understand quite a lot about the way that search engines work, and needs to optimise queries accordingly. It also means that the user needs to store a lot of information as the search progresses, since finding the right content can be a hit-or-miss process.

Searches based around images are also known in the patent literature.

US2010/135597 describes a system and method for enabling image searching and includes an image analysis module that is configured to programmatically analyze individual images in a collection of images in order to determine information about each image in the collection. US7016916 describes a method of searching multimedia data is disclosed in which a search for an image can re-performed by automatically updating weights of features and/or weights of feature elements in the respective feature in an image.

The applicant has recognized the need for an improved searching method and apparatus.

SUMMARY OF THE INVENTION

According to a first aspect of the invention there is provided a data processing system for creating a search query comprising a processor configured to:

receive a first user selected image;

extract a first set of features from said first user selected image;

receive a user selection of a first selected feature of a first type from said first set of features;

receive a second user selected image;

extract a second set of features from said second user selected image;

receive a user selection of a second selected feature of a second type from said second set of features; and

combine said first and second selected features to form a composite image search query.

According to a second aspect of the invention there is provided a method of creating a search query comprising

receiving a first user selected image;

extracting a first set of features from said from first user selected image; receiving a user selection of a first selected feature of a first type from said first set of features;

receiving a second user selected image;

extracting a second set of features from said from second user selected image; receiving a user selection of a second selected feature of a second type from said second set of features;

combining said first and second selected features to form a composite image search query. In both aspects, the system or method may receive a third user selected image, extract a third set of features, receive a user selection of a third selected feature from said third set of features and combine this third selected feature with the first and second selected features to form a further composite image search query. The data processing system may further comprise a feature extractor module that is configured to segment at least one of said selected images into a plurality of objects. The feature processor may additionally output said plurality of objects to said user as said extracted set of features. The first user selected image and the second user selected image may be partial images selected from a single main image. The system may be configured to segment the single main image into a plurality of partial images which may be displayed to a user to select the partial images.

The first and second set of features may be image content based features from the group comprising of at least a shape of an object, a colour of an object, a pattern of an object and a texture of an object. Thus, the first and second types of feature may include shape, colour, pattern and texture. The third selected feature may be of the same type as the first or second selected feature type. The system may apply a weight to each selected feature when combining to form said composite image search query or the system may apply a filter to restrict a search on said composite image search query. The system may also segment at least one of said first user selected image and said second user selected image into a plurality of objects and output said plurality of objects to said user to select at least one of said first feature and said second selected feature. The system may also be configured to search for results which match said first selected feature and output said search results to a user. The system may then receive the second user selected image which is an image selected from within said search results. According to a third aspect of the invention there is provided a query engine for conducting a search comprising a server which is configured to:

receive a first user selected image;

extract a first set of features from said first user selected image;

search for results which match said first selected feature;

output said search results to a user;

receive a second user selected image which is selected from within said search results;

extract a first set of features from said first user selected image;

combine said first and second selected features to form a second composite image search query.

According to a fourth aspect of the invention there is provided a method of conducting a search comprising:

receiving a first user selected image;

extracting a first set of features from said from first user selected image;

receiving a user selection of a first selected feature of a first type from said first set of features;

searching for results which match said first selected feature;

outputting said search results to a user;

receiving a second user selected image which is selected from within said search results;

extracting a second set of features from said from second user selected image; receiving a user selection of a second selected feature of a second type from said second set of features; and

receiving a user selection of a second feature; combining said first and second selected features to form a composite image search query.

In both aspects of the invention, the server or method may be further configured to: receive a third user selected image; extract a third set of features from said third user selected image; receive a user selection of a third selected feature from said third user selected image; and combine said third selected feature with the first and second selected features to form a further composite image search query, the first selected feature may be selected from the group consisting of a shape of an object, a colour of an object, a pattern of an object and a texture of an object. Similarly, the second or third selected feature may be selected from the group consisting of a shape of an object, a colour of an object, a pattern of an object and a texture of an object.

The server may be further configured to receive further user selected images selected from any sets of search results which have been displayed in response to earlier composite image search queries. At least one of said first user selected image and said second user selected image may be segmented into a plurality of objects which may be presented to said user to select at least one of said first selected feature and said second selected feature. The first user selected image may be a partial image from a single main image and the second user selected image may be a partial image from the said single main image. The server may be further configured to apply a weight to any one of selected features when combining to form said composite image. The server may be further configured to receive a user selection of said weight for any one of selected features. The query engine may further comprise a user database to store results of previous searches. Said server may be further configured to apply a filter to restrict a search on said composite image.

According to a fifth aspect of the invention there is provided a query server for creating a search query comprising:

a web server, configured to receive a first user selected image and a second user selected image;

a feature extractor, which extracts a first set of features from said first user selected image and a second set of features from said second user selected image; said web server also configured to receive a user selection of a first selected feature of a first type from said first set of features and a user selection of a second selected feature of a second type from said second set of features; and

an image search engine, which combines said first and second selected features to form a composite image search query.

In each of the above embodiments, the set of features extracted from each of the user selected images are of types that can only be determined by analysis of the image, either in advance or as the composite image search query is created. By combining two distinct and different features extracted from each of the two images, the invention allows the user to search from multiple images to specify the desired image result. By providing this functionality in a query server, an improved apparatus for searching is provided. It will be appreciated that using two images is illustrative and a further user selected image and user selection of a further feature of a third type may be used with said combining step combining all said user selected features to form said composite image search query. It will also be appreciated that such a composite image search query enables a user to select multiple features to refine the search, to create an entirely non-verbal search structure. Unlike prior art devices and methods, the query engine does not require advance preparation of additional desirable search

parameters, such as metadata, for the images to be searched.

By analysing and combining features from extracted entirely different images, or partial images, the invention provides for a powerful and efficient search query that will generate meaningful results within the first iteration of searching. The system does not require the image content to be tagged with metadata in advance in order to generate desirable first round results.

These tools encourage users to adopt an exploratory search method, where the user starts from one point (perhaps an image on a web site) and allows the user to systematically refine a search in a series of steps, possibly through one or more iterations of the steps described above until a desired image or images are identified. A later composite image search query may be processed in a similar manner to the first composite image query and it will be appreciated that there is no limit to the number of subsequent image queries that may be received. The results of subsequent composite image search queries may be displayed together on a single user interface thus allowing a user to selected images from within earlier search results.

According to another aspect of the invention, there is provided a query engine for conducting a search comprising: a processor which is configured to receive a first image query comprising a user selection of a first feature within a first selected image; output said first image query to a search engine which is configured to search for results which match said first image query; receive said first set of search results from said search engine; transfer said first set of search results to a user interface for display to said user; receive a second image query which comprises a user selection of a second feature within a second image selected from within said first set of search results; repeat said outputting and receiving steps to obtain a second set of search results for said second image query; transfer said second set of search results to said user interface display to display said first and second search results together on said user interface and receive a third image query based on a third selected image which is selected from within said first set of search results and said second set of search results.

Said processor may be further configured to repeat said outputting and receiving steps for said third image query to obtain a third set of search results for said third image query. Said processor may be further configured to transfer said third set of search results to said user interface display to display said third set of search results together with said first and said second set of search results. Said processor may be further configured to receive a fourth image query based on a fourth selected image which is selected from within said first set of search results, said second set of search results and said third set of search results. Said processor may be configured to receive a plurality of image queries which each of said plurality of image queries based on at least one image which is selected from any sets of search results which have been displayed in response to earlier image queries

The following features apply to all aspects of the invention.

Each selected feature may be a subsection of said user selected image, e.g. a chair within a picture of a room. The selected feature may be a shape, colour, texture, pattern or other parameter of an object within said user selected image. There may be more than one feature selected from within each selected image for each composite image search. For example, a user selection of a first feature may be received from within said first user selected image and a user selection of a second and third feature may be received from within said second user selected image. The desired result image would therefore be a composite image combining all three features. By combining features from two or more images, the invention allows the user to search from multiple images as well as specifying the feature in each image that should be included in the search. By providing this functionality in the query server, an improved apparatus and method for searching is provided.

At least one of said user selected images may be segmented into a plurality of objects which may be presented to a user, e.g. on a user interface. Said feature may be selected from one of said plurality of objects, e.g. by clicking on said object. A weight may be applied to each selected feature when combining to form said composite image and said weight may be adjusted by said user.

The invention further provides processor control code to implement the above- described systems and methods, for example on a general purpose computer system or on a digital signal processor (DSP). The code is provided on a physical data carrier such as a disk, CD- or DVD-ROM, programmed memory such as non-volatile memory (e.g. Flash) or read-only memory (Firmware). Code (and/or data) to implement embodiments of the invention may comprise source, object or executable code in a conventional programming language (interpreted or compiled) such as C, or assembly code. As will be readily appreciated by the skilled person, such code and/or data may be distributed between a plurality of coupled components in communication with one another.

BRIEF DESCRIPTION OF DRAWINGS

The invention is diagrammatically illustrated, by way of example, in the accompanying drawings, in which:

Figure 1 is a flowchart showing the steps of a method for selecting images;

Figure 2a shows one application of the method of Figure 1 ; Figure 2b shows an alternative application of the method of Figure 1 ;

Figures 3a and 3b are representations of weighting which is an optional feature in the method of Figure 1 ;

Figures 4a and 4b illustrate different ways of selecting an input image for the method of Figure 1 ;

Figure 5a is an illustration of a typical system for implementing the method, and

Figure 5b is a screenshot showing an example of how the browser extension (with Google's Chrome browser) might be implemented.

Figure 6 is a flowchart of an iterative search through multiple search results;

Figures 7a to 7c show graphical user interface at various stages through Figure 6 allowing a user to navigate a search;

Figures 8a and 8b are alternative graphical user interfaces for presenting search results to a user;

Figure 9 is a block diagram of the system for implementing the method of Figure 6;

Figure 10 shows an embodiment for generating a composite image search query; and

Figure 1 1 shows an alternative embodiment for generating a composite image search query.

DETAILED DESCRIPTION OF DRAWINGS Figure 1 shows the steps used by a system to assist users in searching for images. In this example, all the images available to be searched by a user have been previously analysed, indexed and stored that in a feature database. For each image, a number of features of different types have been selected and values for each features of each image have been captured in the database, along with the URL of the image. A number of different algorithms may be applied for the both the feature extraction, searching and result presentation. Examples include:

• Colour matching: for example by comparing histograms of colours using chi- squared distance as described in "A Study of Color Histogram Based Image Retrieval", Rishav Chakravarti, Xiannong Meng, 2009 Sixth International Conference on Information Technology: New Generations

• Colour Coherence using the methods outlined in "Comparing Images Using Color Coherence Vectors" by Greg Pass , Ramin Zabih , Justin Miller in MULTIMEDIA '96 Proceedings of the fourth ACM international conference on Multimedia, pages 65-73

• Shape matching: for example by using a method such as Histogram of oriented gradients (HOG: http://en.wikipedia.org/wiki/Histogram_of_oriented_gradients) or as described in US671 1293.

• Texture matching: for example by using a method such as that described by Haralick in his 1973 paper: "Textural Features for Image Classification", IEEE Transactions on Systems, Man and Cybernetics, Nov 1973, ISSN 0018-9472.

• Pattern matching: for example by matching larger-scale patterns such as

stripes, dots, flowers and checks that appear on clothing and other products, as described in the applicant's UK patent application GB1219650.7 and the applicant's US patent application US13/801 , 027.

The invention is not dependent on these specific methods and could be deployed using a different set of algorithms for matching shape, colour, pattern and texture, or indeed using algorithms for matching a range of other feature types (e.g. automatically extracted objects).

The first step S100, a user to selects an image to form the basis of the search. The system extracts features from the user selected image (step S101 ), as described below. The user then specifies which of the extracted feature(s) within the selected image are to be used in the search (step S102). The features may be one or more of colour, colour coherence, pattern, texture or shape of an image, or a those features presented in a subsection of an image. Colour coherence is a measure of the importance of the colour within an image. For example, some red may be scattered (perhaps invisibly) through an image (say of a human face) and this would have a value for coherence that is less than for an image containing a coherent block of red (say in a rose). These features may be used individually or in combination to refine the next round of search results. The user may specify a part of the image by selecting parts of images, for example by using one of:

• Rectangular selection boxes which are overlaid on the original image (as shown in Fig 2a)

• Polygonal (in which the user selects a number of points on the edge of the area of interest, and a polygon that joins those points is created)

Lasso (in which the user draws free-hand around the area of interest).

Automatic segmentation (in this case, the system automatically segments the image into a number of objects, and the user is able to simply click on one object (or more than one object) to indicate interest, or automatic heuristics are applied to select the most likely object).

As set out above, the feature(s) selected may be part of an image or a feature within part of an image. The user can indicate which features are to be searched, and can combine a partial image with another whole or partial image. The user indication of the feature(s) within a whole or partial image(s) may be, alternatively, a textual description, e.g. "Top Right". For example, the user may say "I want to search for an image that has the colour of this part of image 1 but the shape of this part of image 2". Automated segmentation facilitates such selection. The segmentation would enable the user to select an object within an image (e.g. a car, a dress, a cat or a tree) by simply clicking on the object of interest. In this case, the system could optionally provide a textual description of the selected feature(s) and image(s), for example by displaying a message such as: "You have selected an image of a lady wearing a black dress". The user indication could then be confirmation that the message is in line with their selection.

The next step S104 is to consider whether or not other images are to be added into the search. If additional images are to be used, the method loops back to the first step and repeats the selection of the image and the selection of the feature within the image. If additional images are not to be used, the method combines the selected image(s) and feature(s) at step S106 to create a composite query that can be searched. Creating the composite query may, optionally, comprise creating a composite image made up of the selected image(s) or feature(s). This combination step may be termed 'Clamp and Combine'. 'Clamping and combining' allows the user to select a series of distinct and specific aspects of an image (for example its shape alone, or a combination of colour and texture) that are then "clamped" into the search. This effectively filters the search results with multiple clamps, which when combined provide a more refined and useful end result.

The method may include optional weighting features by the user, enabling the user to indicate a preference for features to be displayed in the next round of search results. As shown at step S108, these weights may be presented to a user. At step S1 10, user input on weighting is received. The user input may be provided in response to the presentation at step S108 or may be independently input.

The method may also include optional domain filtering. At step S1 12, the user also has may impose a structured domain filter on the image. For example, the user might select an image of a dress but restrict the search to the domain of skirts or curtains, to find a different type of item that has similar colours or pattern. Once all the inputs are received, the search is carried out at step S1 14 and the results are output (step S1 16). The output of the searches may be a ranked list as is well known in the art or alternatively, the output may not be ranked.

As an alternative for searching for matching results, the system may generate new images at step S1 14. For example, the user may select an image of a dress and select the colour green to form the composite image to the input to the system, i.e. the user might say "I'm looking for a dress like this, but in green". If a search finds an existing image that combines the required features, this will be presented to the user, but in the case where no such image exists, the system might generate a new composite image that combines the selected features. The results may be the first round of a search process and the user may be queried to indicate whether the search results are acceptable at step S1 18. If the user has found what they are looking for, no further searching is required and the process ends. Otherwise, the images delivered as part of the search results may be used to form the basis of the next round of searching. This may be as simple as a user clicking on one of the images from the search in which case the method returns to step S100.

Alternatively, an iterative search as detailed in relation to Figures 6a to 9 may be initiated.

The selection, or non-selection of features effectively enables a range of features, either together or individually), as part of the search process. Thus it becomes possible for a user to provide non-linguistic, highly intuitive feedback to the image search engine. This is much more like the way humans naturally describe things to one another by pointing and showing, and saying 'more like this bit' or 'similar to this shape'. By iterative repetition of the search steps, a user can 'steer' their way towards a satisfactory end result, without needing to describe the end result in words.

The method described herein encourages a new way of searching for images via an evolutionary navigational process. In other words, a user might start with a query, e.g. a dress, narrow the search by specifying a particular feature, e.g. colour, narrow the search further by combining this with a feature from another image, e.g. the texture of a shirt, and then navigate by clicking on the images that seem closest to the one they are looking for. Each click on an image starts a new search, possibly modulated by the elements included in the original search, and brings the user one step closer to a desirable result. Back-tracking may be facilitated by use of a specially adapted user interface as described in more detail below.

Figure 2a shows one application of the method of Figure 1 . The user selects an image and selects a feature (or more than one feature) from part of an image in accordance with steps S100 and S102 of Figure 1 . In this case, the user has found a photo 10 of a room, and has selected the armchair 12 as being the feature of the first type on which an element of the search is to be based. No additional images are used and the search begins using only one input, the armchair. In this example no optional weighting is applied. Three different results 14 are returned by the search. Each of these results is a different armchair or sofa having similar colour and style to the one selected. The user may then select a second feature of a second image, using one or more of the initial search results, to refine the search.

Figure 2b shows an alternative application of the method of Figure 1 in which a user selects different features of different types from a plurality of images and combines the different features to form a composite query. A user selects the shape 20 from a first image 30, in this case a dress, in accordance with steps S100 and S102 of Figure 1 . The user then uses additional images and method repeats steps S100 and S102 to select the colour 22 from a second image 32, in this case a different dress. Finally, in a third iteration, the pattern 24 from a third image 34, also a different dress, may be selected. In this case, the colour and pattern have been selected from parts of the second and third images, rather than using the colours of the images as a whole, although the latter would also be possible. Accordingly, in this implementation, the composite search query is based on the features of shape, colour and pattern together. In this example, a result image 28, having the selected shape 20, colour 22 and pattern 24, is provided.

Figure 3a illustrates one method of presenting a user with a weighting for a feature. For example, the user may be shown the colours that were identified in the first image that forms the basis of the search. In this case, the image 40 selected is a dress, and the bar 42 above the image shows the relative weights of each colour contained in the image. In this example, a bright red has the highest weighting with a first shade of black having the next highest weighting. The user can adjust the relative weights of the individual colours, for example, by dragging a marker over the colours. Alternatively, a user can remove colours, for example by clicking on the colour and selecting delete. Finally, a user may be able to add in new colours, for example by inputting a textual description (e.g. "I'd like to find an item that is this shade of red, but with a bit of green added in as well") or alternatively via a menu to allow selection of other colours.

The user may be shown the representation of Figure 3a along with their search results. This may help a user to adjust the weighting to remove the unwanted results. The representation of Figure 3a may also be adapted to show other features which could be weighted, for example as shown in Figure 3b. The bar may show the weighting of the shape as well as the colour and other features, such as texture, which enables a user to set the relative importance of each feature, e.g. to say that shape is more important than colour which is more important than texture. A representation of the weighting of the colour (or other feature) from each element forming the composite query may be shown. For example, where the composite query combines the colours of two images, a user may be able to show that the colour of the first image is more important (and thus to be more highly weighted) than that of the second. This could be enabled through a set of sliders that the user can slide to set the relative weights. Figures 4a and 4b illustrate how the first step of the method of Figure 1 may be completed. Thus step S100, which starts a search from an image, may not be the first step in the process. As explained below, the user could start a search by selecting a colour, or more than one colour. In Figure 4a, the user is presented with a colour palette 50 comprising a plurality of colours. A user selects one colour, e.g. by clicking on it and a bar 52 showing the selected colour is presented to the user. A mechanism is also provided for a user to deselect the bar 52, in this case by clicking on the cross button. Once the colour selection has been made, a user is shown images 54 that are largely made up of that single colour, such as red. In Figure 4b, the user has selected a second colour, such as yellow, and is shown images 56 made up of those two colours in combination.

Thus, the first step S100 of Figure 1 may be to select one or more of these images. The system preferably also provides storage so that, having identified images of interest, the user has the ability to save those images, or parts of the images, or specific features of the whole or partial image, for future searches. Hence, the user might see a dress in a desired style, and could indicate to the system: "Find me dresses like this, but in the colour of that pair of shoes I saved last week".

Figure 5a shows a system in which the method may be implemented. The search service is deployed using the normal components of a search engine, which includes at least one query engine 74 to prompt for and respond to queries from users. This system may be formed of many servers and databases distributed across a network, or could be consolidated at a single location or machine. The term 'search engine' can refer to the front end, which is the query engine in this case, and some, all or none of the back end parts used by the query engine, whose functions can be replaced with calls to external services.

A user can make searches via the query engine using an input device 70. The input device may be any suitable device, including computers, laptops, and mobile phones. The input device 70 is connected over a network 72, e.g. a wireless network managed by a network operator, which is in turn connected to the Internet via a WAP gateway, IP router or other similar device (not shown explicitly). Each input device typically comprises one or more processors 84, memory, user interface devices such as keypad, keyboard, microphone, touchscreen, 86, a display and a network connection such as a wireless network radio interface.

The processor 84 of the input device 70 may be configured to create the composite query that is sent to the query server 72 for searching. Thus the processor of the input device may be configured to receive a user selection of at least one image and at least one feature within each image, e.g. from the user interface on the input device 70. The processor 84 may then combine the selections, add any weighting or filters and send the composite query to the query server. Some or all of the steps in creating the composite query may be undertaken by the processor 82 of the query server. In this case, the processor of the query server may be configured to receive a user selection of at least one image and at least one feature within each image from the input device 70. The processor 82 may then combine the selections, add any weighting or filters and search for the resulting composite query. As explained above, the method provides a better query, which initiates the search and thus when the query engine is enabling a user to the input this improved query, the query engine is effectively acting as a more efficient query server.

As shown in Figure 5a, the query engine(s) 74 are connected to an image database 76 and a feature database 78. These are stores of images and features that can be presented to a user on the user interface of the input device 70 for selection. These databases can also be used to store images and features for individual users, for example, as explained with reference to Figures 4a and 4b. Both the image and feature databases 76, 78 are connected to a feature extractor 80. The feature extractor 80 takes images from the image database 76 and automatically segments them into individual features that are then stored in the feature database 78.

The method could be implemented in a number of forms, for example:

• As a browser plug-in / extension. When a user views an image in a user interface 60 they are interested in, they could right-click on the image to reveal a context- sensitive menu 62. Within this menu would be an option to search for similar images. Having selected this, a side-bar 64 would appear showing similar images and providing further options. This is illustrated in Figure 5b.

• As a dedicated web site. • As an addition to an existing e-commerce site.

• As a native app on a mobile phone or other hand-held device (and in this case it could be used to find similar objects to one contained in a photo taken using the device). There are various applications for the described method. For example, with reference to Figure 5b, the method could effectively provide an online shopping assistant. This enables people to search for items that they might otherwise find hard to find. One example of the mechanism might be a tool that a user can click on to indicate they are interested in finding other images similar to one they are viewing on a web page. This could have a commerce aspect: the user might be viewing a picture of a watch, and by clicking on the image they could be shown similar watches for sale, with links to sites (or a single site) selling similar watches.

Clearly, there is a more general application as a tool for helping people find interesting content. Like Figure 5b, this could sit as a side-bar in the web browser, and as the user views a page, the side-bar would update with images similar to the ones on the page. Another application is as a tool to assist designers in finding images (photos, icons, drawings, etc.) that have appropriate colours, patterns or shapes for use in marketing material, web site design and other design elements.

Figure 6 illustrates a way of allowing the user to navigate a search space iteratively, providing the user with a sense of context, location within the search space, and also providing far more fine-grained control over where next to go in the search space. The user starts by entering a query (S200), which could be specified as a keyword query or by pointing to an image (e.g., by uploading it from a phone or by clicking on an image on a web page). As shown in Figure 7a, the user is shown images that meet the search criteria (S202), e.g., by being similar to the query image. The search may be conducted using any known technique including those detailed above in relation to a composite query. The user may then review the search results to see whether or not one of the images from the search results matches expectations (S204). If the correct image is shown, the user can click on it to see it in its original context, for example the page on which it was hosted or the site from which the item pictured can be purchased, and the system can output more details as required (S206). However, if the correct image has not been found, the invention provides the user with a number of choices.

First, the user can initiate a new search from a combination of the search results (S208). This could be achieved simply by selecting one of the displayed images and initiating a new search. The selected image may be very close to what a user is seeking or alternatively, may just be a step closer to the desired result. In this case, the user is embarking on an evolutionary-style process of manual artificial-selection. In other words, perhaps the user is searching for a striped shirt of a particular shade of dark blue, and the system has shown a lot of shirts in various colours. Accordingly, the user selects an image containing a dress of appropriate shape that is pale blue because it is, at least, blue. The user can also select an image of another dress, which not of the desired shape but is of a more desirable dark blue colour. The next set of search results contains a lot of blue dresses, including some that are darker blue, so the user selects one of these darker blue images. The next set of images are all dark blue dresses, and the user can keep following this process until they have narrowed in on the precise item they are searching for.

For example, as shown in Figure 7a, the user has run a search for 'shirts' and selected, as shown in figure 7b, the colour of the first image, all features (colour, pattern and type of object) in the third image and the pattern from the fourth image. A new search is run on this combination and the results of the second search are shown on the user interface (S210). A key difference to a standard set of search results is that the results of the second search are shown on the same user interface as the original search results (in this case below).

The method then loops back to step S204 to determine whether or not the correct image is shown. As before, if one of the search results is suitable, the search is terminated. However, if the search results are not yet sufficient, the user can run another search.

The user can then select a single feature or a combination of features from one or more images in the second set of results and run a new search. For example, the user can select the pattern from the fifth image and the results of this search are shown in Figure 7c. Again the results for the third search are shown with the results from the first and second searches. In this case, the search has returned a variety of different images all having stripes as the predominant pattern.

Such a presentation of results allows the user to follow an iterative search mechanism. For example, after following a thread towards stripes, perhaps the user realises that he is only interested in striped shirts. The user interface of Figure 7c gives quick and easy access to the search results for the previous queries. So the user can point at one of the current cohort of images and say "this colour" and can then point back to an earlier query and say "this pattern" or "this style", creating a new combined query which is effectively illuminating a more focused path through the search space of images. As another alternative, perhaps the user has followed a search thread towards darker dresses and now realises that although the dresses are of the right colour, the dresses are no longer in the right style. Perhaps the user is looking for a ball-gown and the search has produced mainly plain cocktail dresses. The user now wants to be able to say something like "I want dresses that are this colour but have the same style as the dresses I had in the results list for my first query". The iterative search system provides this capability. Similarly, the user may go down a wrong path, perhaps making the dresses too dark, and they can then easily backtrack up a level to see the previous set of dresses and follow a new path from one of those.

This process can be repeated, enabling the user to add many images to their search: "I want something that captures the essence of these 5 images". The search process may be hidden from the user and made automatic: effectively learning from a user's behaviour what kinds of images, colours, shapes, styles or objects are preferable, so that when a user initiates a completely new search this additional information can be taken into account to bias the first set of results. In Figures 7a to 7c only six results are shown at the end of each search because this is the number that can be reasonably shown across a graphical user interface. Figure 8a shows an alternative graphical user interface in which more than six results are shown. The user can scroll along the string of search results to access more than the six results that can reasonably be shown on the interface. Figure 8b shows another graphical user interface in which four search results are presented. Symbols rather than letters are used to depict the features (colour, pattern and type) that may be selected. Although Figures 7a to 8b show linear representations for the search results, this is not the only way to display the results. The system allows a user to navigate a search space by expanding branches of a very large tree, so it may also be possible to show results in a tree-structure or in a number of other possible layouts, such as concentric circles.

Figures 7a to 8b also show only a maximum of four sets of search results on a single page of the graphical user interface. However, all of the earlier sets of search results are also retained so that a user may select feature(s) from image(s) in any previous search. It is expected that a vertical scroll bar will also be included to allow a user to access previous search results. However, it will be difficult for a user to navigate all the previous sets of search results if too many results are presented. Accordingly, the user interface may be enhanced by including a side-bar or other drop zone on the screen into which a user can move individual images. These images may form a set of favourites for a user. Any images moved, for example by simple drag and drop, into this area may be stored for ease of including them in subsequent searches.

Some or all of the images in the drop zone may be combined with some or all of the images in other search result sets. This means, for example, that it is to combine an image from one query with an image from a query that is carried out many queries later. It also could become a mechanism for a user to store all kinds of items that they like, indicating that they like everything about one image i.e., all features, the colour of another image and the pattern of yet another image. The user could then request the that system "carry out a search for an image like this dress, but take into account my entire set of favourites", which would create a very large query, combining features from lots of images or any other items stored in the drop zone.

Figure 9 shows an alternative system diagram in which the system of Figure 5a has been adapted for the iterative search method of Figures 6 to 8b although it will be appreciated that the system of Figure 9 may also be used in other embodiments. The system comprises an input device 70. In the example shown in Figure 9, the input device 70 is a personal computer but it will be appreciated that any suitable computing device, e.g. phone, laptop, etc. may be used. The system further comprises a query engine or server 52, which in turns further comprises a plurality of modules including a web server 54, image search engine 74, a feature extractor 80 and the feature database 78 on a storage medium such as disk. In this example, prior to any searches being conducted, the system analyses at set of images, stored on disk 76, and extracts all of the features that may be later searched by a user. The results of the feature analysis is stored in a feature database 78, along with the URL of the image and any other relevant metadata that may be required by the search engine 74.

The feature extractor extracts multiple features from each image to be indexed using the mechanisms described supra, to create a set of features of different types: colour, colour coherence, shape, pattern, texture, etc. It will be appreciated by one skilled in the art that a number of image based features may be selected, depending upon the nature of the images to be searched, which each type of feature adding a new dimension to the search capabilities of the invention. It will also be appreciated that the additional feature types may be added during operation of the query engine by reprocessing the images and adding the additional analysis to the feature database. A user may input a search query on the input device 50, for example into an application running on a web browser on the PC. The input device may also have local storage that stores the results from each iteration of the search. The query is submitted, via the Internet, to the query engine or server 52. The search query is received at the web server that in turn passes the query to the image search engine 74. The image search engine 74 checks whether the features for the query image are already available. If the query image has not been previously indexed and is not available, the image search engine 74 passes the query image to the feature extractor 58 to extracts features from the query image as described above. Once the image search engine has the required features, these features are then compared with the features for the images in the feature database 78 to find the most similar images.

These images are submitted back, using the URL stored in the feature database and via the web server, to the user's browser as a set of search results.

A key difference in the proposed method is ability to display multiple historic searches. As described above, information about the query and its results may be stored in the local storage on a user input device but in the alternative such storage may be managed by the web browser. For example, on a subsequent query, information from the local storage may be combined with the information from the current query results to generate a new query, which proceeds as above. Alternatively or additionally, a user's query information could be stored by the query server in a user database 64, so that subsequent queries that the user makes (say, from a different PC or just after the local storage has been cleared) could still take into account previous query results or stored information.

In all the embodiments above, the image is preferably a digital image which may be stored in any convenient file format, such as JPEG, GIF, BMP etc. The image may be a photograph, a graphic, a video image or any combination thereof. Each digital image includes image data for an array of pixels forming the image.

In Figures 5a and 9, the server is shown a single computing device with multiple internal components which may be implemented from a single or multiple central processing units, e.g. microprocessors. It will be appreciated that the functionality of the server may be distributed across several computing devices. It will also be appreciated that the individual components may be combined into one or more components providing the combined functionality. Moreover, any of the modules, databases or devices shown in Figures 5a and 9 may be implemented in a general purpose computer modified (e.g. programmed or configured) by software to be a special-purpose computer to perform the functions described herein.

The query engine or server for conducting the search, including servers for indexing, calculating metrics and for crawling, can be implemented using standard hardware. The hardware components of any server typically include: a central processing unit (CPU), an Input/Output (I/O) Controller, a system power and clock source; display driver; RAM; ROM; and a hard disk drive. A network interface provides connection to a computer network such as Ethernet, TCP/IP or other popular protocol network interfaces. The functionality may be embodied in software residing in computer- readable media (such as the hard drive, RAM, or ROM). A typical software hierarchy for the system can include a BIOS (Basic Input Output System) which is a set of low level computer hardware instructions, usually stored in ROM, for communications between an operating system, device driver(s) and hardware. Device drivers are hardware specific code used to communicate between the operating system and hardware peripherals. Applications are software applications written typically in C/C++, Java, assembler or equivalent which implement the desired functionality, running on top of and thus dependent on the operating system for interaction with other software code and hardware. The operating system loads after BIOS initializes, and controls and runs the hardware. Examples of operating systems include Linux™, Solaris™, Unix™, OSX™ Windows XP™ and equivalents.

The following describe examples for creating the composite image used as the search query:

1 . A user is a casual shopper wanting to find some jewellery for his wife. He knows the kinds of things she likes, but has no idea, or vocabulary to describe, what elements such jewellery items have in common. He can recognise the right kind of jewellery, but has no idea how to describe it. Initially, he selects an arbitrary set of jewellery and clicks on the one image that was closest to what he was looking for, allowing the user to navigate towards a desired final result. 2. A user is a shopper with a specific need for a replacement item of jewellery. The shape is toroidal (a circle with a hole in the middle) and the material is quartz, maybe, or some crystalline pink material. The initial composite image would be formed by selected an image and selecting the toroidal shape and by selecting an image and selecting the appropriate colour of pink. The user would then be able to navigate and refine the choices from the initial results described herein.

3. A user is a designer, looking for a good background image to go on a piece of marketing material. As shown in Figure 4a and 4b, the user could start by selecting the three main colours in the palette and navigate through the space of images until a suitable result is found. Ideally, the resulting image should fit with the colour palette but not be too dominant.

4. A user is a casual shopper wanting to buy a coffee table for the lounge. The user uploads a photo of the lounge as the image to be searched. The results will return similar lounges, possibly with coffee tables. Once an image with a suitable table is returned, the user can select the table as the input to the refine the search, with the ultimate aim of finding a place to buy the desired table.

5. A user is an art lover wanting to buy a painting that will look good in a room that already has two paintings. Photos of the two paintings are uploaded to form the composite image for the search. The search results will return other paintings with similar properties (colour, texture, etc.) to the two initial images. 6. A user is a casual shopper looking for a bedspread that matches curtains in a bedroom. Then user may upload an image of the curtains as the image in step S100, and then follow the other steps described herein.

7. A user is a casual window shopper who likes to browse the internet looking at things he might buy one day. Occasionally he will buy something. Starting from a link sent by a friend, he is shown other similar items. Clicking on one of those items provides the image in step S100, and the user follows the other steps described herein.

8. A user is a house-buyer. He uploads a photo of a house he likes that is not for sale as the image in step S100. Features selected from other images returned with an initial search, such as style, age, shape can be used to generate the composite image and a filter can be applied to generate results in the right area.

9. A user is a female shopper browsing the internet looking for new clothes. One day, she sees a dress she likes and clicks on the "I like this" button on the browser add-on. This triggers searching by the system, which returns a collection of similar dresses, and other types of clothing that have similar patterns and colours depending on the features and/or weighting applied by the user. The user would then be able to navigate and refine choices from the initial results by selecting additional images from which to add further distinct features.

10. A user is a shopper who sees an architectural feature on a building. He uploads a photo of the feature to find an object for inside the home (a sculpture, a light- -fitting etc.) that is similar in style. The user would then be able to navigate and refine the choices from the initial results as described herein.

1 1 . A user is redecorating his house, and looking for a bath shower-tap that is similar in shape to an old-fashioned phone handset. The image in step S100 is a picture of a phone that can be combined with the shower category. The user would then be able to navigate and refine the choices from the initial results described herein.

12. A user is building a web site and looking for an icon that will fit with the existing design. The composite image is built from an icon that has the right shape and another that has the right colour palette. A search is thus initiated from these two icons.

Figure 10 shows an embodiment 100 for generating a composite image search query within the system illustrated in Figure 5a. In this example, all the images available to be searched have been analysed, indexed and stored that in the feature database prior to access by the user, as described above with reference to figure 1 . A user selects a first image 101 a and a first set of features associated with the image, having been previously indexed and stored within the feature database 78, are presented as selectable by the user. In this example, the first feature (a) selected by the user is colour, for example, from a histogram of the colours contained within a portion of the image located inside the dress. In this example, this histogram can be represented as a 64-dimensional feature vector (102a) of 64 integer values, each of which represents the extent to which a group of colours occur in the image within the portion of the image selected.

The user also selects a second image 101 b and second set of features of the second image are analysed by the query engine 74. In this example, the second feature (b) selected by the user is the shape of the boot. The shape of the boot within the image may be represented by a variety of vectors of features, where each feature represents a specific type of shape or visual sub-component that could occur in an image. One advantage of this approach is that nature of the feature may be adapted over time as alternative methods for effectively searching features are developed. In this example, the image is reduced to a small size (e.g., 32x32) and then converted into a black-and- white representation, by thresholding the colours at the median colour for the image. The 32x32 representation can then be flattened to a single 1024-dimensional feature vector 102b.

In the embodiment illustrated in Figure 10, a non-exhaustive search is conducted on the feature database using a suitable search algorithm appropriate, such as a Locality Sensitive Hashing (LSH) is employed for each desired feature type. In this example, two searches 103a and 103b are each conducted and two subsets of results, 104a and 104b are returned. Distances from the feature searched are also returned with the each result image. Merge function 105 calculates distances for the alternate feature not already searched, in this example colour for the images returned in subset 104b and shape for the subset in list 104a. The two subsets results lists are then combined to produce a combined results list 106 which ranks images with both features a and b in order of distance from the desired result. The resulting image set can then be sorted, and the items with the smallest average distance to both features a and b can be returned to the user. Search methods such as described "A statistical interpretation of term specificity and its application in retrieval", Karen Sparck Jones, Journal of Documentation, Vol. 28 Iss: 1 , pp.1 1 - 21 or "An Efficient Earth Mover's Distance Algorithm for Robust Histogram Comparison" referenced supra may be utilised. Figure 1 1 shows an alternative embodiment 1 10 for generating a composite image search query within the system illustrated in figure 5a. As with the example described supra with reference to figure 6, a user selects two images 1 1 1 a and 1 1 1 b and selects different features 1 12a and 1 12b from each image to form the basis of a composite image search. Merge Function 1 15 produces a composite search 1 13 by concatenating the vectors representing features a and b. In this example, the features 1 12a and 1 12b are concatenated together, after being mathematically normalised, using well known techniques such as that described by Alfassi "On the normalization of a mass spectrum for comparison of two spectra" Journal of the American Society for Mass Spectrometry Volume 15, Issue 3, March 2004, Pages 385-387, to ensure, for example, that a longer feature vector does not dominate a shorter feature vector. This combined feature vector can then be looked up using a standard k-nearest-neighbour (KNN) or LSH algorithm, to identify the images that are most similar to the composite query 1 13. In this example, a previously built index of concatenated features for each image is searched. No doubt many other effective alternatives will occur to the skilled person. It will be understood that the invention is not limited to the described embodiments and encompasses modifications apparent to those skilled in the art lying within the spirit and scope of the claims appended hereto.

Claims

CLAIMS:

1 . A data processing system for creating a search query comprising a processor configured to:

receive a first user selected image;

extract a first set of features from said first user selected image;

receive a second user selected image;

extract a second set of features from said second user selected image;

2. The data processing system according to claim 1 , wherein said processor is configured to:

receive a third user selected image;

extract a third set of features from said third user selected image;

receive a user selection of a third selected feature from said third user selected image; and

combine said third selected feature with the first and second selected features to form a further composite query.

3. The data processing system according claim 2, wherein said third selected feature is of the same type as the first selected feature type.

4. The data processing system according to any one of claims 1 to 3, further comprising:

a feature extractor module which is configured to segment at least one of said selected images into a plurality of objects and

output said plurality of objects to said query engine to send to said user as said extracted set of features.

5. The data processing system according to any one of claims 1 to 4, wherein the first user selected image and the second user selected image are both partial images selected from a single main image.

6. The data processing system according to any one of claims 1 to 5, wherein the sets of features are image content based features selected from the group comprising a shape of an object, a colour of an object, a pattern of an object and a texture of an object.

7. The data processing system according to any one of claims 1 to 6 wherein said processor is configured to apply a weight to each selected feature when combining to form said composite image search query.

8. The data processing system according to any one of claims 1 to 7 wherein said processor is configured to apply a filter to restrict a search on said composite image search query.

9. The data processing system according to any one of claims 1 to 8 wherein said system further comprises

a query engine which is configured to search for results which match said first selected feature and output said search results to a user; and

wherein said processor is further configured to receive the second user selected image which is an image selected from within said search results.

10. A query engine for conducting a search comprising a server which is configured to:

receive a first user selected image;

extract a first set of features from said first user selected image;

search for results which match said first selected feature;

output said search results to a user;

receive a second user selected image which is selected from within said search results; extract a first set of features from said first user selected image;

1 1 . The query engine according to claim 10 wherein said server is further configured to:

receive a third user selected image;

extract a third set of features from said third user selected image;

combine said third selected feature with the first and second selected features to form a further composite image search query.

12. The query engine according to claim 10 or claim 1 1 wherein said server is further configured to receive further user selected images selected from any sets of search results which have been displayed in response to earlier composite image search queries.

13. The query engine according to any one of claims 10 to 12, wherein said server is further configured to apply a weight to any one of selected features when combining to form said composite image.

14. The query engine according to any one of claims 10 to 13, wherein said server is further configured to receive a user selection of said weight for any one of selected features.

15. The query engine according to any one of claims 10 to 14 further comprising a user database to store results of previous searches.

16. The query engine according to any one of claims 10 to 15, wherein said server is further configured to apply a filter to restrict a search on said composite image search query.

17. A method of creating a search query comprising receiving a first user selected image;

extracting a first set of features from said from first user selected image;

receiving a second user selected image;

combining said first and second selected features to form a composite image search query.

18. The method according to claim 17 comprising the further steps of:

receiving a third user selected image;

extracting a third set of features from said third user selected image;

receiving a user selection of a third selected feature from said third user selected image; and

combining said third selected feature with the first and second selected features to form said composite query.

19. The method according to claim 17 or claim 18, wherein the first selected feature is selected from the group consisting of a shape of an object, a colour of an object, a pattern of an object and a texture of an object.

20. The method according to any one of claims 17 to 19, further comprising segmenting at least one of said first user selected image and said second user selected image into a plurality of objects and presenting said plurality of objects to said user to select at least one of said first selected feature and said second selected feature.

21 . The method according to any one of claims 17 to 20, wherein receiving the first user selected image includes the step of receiving a partial image from a single main image and receiving the second user selected image includes the step of receiving a partial image from the said single main image.

22. The method according to any one of claims 17 to 21 , comprising applying a weight to each selected feature when combining to form said composite image wherein said weight for each selected feature may be received from a user.

23. The method according to any one of claims 17 to 22, comprising applying a filter to restrict a search on said composite image search query.

24. A method of conducting a search comprising

receiving a first user selected image;

extracting a first set of features from said from first user selected image;

searching for results which match said first selected feature;

outputting said search results to a user;

receiving a user selection of a second feature;

25. The method according to claim 24 comprising the further steps of:

receiving a third user selected image;

extracting a third set of features from said third user selected image;

combining said third selected feature with the first and second selected features to form said composite image search query.

26. The method according to claim24 or claim 25, wherein the first selected feature is selected from the group consisting of a shape of an object, a colour of an object, a pattern of an object and a texture of an object.

27. The method according to any one of claims 24 to 26, further comprising segmenting at least one of said first user selected image and said second user selected image into a plurality of objects and presenting said plurality of objects to said user to select at least one of said first feature and said second feature.

28. A method according to any one of claims 24 to 27 comprising applying a weight to any of selected features when combining to form said composite image search query

29. A method according to claim 28 wherein said weight for said selected feature is received from a user.

30. A method according to any one of claims 24 to 29, comprising applying a filter to restrict a search on said composite image.

31. A query server for creating a search query comprising:

32. A non-transitory computer readable medium carrying computer program code to, when running, implement a method of creating a search query, the method comprising:

receiving a first user selected image;

extracting a first set of features from said from first user selected image;

receiving a second user selected image;

extracting a second set of features from said from second user selected image; receiving a user selection of a second selected feature of a second type from said second set of features; combining said first and second selected features to form a composite image search query.

33. A carrier carrying computer program code which when running on a computer causes said computer to implement the method of any one of claims 17 to 23.

34. A non-transitory computer readable medium carrying computer program code to, when running, implement a method of conducting a search, the method comprising: receiving a first user selected image;

extracting a first set of features from said from first user selected image;

searching for results which match said first selected feature;

outputting said search results to a user;

receiving a user selection of a second feature;

35. A carrier carrying computer program code which when running on a computer causes said computer to implement the method of any one of claims 24 to 30.

36. A non-transitory computer readable medium carrying computer program code to, when running, implement a method of creating a search query, the method comprising:

receiving a first user selected image;

outputting said first user selected image;

receiving a first set of features extracted from said from first user selected image;

receiving a user selection of a first selected feature of a first type from said first set of features; receiving a second user selected image;

outputting said second user selected image;

receiving a second set of features extracted from said from second user selected image;

receiving a user selection of a second selected feature of a second type from said second set of features;

37. A non-transitory computer readable medium carrying computer program code to, when running, implement a method of conducting a search, the method comprising: outputting a first user selected image;

outputting a user selection of a first selected feature of a first type from said first set of features;

receiving search results which match said first selected feature to a user; displaying said search results to said user;

outputting a second user selected image which is selected from within said search results;

receiving a second set of features extracted from said from second user selected image; and

outputting a user selection of a second selected feature of a second type from said second set of features whereby said first and second selected features are combined to form a composite image search query.