GB2592033A - Image manipulation - Google Patents

Abstract

An image manipulation method comprises receiving an input image 202 and obtaining a set of image content descriptors for the input image 204. A data store of related images is then accessed and a group of stored images 208,210 based on similarity between the set of image content descriptors of the groups of stored images and the set of image content descriptors of the input image to retrieve a stored image 216 that is used as a reference image when generating a manipulated image by performing a style transfer 218 operation on the input image. The data store is created based on analysis of activities of at least one user in relation to images. An additional invention of creation at least one data store adapted for image manipulation is also claimed.

Description

Image Manipulation The present invention relates to image manipulation.

Image manipulation involves editing images using computer vision and graphics algorithms. Style transfer is a known type of image manipulation that aims to apply a desired style to an input image while preserving the original content of the input image. For example, the input image may be manipulated to adopt the appearance or visual style of another image, called the "reference image".

It has been proven and shown in literature that to obtain the best possible result from style transfer, colour transfer or image transformation based on a reference image it is beneficial to have very similar semantics in the input image and the reference image. An example relating to image colorization can be seen in "Deep Exemplar-based Colorization" by Mingming He, Dongdong Chen, Jing Liao, Pedro V. Sander, Lu Yuan (httpsifarxiv.orcilabs11807.06587v2), where the best result was obtained when both the input and reference images shared semantics and it is possible to use an image retrieval system to enforce this. For a colorization problem if it is desired to colorize a grayscale portrait picture where there are many ambiguities regarding which colour to assign to each of the shades of grey, it is better to solve these ambiguities by imitating the colour distribution found in another portrait, rather than doing so from a different type of picture, e.g. an image of a car or a mountain.

Thus, semantics play an important role in obtaining a realistic result and image retrieval is a good solution for obtaining suitable reference images. However, in the real world it is impossible to store a perfect reference image for each possible input image that an image manipulation system may process.

Embodiments of the present invention aim to address the above technical problems and provide improved image manipulation in an efficient manner.

Given a style transfer algorithm that requires a reference image, embodiments can provide an advantageous/optimal reference image for a particular input image using a statistical system. This can offer better results for the given input image and automatically provide an optimal reference for the input image. Embodiments can maintain an image retrieval data store that is used for style transfer according to user demands or behaviour, thereby reducing the memory requirements for storing the dataset. Additionally, embodiments may create a two level database and, by understanding user needs, can cache part of this database in a user device to minimize network data transfers/consumption. Embodiments can provide a data store relating to reference images that can be used by an image manipulation/style transfer system to obtain the best possible reference image for a specific input image. The data store can be created using user data, e.g. by providing a list of inputs in which they are interested. This can allow the size of the data store to be contained to the user's interest(s). Additionally, the user-based data can be taken into consideration in order to have a smaller version of the overall data store stored on the user's device, thereby reducing/minimizing network usage for all/some of the queries submitted by the user. Thus, embodiments can reduce memory requirements related to maintaining an image database and can also reduce data transfer (network activity) by creating a cache comprising a subset of the database on the user side.

According to a first aspect of the present invention there is provided a computer-implemented image manipulation method comprising: receiving an input image; obtaining a set of image content descriptors (d) for the input image; accessing a data store related to a plurality of stored images, the data store comprising a set of image content descriptors related to each of the plurality of stored images and a set of image capture characteristic descriptors (c) related to each of the plurality of stored images, the stored images being logically arranged in a plurality of groups, wherein the stored images in one said group have a same said set of image content descriptors and different said sets of image capture characteristics; finding at least one said group of stored images in the data store based on similarity between the set of image content descriptors of the groups of stored images and the set of image content descriptors of the input image; retrieving the sets of image capture characteristic descriptors associated with the stored images in the at least one found group of stored images; receiving a selection of a said set of image capture characteristic descriptors from amongst the retrieved sets of image capture characteristic descriptors; retrieving the stored image associated with the selected set of image capture characteristic descriptors, and generating a manipulated image by performing a style transfer operation on the input image using the retrieved stored image as a reference image, wherein the data store is created based on analysis of activities of at least one user in relation to images.

The data store may be created by storing sets of image content descriptors and sets of image capture characteristic descriptors determined to be likely to meet future user demands in relation to image manipulation. For example, the stored set of image content descriptors may comprise a set of image content descriptors associated with a stored image determined as one likely to be used as a reference image in a future style transfer operation (e.g. based on reference images used in historical/previous style transfer operations by the at least one user).

The method may comprise: determining a set of desired said image content descriptors of an image likely to meet future user demands; generating or obtaining a new image having the set of desired image content descriptors; obtaining a set of image capture characteristic descriptors of the new image, and adding the set of desired image content descriptors and the set of image capture characteristic descriptors obtained for the new image to the data store.

The determining the set of desired said image content descriptors may comprise statistical analysis of a plurality of existing stored images. The plurality of existing stored images may comprise images created, stored and/or accessed by a user of the device executing the image manipulation method, and/or images created, stored and/or accessed by a plurality of other users. The image content descriptors may comprise numerical values and the statistical analysis may comprise finding an average value of the image content descriptors of the plurality of stored images.

The set of image capture characteristic descriptors of the new image may be generated from metadata of the new image, and/or using an image analysis process, such as colour analysis.

The new image may be obtained by capturing a photograph using/following the set of desired image content descriptors. Alternatively, the new image may be synthetically generated using the set of desired image content descriptors, e.g. using a 3D renderer or a GAN (Generative Adversarial Network).

The method may comprise adding the image content descriptors (d) of the input image to at least one data store comprising indications of missing image-related data when the similarity between the set of image content descriptors (d) of the groups of stored images and the set of image content descriptors (d) of the input image meets a predetermined threshold. Some embodiments include a first said data store comprising indications of missing image-related data associated with the remote data store, and a second said data store comprising indications of missing image-related data associated with the local data store. The method may further comprise analysing the image content descriptors added to the at least one data store comprising indications of missing image-related data to determine the set of desired said image content descriptors and the set of image capture characteristic descriptors of an image likely to meet future user demands.

The step of accessing the data store may comprise selecting the data store from among a plurality of data stores. The plurality of data stores may comprise a remote data store and a local data store. The local data store may be stored on a user device that executes at least part of the method (or the local data store may be located at a near node of the user device). The selecting of the data store may comprise selecting the remote data store if the local data store is not present or available on the user device. The method may further comprise selecting the remote data store if the method does not initially find at least one said group of stored images in the local data store.

The local data store may comprise a subset of data of the remote data store. The subset may comprise said image content descriptors of stored images that match activities of the user of the user device, e.g. based on historical use or images stored in a user gallery on the user 25 device.

A said image content descriptor may comprise a semantic description of the content of a said image. Examples include: class (e.g. landscape, office, room, portrait, etc); a histogram relating to features identified in the image; semantic embedding extracted from a neural network (e.g. variational auto-encoder). The stored images in one said group may comprise a same said image having different capture characteristics. The image content descriptor may be used as a primary key for searching the data store. In some embodiments the data store comprises a table having columns representing the image content descriptors and rows representing the image capture characteristic descriptors. The step of finding the most similar at least one said stored image may be based on Euclidian or Cosine distance from the input image to a said stored

image in the table.

The set of image capture characteristic descriptors may be obtained from metadata of a said stored image, e.g. location of capture of the image; date/time of capture of the image, etc. The method may further comprise outputting the manipulated image, e.g. for storage or transfer.

According to another aspect of the present invention there is provided a computer-implemented method of creating at least one data store adapted for image manipulation, the method comprising: creating a data store related to a plurality of stored images based on analysis of activities of at least one user in relation to images, wherein the data store comprises a set of image content descriptors related to each of a plurality of stored images and a set of image capture characteristic descriptors related to each of the plurality of stored images, the stored images being arranged in a plurality of groups, wherein the stored images in one said group have a same said set of image content descriptors and different said sets of image capture characteristics.

According to another aspect of the present invention there is provided apparatus comprising a processor configured to at least partially execute one or more methods substantially as described herein. The apparatus may comprise a mobile computing device, such as a smartphone.

According to another aspect of the present invention there is provided apparatus configured to create at least one data store adapted for image manipulation substantially as described herein.

According to yet another aspect of the present invention there is provided a computer-implemented method (e.g. performed by a server) of creating a data store adapted for image manipulation, wherein the data store is created based on analysis of activities of at least one user in relation to images. This method may comprise: receiving, from a remote device, a set of image content descriptors for an input image (or receiving an input image and then obtaining a set of image content descriptors for the input image); accessing a data store related to a plurality of stored images, the data store comprising a set of image content descriptors related to each of the plurality of stored images and a set of image capture characteristic descriptors related to each of the plurality of stored images, the stored images being logically arranged in a plurality of groups, wherein the stored images in one said group have a same said set of image content descriptors and different said sets of image capture characteristics; finding at least one said group of stored images in the data store based on similarity between the set of image content descriptors of the groups of stored images and the set of image content descriptors of the input image; retrieving the sets of image capture characteristic descriptors associated with the stored images in the at least one found group of stored images, and transmitting the sets of image capture characteristic descriptors to the remote device.

According to another aspect of the present invention there is provided computer readable medium (or circuitry) storing a computer program to operate methods substantially as described herein.

According to the present invention, there is provided a method and apparatus as set forth in the appended claims. Other features of the invention will be apparent form the dependent claims, and the description which follows.

For a better understanding of the invention, and to show how embodiments of the same may be carried into effect, reference will now be made, by way of example, to the accompanying diagrammatic drawings in which: Figure 1 schematically illustrates an example computing device configured to execute an embodiment; Figures 2A -2B are flowcharts showing example steps performed by an embodiment, Figure 3 is a chart illustrating an example of semantic classes in an example image that can be determined by an embodiment.

Figure 1 is a block diagram showing a first computing device 100 and a second computing device 110 that are configurable to execute embodiments of the image manipulation method.

The first computing device will normally comprise a user device, e.g. a mobile telephone/smartphone, tablet computer or another type of, typically mobile/portable/handheld, computing/communications device. The user device will include, or be associated with, at least a processor 102, memory 104, a wireless communication unit 106 and a user interface and component 108. The user component interface 108 may comprise a touchscreen in some embodiments. Other components and features of the device will be well-known to the skilled person and need not be described herein in detail.

The second computing device 110 may comprise a server computing device and may be operated by a service provider entity. The server will also include, or be associated with, at least a processor 112, memory 114 and a wireless communication unit 116, as well as other well-known components and features, such as a user interface 118. The user device 100 and the server can communicate via any suitable communications network 120, which can include the internet, a cellular/wireless network, etc. Figures 2A -2B are flowcharts showing steps of an embodiment of an image manipulation method 200 that can be performed by means of software instructions being executed on the user device 100 and/or the server 110. A developer may make a design choice regarding whether software executed on the user device or the server is used to perform certain steps. It will also be appreciated that at least some of the steps shown in the Figures herein may be reordered or omitted. One or more additional steps may be performed in some cases. Further, although the steps are shown as being performed in sequence in the Figures, in alternative embodiments at least some of them may be performed concurrently and/or by different ones of the devices 100, 110, or even other/remote computing devices or a cloud service. It will also be understood that embodiments can be implemented using any suitable software, programming language, data editors, etc, and may be represented/stored/processed using any suitable data structures and formats.

The method 200 will typically be invoked when there is a need for the user device 100 to generate a manipulated image. For instance, a user of the device may cause an application executing on the device 100 to start performing the method 200. The method may be implemented as a feature of a multi-function application (e.g. a photo gallery, image sharing or image editing application), or it may be a stand-alone application. The method may display a user interface on the touchscreen of the user device that can prompt the user for input and also display outputs of the method.

An example use case of the method 200 will now be given for explanatory purposes. In this example the user provides an input image that comprises a natural landscape. The method can obtain a set of image content descriptors for the input image. The image content descriptors may comprise numerical values. For example, an image content descriptor may indicate the percentage of pixels in the image that represents a certain type of content, such as a natural landscape or portrait. Additionally or alternatively, the image content descriptors may comprise counters, such as: the image contains N number of certain features, e.g. persons or animals.

Additionally or alternatively, the image content descriptors may comprise binary values, e.g. the image contains a certain feature (e.g. person or seascape): 1/0. The method can then access a data store containing data relating to groups of stored images. Based on the image content descriptors of the input image and image content descriptors of the stored images held in the data store the method can find a group of stored images that comprises a similar natural landscape (e.g. image content descriptors for the input and stored images in the group indicate that the images both comprise over 90% natural landscape). In one example, one group is found that comprises a first stored image showing its landscape captured in autumn and a second stored image showing its landscape captured in winter. The user can then be presented with data describing the image capture characteristics of the stored images in the found group. For instance, the method may display thumbnail versions of the first and second stored images and/or text describing the capture characteristics of the images, e.g. "autumn" and "winter". The user can then select one of these and the corresponding stored image is then retrieved. For example, if the user selects the "autumn" capture characteristics then the first stored image is retrieved. That retrieved stored image is then used as a reference image for a style transfer process applied to the user's input image. Thus, an output image is generated that is a modified version of the user's input image (which may have been captured in summer, for example) that has characteristics of the autumnal stored image (e.g. a more red/orange colour pallet) applied to it to give a different visual effect that is desired by the user (e.g. to give the impression that the input photograph was captured in the autumn season). As the method aims to ensure that the retrieved stored image used as the reference image is similar to the input image, the output image will be of good quality.

It will be understood that the above use case is exemplary only and many variations are possible. For instance, embodiments are not limited to landscapes and other types of input images and style transfer images can be used. A non-limiting set of examples of other image types that embodiments can process includes: portraits, interiors, sporting events, performances, groups, animals, or even various sub-types of these. Further, the style transfer may involve transferring any image capture characteristic (e.g. lighting properties, colour properties, shadow properties, etc) of the retrieved stored image to the input image so that it gains a similar appearance or visual style to the reference image whilst preserving original content of the input image.

Referring to Figure 2A, at step 202 the user of the device 100 provides an input image that is to be used to generate a manipulated image. In some cases the user may select an image from a plurality of stored images. The images may be stored in the memory 104 of the user device 100 and/or in at least one other storage location (e.g. an external storage device, such as a removable storage media), or may be accessed over a network connection (e.g. using a Cloud-based service). The input image may be selected in any suitable manner. For example, the user may be presented with thumbnail versions of the plurality of images that can be chosen via the user interface, or the user may search for a specific image in some other manner, e.g. by using text-based tags, time/date stamps, etc, associated with the images. Alternatively, the user may use a camera application or device to produce a new input image for the method.

At step 204 embodiments of the method 200 can obtain a set of image content descriptors for the input image. The image content descriptors can comprise a set of interpretable features that describe the content of an image (typically only to a partial extent). In order words, the image content descriptors can have semantic meaning. For instance, an image content descriptor may refer to a known class (e.g. landscape, office, room, portrait, etc) of image.

Alternatively or additionally, an image content descriptor may describe the content of an image by means of a histogram of classes of content in an image (e.g. 80% sky, 20% grass). Figure 3 graphically illustrates the mean histogram of example classes in an example image. Alternatively or additionally, an image content descriptor may comprise semantic embedding extracted from a Neural Network (for example, ones be formed using a variational auto-encoder).

Some embodiments may process a vector d of these image content descriptors.

The image content descriptors for the input image may be generated manually by a user (e.g. by inputting keywords that describe the features/content of an image), and/or may be generated in an automated manner, e.g. using a machine learning technique. Examples include a content analysis method, an image recognition method (including scene classification algorithms, such as CNN, DNN, etc) or the like. Such an automated technique can output data describing the main features/content of the image and, in some case, the associated distributions of those features in relation to other images.

In embodiments the method 200 can aim to find a group of stored images that are similar to the input image. Typically, the group of found stored images comprise different versions of the same (or substantially similar) stored image with different image capture characteristics. One of these can then be selected to retrieve a reference image that is used to perform a style transfer operation on the input image.

For the purpose of finding stored images similar to the input image at least one data store can be accessed by the method 200. The data store contains data relating to a plurality of existing/stored images. The stored images themselves may be stored in the same data store or in a different data store. That is, the data used by embodiments may be stored in more than one database, device and/or location. It should also be understood that the data store(s) can have any suitable structure, format, contents, etc, and need not comprise databases/tables exactly as described for the detailed examples herein.

In embodiments the stored images (and/or related data) can be at least logically arranged in a plurality of groups. Typically, the stored images in one group may all be substantially the same/similar image, but with different image capture characteristics. For example, the group may comprise versions of substantially the same image captured at different times of day, different seasons of the year, using different illumination colours, and so on. The stored images in one group may be associated with a set of image content descriptors. Again, these image content descriptors can comprise a set of interpretable features that describe the content of the image and can have semantic meaning, e.g. may refer to a known class (e.g. landscape, office, room, portrait, etc) of image, etc. These image content descriptors (which may be represented by vector d in some embodiments) will typically be generated in the same manner as the image content descriptors for the input image discussed above.

Each of the stored images in the group can also have an associated set of descriptors for its image capture characteristics, which are also contained in the data store. An image capture characteristics descriptor may comprise an abstract representation of language. The image capture characteristics descriptors may be generated manually by a user (e.g. by inputting keywords that describe the capture characteristics of the image), and/or may be generated in an automated manner. The automated technique may be based on a content analysis method, an image recognition method or the like (which may comprise machine learning techniques) and/or may be extracted from image processing algorithms or from the metadata of the image.

For instance, the image processing may comprise colour analysis, e.g. monotone, high contrast, colourful, etc. The metadata can be used to extract information such as location, date/time, the camera mode used to capture the image (e.g. sunny, cloudy, night, etc), for instance. Some embodiments may process a vector e of these image capture characteristic descriptors. In some embodiments the data store may comprise a table (a, 5) that contains a, e, where a defines a composed primary key and e is a list of Boolean attributes for each of the possible configurations of the images in a particular group that is known in the data store.

At step 206 embodiments may select a data store from a plurality of possible data stores that is to be used to find a group of stored images similar to the input image. It is clearly not practical to have all possible images (captured using all possible image capture characteristics) included in a data store and so embodiments aim to reduce the amount of stored data required by analysing statistics relating to demands/activities of at least one user in relation to images, and using these statistics to generate the at least one data store. At an initial stage data relating to a subset of all images processed by the system can be contained in the data store. When at least one user starts using the system it will be possible to capture statistics of the images he captures, uses or stores (e.g. in a user gallery on the user's device) and use this information to select which image data should be stored. Examples of this type information will be discussed further below.

Additionally, in order to avoid/reduce network data transfer between the user device 100 and the remote server 110 over the communications network 120 it can be beneficial for embodiments to only access a data store that is local to the user device, e.g. a data store contained in the memory 104 of the user device. Alternatively, the local data store may be located on a near node/device that can be accessed by an alternative type of connection (such as BluetoothTM, WifiTM, etc) that has advantages, such as reliability, cost, speed, etc, over the communications network 120. For this reason, it is advantageous for the local data store to contain data relating to a more limited number of images that are most likely to be relevant to the user's needs in relation to image manipulation. Details regarding how embodiments can achieve these aims will be given below in relation to creation of the global and local data stores.

In some embodiments the local data store may comprise a table (a, 5) that is a subset of the remote data store and has been defined using information regarding user demands and content.

Returning to step 206, if a check performed by the method 200 (e.g. executing on the user device 100) indicates that the user device 100 does not contain a local data store/cache that can be accessed by the method 200 then control passes to step 208, where the method accesses a remote data store (e.g. on the server device 110) via the communications network 120. The set of image content descriptors obtained for the input image may be presented as a query to the data store/database.

At step 208 the method 200 can use the remote data store to search for at least one group of stored images that are similar to the input image, e.g. similar in terms of image content descriptor(s), d. Some embodiments use Euclidean or Cosine distance to determine the closest image(s) described by d in the remote data store/table, although alternative techniques may be used, e.g. decision trees (classification trees or boosted trees), random forest, support vector machines or other type of classifier. Some embodiments may set a predetermined distance threshold, e.g. 10, to ensure to a big enough list of image capture characteristics (c's) is returned. Ideally, the distance would be 0, where the system would theoretically contain examples of all possible input images and so could always find a perfect reference image. However, as this is unfeasible, the distance may be tuned according to the number of stored images referenced by the data store and the descriptors used. For example, if an image content descriptor comprises a semantic percentage then an embodiment may determine that two images are sufficiently similar if their contents differ by less than 10%.

To give a simple example to illustrate, the following three image content descriptors may be obtained for the input image at step 204: % natural landscape (numerical value), contains lake (binary value), and contains mountain (binary value). These can be compared with the corresponding image descriptors for stored image groups: Input image Stored image group dl Stored image group d2 Image content 95% 98% 97% descriptor (numerical): % natural landscape Image content 1 1 1 descriptor (binary): contains lake Image content 1 1 0 descriptor (binary): contains mountain In the above example case, the stored image group d2 may be found as the closest found image group to the input image.

At step 209 the method 200 can retrieve image capture characteristics (c's) relating to existing stored images that look very similar to the input image (i.e. ones close in the d space) and which were found at the step 208. The user will later be able to select his preferred image capture characteristics (c) from the list of all these known configurations/image capture characteristics (c's), e.g. after data representing c is transferred back to the user device 100.

Returning to step 206, if the check performed by the method 200 indicates that the user device 100 does contain a local data store/cache that can be accessed by the method then control passes to step 210, where the method accesses a local data store. At the step 210 the method (e.g. executing on the user device 100) can use the local data store to search for at least one group of stored images that are similar to the input image in terms of image content descriptor(s), d. Typically, this will be done in a similar manner to the finding operation performed on the remote data store at the step 208, e.g. use Euclidean or Cosine distance to determine which are the closest image described by d in the local data store/table. However, when accessing the local data store the method might search for data relating to stored images that are at a lower predetermined distance threshold, e.g. 1, because the local data store will contain data relating to a smaller selection/spectrum of stored images than the remote data store.

At step 212 the method checks whether an empty retrieval occurred at the step 210. This may happen if there is no stored image having a distance lower than the predetermined threshold (e.g. 1) at the step 210, in which case the method may inform the user of the situation and terminate or request further input. If empty retrieval has occurred then control can pass to the step 208, where the method (e.g. executing on the user device 100) accesses/connects to the remote data store and searches for data relating to similar images after failing to find any in the local data store.

If empty retrieval did not occur then control passes to step 213, where the method 200 can retrieve from the local data store image capture characteristics (c's) associated with the existing stored images that look very similar to the input image to later provide a list of all these possible configurations/image capture characteristics (c's).

Following step 209 or step 213, control can pass to step 214. At the step 214 the method may determine whether data relating to the retrieved set of image capture characteristics (c's) is to be used for insertion into a data store, and/or for retrieving a reference image for a style transfer operation. This determination may be made based on an indication included in the query/instruction relating to the input image so that the same starting point can be used for both types of operations. For example, the user may provide input indicating how the input image is to be used, or an application including the method 200 may provide a user setting for indicating that all, or only certain types of, new images captured by the user device's camera or downloaded by the user's device, etc, may be submitted for insertion in the local and/or remote data store, and so on.

If the method 200 is to retrieve a reference image for the style transfer operation then at step 215 the method 200 can present to the user the retrieved set of image capture characteristics (c's) so that he can make a selection from amongst them. For example, if the user provides input image dl then the method may retrieve the image capture characteristics relating to at least one group of stored images that is/are very close in the d space. If the method finds two groups of stored images d2 and d3 that are very close to dl in the d space and the user selects the set of image capture characteristics (c's) associated with the stored image d2 then that stored image d2 will be retrieved at step 216 and used for the style transfer operation. In the event that only one group of stored images is found at step 209 (or step 213) then an automatic selection of the image capture characteristics of that group may be made, or the user may be asked to confirm the selection of that set of image capture characteristics.

At step 218 embodiments can use the image retrieved at step 216 as a reference image for performing a style transfer operation on the input image. Examples of suitable style transfer operations that can use the input image and reference image as inputs include GANs, Generative Models, or Histogram Transfer, but it will be understood that other techniques may be used. For instance, some embodiments may use trained neural networks to perform the style transfer operation. Thus, the output image comprises a manipulated version of the input image having a style of the retrieved stored image applied to it by means of an embodiment of the method described herein. Following the completion of the style transfer operation a user may perform a further action using the manipulated image, e.g. select it for storage, display or sharing using a menu, for example.

Returning to the step 214, if the method 200 determines that data relating to the retrieved set of image capture characteristics (c's) is to be used for insertion into a data store then control passes to step 220 of Figure 2B. At the step 220, the method checks whether a stored image (d) found at the step 208 or the step 210 was a perfect/very close match to the input image. Finding a stored image (d) does not always mean that the method was able to find a perfect match for the input image; rather, it means that the input image was close to the found stored image. A new entry d, c will be added to the remote data store if: -The input image is within a predetermined distance of the closest stored image/group (d) in the data store -see steps 220 to 222 to 224, or -The input image is very close to a known stored image/group (d), but the data store does not contain the specified combination of retrieved image capture characteristic descriptors (c) for that stored image/group (d) -see steps 220 to 226 to 228.

The distances used for these determinations may be selected/tuned according to various factors, e.g. the number of stored images referenced by the data store and the descriptors used. For example, if the input shares less than 10% of its semantic content with one of the stored images then it may be determined that the data store does not contain any sufficiently close stored image in terms of d space.

If the method is unable to add the new d, c to the data store (following steps 220 to 226, or following steps 220 to 222) then at step 230 the image content descriptors (obtained at the step 204) for the input image are added to at least one data store comprising indications of missing image-related data. In the illustrated example there is such a missing image-related data store 232 associated with the remote data store, and another such missing image-related data store 234 associated with the local data store. For example, embodiments may add data to the remote data store if already connected to it. If there is no existing connection then embodiments may add to the local data store and may later (e.g. periodically or when a connection is set up for searching the remote data store, for instance) copy or transfer the data to the remote data store. Periodically (and/or on demand, and/or on an event-driven basis), embodiments may analyse the missing image data store(s) 232, 234 to determine what new stored image-related data would be beneficial to add to the remote data store and/or the local data store.

With regards to the missing image-related data data store 232 associated with the remote data store, at step 236 an embodiment may extract statistics from that missing data store 232. In some embodiments the missing image data data store 232 can take the form of a "global missed table" that comprises the image content descriptors (d) obtained (at the step 204) for one or more input image for which the system was unable to find a sufficiently similar existing stored image. This data store may optionally contain additional information. For instance, in some embodiments it may contain a counter so that the store can comprise a set of image content descriptor (d's) and a counter that sums up every time that a sufficiently close d vector was not found in the remote data store.

The analysis/extraction/generation of the statistics at the step 236 will be intended to help identify new images (e.g. photos) that meet likely future user demands so that information relating to such images can be added to the remote data store. These new images can also be stored and made available to embodiments so that they can be used as reference images when needed by a user. The new images may be found/generated by a human administrator user or may be done algorithmically. For example, an administrator may review the information in the missing image related data store 232 and determine the appropriate semantic classes and then determine what type(s) of stored image-related information should be added to the remote data store. The missing image related data store 232 will also be updated (e.g. relevant entry deleted) when the new image data has been created and added.

The statistical analysis performed at the step 236 may be intended to avoid generating a new sample for each d entry in the data store 232. The statistical analysis performed at the step 236 may involve various techniques, such as finding an average image of some or all of the images represented in the data store 232. For example, for entries dl and d2 (each being an image content descriptor indicating the percentage of the image that comprises water, natural landscape and artificial landscape) in the data store 232, an average image dsnry may be computed as follows: * dl = 0% water, 10% nature, 90% artificial * d2 = 5% water, 15% nature, 80% artificial * dsummary: 2,5% water, 12,5% nature, 85% artificial Some embodiments may look at N centroids of the data distribution in the missing data store 232. It can be the mean of all data or sub-clusters of it, depending of the distance between d's stored in the missing image related data store 232. N may be a manually tuned value. For example, given a list of 200 images, it may not be feasible to produce data store entries for future searches for all of these. If the average is computed then precision may be lost because a single example would be generated from 200 inputs. Embodiments may be intended to try to represent the 200 inputs in a better way without needing to generate new samples for each of them. Thus, N clusters may be defined in order to work with N centroids instead of 200 or 1. There are many metrics that will determine how many clusters will be used in this clustering method, e.g. how many administrator users may be available for the task of manually generating samples, the possibility of analysing the standard deviation of the d contained in the missing image related data store and selecting an N proportionally. It will be understood that alternatives to finding an average may be used by other embodiments, e.g. finding the mode, the mean, finding N-clusters in these and keeping the centroids, and so on. Any method used here will have the same intention, i.e. to find missing points in the d data store so that these can be covered based on statistics of usage.

At step 238 embodiments may generate a new set d, c based on the analysis of the step 236. This will typically involve creating an image having the desired image content descriptors (e.g. as indicated by d.m.). This may be done manually by a photographer or artist, for example, using the desired image content descriptors as guidelines. The capture characteristics of the new image can then be analysed to generate a set of image capture characteristic descriptors (c) to be associated with that image. This may be done by manual labelling when generating/capturing the new image. Additionally or alternatively, the metadata of the new image may be analysed to generate the capture characteristic descriptors, which may include, for example, time of the day or even the camera mode that was used to capture the picture (sunny, cloudy, night...), and so on. Image analysis techniques, such as colour analysis may be used. In some cases it may be possible to synthetically generate this image instead of having a photographer capture it. For instance, a realistic 3D renderer or a GAN can be used to generate scenes that match the desired d's and automatically set possible c's.

At step 240 the new set of image content descriptors (d) and image capture characteristic descriptors (c) generated at the step 238 can be added to the remote data store.

Some embodiments may also access a local image data store 242 (e.g. a user gallery) on the user's device 100 in order to determine what new stored image-related data would be beneficial to add to the remote data store and/or the local data store. Embodiments may obtain and create image content descriptors (d) and image capture characteristic descriptors (c) whenever an image is added to the user gallery, or may seek user input/permission to confirm that this should be done. At step 244, embodiments can check if c is available for the image being processed. Depending on the application, c might not be available from the user gallery.

It might be available if c is fully generated using metadata or can be estimated from image content. However, for some applications the information might not be available If the c is available then control can pass to the step 224 (where d, c is added to the local and/or the remote data store); otherwise the image content descriptors (d) of the image are added to the missing image-related data data store 234 associated with the local data store.

With regards to the missing image-related data data store 234 associated with the local data store, at step 246 an embodiment may extract statistics from that missing image related data store 234. In some embodiments the missing image related data data store 234 can take the form of a "local missed table" that comprises the image content descriptors (d) obtained for one or more input image for which the system was unable to find a sufficiently similar existing stored image. This data store may optionally contain additional information, e.g. a counter.

The extraction/generation of the statistics at the step 246 may be similar to the process performed at the step 236, but is done in relation to the local missed data store 234 rather than the missing image-related data store 232 that is associated with the remote data store. Again, the intention is to help identify new images that meet likely future user demands so that information relating to such images can be added to data stores. In some cases, the operations related to the data store 234 may be intended to update only the local data store that is associated with a particular user of the user device, rather than to update the remote data store, the contents of which may be shared with other users.

At step 248 embodiments may generate a new set d, c based on the analysis of the step 246. The remote data store will be updated over time and embodiments can use it to update the local data store accordingly. Embodiments can retrieve d, c from the remote data store so that the user can have quick access (with little/no latency and bandwidth free) to the local data store that has been shaped to his predicted image-related preferences in relation to style transfer operations. At step 250 the new set of d, c generated can be added to the local data store.

With the addition of multiple users of the system some embodiments can obtain statistics relating to the multiple users' demands. For example: * 80% of the users try to retrieve images of flowers.

* Or, this is the mean histogram of classes in an image The server 110 may connect with multiple user devices and collect and process image-related data as discussed above from each of the devices. The data can be generally treated in the same way as done in the case of a single user of the user device 100. This data can be combined/used to update the remote data store and/or the local data store associated with each user/device. Information regarding the users of the multiple devices may be collected and used to determine groups of users whose data can be used together.

It is understood that according to an exemplary embodiment, a computer readable medium storing a computer program to operate a method according to the foregoing embodiments is provided.

Attention is directed to any papers and documents which are filed concurrently with or previous to this specification in connection with this application and which are open to public inspection with this specification, and the contents of all such papers and documents are incorporated herein by reference.

All of the features disclosed in this specification (including any accompanying claims, abstract and drawings), and/or all of the steps of any method or process so disclosed, may be combined in any combination, except combinations where at least some of such features and/or steps are mutually exclusive.

Each feature disclosed in this specification (including any accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise. Thus, unless expressly stated otherwise, each feature disclosed is one example only of a generic series of equivalent or similar features.

The invention is not restricted to the details of the foregoing embodiment(s). The invention extends to any novel one, or any novel combination, of the features disclosed in this specification (including any accompanying claims, abstract and drawings), or to any novel one, or any novel combination, of the steps of any method or process so disclosed.

Claims (17)

1. CLAIMS1. A computer-implemented image manipulation method comprising: receiving (202) an input image; obtaining (204) a set of image content descriptors for the input image; accessing (206) a data store related to a plurality of stored images, the data store comprising a set of image content descriptors related to each of the plurality of stored images and a set of image capture characteristic descriptors related to each of the plurality of stored images, the stored images being logically arranged in a plurality of groups, wherein the stored images in one said group have a same said set of image content descriptors and different said sets of image capture characteristics; finding (208, 210) at least one said group of stored images in the data store based on similarity between the set of image content descriptors of the groups of stored images and the set of image content descriptors of the input image; retrieving (209, 213) the sets of image capture characteristic descriptors associated with the stored images in the at least one found group of stored images; receiving a selection (214) of a said set of image capture characteristic descriptors from amongst the retrieved sets of image capture characteristic descriptors; retrieving the stored image (216) associated with the selected set of image capture characteristic descriptors, and generating (218) a manipulated image by performing a style transfer operation on the input image using the retrieved stored image as a reference image, wherein the data store is created based on analysis of activities of at least one user in relation to images.
2. 2. A method according to claim 1. wherein the data store is created by storing sets of image content descriptors and sets of image capture characteristic descriptors determined to be likely to meet future user demands in relation to image manipulation.
3. 3. A method according to claim 2, wherein the stored set of image content descriptors comprise a set of image content descriptors associated with a stored image determined as one likely to be used as a reference image in a future style transfer operation based on reference images used in historical/previous style transfer operations by the at least one user.
4. 4 A method according to any preceding claim, further comprising: determining (236, 246) a set of desired said image content descriptors of an image likely to meet future user demands; generating or obtaining (238, 248) a new image having the set of desired image content descriptors; obtaining (238, 248) a set of image capture characteristic descriptors of the new image, and adding (240, 250) the set of desired image content descriptors and the set of image capture characteristic descriptors obtained for the new image to the data store.
5. 5. A method according to claim 4, wherein the determining (236, 246) the set of desired image content descriptors includes statistical analysis of a plurality of existing stored images.
6. 6. A method according to claim 5, wherein the plurality of existing stored images comprise images created, stored and/or accessed by a user of the device executing the image manipulation method, and/or images created, stored and/or accessed by a plurality of other users.
7. 7. A method according to claim 5 or 6, wherein the image content descriptors comprise numerical values and the statistical analysis comprises finding an average value of the image content descriptors of the plurality of stored images.
8. 8. A method according to any of claims 4 to 7, wherein the set of image capture characteristic descriptors of the new image is generated using metadata of the new image, and/or using an image analysis process.
9. 9. A method according to any preceding claims, further comprising adding (230) the image content descriptors of the input image to at least one data store comprising indications of missing image-related data when the similarity between the set of image content descriptors of the groups of stored images and the set of image content descriptors of the input image meets a predetermined threshold.
10. 10. A method according to claim 9, including a first said data store (232) comprising indications of missing image-related data associated with the remote data store, and a second said data store (234) comprising indications of missing image-related data associated with the local data store.
11. 11. A method according to claim 9, when dependent upon claim 4, further comprising analysing (236, 246) the image content descriptors added to the at least one data store comprising indications of missing image-related data to determine the set of desired said image content descriptors and the set of image capture characteristic descriptors of an image likely to meet future user demands.
12. 12. A method according to any preceding claim, wherein the step of accessing the data store comprises selecting (206) the data store from among a plurality of data stores.
13. 13. A method according to claim 12, wherein the plurality of data stores comprises a remote data store and a local data store stored on a user device that executes at least part of the method, and wherein the selecting (206) of the data store comprises selecting the remote data store if the local data store is not present or available on the user device, or initially selecting the local data store and then selecting (212) the remote data store if the method does not initially find at least one said group of stored images in the local data store.
14. 14. A method according to claim 13, wherein the local data store comprises a subset of data of the remote data store, wherein the subset comprises said image content descriptors of stored images selected based on activities of the user of the user device.
15. 15. A computer-implemented method of creating at least one data store adapted for image manipulation, the method comprising: creating (240, 250) a data store related to a plurality of stored images based on analysis of activities of at least one user in relation to images, wherein the data store comprises a set of image content descriptors related to each of a plurality of stored images and a set of image capture characteristic descriptors related to each of the plurality of stored images, the stored images being arranged in a plurality of groups, wherein the stored images in one said group have a same said set of image content descriptors and different said sets of image capture characteristics.
16. 16. Apparatus (100, 110) comprising a processor (102, 112) configured to execute a method according to claim 1 or claim 15.
17. 17. A computer readable medium storing a computer program to operate a method according to claim 1 or claim 15.