WO2016201547A1

WO2016201547A1 - A computer-implemented method of aggregating and presenting digital photos from numerous sources

Info

Publication number: WO2016201547A1
Application number: PCT/CA2016/000170
Authority: WO
Inventors: Marc-Antoine Benglia; Sylvain FETIVEAU; Mostafa HOSSEINI
Original assignee: Kwilt Inc.
Priority date: 2015-06-18
Filing date: 2016-06-20
Publication date: 2016-12-22
Also published as: CA2894608A1

Abstract

A computer-implemented method of providing a client (e.g. an iPhone) with the ability to seamlessly aggregate all of a user's digital photos from both local sources (e.g. local memory on the iPhone) and remote sources (e.g. social media services like Facebook), and displaying relevant grouping (s) of same based on the user's desired search filter (e.g. grouped by location) without the user ever having to know where the photos are actually stored. The user is simply required to provide the app of the present invention with one-time secure access details to all of his/her sources (e.g. username and password to Facebook) and the app is thereafter able to aggregate and display photos on the client based on the user's search criteria. The photo groupings can simply be displayed on the client as desired for viewing purposes, or the relevant groupings can be displayed during the use of a keyboard for messaging purposes wherein any photo of choice can be readily located regardless of source and attached as part of a message to be sent to another person.

Description

A COMPUTER-IMPLEMENTED METHOD OF AGGREGATING AND PRESENTING

DIGITAL PHOTOS FROM NUMEROUS SOURCES

1. FIELD OF THE INVENTION

The present invention relates to the field of digital photos. In particular, the present invention relates to the aggregation of a user' s digital photos from local and on-line sources and the presentation of same on a user's device.

2. BACKGROUND OF THE INVENTION

With the transition of photography from analog to digital and the availability of cameras on handheld phones and other mobile devices, the acquisition and use of personal and professional digital photos has risen exponentially in recent years and this will only continue into the foreseeable future. This growth has also coincided with the growth of digital storage devices as well as cloud storage services that offer enormous storage capacities for storing a user's photos.

Many solutions have appeared that desire and intend to be the central repository for a user's digital files and documents, including photos. In this respect, both small and big

corporations have continued to expand their storage capacities and capabilities, and often reduce pricing models to attract customers in the hope of becoming the final and dominant

solution for digital storage. As a result, many users have switched time and again from one storage provider to another (as well as from local to cloud storage) as conditions have changed and favourable pricing offers have arisen. This has created a scattered digital world and users have found themselves with their digital data distributed amongst a series of storage providers and personal backup archives.

In addition to a user's personal photos, the stream of photos posted and shared by friends through social media and community groups has contributed more and more to the collection of photos that a user may wish to store and/or have access to. Such photos often live on the social media networks of their origin.

The foregoing has created the need for a facility that can provide a user with the means of finding and accessing their vast collection of photos without the hassle of maintaining the knowledge of where the sought-after photo actually resides and how to get to it. Many applications have tried to address this need by providing mechanisms to copy a user's photos from various sources to their own central repository in the hope of becoming the final solution for a user's digital storage.

However, this has resulted in yet more providers being added to the collection of storage providers for digital storage.

An eloquent solution is therefore needed that would not comprise the copying of current photos from various sources for storage in yet another repository, but that would instead maintain a reference to the original source of a user's various photos, while allowing the user to find and access sought-after photos at any time using a unified engine that acts as a "one-stop shop" for retrieving the desired photo and accompanying data without ever having to know where the photo is actually stored. 3. SUMMARY OF THE INVENTION

The present invention provides a computer-implemented method of providing a client (e.g. a user's PC or portable device, such as an iPhone) with the ability to seamlessly aggregate all of a user's digital photos from both local sources (e.g. local memory on a PC or iPhone or local storage device) and remote sources (e.g. cloud storage services like Google Drive or social media services like Facebook) , and displaying relevant grouping (s) of same based on the user's desired search filter (e.g. grouped by date or location) without the user ever having to know where the photos are actually stored. The user is simply required to provide the computer-implemented application ("app") of the present invention with one-time secure access details to all of his/her sources (e.g. username and password to Google Drive and Facebook) and the app is thereafter able to look after all aggregation and displays on the client based on the user's search/filter criteria. The photo groupings can simply be displayed on the client as desired for viewing purposes, or the relevant groupings can be displayed during the use of a keyboard for messaging purposes wherein any photo of choice can be readily located regardless of source and attached as part of a message to be sent to another person. In so doing, the app does not make yet another copy of the digital photo for storage in yet another database, but instead provides a link to the photos in the various sources and saves information about the links and data about the photos in a database. The present invention thereby provides a user with searchable ease of access to all of his/her photos regardless of source, without the user having to worry about keeping track of sources ever again. In view of the foregoing, one potential aspect of the present invention is to provide a mechanism in which local photos stored on a client device are aggregated and displayed seamlessly with remote photos from 3^rd party providers, thereby relieving a user from having to maintain the knowledge of which photo is stored where and whether it is local or remote.

Another potential aspect of the present invention is to provide a mechanism that maintains an up-to-date index of the entire digital data set (photo library) of a user from his/her various local and remote sources in order to optimize the procedure of processing and searching amongst the entire data set.

Another potential aspect of the present invention is to provide a means of categorizing and grouping photos stored in local and remote locations based on their existing meta data in order to facilitate searches based on different criteria.

Another potential aspect of the present invention is providing the ability for a user to locate photos whether stored in local or remote locations by allowing the user to navigate through groups and sub groups of photos that have been aggregated according to search criteria in a way that minimizes the effort of narrowing down the intended set of photo (s) the user is seeking.

Another potential aspect of the present invention is to provide a mechanism in which the credentials and authentication

information for accessing 3^rd party storage providers is entered once and maintained as long as the user has provided the proper authorization for accessing the 3^rd party provider and maintains that authorization. This eliminates the need for the user to reenter such credentials into various outlets for later access.

A further potential aspect of the present invention is to provide a unified mechanism to find and retrieve information about any photo from any source without the need to adapt to upgrades and changes to SDKs from 3^rd party sources and to shield the user and the software utilizing the mechanism from such changes, thereby resulting in less changes on the end point software and concentrating the management of communicating with 3^rd party sources on a central cloud service.

Another potential aspect of the present invention is to have the capability of receiving new information and data that may become available at a later time or as users backup, share and store more photos in their local storage or third party service

providers of choice.

Another potential aspect of the present invention is providing a mechanism in which an optimum amount of data is transferred between the cloud service and the client and between the cloud service and the 3^rd party service providers for the purpose of presenting the entire data set of photos to the client. Another potential aspect of the present invention is providing a means of reducing the wait time for users to have access to photos, and allowing the processing of current available data while updates are retrieved in the background. Another potential aspect of the present invention is eliminating the need for any software integration or use of SDKs by outlets and software intending to use the method of the present invention by providing the results through a universally adopted keyboard interface instead.

In one embodiment, the present invention may comprise a

computer-implemented method for aggregating information about a user' s digital photos stored on one or more electronic sources and presenting a user-chosen subset of said photos on an

electronic device of the user, comprising the steps of: making an inventory of the user's digital photos; categorizing the photos into one or more potential groups based on pre-defined criteria; displaying the one or more potential groups to the user for selection by the user; filtering the photos into a set representing the group selected by the user; and making the set of filtered photos available for consumption by the user. The one or more electronic sources may comprise the electronic device of the user, one or more remote electronic sources, or the electronic device of the user and one or more remote

electronic sources. The one or more remote electronic sources may comprise one or more cloud storage services, such as Google Drive, Facebook, Flickr, Googlet, Instagram, Twitter,

Photobucket, and Onedrive. Presenting the user-chosen subset of the photos may comprise displaying the photos in association with a keyboard interface. Making the inventory of the user's digital photos may comprise storing attributes of each of the user's digital photos in a database. The attributes may comprise one or more or all of: a unique identifier of each photo; an identification of the user each photo belongs to; an

identification of the electronic source where each photo

resides; a unique identifier by which each photo can be

referenced on the electronic source where each photo resides; the time a file containing each photo was created; and a URL or information a URL can be constructed from that may point to an original version of each photo. The pre-defined criteria may be any one or more of date, location, source, or user-defined search criteria. Displaying the one or more potential groups to the user for selection by the user may comprise displaying random thumbnails of a subset of photos within the one or more potential groups. Making the set of filtered photos available for consumption by the user may comprise: allowing the user to share the filtered photos with other users; allowing the user to back up the filtered photos to a storage device; allowing the user to download the filtered photos to the electronic device of the user; and allowing the user to view the filtered photos as a slideshow on the electronic device of the user. In another embodiment, the present invention may comprise a computer program product comprising a computer readable memory storing computer executable instructions thereon that when executed by a computer perform the method steps described in this paragraph.

In yet another embodiment, the present invention may comprise a computer readable memory having recorded thereon statements and instructions for execution by a computer for aggregating

information about a user' s digital photos stored on one or more electronic sources and presenting a user-chosen subset of said photos on an electronic device of the user, said statements and instructions comprising: code means for making an inventory of the user's digital photos; code means for categorizing the photos into one or more potential groups based on pre-defined criteria; code means for displaying the one or more potential groups to the user for selection by the user; code means for filtering the photos into a set representing the group selected by the user; and code means for making the set of filtered photos available for consumption by the user. The one or more electronic sources may comprise the electronic device of the user, one or more remote electronic sources, or the electronic device of the user and one or more remote electronic sources. The one or more remote electronic sources may comprise one or more cloud storage services, such as Google Drive, Facebook, Flickr, Google+, Instagram, Twitter, Photobucket, and Onedrive . Presenting the user-chosen subset of the photos may comprise displaying the photos in association with a keyboard interface. The code means for making the inventory of the user' s digital photos may comprise code means for storing attributes of each of the user's digital photos in a database. The attributes may comprise one or more or all of: a unique identifier of each photo; an identification of the user each photo belongs to; an identification of the electronic source where each photo

resides; a unique identifier by which each photo can be

referenced on the electronic source where each photo resides; the time a file containing each photo was created; and a URL or information a URL can be constructed from that may point to an original version of each photo. The pre-defined criteria may be any one or more of date, location, source, or user-defined search criteria. The code means for displaying the one or more potential groups to the user for selection by the user may comprise code means for displaying random thumbnails of a subset of photos within the one or more potential groups. The code means for making the set of filtered photos available for consumption by the user may comprise code means for: allowing the user to share the filtered photos with other users; allowing the user to back up the filtered photos to a storage device; allowing the user to download the filtered photos to the

electronic device of the user; and allowing the user to view the filtered photos as a slideshow on the electronic device of the user . 4. BRIEF DESCRIPTION OF THE DRAWINGS

The embodiment of the invention will now be described, by way o example, with reference to the accompanying drawings in which:

Figure 1 is a Master Flow Diagram showing the entire ecosystem at work from a very high level with emphasis on core components of the ecosystem.

Figure 2 is a flow diagram of the Central Server and Data Setup showing the sub-components of the Central Server and the

interaction between those sub-components, as well as the

interaction between the Central Server and 3^rd party service providers .

Figure 3 is a flow diagram of Import User Data showing the flow of using 3^rd party SDKs and facilities to retrieve information about photo collections residing on those providers, and

demonstrating how the information is kept up-to-date at regular intervals .

Figure 4 is a flow diagram of Main Client Flow showing a very high level view of the main thread on the client displaying the results of photo aggregation to the user while waiting for information to be fetched and processed in the background.

Figure 5 displays Client data structures, showing the various data structures needed on the client in order to build up a data model that feeds the display that represents the entire

collection of a user's photos from local and remote sources. Figure 6 is a flow diagram of the Interaction of data models on the client, showing the relationship between various data models as well as the interaction between the entities that update the data .

Figure 7 is a flow diagram of the Item collection data model, the data model representing items (photos) grouped together using a specific criteria such as date or location. Figure 8 is a flow diagram of the Group list data model, the data model representing the list of possible groups extracted by grouping all items based on a specific criteria such as date or location . Figure 9 is a flow diagram of the Item list data model, the data model representing the flat list of the entire data set (entire list of photos) including all the details of each item (photo) .

Figure 10 is a flow diagram showing the availability of photos from various sources through a keyboard outlet, demonstrating how the provided facility of photo aggregation is used through the keyboard interface with zero integration effort and by any application . Figure 11 shows the general Database table structure of the

Storage Provider Table, Files Table and User Table in the database on the Central Server.

Figure 12 shows samples of data that may be stored in the database tables on the central server with examples of values in each field. Figure 13 is an overview of the general hardware components that make up the central server.

5. DETAILED DESCRIPTION OF THE INVENTION

The details of the present invention are given in the following description which is presented as a means for one who is skilled in the art to create and use the invention for its intended purpose as described herein. Other embodiments of this invention may be apparent to those who are skilled in the art, and this document intends the patentable technology to apply to all such potential applications.

5.1 Definitions

The following definitions should assist persons skilled in the art in understanding the nature of the present invention as detailed in the disclosure that follows.

"API" is an acronym for Application Programming Interface and in this context is a set of rules, syntaxes and methods that allow software developers to communicate via a service to perform a specific task.

"Binding" is the term describing the one to one relationship between a storage provider and a user. When the system is made aware that a certain storage provider belongs to a certain user, a binding has taken place.

"Central Server" is the term used to refer to a cloud server built using an operating system running various software

components and services, as well as the logic and software implemented in order to allow the central server to perform the various actions needed by the described invention. The cloud server is available on the internet and can be accessed via standard communication protocols but preferably the well known HTTPS protocol.

"Client" generally refers to a PC or mobile communication device (e.g. a smartphone such as an iPhone) running an operating system and capable of running applications, that connects to the internet and interfaces with the user in a graphical format. However the scope of this term can be expanded to include any software platform on a device that is capable of running

applications, connecting to the internet and interfacing with the user in a graphical format. In this context of the present disclosure, the client is running a software application that is utilizing the service described.

"filter" is the criteria that will separate a subset of photos from the rest of photos by a common property such as date-taken, location-taken, source , etc.

"Grouping" is the term used to describe gathering a set of photos that share a common property. Examples would be grouping by date, which would group photos by the date they were taken, or grouping by location, which would group photos by the

geographical location they were taken.

"index" refers to a collection of records containing the

information about photos. "local photos" refers to photos that reside on the client and that are stored on its memory. Typically these photos are generated by the client taking photos using its built-in camera. "meta data" refers to the properties of photos. This includes a wide range of properties such as geolocation of the photo, width, height, time taken, originating camera, etc.

"REST" is an abbreviation for "Representational State Transfer" and is a software architecture style consisting of guidelines and best practices for creating scalable web services.

"SDK" is an acronym for Software Development Kit and refers to a set of software components and libraries that allow software developers to integrate a feature or a service into a parent software application.

"Storage Provider" is a service, typically on the internet, that allows the user to store photos on the platform it provides and allows users to access the stored photos at any given time.

"throbber" is an animated icon reflecting the fact that the application is processing data in the background and the user needs to wait for the information to be ready.

"thumbnail" is a smaller representation of a photo.

"User" refers to a human operating a "client" and running applications that utilize the described facilities.

"User account" are the credentials by which a user is identified and authenticated to the system. 5.2 Exemplary embodiments - overview

The present invention is a system comprised of many independent software components, database structures, automation scripts, hardware components, manual processes and analytical procedures and control interfaces. The system imports and gathers photos and information about the photos from various storage providers as well as user account information and stores this information in a database 103. The system running on the client 104 makes queries to the database and presents the results to the user for consumption through general viewing on a client screen or in association with a keyboard interface. Fig.l ("Master Flow

Diagram") illustrates this high level process and the following provides a brief explanation and high level overview of the process. All the remaining figures in this document will be explained in greater detail in sections 5.2.1 (Exemplary

methods) and 5.2.2 (Exemplary data structures).

The first step in realizing this invention is the setup of the Central Server 100 and the required Data structures as

illustrated in Fig. 2 ("Central Server and Data Setup") . The database 103 is setup by creating the required tables with predefined schema on a database server. The remaining components are deployed by automated scripts in the form of standard software packages and installed on either the same server computer as the database server or on another instance meant to execute the code of the cloud server engine. Some parameters such as memory capacity and CPU power of the deployed server could be altered from the defaults by a person skilled in the art if needed. There is no initial data needed to be imported and all the data gets populated as users create accounts and add details about the storage providers they utilize (e.g. Google Drive and Facebook) through the interface of the client 104. After the system has been initialized it is ready to receive data from the client 104. In an exemplary embodiment of the present invention the user will create an account by choosing an email address and a password and the client 104 will transfer this information to the central server 100. If there is no duplicate found and if the criteria passes sanity tests a record for the user will be created with the given information and thereby the identity of the user is created in the system as a unique "user account". The procedure of creating a user account, and in further usage, the procedure of logging in to a user account is handled by the authentication module 201 as shown in Fig. 2, and the information is stored in the user table 202.

The next step in the present invention is the process of

"binding" through which the user is taken to the user interface of a 3^rd party storage provider (e.g. Google Drive or Facebook) via a series of information exchanges in order to authorize the central server 100 to have access to the storage provider and as a result to have access to the photos stored on the storage provider on behalf of the user. In Fig. 2, the 3^rd party storage provider authentication module 204 is the component responsible for this procedure and the information is stored in the storage provider table 206. Figure 3 illustrates the process of binding wherein an automation script is executed on the indexer 207 through which the entire data set residing on the storage provider is processed and relevant information is extracted and saved in the database 103. The binding and data import

procedures may be repeated for several storage providers. Figs 4 to 9 illustrate all the details of the processes and the data structures on the client 104 side that come together to retrieve the indexed data on the central server 100, and that sort, group and present the data to the user and as a result provide access to the entire set of photos the user has access to on various platforms.

Fig 10 illustrates how the client side of the invention may take the form of a keyboard extension in order to provide the

intended service with zero integration effort inside any

software application that utilizes the keyboard interface.

Following is a more detailed explanation of all figures and processes of an exemplary method of the present invention.

5.2.1 Exemplary methods

The first step in the process of the present invention is the configuration and initialization of the central server database 103. The database is the core of many operations and several processes will directly or indirectly interact with it. For an exemplary embodiment of the present invention, this database is a MySQL 5.6.1 database which is an open source rational

database. The choice of database is not exclusive to the

implementation of the present invention and many other alternate embodiments are possible.

The initialization of the database involves importing a predefined schema which creates three main tables (see Figs. 11 and 12) :

1) User Table 202: This table contains the information related to each individual user and contains the following fields (not all of which are necessary for the working of the invention) :

Id: A unique identifier assigned to the user;

first_name : The first name of the user;

last_name: The last name of the user;

Email: email address of a user;

email_status : Indicates whether the email has been

verified;

last_content_update : Timestamp of the latest update applied to information about the files belonging to this user;

creation_timestamp : Timestamp of when the user was created; creation_ipv4 : IP address of user when the user was created in numerical format;

service_level : bitmask of services enabled for the user; locale: The locale identifier of the language setting of the client through which the user was created;

country_code : Country code of where the user creation was initiated from;

email_opt_in : Whether user has agreed to receive emails or not; and

Passwordcrypt : password of the user in crypt format. ) Storage Provider Table 206: This table contains the

information related to each storage provider a user is bound to and contains the following fields (not all of which are necessary for the working of the invention) :

Id: A unique identifier of the record of binding;

Account_id: The id of the user this storage provider belongs to;

Product_id: Determines the type of storage provider;

Friendly_name : A name that the user will optionally give to a storage provider to identify it more easily; identifier: The field that identifies this storage provider on the 3rd party storage provider;

account_holder : Information of the user for this storage provider on the 3rd party storage provider;

creation_timestamp : Timestamp of when this storage provider was added to the database;

Coordinates: A field containing all the necessary

information to connect to and authenticate with the storage provider;

Permanent status: A field used to mark the status of the storage provider, for example, if it cannot be indexed anymore due to invalid coordinates.

Last_index_end_timestamp : A date/time field determining the last time the contents of the storage provider were brought up to date in the files' table;

Last_indexing_state : A field that determines what the last known state of the indexing is;

Current_indexing_start_timestamp : A date/time field

determining when the start time of the current active indexing was; and

Last_index_change_timestamp : A timestamp field that records the last time a change was made to the index of files. ) Files Table 205: This table holds information about each individual photo as well as folders containing photos. It contains the following fields (not all of which are

necessary for the working of the invention) :

Id: Unique identifier of the photo;

Account_id: The id of the user the photo belongs to;

Parent_id: The id of the folder the photo resides in;

Storage_provider_id : The id of the storage provider this photo resides on; type: The type of the item (e.g. file or folder);

deleted: Whether this item has been deleted from the storage provider;

modified_time : The timestamp of the last modification to the file on the storage provider;

favorite: Flag to indicate whether the item has been marked as favorite;

hidden: Flag to indicate whether the item has been marked as hidden;

sorting_time : The timestamp used to determine the best time to be used for sorting the item;

tags: Any known tags or topics related to the item;

Sp_id: A unique identifier by which the photo can be uniquely referenced on the storage provider;

Parent_sp_id : A unique identifier by which the containing folder of the photo can be referenced on the storage provider;

Name: The name given to a photo - usually the filename; Created_time : The time the file containing the photo was created;

Description: Any extra description provided for the photo; Nb_images: In case of folders, the number of photos

residing in the folder;

Size: The size (in bytes) of the photo;

Mime_type: The mime-type of the photo, usually determining what format it is in;

Sp_source: A URL or information a URL can be constructed from that would point to the original version of the photo; Sp_thumbnail : A URL or information a URL can be constructed from that would point to the thumbnail of the photo;

City: The city in which the photo was taken;

State: The state in which the photo was taken; Country: The country in which the photo was taken; Time_taken : The time the photo was taken;

Height: The height of the photo in pixels; and

Width: The width of the photo in pixels.

Once the database is created the next step involves deploying the engine that is responsible for receiving all requests from the client 104, processing all actions and carrying out all procedures on the cloud server. For an exemplary embodiment of the present invention, this engine is implemented in PHP 5.6. The parts that honor requests from the client are executed via the PHP library module of the Apache open-source web server and the parts that run in a standalone manner are executed by the PHP command line interface. The choice of the interpreter is not exclusive to the implementation of the present invention and many other alternate embodiments are possible.

The actual PHP code is deployed via standard software packaging mechanisms. In the exemplary embodiment of this invention this packaging mechanism is based on RPMs (RedHat Package

Management) .

Once the engine is deployed and the web server is ready to receive requests from the client 104, the entry point into the system would be creating a user account in the user table 202 which is done via the signup process presented by the client to the user. This process is a typical signup process utilized in many services offered on the internet and follows the same basic rules of uniqueness and security. This information is sanitized by the client to be in accordance with the rules of account creation and sent to the central server's authentication module 201 in which after validation the account is created and the information is stored in the user table 202. The next step in utilizing the service is performing a binding to a storage provider. At the start of the binding process, the user chooses from a list of supported storage providers offered by the client. The choice is communicated with the central server' s Single Signon module 203 and the response from the central server comes as a URL pointing to the relevant storage provider's 3^rd party storage provider authentication module 204. The URL contains all the necessary information about the request for binding which the storage provider will use to process the authorization. The 3^rd party storage provider authentication module 204 asks the user for the credentials and upon

authentication, asks for confirmation from the user that the central server 100 is authorized to access the user's data. Once the user confirms the authorization, an access token is provided which the client 104 will then send to the central server 100. The central server 100 upon receiving the access token, creates a record in the storage provider table 206 with a unique Id, and stores the access token in the Coordinates field, as well as any other necessary information that describes the bound storage provider in that record, such as Friendly_name and Product_id. One of the fields of the record is the account id of the user. Therefore at this stage there is a "binding" between a storage provider and a user. The initial state of the storage provider record will indicate that it is fresh and has not been processed due to

last_index_end_timestamp and last_indexing_state fields being null initially. This brings in the indexer script which will now start the process of indexing the storage provider.

Fig. 3 illustrates the indexing procedure which is the process of getting access to the storage provider and reading in all the information about photos stored on that service. The result is stored in as many records as needed in the files table 205. The process involves the main indexer 207 going through all records of binding of the storage provider table 206 and processing them accordingly. The record is fetched (step 301) and based on the type of storage provider it relates to (based on the product_id field) an indexer object, a derivative of the base indexer class, capable of processing the target storage provider is instantiated (step 302) and takes over the process. The indexer object uses the coordinates field, which includes all the information required to identify and communicate with the storage provider, to authenticate to the storage provider. This typically includes the username identifying the user on the storage provider as well as an access token obtained through the binding process. The indexer 207 then makes the necessary API calls to the 3^rd party storage provider authentication module 204 and logs in (step 303) . At this point the indexer is ready to import data from the 3^rd party storage provider source 208. Two scenarios are possible at this point. Either the storage

provider has been processed before and therefore has entries in the files table 205 already or it is dealing with the first indexing (step 304) . In the case of first indexing, the entire data set on the storage provider needs to be processed (step

305) . Depending on how the 3^rd party storage provider has

designed their API, this process involves either traversing a hierarchy and retrieving information level by level or

processing a flat list using paged results. Section 5.3.1

contains sample code for this process for a specific storage provider. Regardless of the method, as the indexer proceeds through the hierarchy or pages of data, it will obtain

information about individual photos and their properties. The information about each individual photo is then inserted in the files table 205 via the module in step 306. Depending on the storage provider as the information about the photos are obtained, some meta-data is also provided. From this meta-data the fields that are relevant to the system are inserted into the record of the photo in the database via module/step 307. Not all storage providers provide all the meta-data that the system needs. In order to extract this portion of information, the meta-data embedded in the header of the photo needs to be parsed. When the condition is true for the photo in question (step 308) the meta-data extractor module 209 is utilized. This module performs a partial download from the beginning of the photo source (step 309), enough to obtain a header that includes all the necessary information. This information is then parsed and inserted into the record of the photo in the database (step 310) .

Once the first indexing is complete, the storage provider record in the database should be updated to reflect this new status. This involves updating the last_indexing_end_timestamp and last_indexing_state to the time the indexing was completed and to the cursor pointing to the last file that was indexed (or to a cursor provided by the storage provider, depending on the API), respectively (step 311) . As noted above, the second scenario is the case where the indexer is dealing with a storage provider that has already been indexed. The process in this case is to discover the changes that happened since the last index and apply them to the

relevant records. To do so, the cursor or timestamp (depending on the 3^rd party SDK) from the last indexing state is read from the storage provider record (step 312) . Then this cursor or timestamp is passed on to the storage provider' s API as an argument and a request is made to retrieve the list of all changes that happened since that cursor/timestamp was generated. The result is a list of photos and their changes that go through the same procedure from point A in Fig. 3.

By covering these two main scenarios and having the indexer 207 perform the procedure illustrated in Fig.3 in a loop, the end result is a files table 205 containing an up-to-date

representation of the entire data set (all photos) of all users which is one of the key aspects of the present invention.

There is an additional but specific method of creating and maintaining a complete index of files from a storage provider where the storage provider in question has been enhanced with an engine that moves the logic and functionality of Fig. 3 to the storage provider itself and as a result the indexing

functionality of the cloud server is replaced with the transfer of the up-to-date index, which holds all the extracted

information, from the storage provider to the indexer 207 to be merged into the files table 205. An example of such

implementation is the deployment of the KwiltStreamer engine on NETGEAR routers where the KwiltStreamer engine implements the logic in Fig. 3 and then transfers the index to the indexer 207. The benefit of this method is increased speed in indexing due to the indexing logic being on the same entity where the storage is located plus the offloading of processing from the cloud server to the individual storage providers (e.g. routers).

At this stage this disclosure proceeds with the explanation of the mechanisms involved on the client 104 side that will utilize the information from the central server 100. For an exemplary embodiment of this present invention, all software modules on the client are implemented in Objective-C and compiled for Apple's iOS devices. However the choice of the programming language and device is not exclusive to the implementation of the present invention and many other alternate embodiments are possible.

Fig. 4 ("Main Client Flow") illustrates a simple but crucial principal of how the client handles data and reflects it on its display. Since the presentation of this service may be done as a collage of groupings of photos or through the keyboard

interface, for instance, it is crucial for the display to be fast and responsive. This flow makes sure that the display is initiated as soon as possible (step 401) to the user to interact with and all updates and data processing are handled in the background and only made available to the display when they can be presented with no delay. As the main flow on the client starts a first time throbber is displayed 402 to allow for the initialization of the engine. While the throbber displays the application waits for the data that can be made available as soon as possible which is the group list of local photos 403.

This involves steps 901, 902, 801, 802 and 803 (see Figs. 8 and 9) to be completed and is the minimal possible processing that is needed to get the first display available to the user. Once the grouping of the local photos is displayed 404 the

application goes into a loop in which it waits for new data to become available in the background but allows the user to interact with the available data.

Fig. 6 ("Interaction of data models on the client") illustrates how different components, that are responsible for processing different data models, and the updates applied to them, interact with each other and eventually with the display. The Item List Data Model 604 is the component that represents the entire list of photos (local and remote) and their properties. It is

responsible for initializing this list and keeping it up to date. Module 605 named the CS Poller is the component that periodically checks for updates to remote photo details on the central server 100. When it is informed that a change is

available, it informs the item list data model 604 which takes care of retrieving the change. Module 606 is the module

monitoring changes to the local photos. This typically is a module readily available on the operating system of the client. When it is informed that a change is available to local photos, it informs the item list data model 604 which takes care of retrieving the change. Whenever the item list data model 604 processes new information from either local or remote photos, it informs the group list data model 602. The group list data model 602 is responsible for grouping photos and maintaining a list of groups based on the criteria chosen by the user. Upon receiving updates from the item list data model 604, it updates its groupings and informs the display 601. The details of the group list data model 602 are explained in Fig. 8 and will be explained in detail. The item collection data model 603, is the module representing a collection of photos grouped by a specific criteria chosen by the user through interaction with the interface. It receives updates from the item list data model 604 and informs the display 601 when it is ready to provide the data it is

representing. The details of this module are illustrated in Fig. 7 and will be explained in detail.

Fig. 7 illustrates the mechanism of the item collection data model 603 which is the data model that represents a list of photos grouped by a specific criteria such as date, location, source, etc. It is the data model that will be used by the display when the user has narrowed down their search criteria using the group views and is at this point viewing the actual photos of the group they have chosen. It is initiated by assuming a default filter (if none is given) (step 701) and immediately enters an idle state (Notification loop) waiting for data to become available by the item list data model 604 or for the user to change the grouping criteria. The events that trigger an action in this module are:

• Local list change

• Remote list change

• User filter change

When a "local list change" event is received (including the availability of the list of local photos for the first time) , this module updates its list with the local photos that have been affected by the change and are part of the collection (step 702) . When a "remote list change" event is received (including the availability of the list of remote photos for the first time) , this module updates its list with the remote photos that have been affected by the change and are part of the collection (step 703) . When a "user filter change" event is received, this module recollects its list based on the new filter that has been chosen by the user (step 704) . After every change that alters the data the item collection data model 603 represents, it sends a notification to the display informing it that new data is available for display (step 705) .

Fig. 8 illustrates the mechanism of the Group list data model 602 which is the data model that represents a list of groups which the available data set can be grouped into based on a specific filter the user has selected. It is the data model that is used when the display is showing all available groups into which the photos can be categorized given a selected filter. Once initiated it immediately goes into an idle state

(Notification Loop) waiting for a list of local items to become available (step 801) . Once the list of local photos (and their properties) are ready (see step 902 in Fig. 9) it initially creates a group list just comprised of local photos (step 802) . As soon as this information is available it sends a notification to the display informing it that a group list is available to be displayed (step 803) . It then checks for the availability of a previously cached grouping of remote photos (step 804) . If a cached group is available it will load it into memory (step 805) and merge it into a combined group list which now includes a grouping of both local and remote photos (step 806) . A

notification is then sent to the display informing it that new information is available (step 807) and it goes into an idle mode and waits for events (step 808) . If a remote group list cache is not available in the check point at step 804, it goes into idle mode and waits for events. In idle mode (step 808) there are three events that will trigger an action:

• Local list change

• Remote list change

• User filter change

When a "local list change" event is received, this module creates a new local group list (step 809) and then updates the combined group list (step 810) and sends a message to the display informing it of new data being available (step 807) and then goes back to idle mode (step 808) . When a "remote list change" event is received, this module creates a new remote group list (step 811) , saves the new remote group list in cache for future use (step 812) and then merges the remote group list into a combined group list (step 806) and sends a notification to the display informing it that new information is available (step 807) and then goes back to idle mode (step 808) . When a "user filter change" event is received, this module creates a new local group list (step 813) , adds the local group list into a new combined group list (step 814), creates a new remote group list (step 815), saves the remote group list into cache (step 812) and then merges the remote group list into a combined group list (step 806) and sends a notification to the display

informing it that new information is available (step 807) and then goes back to idle mode (step 808) .

Fig . 9 illustrates the mechanism of the item list data model 604 which is the data model that represents the complete list of photos and their properties. This data model is not used

directly in the display but feeds into the group list data model 602 and item collection data model 603 (see Fig . 6) . This module initiates its flow by retrieving the list of local photos (step 901) . This step utilizes the operating system's API in order to get this list. Once the list is available it sends a

notification to the group list data model 602 in step 902 to allow that module to begin its action of grouping. In the next step, it checks to see if a cache of the remote list of photos is available (step 903) . If there is, it loads the cache into memory for the next step (904) but if the cache is not

available, it sends a request to the query handler 210 on the central server 100 in step 905 to retrieve the list of remote photos. Section 5.3.2 contains sample code for this call to the central server. Once it receives the list, it saves the

timestamp of the response and also saves the list into cache (step 906) . Once the list of remote photos is available, it sends a notification to the item collection data model 603 and the group list data model 602 to inform them that the data is available for processing (step 907) . After this step the module goes into idle mode and waits for events (step 909) . The events that can trigger an action in this state are:

• Local item change

• Remote item change

When a "local item change" event is received, the module

retrieves the new list of local items (step 908) and sends a notification to the item collection data model 603 and the group list data model 602 to inform them that the data is available for processing (step 907) . It then goes back to idle mode (step 909) . When a "remote item change" event is received, the module sends a request to the query handler 210 on the central server in step 905 to retrieve the list of remote photos. Once it receives the list, it saves the timestamp of the response and also saves the list into cache (step 906) . Once the list of remote photos is available, it sends a notification to the item collection data model 603 and the group list data model 602 to inform them that the data is available for processing (step

907) . It then goes back to idle mode (step 909) .

Fig. 10 illustrates one way in which the user may use the system and how the end result is activated and utilized on the client. The host application, which is an arbitrary application on the client that uses photos in some way and uses the keyboard interface as well, activates the keyboard interface when it is ready to receive text and image input from the user (step 1001). The operating system is the entity that allows the user to choose the keyboard of their choice and in the case of the present invention, they will choose the custom keyboard utilizing the mechanism. Once the custom keyboard launches it reads in the data available from the described data models (step 1002) and displays the high level grouping of photos to the user (step 1003) . This is the starting point for the user in

narrowing down their choice. In an exemplary embodiment of the present invention, this grouping is:

• By timeline : Starting with the years

• By location: Starting with Continents

• By Sources: Starting with the list of storage providers .

At this level the user chooses one of these groups and arrives at any of the following sub groups (step 1004) :

• By timeline: By months of year chosen

• By location: By countries in the continent chosen · By sources: By folders of the chosen source

If the user chooses to go back they will arrive at the high level grouping again (step 1003) . If the user chooses one of the sub groups they will arrive at the following sub groups:

• By timeline: Days of the month chosen

· By location: Cities of the country chosen

• By sources: The sub folders of the folder chosen and the list of photos available in the chosen folder.

From this point onwards the user has the ability to select a particular photo. When they do so, the photo is copied to the clipboard of the operating system (step 1005) and they can then select the input field of the host application and paste the copied image to that input field and by this means transfer the photo to the host application to be used. With the whole system at work as described above and with the user utilizing the system in the way described above, it can be seen that the task of having a single entry point to all the photos the user has access to from a common interface allows the keyboard to easily function with adding photos from any source to messages.

5.2.2 Exemplary data structures

Both the Central Server 100 and the client 104 have their own data structures they use to handle the photo data. Each will b explained here.

Fig. 5 shows the high level data structures on the client used by the data models described in the previous section and in

Figs. 7 to 9. The data structures are divided into two main categories: Groups and Items.

The "Group" categories represent data structures containing the information about the extracted groups of photos. When a list of photos is processed, depending on the active filter, a list of groups is generated that those photos are divided into. For example, with the timeline filter, the list of photos are grouped into photos that fall into a certain date. The

individual list of dates that photos fall into make up the group list. There are two group lists generated: the remote group list 501 and the local group list 502. The remote group list 501 is made up of a list of group entries related to remote items. Each group entry contains a value reflecting the group' s value such as a specific date and a pointer to a cover photo. The cover photo is a thumbnail of one of the members of the group and may be chosen randomly. Each remote group list also has a

"timestamp" field determining the time it was generated and a "type" which states the filter used to generate the group. A remote group list is saved to the disk cache when it is generated for faster retrieval at subsequent launches (See Fig. 8) . The local group list 502 is the same as the remote group list 501 except it is derived from local photos and does not have the timestamp field. The reason a timestamp field is not needed for the local group list 502 is that it does not get cached because it is derived from the list of local photos on the device every time. The combined group list 503 is a merged and sorted collection of the local and remote groups list.

Therefore it is a list of pointers to individual group entries within the local and remote groups list. It holds one additional "type" field which determines the filter in effect for the current grouping. The "Items" category represents data structures containing the information about the individual photos. There are two "items" data structures: the local items 504 and the remote items 505. The local items 504 is an array of items, with each item

containing information about one individual local photo. The remote items 505 is an array of items, with each item containing information about one individual remote photo. The remote items data structure also has a "timestamp" field reflecting the timestamp of the response from the query handler 210 that included the remote items. The remote items 505 always gets saved to the cache for faster retrieval on subsequent launches

(See step 906 in Fig. 9) . The item collection data structure 506 is a sorted array of pointers to items grouped together based on a specific property. This data structure is used when the user is presented with the photos belonging to a group which has been narrowed down by the user. For example, if the user has narrowed down their choice of photos to photos from May 25 2015, this data structure will be an array of all item pointers for those items that were taken on May 25 2015.

Fig. 11 displays a diagram of the central server's database 103 table structure, and represents an exemplary database layout for the present invention, showing the layout of user table 202, files table 205 and storage provider table 206. This database serves as the storage location for all user accounts, the storage providers those users are bound to, and the list of photos extracted from all those storage providers along with all the meta-data of the photos. Any field in the tables with the "_id" suffix is a table key that is used to identify a record in a table and may be used to cross-reference records in other tables. A "one to many relationship" means that the two

connected tables are joined such that a single record in the first table corresponds to 1 or more records in the second table. The "primary key" identifying some of the fields are the keys that guarantee the uniqueness of records and are used to index the tables for faster queries. The database tables, fields and field formats represented in Fig. 11 are neither meant to be an exclusive nor exhaustive implementation of the present invention and other database tables, fields and field formats potentially exist which are compatible with the present

invention .

The User table 202 stores the information about the user. It has a one-to-many relationship with the storage provider table 206 meaning one user can have multiple storage providers bound to them and it has a one-to-many relationship with the files table 205 meaning the user may be the owner of many files recorded in the system. The "ID" is a unique number given to each user to identify them in the system. The "email" field is an email address used as the username which the user will provide as a key to identify themselves to the system. This information accompanies a "password" at the time of submission which will be compared to a "password crypt" field in the user table 202 to authenticate the user. Once the user is

authenticated all the information they provide is marked with their "ID" as a key connecting that information to the user. "last_content_update" is a timestamp field that is updated whenever any information related to the files or storage

providers belonging to the user are modified. The following fields are for the purpose of gathering demographic and personal information about the users and are not crucial to the

functionality of the present invention: first_name, last_name, email_status , creation_timestamp, creation_ipv4 , service_level , locale, country_code, email_opt_in .

The Storage provider table 206 stores the information about each storage provider that the user binds to their account. It has a one-to-many relationship with the files table 205 meaning many files recorded in the system may belong to one storage provider. Each storage provider record in the storage provider table holds enough information to bind that storage provider to one user and enough information for the central server 100 to authenticate against the storage provider and access photos that belong to the user on that storage provider plus the detailed information about those photos. The "ID" is a unique numerical identifier that identifies the storage provider in the system. The

"account_id" field is the unique identifier of the user that this storage provider belongs to. It points to a user record in the user table 202. "product_id" is an id that defines the type of the storage provider. Each of the storage providers supported by the central server 100, for example Google Drive, Dropbox, Flickr, etc, have a unique id. "friendly_name" is a name that the user will optionally give to a storage provider to identify it more easily. This field is particularly useful when a user has multiple accounts on the same storage provider since they can use this field to assign more recognizable names to those storage providers, "coordinates" is a line-feed separated set of values which are used to authenticate against a storage

provider. This field could be a combination of id, username, password, access tokens, etc. When a user goes through the binding process all necessary information is recorded and stored in this field to enable the central server 100 to find the storage provider and authenticate to it. "permanent_status" is a field used to mark a storage provider as being in a permanently unrecoverable condition, until the user takes action. This field is populated in cases where the coordinates have been

invalidated in a way that can't be used by the central server 100 to authenticate and extract information. It is a way to relieve the indexer from trying to authenticate when the last attempt had already failed and no change was made to coordinates suggesting that the coordinates would still be invalid.

"last_index_end_timestamp" is a timestamp field that is set when the indexing process finishes. It states the time when the indexing last finished and is used to make decisions on when a next indexing would be required. "last_indexing_state" is a field that determines what the status of the last_indexing is or was and for certain storage providers can hold information on where an indexing needs to resume from. In other words it can act as a cursor into the indexing of certain storage providers . "current_indexing_start_timestamp" is a timestamp field that would be set if the storage provider is currently being indexed. It will be set to the date/time the indexing started. It can be used to determine how long the process of indexing has taken. "last_index_change_timestamp" is a timestamp field that records the last time a change was made to the index of files (in the files table 205) that belong to this storage provider. The

"identifier" field contains the unique identifier of the storage provider as it is known on the 3rd party storage provider. The "account_holder" field contains information about the user as they are known to the 3rd party storage provider. Depending on the storage provider it can be a username, user id, email, name, etc. The "creation_timestamp" field determines when the storage provider was added to the database. In other words it holds the time when the binding took place.

The files table 205 is the table that holds the records of all indexed files for all storage providers and all users. Each "file" record has an "id" that is a unique identifier of the file record in the system. All references to the file are done via this id. "account_id" is the id of the user to which the file belongs. "storage_provider_id" is the id of the storage provider from the storage provider table 206 from which the file was indexed. In some storage providers there is a hierarchy and files can be grouped under folders and subfolders. For these storage providers, in order to maintain the hierarchy, the id of the folder to which the files belong is stored in a field named "parent_id". Naturally if there is no hierarchy or if the file is at the top of the hierarchy (i.e. is at the root level), the parent_id will be cleared (set to null ) . Each file record refers to a file on the storage provider. This reference is stored in the "sp_id" field which when provided to the storage provider via its API will uniquely identify the file.

"parent_sp_id" is the reference to the parent of the file on the storage provider, "name" is the name given to the file. Usually this is the same as the filename, "created time" is the creation timestamp of the file. "nb_images" holds the number of files inside folders and is relevant for records that represent folders in storage providers that have a hierarchy, "size" is the size in bytes of the file, "mime-type" is the mime type of the file determining its format. "sp_source" holds the URL that can be used to fetch the file from the storage provider or it holds enough information to construct the URL that can be used to fetch the file from the storage provider. "sp_thumbnail" holds the URL that can be used to fetch the thumbnail of the file from the storage provider or it holds enough information to construct the URL that can be used to fetch the thumbnail of the file from the storage provider, "city", "state" and "country" are fields that determine the location the photo was taken.

"time_taken" is the date/time the photo was taken, "height" and "width" are the height and width in pixels of the photo

respectively. "Type" determines the type of the item on the storage provider. It can be an image, a folder, an album, etc. The field "deleted" is a boolean flag that states whether the item it corresponds to has been deleted from the storage

provider and therefore does not exist, "modified_time" holds the timestamp of when the item was last modified on the storage provider, "favorite" is the field that determines whether the item has been added to the user's favorite list, "hidden" indicates whether the user has chosen to hide this item from being displayed. "sorting_time" contains the time to be used for sorting the item when grouped with other items and displayed in a chronological order, "tags" is a field that holds additional keywords assigned or extracted for the item either by the 3rd party storage provider or by the central server 100 as a result of an analyzing process such as image recognition. 5.2.3 Exemplary Apparatus

Examples of well known computer systems that may be suitable fo use in the present system with aspects of the subject matter described herein include, but are not limited to, personal computers, server computers, laptop devices, smart phones, mobile devices, multiprocessor systems, network personal computers, mainframe computers, distributed computing

environments where tasks are performed by remote processing linked through a communications network, remote systems, etc., and the like. In a distributed computing environment, program components or modules may be located in both local and remote computer storage media including memory storage devices. Remote systems may involve remote computers, such as personal

computers, a router, a server, a network personal computer, a peer device or other common network node, and the like.

In addition, components of suitable platforms may include, but are not limited to, a central processing unit, a system memory, and a system bus that integrates various system components, including the system memory to the processing unit. The system memory may include computer storage media in the form of volatile or non-volatile memory, such as read-only memory ( ROM ) , random access memory (RAM) , EEPROM, flash memory, CDROM, DVD, magnetic storage devices, etc., which can be used to store the desired information and is accessible by the computer system. The system bus may be any of several types of bus structures including a memory controller, a peripheral bus, and a local bus of various bus architectures. Suitable platforms also include a variety of computer-readable media comprising, for example, computer communication media and computer storage media. Computer storage media may include both removable and non-removable media, or volatile or non-volatile media alike, for information storage of computer-readable instructions, program modules, data structures or other data, etc. Computer communication media generally comprises any information delivery media for delivering media information to a user, computer- readable instructions, program modules, data structures or other data, etc.

For an exemplary embodiment of the present invention, Fig. 13 is a diagram that illustrates the key hardware components that may be used for the central server 100 architecture. The Query

Handler 210, the Indexer 207 and the Database 103 are systems that may exist within a cloud computing environment, meaning that their physical components are potentially spatially

distributed across many physical machines. Each of these systems may consist of a CPU, Memory and Storage, in addition to all other required components for the typical operation of such computing system (as detailed above) .

In an exemplary embodiment of the present invention, the query handler 210, indexer 207, and Database server 103 are connected together via a Fiber Channel over Ethernet (FCoE) connection and all are similarly connected to an Internet Firewall 1304 which provides security from unauthorized external access. The

firewall is connected to the Internet 1305 and the entire system is accessible via the Internet from the client.

5.2.4 Alternatives and Extensions

The present invention is designed to provide access to all photos from various sources through a display that can, for instance, show various groupings of photos for viewing by a user, or, for instance, through a universally adapted interface the keyboard. However, photos are just a subset of media documents that users consume. Audio and Video files are also good candidates to be considered in a similar implementation the present invention but with the outlet being suitable for consuming audio and/or video.

Another potential alternative to the present invention is to consider photos or media that are public and that are not necessarily owned by a user or protected by their credentials. In the present invention as generally described all the photos have the common attribute of being owned by the same user. In an alternative implementation the common attribute of photos grouped together might be the topic of the photos or hashtags assigned to photos from publicly available sources, thereby providing access to all photos with the same subject from various sources and through, for instance, a universally adapted interface, the keyboard.

5.2.5 Example implementation

In the following section we will describe an example

implementation of the present invention using Apple's iOS platform as the deployment target and will demonstrate how the present invention has significantly made it easier and more convenient to consume photos from various sources and will compare it to the procedure required in the absence of this mechanism.

In this example an iOS application has been written to implement the client side of the system (see Fig. 4 to Fig. 9) in the form of an iOS keyboard extension. With the launch of the application, the user first creates an account and logs in via the signup/signin screen.

The user record gets created in the user table 202 and the user logs in via the authentication module 201.

In the next step the user binds the application to all the accounts they have on various storage providers.

< PHOTO STREAMS O

SOCIAL : ;

f ' ACEBOOK

Θ INSTAGRAM

CLOUD

■ GOOGLE DRIVE ONEDRIVE

This goes through the 3^r party storage provider authentication module 204 and authorizes the application and the central server 100 to access all the photos the user has on those platforms. In the background on the cloud, the indexing begins for each storage provider, which procedure has been described with respect to Fig. 3. and the database gets populated.

At this point the user can now start utilizing the system and benefiting from it. In this example the user launches the email application in order to send photos to a friend. The photos are in different locations: 1. Some photos are on the iOS device; 2. Some photos have been posted to facebook; and 3. Some photos have been synced to dropbox. All the user needs to do when they start typing the body of the email is to select the keyboard that implements the system. The keyboard launches and performs all the procedures shown in relation to Figs. 4 to 9 and

eventually from the presentation layer takes the user through the procedure shown in Fig. 10. The user is presented with a high level grouping of all their photos from various sources in this example starting by year. They select 2014, then the month,

The user is presented with all photos from their local device, Facebook, dropbox, etc that have been taken on June 12^th 2014 in one place. They can now select the desired photo (s) with one tap to copy and paste them in the body of the email with another tap .

To compare the simplicity of the above procedure with what would have been needed in the absence of this keyboard interface the steps below are listed for the exact same two pictures (one from Facebook and one from Dropbox) . The user needs to leave the mail application, launch their Facebook application, browse their albums, find the photo, save it locally to the device, go back to the mail application, tap and hold on the body of the email, select "Insert photo", be presented with the entire camera roll, find the picture saved from Facebook, and select it to be added to the email. For the next picture which is on Dropbox, the user needs to leave the mail application again, launch their Dropbox application, browse to their albums, find the photo, save it locally to the device, go back to the mail application, tap and hold on the body of the email, select "Insert photo", be presented with the entire camera roll, find the picture saved from Dropbox and select it to be added to the email. The

tediousness of this procedure is obvious compared to the

procedure offered by the example keyboard extension.

To demonstrate the power of grouping photos by other types of meta-data which is readily available due to the indexing that has taken place on the central server 100, the user now composes another email to send pictures from a certain location. Again they launch the keyboard and select the "location" filter. They are presented by a high level grouping of photos by continent. This is using the data from photos that come from various sources without the user needing to know what the source is. The user selects "Europe", then "Germany" and then "Nordlingen" . They are presented with four photos each coming from a different source. Selecting all four is a matter of copying and pasting from the same interface without leaving the mail application.

Another example of an implementation of the present invention is a photo browser and editor capable of grouping all photos from various sources based on a common criteria. In this example the account creation, authentication and binding of storage providers is the same as the previous example however the consumption takes the form of a full screen photo browser.

TIMELINE A Q Photos from various sources grouped by date

i MMt

Photos from various sources grouped by location M»

Photos from various sources that have the same location Sample Code

In this section samples of some of the code written to implement the present invention will be provided. Code sample #1 will be from the central server 100 and code sample #2 will be from the client. This code will allow the person skilled in the art to readily understand how to program the other key modules that make up the present invention, as previously described.

5.3.1 Code Sample #1

This sample code is from the central server 100 and has been chosen from the "indexer" module 207, whose functionality has been described with respect to Fig. 3.

In particular, this snippet is the code from step 305 where the indexer makes an API call to the storage provider to fetch the index of photos. The object returned is a PHP object

representing the fetched data. The results of the fetched data are then parsed and inserted into the database to form the index of the files on the storage provider, which in this sample is Flickr .

* This function supports calling the Flickr API.

*

* @param array $query_params Defines which Flickr API function to call and provides the necessary values to do so.

*

^* ©throws HipPlayException in the event we received self::NUM_RANDOM_BACKOFF_ATTEMPTS + 1 errors from Flickr in a row.

* ©return Array of json-decoded results from Flickr.

^*/

private function callFlickr($query_params)

{

// Note: Intentionally not checking the parameters here.

// This is a support function for other functions in this same class (private) // and we will assume the parameter is an array with at least the method,

// format and nojasoncallback indexes defined.

// Get the signed URL

$base_url = 'https://api.flickr.com/services/rest/';

$signed_url = $this->oauth->getSignedURL('GET', $base_url, $query_params);

if (!isset($this->ch)) {

$this->ch = curljnitO;

curl_setopt($this->ch, CURLOPT_ENCODING, "gzip, deflate");

curl_setopt($this->ch, CURLOPT_RETURNTRANSFER, true);

curl_setopt($this->ch, CURLOPT_USERAGENT, 'Kwilt'); // this field can't be empty

}

curl_setopt($this->ch, CURLOPT_URL, $signed_url);

$nb_attempts = 0;

while (true) {

$nb_attempts++;

$exception = null;

// Make the request to Flickr

$response_body = curl_exec($this->ch);

// Grab the status code

$status_code = curl_getinfo($this->ch, CURLINFO_HTTP_CODE);

// Parse the body

$flickrData = json_decode($response_body, true);

// Check for errors

if ($status_code !== 200) {

$exception = new HipPlayException("Flickr error with HTTP status "' . $status_code . "', here is body: " $response_body, HipPlayException::FATAL_ERROR);

} else if(is_null($flickrData) || !is_array($flickrData)){

$exception = new HipPlayException('Flickr did not return a valid json array. Body: ' .

print_r($response_body, true), HipPlayException::FATAL_ERROR);

} else if (!array_key_exists('stat\ $flickrData) || $flickrData['stat'] !== 'ok') {

// See https://www.flickr.com/services/api/flickr.photos.recentlyUpdated.html for error codes if (array_key_exists('code\ SflickrData) && ($flickrData['code'] == 98 || $flickrData['code'] == 99)) { // need to re-authenticate

$exception = new HipPlayException('Need to re-authenticate with Flickr. Body: ' .

print_r($response_body, true), HipPlayException::OAUTH_ACCESS_DENIED);

} else {

$exception = new HipPlayException('Flickr results status not "ok". Body: ' . print_r($response_body, true), HipPlayException::FATAL_ERROR);

}

// If there was an error, backoff and try again or throw the error

if(isset($exception)){

if($nb_attempts > self::NUM_RANDOM_BACKOFF_ATTEMPTS) {

curl_close($this->ch);

throw $exception;

}

$this->exponentialRandomSleep($nb_attempts); // random backoff before trying again

} else {

break;

}

} // end while(true)

// Done! return $flickrData;

}

5.3.2 Code Sample #2

In this code sample, the call that is sent from the client to the query handler 210 is shown. This is the call that happens in step 905 (see Fig. 9) to fetch the list of remote photos from the central server. It allows for passing of a few arguments to fine tune the request and, for example, to exclude certain storage providers which the user may have turned off. T e response in this example is in XML so the XML parsing takes place to extract all the attributes of the fetched data set. As shown in the code, the attributes have a close correlation with the fields of the files table 205. When all results have been parsed and processed and internal objects have been created, these attributes can be used to group the dataset using

different filters. - (NSArray *)getltemsWithExcludelDs:(NSArray *)excludelDs

limit:(NSUInteger)limit

offset:(NSUInteger)offset

force:(BOOL)force

lastModifiedDate:(inout NSString * autoreleasing *)lastModifiedDate

useCached:(out BOOL *)useCached

error:(NSError * autoreleasing *)error {

NS utableString *queryString = [NSMutableString

stringWithString:@"type=image&fields=id, sorting _time,thumbnail_url,storage_provider_id, latitude, longitude"];

if ([HPNetworkHelpers getNetworkConnection] == ReachableViaWiFi) {

NSString "ipAddress = [CustomRoute locallPAddress];

NSString "netmask = [CustomRoute localNetmask];

[queryString appendFormat:@"&localip=%@&netmask=%@", ipAddress, netmask];

[queryString appendFormat:@"&limit=%lu", (unsigned long) limit];

[queryString appendFormat:@"&offset=%lu", (unsigned long) offset];

NSString *requestUrl = [NSString stringWithFormat:@"%@%@?%@", HIPPLAYDO AIN, API_REST_INDEX, queryString];

NSString "Jast odifiedDate = force ? nil : (lastModifiedDate ? *lastModifiedDate : nil);

xmlDoc 'document = NULL;

NSString *outLastModifiedDate = nil;

NSError *outError = nil;

BOOL returnNil = NO;

©autoreleasepool {

NSURLResponse 'response = nil;

NSData *dataQueryResult = [[HPKwiltServer sharedlnstance] getSynchronousRequest:requestUrl

last odifiedDate:_lastModifiedDate responseOut&response errorOut:&outError];

NSInteger statusCode = 0;

if ([response isKindOfClass:[NSHTTPURLResponse class]]) {

NSHTTPURLResponse *httpResponse = (NSHTTPURLResponse ^*) response;

statusCode = httpResponse. statusCode;

if (statusCode != HTTP_CODE_NOT_MODIFIED) {

NSDictionary *allHeaderFields = [httpResponse allHeaderFields];

outLastModifiedDate = [allHeaderFields objectForKey:@"Last-Modified"];

}

BOOL outUseCached = (statusCode == HTTP_CODE_NOT_MODIFIED);

if (useCached)

"useCached = outUseCached;

if (outUseCached || !dataQueryResult)

returnNil = YES;

else

document = xmlReadMemory((const char *) [dataQueryResult bytes], (int) [dataQueryResult length], NULL, NULL, (XML_PARSE_HUGE | XML_PARSE_NOCDATA | XML_PARSE_NOBLANKS));

dataQueryResult = nil;

}

if (lastModifiedDate)

*lastModifiedDate = outLastModifiedDate;

if (error)

'error = outError;

if (returnNil || document == NULL)

return nil;

NSMutableArray ^*retVal = nil;

xmlNodePtr rootNode = xmlDocGetRootElement(document);

if (rootNode != NULL) {

if (strcmp((const char *) rootNode->name, "errors") == 0) {

if (error) {

NSString 'message = @"";

NSString ^*code = @"";

xmlNodePtr firstChild = rootNode->children;

if (firstChild != NULL) {

xmlAttrPtr curProp = firstChild->properties;

while (curProp != NULL) {

xmlChar *valueXml = xmlNodeGetContent(curProp->children);

if (valueXml == NULL) {

curProp = curProp->next;

continue;

}

if (PROP_NAME("code"))

code = [NSString stringWithUTF8String:(char *) valueXml];

else if (PROP_NAME("msg"))

message = [NSString stringWithUTF8String:(char ^*) valueXml];

xmlFree(valueXml);

curProp = curProp->next;

}

NSString *desc = [NSString stringWithFormat:@"Error %@: %@", code, message];

if (error)

^•error = [NSError errorWithDomain:ErrorDomain code:0 userlnfo:@{

@"code" : code,

@"msg" : message,

@"desc" : desc

}];

} }

else if (strcmp((const char *)rootNode->name, "items") == 0)

{

NSDictionary "remoteltemsCache = [[HPItemCache sharedlnstance] readRemoteltemsCache];

retVal = @[].mutableCopy;

NSDateFormatter *dateFormatter = [[NSDateFormatter alloc] init];

[dateFormatter setDateFormat:@"yyyy-MM-dd HH:mm:ss"];

NSString *localldentProperty = ([[HPAssetsManager sharedlnstance] phLibrary]) ? @"localldentifier" : @"thumbnailUrl"; xmlNodePtr curChild = rootNode->children;

HPImageltem *imageltem = nil;

while (curChild != NULL)

{

if (strcmp((const char *)curChild->name, "item") != 0)

{

curChild = curChild->next;

continue;

}

@autoreleasepool {

if (imageltem == nil)

imageltem = [HPImageltem new];

double latitude = 0, longitude = 0;

xmlAttrPtr curProp = curChild->properties;

while (curProp != NULL)

{

xmlChar "valueXml = xmlNodeGetContent(curProp->children);

if (valueXml == NULL)

{

curProp = curProp->next;

continue;

}

char 'value = (char *)valueXml;

if (PROP_NAME("id"))

imageltem.lD = [NSString stringWithUTF8String:value];

else if (PROP_NAME("parent_id"))

imageltem. parentID = atol(value);

else if (PROP_NAME("name"))

imageltem. name = [NSString stringWithUTF8String:value];

else if (PROP_NAME("size"))

imageltem.fileSize = @(atol(value));

else if (PROP_NAME("source_url"))

imageltem. sourceUrl = [NSString stringWithUTF8String:value];

else if (PROP_NA E("sorting_time"))

imageltem. sortingTime = [dateFormatter dateFromString:[NSString stringWithUTF8String:value]];

else if (PROP_NA E("thumbnail_url"))

imageltem. thumbnailUrl = [NSString stringWithUTF8String:value];

else if (PROP_NAME("friendly_name"))

imageltem.friendlyName = [NSString stringWithUTF8String:value];

else if (PROP_NAME("storage_provider_id"))

imageltem. storageProviderlD = [NSString stringWithUTF8String:value];

else if (PROP_NAME("created_time"))

imageltem. createdTime = [dateFormatter dateFromString:[NSString stringWithUTF8String:value]];

else if (PROP_NAME("modified_time"))

imageltem. modifiedTime = [dateFormatter dateFromString:[NSString stringWithUTF8String;value]];

else if (PROP_NAME("width"))

imageltem. size = CGSizeMake(atof(value), imageltem. size. height);

else if (PROP_NAME("height"))

imageltem. size = CGSizeMake(imageltem. size. width, atof(value));

else if (PROP_NAME("geolocation"))

imageltem. location = [HPLocation locationFromDelimitedString:[NSString stringWithUTF8String:value] delimiter:@"|"];

else if (PROP_NA E("latitude"))

latitude = [[NSString stringWithUTF8String:value] doubleValue]; else if (PROP_NAME("longitude"))

longitude = [[NSString stringWithUTF8String:value] doubleValue];

else if (PROP_NAME("timeJaken"))

imageltem.timeTaken = [dateFormatter dateFromString:[NSString stringWithUTF8String:value]];

else if (PROP_NAME("user_albums"))

imageltem.userAlbumlDs = [[NSString stringWithUTF8String:value] componentsSeparatedByString:®","]; else if (PROP_NA E("hidden"))

imageltem.hiddenTime = [dateFormatter dateFromString:[NSString stringWithUTF8String:value]];

else if (PROP_NA E("favorite"))

imageltem.favoriteTime = [dateFormatter dateFromString:[NSString stringWithUTF8String:value]]; else if (PROP_NAME("duid"))

imageltem.DUID = [NSString stringWithUTF8String:value];

else if (PROP_NA E("localid"))

imageltem.locallD = [NSString stringWithUTF8String:value];

else if (PROP_NA E("mime_type"))

imageltem.mimeType = [NSString stringWithUTF8String:value];

xmlFree(valueXml);

curProp = curProp->next;

}

if (latitude II longitude)

imageltem. location = [HPServerConnectionModel geocodeLat: latitude lon:longitude];

if ([imageltem.DUID isEqualToString:uniqueDevicelD])

{

if ([imageltem.locallD hasPrefix:STORAGE_LOCAL_DLNA])

{

[imageltem setlsStoredLocal:NO];

[imageltem setlsDLNA:YES];

}

else

{

[imageltem setValue:imageltem.locallD forKeyilocalldentProperty];

[imageltem setlsStoredLocakYES];

}

NSString *itemKey = [[HPItemCache sharedlnstance] getldentifienimageltem];

HPImageltem *oldltem = remoteltemsCache[itemKey];

if (old Item) {

[retVal addObject: imageltem];

// id,sorting_time,thumbnail_url,storage_provider_id,geolocation

oldltem.lD = imageltem. ID;

oldltem.sortingTime = imageltem. sortingTime;

oldltem.thumbnailUrl = imageltem.thumbnailUrl;

oldltem.storageProviderlD = imageltem. storageProviderlD;

oldltem. location = imageltem. location;

[retVal addObject:oldltem];

} else {

[retVal addObject:imageltem];

imageltem = nil;

}

xmlNodePtr tmpChild = curChild;

curChild = curChild->next;

xmlFreeNode(tmpChild);

} rootNode->children = NULL; // this shouldn't leak memory since we have already freed all the children rootNode->last = NULL;

}

xmlFreeDoc(document);

xmlCleanupParserO;

return retVal; Although specific embodiments of the invention have been described, it will be apparent to one skilled in the art that variations and modifications to the embodiments may be made within the scope of the following claims.

Claims

CLAIMS:

A computer-implemented method for aggregating information about a user' s digital photos stored on one or more electronic sources and presenting a user-chosen subset of said photos on an electronic device of the user, comprising the steps of: making an inventory of the user's digital photos; categorizing the photos into one or more potential groups based on pre-defined criteria; displaying the one or more potential groups to the user for selection by the user; filtering the photos into a set representing the group selected by the user; and making the set of filtered photos available for

consumption by the user.

The computer-implemented method of claim 1 wherein the one or more electronic sources comprises the electronic device of the user.

The computer-implemented method of claim 1 wherein the one or more electronic sources comprises one or more remote electronic sources.

4. The computer-implemented method of claim 1 wherein the one or more electronic sources comprises the electronic device of the user and one or more remote electronic sources .

5. The computer-implemented method of claim 3 or 4 wherein the one or more remote electronic sources comprises one or more cloud storage services . 6. The computer-implemented method of claim 5 wherein the one or more cloud storage services is one or more of Google Drive, Facebook, Flickr, Googlet, Instagram, Twitter, Photobucket, and Onedrive. 7. The computer-implemented method of any one of claims 1 to

6 wherein presenting the user-chosen subset of said photos comprises displaying the photos in association with a keyboard interface. 8. The computer-implemented method of any one of claims 1 to

7 wherein making the inventory of the user' s digital photos comprises storing attributes of each of the user's digital photos in a database. 9. The computer-implemented method of claim 8 wherein the attributes may comprise one or more of: a unique

identifier of each photo; an identification of the user each photo belongs to; an identification of the

electronic source where each photo resides; a unique identifier by which each photo can be referenced on the electronic source where each photo resides; the time a file containing each photo was created; and a URL or information a URL can be constructed from that may point to an original version of each photo.

10. The computer-implemented method of claim 8 wherein the attributes comprises: a unique identifier of each photo; an identification of the user each photo belongs to; an identification of the electronic source where each photo resides; a unique identifier by which each photo can be referenced on the electronic source where each photo resides; and a URL or information a URL can be

constructed from that may point to an original version of each photo.

11. The computer-implemented method of any one of claims 1 to 10 wherein the pre-defined criteria is any one or more of date, location, source, or user-defined search

criteria .

12. The computer-implemented method of any one of claims 1 to 11 wherein displaying the one or more potential groups to the user for selection by the user comprises

displaying random thumbnails of a subset of photos within the one or more potential groups.

13. The computer-implemented method of any one of claims 1 to 12 wherein making the set of filtered photos available for consumption by the user comprises: allowing the user to share the filtered photos with other users; allowing the user to back up the filtered photos to a storage device; allowing the user to download the filtered photos to the electronic device of the user; and allowing the user to view the filtered photos as a slideshow on the electronic device of the user.

14. A computer readable memory having recorded thereon statements and instructions for execution by a computer for aggregating information about a user' s digital photos stored on one or more electronic sources and presenting a user-chosen subset of said photos on an electronic device of the user, said statements and instructions comprising: code means for making an inventory of the user' s digital photos; code means for categorizing the photos into one or more potential groups based on pre-defined criteria; code means for displaying the one or more potential groups to the user for selection by the user; code means for filtering the photos into a set

representing the group selected by the user; and code means for making the set of filtered photos

available for consumption by the user.

15. The computer readable memory of claim 14 wherein the one or more electronic sources comprises the electronic device of the user.

16. The computer readable memory of claim 14 wherein the one or more electronic sources comprises one or more remote electronic sources.

17. The computer readable memory of claim 14 wherein the one or more electronic sources comprises the electronic device of the user and one or more remote electronic sources .

18. The computer readable memory of claim 16 or 17 wherein the one or more remote electronic sources comprises one or more cloud storage services .

19. The computer readable memory of claim 18 wherein the one or more cloud storage services is one or more; of Google Drive, Facebook, Flickr, Google+, Instagram, Twitter, Photobucket, and Onedrive .

20. The computer readable memory of any one of claims 14 to 19 wherein presenting the user-chosen subset of said photos comprises displaying the photos in association with a keyboard interface.

21. The computer readable memory of any one of claims 14 to 20 wherein the code means for making the inventory of the user' s digital photos comprises code means for storing attributes of each of the user' s digital photos in a database.

22. The computer readable memory of claim 21 wherein the attributes may comprise one or more of: a unique

23. The computer readable memory of claim 21 wherein the attributes comprises: a unique identifier of each photo; an identification of the user each photo belongs to; an identification of the electronic source where each photo resides; a unique identifier by which each photo can be referenced on the electronic source where each photo resides; and a URL or information a URL can be

constructed from that may point to an original version of each photo.

24. The computer readable memory of any one of claims 14 to 23 wherein the pre-defined criteria is any one or more of date, location, source, or user-defined search

criteria .

25. The computer readable memory of any one of claims 14 to 24 wherein the code means for displaying the one or more potential groups to the user for selection by the user comprises code means for displaying random

thumbnails of a subset of photos within the one or more potential groups.

26. The computer readable memory of any one of claims 14 to 25 wherein the code means for making the set of filtered photos available for consumption by the user comprises code means for: allowing the user to share the filtered photos with other users; allowing the user to back up the filtered photos to a storage device; allowing the user to download the filtered photos to the

electronic device of the user; and allowing the user to view the filtered photos as a slideshow on the electronic device of the user.

27. A computer program product comprising a computer readable memory storing computer executable instructions thereon that when executed by a computer perform the method steps of any one of claims 1 to 13.