CA2894608A1 - A computer-implemented method of aggregating and presenting digital photos from numerous sources - Google Patents

Abstract

A computer-implemented method of providing a client (e.g. an iPhone) with the ability to seamlessly aggregate all of a user's digital photos from both local sources (e.g. local memory on the iPhone) and remote sources (e.g. social media services like Facebook), and displaying relevant grouping(s) of same based on the user's desired search filter (e.g. grouped by location) without the user ever having to know where the photos are actually stored. The user is simply required to provide the app of the present invention with one-time secure access details to all of his/her sources (e.g. username and password to Facebook) and the app is thereafter able to aggregate and display photos on the client based on the user's search criteria. The photo groupings can simply be displayed on the client as desired for viewing purposes, or the relevant groupings can be displayed during the use of a keyboard for messaging purposes wherein any photo of choice can be readily located regardless of source and attached as part of a message to be sent to another person.

Description

A COMPUTER-IMPLEMENTED METHOD OF AGGREGATING AND PRESENTING
DIGITAL PHOTOS FROM NUMEROUS SOURCES
1. FIELD OF THE INVENTION
The present invention relates to the field of digital photos. In particular, the present invention relates to the aggregation of a user's digital photos from local and on-line sources and the presentation of same on a user's device.

2. BACKGROUND OF THE INVENTION
With the transition of photography from analog to digital and the availability of cameras on handheld phones and other mobile devices, the acquisition and use of personal and professional digital photos has risen exponentially in recent years and this will only continue into the foreseeable future. This growth has also coincided with the growth of digital storage devices as well as cloud storage services that offer enormous storage capacities for storing a user's photos.
Many solutions have appeared that desire and intend to be the central repository for a user's digital files and documents, including photos. In this respect, both small and big corporations have continued to expand their storage capacities and capabilities, and often reduce pricing models to attract customers in the hope of becoming the final and dominant solution for digital storage. As a result, many users have switched time and again from one storage provider to another (as well as from local to cloud storage) as conditions have changed and favourable pricing offers have arisen. This has created a scattered digital world and users have found themselves with their digital data distributed amongst a series of storage providers and personal backup archives.
In addition to a user's personal photos, the stream of photos posted and shared by friends through social media and community groups has contributed more and more to the collection of photos that a user may wish to store and/or have access to. Such photos often live on the social media networks of their origin.
The foregoing has created the need for a facility that can provide a user with the means of finding and accessing their vast collection of photos without the hassle of maintaining the knowledge of where the sought-after photo actually resides and how to get to it. Many applications have tried to address this need by providing mechanisms to copy a user's photos from various sources to their own central repository in the hope of becoming the final solution for a user's digital storage.
However, this has resulted in yet more providers being added to the collection of storage providers for digital storage.
An eloquent solution is therefore needed that would not comprise the copying of current photos from various sources for storage in yet another repository, but that would instead maintain a reference to the original source of a user's various photos, while allowing the user to find and access sought-after photos at any time using a unified engine that acts as a "one-stop shop" for retrieving the desired photo and accompanying data without ever having to know where the photo is actually stored.

3. SUMMARY OF THE INVENTION
The present invention provides a computer-implemented method of providing a client (e.g. a user's PC or portable device, such as an iPhone) with the ability to seamlessly aggregate all of a user's digital photos from both local sources (e.g. local memory on a PC or iPhone or local storage device) and remote sources (e.g. cloud storage services like Google Drive or social media services like Facebook), and displaying relevant grouping(s) of same based on the user's desired search filter (e.g. grouped by date or location) without the user ever having to know where the photos are actually stored. The user is simply required to provide the computer-implemented application ("app") of the present invention with one-time secure access details to all of his/her sources (e.g. username and password to Google Drive and Facebook) and the app is thereafter able to look after all aggregation and displays on the client based on the user's search/filter criteria. The photo groupings can simply be displayed on the client as desired for viewing purposes, or the relevant groupings can be displayed during the use of a keyboard for messaging purposes wherein any photo of choice can be readily located regardless of source and attached as part of a message to be sent to another person. In so doing, the app does not make yet another copy of the digital photo for storage in yet another database, but instead provides a link to the photos in the various sources and saves information about the links and data about the photos in a database. The present invention thereby provides a user with searchable ease of access to all of his/her photos regardless of source, without the user having to worry about keeping track of sources ever again.

In view of the foregoing, one aspect of the present invention is to provide a mechanism in which local photos stored on a client device are aggregated and displayed seamlessly with remote photos from 3r1 party providers, thereby relieving a user from having to maintain the knowledge of which photo is stored where and whether it is local or remote.
Another aspect of the present invention is to provide a mechanism that maintains an up-to-date index of the entire digital data set (photo library) of a user from his/her various local and remote sources in order to optimize the procedure of processing and searching amongst the entire data set.
Another aspect of the present invention is to provide a means of categorizing and grouping photos stored in local and remote locations based on their existing meta data in order to facilitate searches based on different criteria.
Another aspect of the present invention is providing the ability for a user to locate photos whether stored in local or remote locations by allowing the user to navigate through groups and sub groups of photos that have been aggregated according to search criteria in a way that minimizes the effort of narrowing down the intended set of photo(s) the user is seeking.
Another aspect of the present invention is to provide a mechanism in which the credentials and authentication information for accessing 3rd party storage providers is entered once and maintained as long as the user has provided the proper authorization for accessing the 3rd party provider and maintains that authorization. This eliminating the need for the user to re-enter such credentials into various outlets for later access.

4 A further aspect of the present invention is to provide a unified mechanism to find and retrieve information about any photo from any source without the need to adapt to upgrades and changes to SDKs from 3r1 party sources and to shield the user and the software utilizing the mechanism from such changes, thereby resulting in less changes on the end point software and concentrating the management of communicating with 3rd party sources on a central cloud service.
,Another aspect of the present invention is to have the capability of receiving new information and data that may become available at a later time r as users backup, share and store more photos in their local storage or third party service providers of choice.
Another aspect of the present invention is providing a mechanism in which an optimum amount of data is transferred between the cloud service and the client and between the cloud service and the 3r1 party service providers for the purpose of presenting the entire data set of photos to the client.
Another aspect of the present invention is providing a means of reducing the wait time for users to have access to photos, and allowing the process ing of current available data while updates are retrieved in the background.
Another aspect of the present invention is eliminating the need for any software integration or use of SDKs by outlets and software intending to use the method of the present invention by

5 providing the results through a universally adopted keyboard interface instead.
4. BRIEF DESCRIPTION OF THE DRAWINGS
The embodiment of the invention will now be described, by way of example, with reference to the accompanying drawings in which:
Figure 1 is a Master Flow Diagram showing the entire ecosystem at work from a very high level with emphasis on core components of the ecosystem.
Figure is a flow diagram of the Central Server and Data Setup showing the sub-components of the Central Server and the interaction between those 3ub-components, as well as the interaction between the Central Server and 3rd party service providers.
Figure 3 is a flow diagram of Import User Data showing the flow of using 3rd party SDKs and facilities to retrieve information about photo collections residing on those providers, and demonstrating how the information is kept up-to-date at regular intervals.
Figure 4 is a flow diagram of Main Client Flow showing a very high level view of the main thread on the client displaying the results of photo aggregation to the user while waiting for information to be fetched and processed in the background.
Figure 5 displays Client data structures, showing the various data structures needed on the client in order to build up a data

6 model that feeds the display that represents the entire collection of a user's photos from local and remote sources.
Figure 6 is a flow diagram of the Interaction of data models on the client, showing the relationship between various data models as the components that build the entire data model as well as the interaction between the entities that update the data.
Figure 7 is a flow diagram of the Item collection data model, the data model representing items (photos) grouped together using a specific criteria such as date or location.
Figure 8 is a flow diagram of the Group list data model, the data model representing the list of groups possible by grouping all items based on a specific criteria such as date or location.
Figure 9 is a flow diagram of the Item list data model, the data model representing the flat list of the entire data set (entire list of photos) including all the details of each item (photo).
Figure 10 is a flow diagram showing the availability of photos from various sources through a keyboard outlet, demonstrating how the provided facility of photo aggregation is used through the keyboard interface with zero integration effort and by any application.
Figure 11 shows the general Database table structure of the Storage Provider Table, Files Table and User Table in the database on the Central Server.

7 Figure 12 shows samples of data that may be stored in the database tables on the central server with examples of values in each field.
Figure 13 is an overview of the general hardware components that make up the central server.
5. DETAILED DESCRIPTION OF THE INVENTION
The details of the present invention are given in the following description which is presented as a means for one who is skilled in the art to create and use the invention for its intended purpose as described herein. Other embodiments of this invention may be apparent to those who are skilled in the art, and this document intends the patentable technology to apply to all such potential applications.
5.1 Definitions The following definitions should assist persons skilled in the art in understanding the nature of the present invention as detailed in the disclosure that follows.
"API" is an acronym for Application Programming Interface and in this context is a set of rules, syntaxes and methods that allow software developers to communicate via a service to perform a specific task.
"Binding" is the term describing the one to one relationship between a storage provider and a user. When the system is made aware that a certain storage provider belongs to a certain user, a binding has taken place.

8 "Central Server" is the term used to refer to a cloud server built using an operating system running various software components and services, as well as the logic and software implemented in order to allow the central server to perform the various actions needed by the described invention. The cloud server is available on the internet and can be accessed via standard communication protocols but preferably the well known HTTPS protocol.
"Client" generally refers to a PC or mobile communication device (e.g. a smartphone such as an iPhone) running an operating system and capable of running applications, that connects to the internet and interfaces with the user in a graphical format.
However the scope of this term can be expanded to include any software platform on a device that is capable of running applications, connecting to the internet and interfacing with the user in a graphical format. In this context of the present disclosure, the client is running a software application that is utilizing the service described.
"filter" is the criteria that will separate a subset of photos from the rest of photos by a common property such as date-taken, location-taken, source , etc.
"Grouping" is the term used to describe gathering a set of photos that share a common property. Examples would be grouping by date, which would group photos by the date they were taken, or grouping by location, which would group photos by the geographical location they were taken.

9 "index" refers to a collection of records containing the information about photos.
"local photos" refers to photos that reside on the client and that are stored on its memory. Typically these photos are generated by the client taking photos using its built-in camera.
"meta data" refers to the properties of photos. This includes a wide range of properties such as geolocation of the photo, width, height, time taken, originating camera, etc.
"REST" is an abbreviation for "Representational State Transfer"
and is a software architecture style consisting of guidelines and best practices for creating scalable web services.
"SDK" is an acronym for Software Development Kit and refers to a set of software components and libraries that allow software developers to integrate a feature or a service into a parent software application.
"Storage Provider" is a service, typically on the internet, that allows the user to store photos on the platform it provides and allows users to access the stored photos at any given time.
"throbber" is an animated icon reflecting the fact that the application is processing data in the background and the user needs to wait for the information to be ready.
"thumbnail" is a smaller representation of a photo.
"User" refers to a human operating a "client" and running applications that utilize the described facilities.

"User account" are the credentials by which a user is identified and authenticated to the system.
5.2 Exemplary embodiments ¨ overview The present invention is a system comprised of many independent software components, database structures, automation scripts, hardware components, manual processes and analytical procedures and control interfaces. The system imports and gathers photos and information about the photos from various storage providers as well as user account information and stores this information in a database. The system running on the client 104 makes queries to the database and presents the results to the user for consumption through general viewing on a client screen or in association with a keyboard interface. Fig.1 ("Master Flow Diagram") illustrates this high level process and the following provides a brief explanation and high level overview of the process. All the remaining figures in this document will be explained in greater detail in sections 5.2.1 (Exemplary methods) and 5.2.2 (Exemplary data structures).
The first step in realizing this invention is the setup of the Central Server 100 and the required Data structures as illustrated in Fig. 2 ("Central Server and Data Setup"). The database is setup by creating the required tables with pre-defined schema on a database server. The remaining components are deployed by automated scripts in the form of standard software packages and installed on either the same server computer as the database server or on another instance meant to execute the code of the cloud server engine. Some parameters such as memory capacity and CPU power of the deployed server could be altered from the defaults by a person skilled in the art if needed. There is no initial data needed to be imported and all the data gets populated as users create accounts and add details about the storage providers they utilize (e.g. Google Drive and Facebook) through the interface of the client 104.
After the system has been initialized it is ready to receive data from the client 104. _n an exemplary embodiment of the present invention the user will create an account by choosing an email address and a password and the client 104 will transfer this information to the central server 100. If there is no duplicate found and if the criteria passes sanity tests a record for the user will be created with the given information and thereby the identity of the user is created in the system as a unique "user account". The procedure of creating a user account, and in further usage, the procedure of logging in to a user account is handled by the authentication module 201 as shown in Fig. 2, and the information is stored in the user table 202.
The next step in the present invention is the process of "binding" through which the user is taken to the user interface of a 3rd party storage provider (e.g. Google Drive or Facebook) via a series of information exchanges in order to authorize the central server 100 to have access to the storage provider and as a result to have access to the photos stored on the storage provider on behalf of the user. In Fig. 2, the 3rd party storage provider authentication module 204 is the component responsible for this procedure and the information is stored in the storage provider table 206.After the process of binding takes place an automation script is executed on the indexer 207 through which the entire data set residing on the storage provider is processed and relevant information is extracted and saved in the database. Fig. 3 illustrates this process. The binding and data import procedures may be repeated for several storage providers.

Figs 4 to 9 illustrate all the details of the processes and the data structures on the client 104 side that come together to retrieve the indexed data on the central server 100, and that sort, group and present the data to the user and as a result provide access to the entire set of photos the user has access to on various platforms.
Fig 10 illustrates how the client side of the invention may take the form of a keyboard extension in order to provide the intended service with zero integration effort inside any software application that utilizes the keyboard interface.
Following is a more detailed explanation of all figures and processes of an exemplary method of the present invention.
5.2.1 Exemplary methods The first step in the process of the present invention is the configuration and initialization of the central server database 103. The database is the core of many operations and several processes will directly or indirectly interact with it. For an exemplary embodiment of the present invention, this database is a MySQL 5.6.1 database which is an open source rational database. The choice of database is not exclusive to the implementation of the present invention and many other alternate embodiments are possible.
The initialization of the database involves importing a pre-defined schema which creates three main tables:
1) User Table 202: This table contains the information related to each individual user and contains the following fields:

Email: email address of a user;
Password: password of the user in crypt format; and Id: A unique identifier assigned to the user.
2) Storage Provider Table 206: This table contains the information related to each storage provider a user is bound to and contains the following fields:
Id: A unique identifier of the record;
Account id: The id of the user this storage provider belongs to;
Product id: Determines the type of storage provider;
Friendly_name: A name that the user will optionally give to a storage provider to identify it more easily.
Coordinates: A field containing all the necessary information to connect to and authenticate with the storage provider;
Permanent status: A field used to mark a storage provider as falling into a state where it cannot be indexed anymore for example invalid coordinates.
Last index end timestamp: A date/time field determining the _ _ last time the contents of the storage provider were brought up to date in the 'files' table;
Last indexing state: A field that determines what the last known state of the indexing is;
Current indexing start timestamp: A date/time field determining when the start time of the current active indexing was; and Last index change timestamp: A timestamp field that records the last time a change was made to the index of files.

3) Files Table 205: This table holds information about each individual photo as well as folders containing photos. It contains the following fields:
Id: Unique identifier of the photo;
Account id: The id of the user the photo belongs to;
Parent id: The id of the folder the photo resides in;
Storage_provider id: The id of the storage provider this photo resides on;
Sp_id: A unique identifier by which the photo can be uniquely referenced on the storage provider;
Parent_sp_id: A unique identifier by which the containing folder of the photo can be referenced on the storage provider;
Name: The name given to a photo - usually the filename;
Created time: The time the file containing the photo was created;
Description: Any extra description provided for the photo;
Nb images: In case of folders, the number of photos residing in the folder;
Size: The size (in bytes) of the photo;
Mime type: The mime-type of the photo, usually determining what format it is in;
Sp_source: A URL or information a URL can be constructed from that would point to the original version of the photo;
Sp_thumbnail: A URL or information a URL can be constructed from that would point to the thumbnail of the photo;
City: The city in which the photo was taken;
State: The state in which the photo was taken;
Country: The country in which the photo was taken;
Time taken: The time the photo was taken;
Height: The height of the photo in pixels; and Width: The width of the photo in pixels.

Once the database is created the next step involves deploying the engine that is responsible for receiving all requests from the client 104, processing all actions and carrying out all procedures on the cloud server. For an exemplary embodiment of the present invention, this engine is implemented in PHP 5.6.
The parts that honor reque-ts from the client are executed via the php library module of the Apache open-source web server and the parts that run in a standalone manner are executed by the PHP command line interface. The choice of the interpreter is not exclusive to the implementation of the present invention and many other alternate embodiments are possible.
The actual PHP code is deployed via standard software packaging mechanisms. In the exemplary embodiment of this invention this packaging mechanism is based on RPMs (RedHat Package Management).
Once the engine is deployed and the web server is ready to receive requests from the client 104, the entry point into the system would be creating a user account in the user table which is done via the signup process presented by the client to the user. This process is a typical signup process utilized in many services offered on the internet and follows the same basic rules of uniqueness and security. This information is sanitized by the client to be in accordance with the rules of account creation and sent to the central server's authentication module 201 in which after validation the account is created and the information is stored in the user table 202.
The next step in utilizing the service is performing a binding to a storage provider. At the start of the binding process, the user chooses from a list of supported storage providers offered by the client. The choice is communicated with the central server's Single Signon module 203 and the response from the central server comes as a URL pointing to the relevant storage provider's 3rd party storage provider authentication module 204 .
The URL contains all the necessary information about the request for binding which the storage provider will use to process the authorization. The 3rd party storage provider authentication module 204 asks the user for the credentials and upon authentication, asks for cunfirmation from the user that the central server 100 is authorized to access the user's data. Once the user confirms the authorization, an access token is provided which the client 104 will then send to the central server 100.
The central server 100 upon receiving the access token, creates a record in the storage provider table 206 and stores the access token as well as any other necessary information that describes the bound storage provider in that record. One of the fields of the record is the account id of the user. Therefore at this stage there is a "binding" between a storage provider and a user.
The initial state of the storage provider record will indicate that it is fresh and has not been processed due to last index end timestamp and last indexing state fields being _ _ null. This brings in the indexer script which will now start the process of indexing the storage provider.
Fig. 3 illustrates the indexing procedure which is the process of getting access to the storage provider and reading in all the information about photos stored on that service. The result is stored in as many records as needed in the files table 205. The process involves the main indexer 207 going through all rows of the storage provider table 206 and processing them accordingly.
The record is fetched (step 301) and based on the type of storage provider it relates to (based on the product_id field) an indexer object, a derivative of the base indexer class, capable of processing the target storage provider is instantiated (step 302) and takes over the process. The indexer object uses the coordinates field, which includes all the information required to identify and communicate with the storage provider, to authenticate to the storage provider. This typically includes the username identifying the user on the storage provider as well ab an access token obtained through the binding process. The indexer then makes the necessary API calls to the 31d party storage provider authentication module 204 and logs in (step 303). At this point the indexer is ready to import data from the 3rd party storage provider source 208. Two scenarios are possible at this point. Either the storage provider has been processed before and therefore has entries in the files table already or it is dealing with the first indexing (step 304). In the case of first indexing, the entire data set on the storage provider needs to be processed (step 305).
Depending on how the 3' party storage provider has designed their API, this process involves either traversing a hierarchy and retrieving information level by level or processing a flat list using paged results. Section 5.3.1 contains sample code for this process for a specific storage provider. Regardless of the method, as the indexer proceeds through the hierarchy or pages of data, it will obtain information about individual photos and their properties. The information about each individual photo is then inserted in the files table 205 via the module in step 306.
Depending on the storage provider as the information about the photos are obtained, some meta-data is also provided. From this meta-data the fields that are relevant to the system are inserted into the record of the photo in the database via module 307. Not all storage providers provide all the meta-data that the system needs. In order to extract this portion of information, the meta-data embedded in the header of the photo needs to be parsed. When the condition is true for the photo in question (step 308) the meta-data extractor module 209 is utilized. This module performs a partial download from the beginning of the photo source (step 309), enough to obtain a header that includes all the necessary information. This information is then parsed and inserted into the record of the photo in the database (step 310).
Once the first indexing is complete, the storage provider record in the database should be updated to reflect this new status.
This involves updating the last_indexing_end_timestamp and last indexing state to the time the indexing completed and to the cursor pointing to the last file that was indexed (or to a cursor provided by the storage provider, depending on the API), respectively (step 311).
The second scenario is the case where the indexer is dealing with a storage provider that already has been indexed. The process in this case is to discover the changes that happened during the last index and apply them to the relevant records. To do so, the cursor or timestamp (depending on the 3rd party SDK) from the last indexing state is read from the storage provider record (step 312). Then this cursor or timestamp is passed on to the storage provider's API as an argument and a request is made to retrieve the list of all changes that happened since that cursor/timestamp was generated. The result is a list of photos and their changes that go through the same procedure from point A in Fig. 3.

By covering these two main scenarios and having the indexer 207 perform the procedure illustrated in Fig.3 in a loop, the end result is a files table containing an up-to-date representation of the entire data set (all photos) of all users which is one of the key aspects of the present invention.
At this stage this disclosure proceeds with the explanation of the mechanisms involved on the client 104 side that will utilize the information from the central server 100. For the exemplary embodiment of this present invention, all software modules on the client are implemented in Objective-C and compiled for Apple's iOS devices. However the choice of the programming language is not exclusive to the implementation of the present invention and many other alternate embodiments are possible.
Fig. 4 ("Main Client Flow") illustrates a simple but crucial principal of how the client handles data and reflects it on the display. Since the presentation of the facility may be done as a collage of groupings of photos or through the keyboard interface, for instance, it is crucial for the display to be fast and responsive. This flow makes sure that the display is made available as soon as possible to the user to interact with and all updates and data processing are handled in the background and only made available to the display when they can be presented with no delay. As the main flow on the client starts a first time throbber is displayed 402 to allow for the initialization of the engine that is inevitable. While the throbber displays the application waits for the data that can be made available as fast as nossible which is the group list of local photos 403. This involves steps 901, 902, 801, 802 and 803 to be completed and is the minimal possible processing that is needed to get the first display available to the user. Once the grouping of the local photos is displayed 404 the application goes into a loop in which it waits for new data to become available in the background but allows the user to interact with the available data.
Fig. 6 ("Interaction of data models on the client") illustrates how different components, each responsible for processing different data models and the updates required for each interact with each other and eventually with the display. The Item List Data Model 604 is the component that represents the entire list of photos (local and remote) and their properties. It is responsible for initializing this list and keeping it up to date. Module 605 named the CS Poller is the component that periodically checks for updates to remote photos on the central server 100. When it is informed that a change is available, it informs the item list data model 604 which takes care of retrieving the change. Module 606 is the module monitoring changes to the local photos. This typically is a module readily available on the operating system of the client. When it is informed that a change is available to local photos, it informs the item list data model 604 which takes care of retrieving the change.
Whenever the item list data model 604 processes new information from either local or remote, it informs the group list data model 602. This procedure is illustrated in detail in Fig. 8 and will be explained in detail. The group list data model 602 is responsible for grouping photos and maintaining a list of groups based on the criteria chosen by the user. Upon receiving updates from the item list data mo,iel 604, it updates its groupings and informs the display 601. The details of the group list data model 602 are explained in Fig. 8 and will be explained in detail. The item collection data model 603, is the module representing a collection of photos grouped by a specific criteria chosen by the user through interaction with the interface. It receives updates from the item list data model 604 and informs the display 601 when it is ready to provide the data it is representing. The details of this module are illustrated in Fig. 7 and will be explained in detail.
Fig. 7 illustrates the mechanism of the item collection data model which is the data model that represents a list of photos grouped by a specific criteria such as date, location, source, etc. It is the data model that will be used by the display when the user has narrowed down their search criteria using the group views and is at this point viewing the actual photos of the group they have chosen. It is initiated by assuming a default filter (if none is given) and immediately enters an idle state waiting for data to become available by the item list data model 604 or for the user to change the grouping criteria. The events that trigger an action in this module are:
= Local list change = Remote list change = User filter change When a "local list change" event is received (including the availability of the list of local photos for the first time), this module updates its list with the local photos that have been affected by the change and are part of the collection (step 702). When a "remote list change" event is received (including the availability of the list of remote photos for the first time), this module updates its list with the remote photos that have been affected by the change and are part of the collection (step 703). When a "user filter change" event is received, this module recollects its list based on the new filter that has been chosen by the user (step 704). After every change that alters the data the item collection data model 603 represents, it sends a notification to the display informing it that new data is available for display (step 705).
Fig. 8 illustrates the mechanism of the Group list data model 602 which is the data model that represents a list of groups which the available data set can be grouped into based on a specific filter the user has selected. It is the data model that is used when the display is showing all available groups in which the photos can be categorized into given a selected filter. Once initiated it immediately goes into an idle state waiting for a list of local items to become available (step 801). Once the list of local photos (and their properties are ready (see 902) it initially creates a group list just comprising of local photos (step 802). As soon as this information is available it sends a notification to the display informing it that a group list is available to be displayed (step 803; also see 402). It then checks for the availability of a previously cached grouping of remote photos (step 804). If a cached group is available it will load it into memory (step 805) and merge it into a combined group list which now includes a grouping of both local and remote photos (step 806). A
notification is then sent to the display informing it that new information is available (step 807) and it goes into an idle mode and waits for events (step 808). If a remote group list cache is not available in the check point at step 804, it goes into idle mode and waits for events. In idle mode (step 808) there are three events that will trigger an action:
= Local list change = Remote list change = User filter change When a "local list change" event is received, this module creates a new local group list (step 809) and then updates the combined group list (step 810) and sends a message to the display informing it of new data being available (step 807) and then goes back to idle mode (step 808). When a "remote list change" event is received, this module creates a new remote group list (step 811), saves the new remote group list in cache for future use (step 812) and then merges the remote group list into a combined group list (step 806) and sends a notification to the display informing it that new information is available (step 807) and then goes back to idle mode (step 808). When a "user filter change" event is received, this module creates a new local group list (step 813), adds the local group list into a new combined group list (step 814), creates a new remote group list (step 815), saves the remote group list into cache (step 812) and then merges the remote group list into a combined group list (step 806) and sends a notification to the display informing it that new information is available (step 807) and then goes back to idle mode (step 808).
Fig. 9 illustrates the mechanism of the item list data model 604 which is the data model that represents the complete list of photos and their properties. This data model is not used directly in the display but feeds into the group list data model 602 and item collection data model 603 (see Fig. 6). This module initiates its flow by retrieving the list of local photos (step 901). This step utilizes the operating system's API in order to get this list. Once the list is available it sends a notification to the group list data model 602 in step 902 to allow that module to begin its action of grouping. In the next step, it checks to see if a cache of the remote list of photos is available (step 903). If there is, it loads the cache into memory for the next step but if the cache is not available, it sends a request to the query handler 210 on the central server in step 905 to retrieve the list of remote photos. Section 5.3.2 contains sample code for this call to the central server. Once it receives the list, it saves the timestamp of the response and also saves the list into cache (step 906). Once the list of remote photos is available, it sends a notification to the item collection data model 603 and the group list data model 602 to inform them that the data is available for processing (step 907). After this step the module goes into idle mode and waits for events (step 909). The events that can trigger an action in this state are:
= Local item change = Remote item change When a "local item change" event is received, the module retrieves the new list of local items (step 908) and sends a notification to the item collection data model 603 and the group list data model 602 to inform them that the data is available for processing (step 907). It then goes back to idle mode (step 909). When a "remote item change" event is received, the module sends a request to the query handler 210 on the central server in step 905 to retrieve the list of remote photos. Once it receives the list, it saves the timestamp of the response and also saves the list into cache (step 906). Once the list of remote photos is available, it sends a notification to the item collection data model 603 and the group list data model 602 to inform them that the data is available for processing (step 907). It then goes back to idle mode (step 909).

Fig. 10 illustrates one way in which the user may use the system and how the end result is activated and utilized on the client.
The host application, which is an arbitrary application on the client that uses photos in some way and uses the keyboard interface as well, activates the keyboard interface when it is ready to receive text and image input from the user (step 1001).
The operating system is the entity that allows the user to choose the keyboard of their choice and in the case of this present invention, they will choose the custom keyboard utilizing the mechanism. Once the custom keyboard launches it reads in the data available from the described data models (step 1002) and displays the high level grouping of photos to the user (step 1003). This is the starting point for the user in narrowing down their choice. In an exemplary embodiment of the present invention, this grouping is:
= By timeline : Starting with the years = By location: Starting with Continents = By Sources: Starting with the list of storage providers.
At this level the user chooses one of these groups and arrives at any of the following sub groups (step 1004):
= By timeline: By months of year chosen = By location: By countries in the continent chosen = By sources: By folders of the chosen source If the user chooses to go back they will arrive at the high level grouping again (step 1003). If the user chooses one of the sub groups they will arrive at the following sub groups:
= By timeline: Days of the month chosen = By location: Cities of the country chosen = By sources: The sub folders of the folder chosen and the list of photos available in the chosen folder.
From this point onwards the user has the ability to select a particular photo. When they do so, the photo is copied to the clipboard of the operating system (step 1005) and they can then select the input field of the host application and paste the copied image to that input field and by this means transfer the photo to the host application to be used.
With the whole system at work as described above and with the user utilizing the system in the way described above, it can be seen that the task of having a single entry point to all the photos the user has access to from a common interface allows the keyboard to easily function with adding photos from any source to messages.
5.2.2 Exemplary data structures Both the Central Server 100 and the client 104 have their own data structures they use to handle the photo data. Each will be explained here.
Fig. 5 shows the high level data structures on the client used by the data models described in the previous section and in Fig.
7 to Fig.9. The data structures are divided into two main categories: Groups and Items.
The "Group" categories represent data structures containing the information about the extracted groups of photos. When a list of photos is processed, depending on the active filter, a list of groups is generated that those photos are divided into. For example, with the timeline filter, the list of photos are grouped into photos that fall into a certain date. The individual list of dates that photos fall into make up the group list. There are two group lists generated: the remote group list 501 and the local group list 502. The remote group list 501 is made up of a list of group entries related to remote items. Each group entry contains a value reflecting the group's value such as a specific date and a pointer to a cover photo. The cover photo is a thumbnail of one of the members of the group and is chosen randomly. Each remote group list also has a "timestamp"
field determining the time it was generated and a "type" which states the filter used to generate the group. A remote group list is saved to the disk cache when it is generated for faster retrieval at subsequent launches (See Fig. 8). The local group list 502 is the same as the remote group list 501 except it is derived from local photos and does not have the timestamp field.
The reason a timestamp field is not needed for the local group list 502 is that it does not get cached and every time it is derived from the list of photos refreshed from the current list of local photos on the device. The combined group list 503 is a merged and sorted collection of the local and remote groups list. Therefore it is a list of pointers to individual group entries within the local and remote groups list. It holds one additional "type" field which determines the filter in effect for the current grouping.
The "Items" category represents data structures containing the information about the individual photos. There are two "items"
data structures: the local items 504 and the remote items 505.
The local items 504 is an array of HPItem objects, which each HPItem objects containing information about one individual local photo. The remote items 505 is an array of HPItem objects, with each HPItem object containing information about one individual remote photo. The remote iLems data structure also has a "timestamp" field reflecting the timestamp of the response from the query handler 210 that included the remote items. The remote items 505 always gets saved to the disk cache for faster retrieval on subsequent launches (See Fig. 9). The item collection data structure 506 is a sorted array of pointers to HPItems grouped together based on a specific property. This data structure is used when the user is presented with the photos belonging to a group which has been narrowed down by the user.
For example, if the user has narrowed down their choice of photos to photos from May 25 2015, this data structure will be an array of all HPItem pointers for those HPItems that were taken on May 25 2015.
Fig. 11 displays a diagram of the central server's database 103 table structure, and represents an exemplary database layout for the present invention, showing the layout of user table 202, files table 205 and storage provider table 206. This database serves as the storage location for all user accounts, the storage providers those users are bound to, and the list of photos extracted from all those storage providers along with all the meta-data of the photos. Any field in the tables with the "Id" suffix is a table key that is used to identify a record in a table and may be used to cross-reference records in other tables. A "one to many relationship" means that the two connected tables are joined such that a single record in the first table corresponds to 1 or more records in the second table. The "primary key" identifying some of the fields are the keys that guarantee the uniqueness of records and are used to index the tables for faster queries. The database tables, fields and field formats represented in Fig. 11 are neither meant to be an exclusive nor exhaustive implementation of the present invention and other database tables, fields and field formats potentially exist which are compatible with the present invention.
The User table 202 stores the information about the user. It has a one-to-many relationship with the storage provider table 206 meaning one user can have multiple storage providers bound to them and it has a one-t-)-many relationship with the files table 205 meaning the user may be the owner of many files recorded in the system. The "user id" is a unique number given to each user to identify them in the system. The "email" field is an email address used as the username which the user will provide as a key to identify themselves to the system. This information accompanies a "password" at the time of submission which will be compared to a "password crypt" field in the "user table" to authenticate the user. Once the user is authenticated all the information they provide are marked with their "user_id"
as a key connecting that information to the user.
The Storage provider table 206 stores the information about each storage provider that the user binds to their account. It has a one-to-many relationship with the files table 205 meaning many files recorded in the system may belong to one storage provider.
Each storage provider record in the storage provider table holds enough information to bind that storage provider to one user and enough information for the central server 100 to authenticate against the storage provider and access photos that belong to the user on that storage provider plus the detailed information about those photos. The "storage_provider_id" is a unique numerical identifier that identifies the storage provider in the system. The "account_id" field is the unique identifier of the user that this storage provider belongs to. It points to a user record in the user table. "product id" is an id that defines the type of the storage provider. Each of the storage providers supported by the central server 100, for example Google Drive, Dropbox, Flickr, etc, have a unique id. "friendly name" is a name that the user will optionally give to a storage provider to identify it more easily. This field is particularly useful when a user has multiple accounts on the same storage provider since they can use this field to assign more recognizable names to those storage providers. "coordinates" is a line-feed separated set of values which are used to authenticate against a storage provider. This field could be a combination of id, username, password, access tokens, etc. When a user goes through the binding process all necessary information is recorded and stored in this field to enable the central server 100 to find the storage provider and authenticate to it. "permanent_status" is a field used to mark a storage provider as being in a permanently unrecoverable condition, until the user takes action. This field is populated in cases where the coordinates have been invalidated in a way that can't be used by the central server 100 to authenticate and extract information. It is a way to relieve the indexer from trying to authenticate when the last attempt had already failed and no change was made to coordinates suggesting that the coordinates would still be invalid.
"last index end timestamp" is a timestamp field that is set when _ _ the indexing process finishes. It states the time when the indexing last finished and is used to make decisions on when a next indexing would be required. "last indexing state" is a field that determines what the status of the last indexing is or was and for certain storage providers can hold information on where an indexing needs to resume from. In other words it can act as a cursor into the indexing of certain storage providers.
"current indexing start timestamp" is a timestamp field that =
would be set if the storage provider is currently being indexed.
It will be set to the date/time the indexing started. It can be used to determine how long the process of indexing has taken.
"last index change timestamp" is a timestamp field that records the last time a change was made to the index of files (in the files table 205) that belong to this storage provider.
The files table 205 is the table that holds the records of all indexed files for all storage providers and all users. Each "file" record has an "id" that is a unique identifier of the file record in the system. All references to the file are done via this id. "account id" is the id of the user to which the file belongs. "storage provider id" is the id of the storage provider from the storage provider table 206 from which the file was indexed. In some storage providers there is a hierarchy and files can be grouped under folders and subfolders. For these storage providers, in order to maintain the hierarchy, the id of the folder to which the files belong is stored in a field named "parent_id". Naturally if there is no hierarchy or if the file is at the top of the hierarchy (i.e. is at the root level), the parent_id will be cleared (set to null ). Each file record refers to a file on the storage provider. This reference is stored in the "sp id" field which when provided to the storage provider via its API will uniquely identify the file.
"parent_sp id" is the reference to the parent of the file on the storage provider. "name" is the name given to the file. Usually this is the same as the filename, created time" is the creation timestamp of the file. "nb_images" holds the number of files inside folders and is relevant for records that represent folders in storage providers that have a hierarchy. "size" is the size in bytes of the file. "mime-type" is the mime type of the file determining its format. "sp_source" holds the URL that can be used to fetch the file from the storage provider or it holds enough information to construct the URL that can be used to fetch the file from the storage provider. "sp thumbnail"
holds the URL that can be used to fetch the thumbnail of the file from the storage provider or it holds enough information to construct the URL that can be used to fetch the thumbnail of the file from the storage provider. "city", "state" and "country"
are fields that determine the location the photo was taken.
"time taken" is the date/time the photo was taken. "height" and "width" are the height and width in pixels of the photo respectively.
5.2.3 Exemplary Apparatus Examples of well known computer systems that may be suitable for use in the present system with aspects of the subject matter described herein include, but are not limited to, personal computers, server computers, laptop devices, smart phones, mobile devices, multiprocessor systems, network personal computers, mainframe computers, distributed computing environments where tasks are performed by remote processing linked through a communications network, remote systems, etc., and the like. In a distributed computing environment, program components or modules may be located in both local and remote computer storage media including memory storage devices. Remote systems may involve remote computers, such as personal computers, a router, a server, a network personal computer, a peer device or other common network node, and the like.
In addition, components of suitable platforms may include, but are not limited to, a central processing unit, a system memory, and a system bus that integrates various system components, including the system memory to the processing unit. The system memory may include computer storage media in the form of volatile or non-volatile memory, such as read-only memory ( ROM
), random access memory (RAM) , EEPROM, flash memory, CDROM, DVD, magnetic storage devices, etc., which can be used to store the desired information and is accessible by the computer system. The system bus may be any of several types of bus structures including a memory controller, a peripheral bus, and a local bus of various bus architectures. Suitable platforms also include a variety of computer-readable media comprising, for example, computer communication media and computer storage media. Computer storage media may include both removable and non-removable media, or volatile or non-volatile media alike, for information storage of computer-readable instructions, program modules, data structures or other data, etc. Computer communication media generally comprises any information delivery media for delivering media information to a user, computer-readable instructions, program modules, data structures or other data, etc.
For an exemplary embodiment of the present invention, Fig. 13 is a diagram that illustrates the key hardware components used for the central server 100 architecture. The Query Handler 210, the Indexer 207 and the Database 103 are systems that exist within a cloud computing environment, meaning that their physical components are potentially spatially distributed across many physical machines. Each of these systems consists of a CPU, Memory and Storage, in addition to all other required components for the typical operation of such computing system (as detailed above).
In an exemplary embodiment of the present invention, the query handler 210, indexer 207 and Database server 103 are connected together via a Fiber Channel over Ethernet (FCoE) connection and all are similarly connected to an Internet Firewall 1304 which provides security from unauthorized external access. The firewall is connected to the Internet 1305 and the entire system is accessible via the Internet from the client.
5.2.4 Alternatives and Extensions The present invention is designed to provide access to all photos from various sources through a display that can, for instance, show various groupings of photos for viewing by a user, or, for instance, through a universally adapted interface, the keyboard. However, photos are just a subset of media documents that users consume. Audio and Video files are also good candidates to be considered in a similar implementation of the present invention but with the outlet being suitable for consuming audio and/or video.
Another potential alternative to the present invention is to consider photos or media that are public and that are not necessarily owned by a user or protected by their credentials.
In the present invention as generally described all the photos have the common attribute of being owned by the same user. In an alternative implementation the common attribute of photos grouped together might be the topic of the photos or hashtags assigned to photos from publicly available sources, thereby providing access to all photos with the same subject from various sources and through, for instance, a universally adapted interface, the keyboard.

5.2.5 Example implementation In the following section we will describe an example implementation of the present invention using Apple's iOS
platform as the deployment target and will demonstrate how the present invention has significantly made it easier and more convenient to consume photos from various sources and will compare it to the procedure required in the absence of this mechanism.
In this example an iOS application has been written to implement the client side of the system (see Fig. 4 to Fig. 9) in the form of an iOS keyboard extension. With the launch of the application, the user first creates an account and logs in via the signup/signin screen.
Gani, 2.00 AM
LOGIN CREATE ACCOUN 1 kwiet =
-. 1111 PIAMO STREAMS
::111111111111111111111, PREFRENM
=IN 111111111111111117 03 fit-COUNT
ANL ci IARE Tot LOVE
ForgW your poortl' Ire tin P
411.
CREATE ACCOUNT m I GOUT >
KCIP:
LOG IN
The user record gets created in the user table 202 and the user logs in via the authentication module 201.
In the next step the user binds the application to all the accounts they have on variJus storage providers.

cer,T,v ? od o1Met 11)04^ ".= WO C.414'; 1'7* 21>
11):"V
< A01)11.1010 STREAMS -47 õ '`" "P941' e =
:4' ***,S*
,43,4';µ,44:11111 f FACCBCKIK 1'1 eir "';1**=,1*., Ww"mv r ipHoNgtiva - co FL/CKR et II01 INSTAGRA4 is le T T
ika,X 11 g+ GOOC,L, 0 U.

s6.06L4 DRIVE =pink INSTAGRAM 0 t... g 416 ONE ()RIVE

This goes through the 3' party storage provider authentication module 204 and authorizes the application and the central server 100 to access all the photos the user has on those platforms. In the background on the cloud, the indexing begins for each storage provider, which procedure which has been described in Fig. 3. and the database gets populated.
At this point the user can now start utilizing the system and benefiting from it. In this example the user launches the email application in order to send photos to a friend. The photos are in different locations: 1. Some photos are on the iOS device; 2.
Some photos have been posted to facebook; and 3. Some photos have been synced to dropbox. All the user needs to do when they start typing the body of the email is to select the keyboard that implements the system. The keyboard launches and performs all the procedures shown in Fig. 4 to Fig. 9 and eventually from the presentation layer takes the user through the procedure shown in Fig. 10. The user is presented with a high level grouping of all their photos from various sources in this example starting by year. They select 2014, then the month, for example June, then the day, for example 12th.
IIIIII!!!!!!!!!111111.1;i!!1:11iIIN
mraulay-i-ti Birthday Ptottimili..1.3õ, = n=-.
=
' .
. , 1 õ.
: : =
. .
. .
. , JO,- AWe; = = .1t4rair .sMf7 =1011, Al -.= :. = = = eThaV; :;<.rn 1 =
= = = : === = = " i' =COM L .
Birthday Pictures Birthday Pictures 1 ' =
Birthday Pictures , , = .
. , !
=
=
11.4117a . kb r: .,::: TIMEUNE - At : <
17**JUN!..i12 9 w .. õ. . . ..,.
.. ..
, , . .. ..
71',,,,...."1.:',.. = = tF," -..,,A:i = 46 10, ' iz,',.., --..eigia . T 1 ',.-w ' - :.:
,- 11 .õ
&.
1111111',,; 4', al =i;7'',',..- . - .4,1- = õ rip., =
.
The user is presented with all photos from their local device, Facebook, dropbox, etc that have been taken on June 12th 2014 in one place. They can now select the desired photo(s) with one tap to copy and paste them in the body of the email with another tap.
, Birthday Pictures . ' _ =,.>
kolth.y, * :4-- . *3. - - 'A =
ilift= = >, =.,--õa= 00..-- ' = =
trcli .= .= 2014 JUNE 12-: . -11, # tiolw.IA ' 1115411111 Yk. d'I'' , t e To compare the simplicity f the above procedure with what would -have been needed in the absence of this keyboard interface the steps below are listed for the exact same two pictures (one from Facebook and one from Dropbox). The user needs to leave the mail application, launch their Facebook application, browse their albums, find the photo, save it locally to the device, go back to the mail application, tap and hold on the body of the email, select "Insert photo", be presented with the entire camera roll, find the picture saved from Facebook, and select it to be added to the email. For the next picture which is on Dropbox, the user needs to leave the mail application again, launch their Dropbox application, browse to their albums, find the photo, save it locally to the device, go back to the mail application, tap and hold on the body of the email, select "Insert photo", be presented with the entire camera roll, find the picture saved from Dropbox and select it to be added to the email. The tediousness of this procedure is obvious compared to the procedure offered by the example keyboard extension.
To demonstrate the power of grouping photos by other types of meta-data which is readily available due to the indexing that has taken place on the central server 100, the user now composes another email to send pictures from a certain location. Again they launch the keyboard and select the "location" filter. They are presented by a high level grouping of photos by continent.
This is using the data from photos that come from various sources without the user needing to know what the source is. The user selects "Europe", then "Germany" and then "Nordlingen".
They are presented with four photos each coming from a different source. Selecting all four is a matter of copying and pasting from the same interface without leaving the mail application.
, . .Z=,,, mPkture s ftworfttOW -'-:: MEM Ra I. ,, ,, , ¨.sd I Place Pictures is, Place4 Picumest Places! Ptctures iF LOCATION . < LOCATION If < GERMANY, NORDLINGEN =_ .
, . .,.
4/3$

Reese Pictures = . - =
VI!
:YiNnt -:3!44 _ ;
:=
=
#40 Vj' !;,,,fiL. =
= ' ' &VP"-=;':t*r "ttg - Mr/.
:;$
.1111110 Another example of an implementation of the present invention is a photo browser and editor capable of grouping all photos from various sources based on a common criteria. In this example the account creation, authentiCation and binding of storage providers is the same as the previous example however the consumption takes the form of a full screen photo browser.
=

44=63-0t 14,, ' s:40 Am I V # alp I ==
2PHati S A, I 1 ,' ' õ. µ. =;:.'"' , 'Al. =

2020. OCTOBER 19 s ftsfros ,...
r'lb , ., ,,,, r - t 1111111111111115001MMINEMailla 'TIMELINE A 0 EMMCUM73 Photos from various sources grouped by date ...õ
,.,,,.. v ,.' , = , I
`'=' Ir"'t:301=': "'--:' ,,,--, = ' , -vorwvier.07 -v--,- :
.
,,,t ,,,, ..,,,,,,g.,...,,,,, , -r ,,,,,,,,,,.
9aU =<
=;,:7"'...-.7',..,40v ''"
PLACES A
Photos from various sources grouped by location MICE=11133:31111ECKE3 t=.1 i.
r4 1 111 i t irea.1 - I
-'te .,44tiiikoert IF
OP' C. A.. . AW. *N =
Photos from various sources that have the same location 5.3 Sample Code In this section samples of some of the code written to implement the present invention will be provided. Code sample #1 will be from the central server 100 and code sample #2 will be from the client. This code will allow the person skilled in the art to readily understand how to program the other key modules that make up the present invention, as previously described.
5.3.1 Code Sample #1 This sample code is from the central server 100 and has been chosen from the "indexer" module, whose functionality has been described in Fig. 3.

In particular, this snippet is the code from step 305 where the indexer makes an API call to the storage provider to fetch the index of photos. The object returned is a PHP object representing the fetched data. The results of the fetched data are then parsed and inserted into the database to form the index of the files on the storage provider, which in this sample is Flickr.
/**
*This function supports calling the Flickr API.
*@param array $query_params Defines which Flickr API function to call and provides the necessary values to do so.
* @throws HipPlayException in the event we received self::NUM_RANDOM_BACKOFF_ATTEMPTS + 1 errors from Flickr in a row.
* @return Array of json-decoded results from Flickr.
*/
private function callFlickr($query_params) //Note: Intentionally not checking the parameters here.
// This is a support function for other functions in this same class (private) // and we will assume the parameter is an array with at least the method, /1 format and nojasoncallback indexes defined.
// Get the signed URL
$base url = 'https://api.flickr.com/services/rest/';
$signe-d_url = $this->oauth->getSignedURL('GET', $base_url, $query_params);
if (!isset($this->ch)) $this->ch = curl_init();
curl_setopt($this->ch, CURLOPT_ENCODING, "gzip, deflate");
curl_setopt($this->ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($this->ch, CURLOPT_USERAGENT, 'Kwilt); // this field can't be empty curl_setopt($this->ch, CURLOPT_URL, $signed_ur1);
$nb_attempts = 0;
while (true) {
$nb_attempts++;
$exception = null;
// Make the request to Flickr $response_body = curl_exec($this->ch);
I/ Grab the status code $status_code = curl_getinfo($this->ch, CURLINFO_HTTP_CODE);
// Parse the body $flickrData = json_decode(Sresponse_body, true);
// Check for errors if ($status_code !== 200) {

$exception = new HipPlayException("Flickr error with HTTP status '" .
$status_code . ", here is body:
$response_body, HipPlayException::FATAL_ERROR);
1 else if(is null($flickrData) II lis_array($flickrData)){
$excepon = new HipPlayException('Flickr did not return a valid json array.
Body:
print_r($response_body, true), HipPlayException::FATAL_ERROR);
}else if (!array_key_exists('stat', $flickrData) II $flickrDatarstatl !==
'ok') // See https://www.flickr.com/services/api/flickr.photos.recentlyUpdated.html for error codes if (array_key_exists(code', $flickrData) && ($flickrData[code] == 98 $flickrDatarcode'l == 99)) {
II need to re-authenticate $exception = new HipPlayException('Need to re-authenticate with Flickr. Body:
print_r($response_body, true), HipPlayException::0AUTH_ACCESS_DENIED);
}else {
$exception = new HipPlayException('Flickr results status not "ok". Body: ' .
print_r($response_body, true), HipPlayException::FATAL_ERROR);

// If there was an error, backoff and try again or throw the error if(isset($exception)){
= 20 if($nb attempts > self: :NUM
curF close($this->ch);
throw $exception;

$this->exponentialRandomSleep($nb_attempts); // random backoff before trying again } else {
break;

}// end while(true) // Done!
return $flickrData;

5.3.2 Code Sample #2 In this code sample, the call that is sent from the client to the query handler 210 is shown. This is the call that happens in step 905 to fetch the list of remote photos from the central server. It allows for passing of a few arguments to fine tune the request and, for example, to exclude certain storage providers which the user may have turned off. The response in this example is in XML so the XML parsing takes place to extract all the attributes of the fetched data set. As shown in the code, the attributes have a close correlation with the fields of the files table 205. When all results have been parsed and processed and internal objects have been created, these attributes can be used to group the dataset using different filters.

- (NSArray igetItemsWithExcludelDs:(NSArray lexcludelDs limit:(NSUInteger)limit offset:(NSUInteger)offset force:(BOOL)force lastModifiedDate:(inout NSString * autoreleasing *)lastModifiedDate useCached: (out BOOL *)useCached error:(NSError * autoreleasing *)error NSMutableString *queryString = [NSMutableString stringWithString:@"type=image&fields=id,sorting_time,thumbnail url,storage_provider_id,latitude,longitudel;
if ([HPNetworkHelpers getNetworkConnection] == ReachableViaWiFi) NSString *ipAddress = [CustomRoute localIPAddress];
NSString *netmask = [CustomRoute localNetmask];
[queryString appendFormat;@"&localip=%@&netmask=%@", ipAddress, netmask];

[queryString appendFormat:@"&limit=%lu", (unsigned long) limit];
[queryString appendFormat:@"&offset=%lu", (unsigned long) offset];
NSString *requestUrl = [NSString stringWithFormat:@"%@%@?%@", HIPPLAYDOMAIN, API_REST_INDEX, queryString];
NSString * lastModifiedDate = force ? nil: (lastModifiedDate ?
*lastModifiedDate : nil);
xmlDoc *document = NULL;
NSString *outLastModifiedDate = nil;
NSError *outError = nil;
BOOL returnNil = NO;
@autoreleasepool NSURLResponse *response = nil;
NSData *dataQueryResult = [[HPKwiltServer sharedlnstance]
getSynchronousRequest:requestUrl lastModifiedDate: lastModifiedDate response0u1:8Tresponse errorOut&outError];
NSInteger statusCode = 0;
if ([response isKind0fClass:INSHTTPURLRe5p0nse class)]) {
NSHTTPURLResponse*httpResponse = (NSHTTPURLResponse *) response;
statusCode = httpResponse.statusCode;
if (statusCode != HTTP CODE NOT MODIFIED) {
NSDictionary *allHeaderFielcTs = [httpResponse allHeaderFields];
outLastModifiedDate = [allHeaderFields objectForKey:@"Last-Modified"];

BOOL outUseCached = (statusCode == HTTP_CODE_NOT_MODIFIED);
if (useCached) *useCached = outUseCached;
if (outUseCached II !dataQueryResult) returnNil = YES;
else document = xmlReadMemory((const char *) [dataQueryResult bytes], (int) [dataQueryResult length], NULL, NULL, (XML_PARSE_HUGE I XML_PARSE_NOCDATA XML_PARSE_NOBLANKS));
dataQueryResult = nil;

if (lastModifiedDate) *lastModifiedDate = outLastModifiedDate;
if (error) *error = outError;
if (returnNil II document == NULL) return nil;
NSMutableArray *retVal = nil;
xmlNodePtr rootNode = xmlDocGetRootElement(document);

if (rootNode != NULL) {
if (strcmp((const char *) rootNode->name, "errors") == 0) {
if (error) {
NSString *message =
NSString *code =
xmlNodePtr firstChild = rootNode->children;
if (firstChild != NULL) {
xmlAttrPtr curProp = firstChild->properties;
while (curProp I= NULL) {
xmlChar *valueXml = xmlNodeGetConient(curProp->children);
if (valueXml == NULL) {
curProp = curProp->next;
continue;

if (PROP_NAME("code")) code = [NSString stringWithUTF8String:(char *) valueXml];
else if (PROP_NAME('msg")) message = [NSString stringWithUTF8String:(char *) valueXml];
xmlFree(valueXml);
curProp = curProp->next;

NSString *desc = [NSString stringWithFormat:@"Error %@: %@", code, message];
if (error) *error = [NSError errorWithDomain:ErrorDomain code:0 userInfo:@{
@"code" : code, @"msg" : message, @"desc" : desc }1;

else if (strcmp((const char *)rootNode->name, "items") == 0) NSDictionary*remotellemsCache = [IHPItemCache sharedInstance]
readRemoteltemsCache];
retVal = @B.mutableCopy;
NSDateFormatter *dateFormatter = [[NSDateFormatter alloc] init];
[dateFormatter setDateFormat:@"yyyy-MM-dd HH:mm:ss"];
NSString *localldentProperty = ([[HPAssetsManager sharedlnstance] phLibrary]) ? @localldentifier" : @"thumbnailUr1";
xmlNodePtr curChild = rootNode->children;
HPImageltem *imageltem = nil;
while (curChild != NULL) if (strcmp((const char lcurChild->name, 'item') != 0) curChild = curChild->next;
continue;

@autoreleasepool if (imageltem == nil) imageltem = [HPImageltem new];
double latitude = 0, longitude = 0;
xmlAttrPtr curProp = curChild->properties;
while (curProp != NULL) xmlChar *valueXml = xmlNodeGetContent(curProp->children);
if (valueXml == NULL) curProp = curProp->next;
continue;

char *value = (char lvalueXml;
if (PROP NAME(id")) imageltem.ID = [NSString stringWithUTF8String:value];
else if (PROP_NAME('parent_id")) imageltem.parentID = atol(value);
else if (PROP_NAME("name")) imageltem.name = [NSString stringWithUTF8String:value];
else if (PROP NAME("size")) imageltem.fileSize = @(atol(value));
else if (PROP_NAMECsource urn) imageltem.sourceUrl = [NSString stringWithUTF8String:value];
else if (PROP_NAME("sorting_time")) imageltem.sortingTime = [dateFormatter dateFromString:[NSString stringWithUTF8String:value]];
else if (PROP NAME("thumbnail url")) imageltem.thumbnailUrl = [NSString stringWithUTF8String:value];
else if (PROP NAME("friendly_name")) imageltem.friendlyName = [NSString stringWithUTF8String:value];
else if (PROP NAME("storage_provider id")) imageltem.storageProviderlD = [NSString stringWithUTF8String:value];
else if (PROP_NAME("created time")) imageltem.createdTime = [dateFormatter dateFromString:[NSString stringWithUTF8String:value]];
else if (PROP_NAME("modified_time")) imageltem.modifiedTime = [dateFormatter dateFromString:[NSString stringWithUTF8String:value]];
else if (PROP_NAME('width")) imageltem.size = CGSizeMake(atof(value), imageltem.size.height);
else if (PROP_NAME("height")) imageltem.size = CGSizeMake(imageltem.size.width, atof(value));
else if (PROP NAME("geolocation")) imageltem.location = [HPLocation locationFromDelimitedString:[NSString stringWithUTF8String:value]
delimiter@T];
else if (PROP NAME("latitude")) latitude = [[NSString stringWithUTF8String:value] doubleValue];
else if (PROP NAME('Iongitude")) longitude = [[NSString stringWithUTF8String:value] doubleValue];
else if (PROP NAME("time taken")) imageltem.timeTaken = [dateFormatter dateFromString:[NSString stringWithUTF8String:value]];
else if (PROP_NAME("user_albums")) imageltem.userAlbumIDs = [[NSString stringWithUTF8String:value]
componentsSeparatedByString:@","];
else if (PROP NAME("hidden")) imageltem.hiddenTime = [dateFormatter dateFromString:[NSString stringWithUTF8String:value]];
else if (PROP NAME("favorite")) imageltem.favoriteTime = [dateFormatter dateFromString:[NSString stringWithUTF8String:value]];
else if (PROP NAME("duid")) imageltem.bUID = [NSString stringWithUTF8String:value];
else if (PROP NAME("localid")) imageltem.ibcalID = [NSString stringWithUTF8String:value];
else if (PROP_NAME('rnime_type")) imageltem.mimeType = [NSString stringWithUTF8String:value];
xmlFree(valueXml);
curProp = curProp->next;
if (latitude II longitude) imageltem.location = [HPServerConnectionModel geocodeLat:latitude Ion:longitude];
if ([imageltem.DUID isEqualToString:uniqueDevicelD]) if (fimageltem.localID hasPrefix:STORAGE_LOCAL_DLNAD
[imageltem setIsStoredLocal:N01;
[imageltem setIsDLNA:YES];
else [imageltem setValue:imageltem.localID forKey:localldentProperty];
[imageltem setIsStoredLocal:YES];

NSString *itemKey = [[HPItemCache sharedInstance] getldentifierimageltem];
HPImageltem *oldltem = remoteltemsCache[itemKey];
if (old Item) [retVal addObjectimageltem];
// id,sorting_time,thumbnail_url,storage_provider_id,geolocation oldltem.ID = imageltem.ID;
oldltem.sortingTime = imageltem.sortingTime;
oldltem.thumbnailUrl = imageltem.thumbnailUrl;
oldltem.storageProviderlD = imageltem.storageProviderlD;
oldltem.location = imageltem.location;
[retVal addObjectoldltem];
}else {
[retVal addObjectimageltem];
imageltem = nil;

xmlNodePtr tmpChild = curChild;
curChild = curChild->next;
xmlFreeNode(tmpChild);

rootNode->children = NULL; // this shouldn't leak memory since we have already freed all the children rootNode->last = NULL;

xmlFreeDoc(document);
xmlCleanupParser();
return retVal;

Although specific embodiments of the invention have been described, it will be apparent to one skilled in the art that variations and modifications to the embodiments may be made within the scope of the following claims.

Claims (2)

We claim:

1.A computer-implemented method of providing an electronic client with the ability to seamlessly aggregate all of a user's digital photos from both local and remote sources, and display relevant groupings of same based on a desired search filter executed by the user, without the user ever having to know the sources where the photos are actually stored.

2.The computer-implemented method of claim 1, wherein the relevant groupings are displayed on the client as a collage of groupings for viewing purposes or wherein the relevant groupings are displayed during the use of a keyboard for messaging purposes wherein any particular photo within a groupings can be readily located regardless of source and attached as part of a message to be sent to another client.