US20170206225A1 - Semantic indexing by augmented object association - Google Patents

Semantic indexing by augmented object association Download PDF

Info

Publication number
US20170206225A1
US20170206225A1 US15/410,658 US201715410658A US2017206225A1 US 20170206225 A1 US20170206225 A1 US 20170206225A1 US 201715410658 A US201715410658 A US 201715410658A US 2017206225 A1 US2017206225 A1 US 2017206225A1
Authority
US
United States
Prior art keywords
user
index
various embodiments
media
data
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US15/410,658
Other languages
English (en)
Inventor
Ragy Farouk Eleish
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Regwez Inc
Original Assignee
Regwez Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Regwez Inc filed Critical Regwez Inc
Priority to US15/410,658 priority Critical patent/US20170206225A1/en
Assigned to Regwez, Inc. reassignment Regwez, Inc. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: ELEISH, RAGY FAROUK
Publication of US20170206225A1 publication Critical patent/US20170206225A1/en
Priority to US15/952,533 priority patent/US20180278600A1/en
Abandoned legal-status Critical Current

Links

Images

Classifications

    • G06F17/30268—
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
    • G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • G06F16/583—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
    • G06F16/5846—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using extracted text
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
    • G06F16/2468—Fuzzy queries
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/248—Presentation of query results
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/28—Databases characterised by their database models, e.g. relational or object models
    • G06F16/289—Object oriented databases
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/40—Information retrieval; Database structures therefor; File system structures therefor of multimedia data, e.g. slideshows comprising image and additional audio data
    • G06F16/41—Indexing; Data structures therefor; Storage structures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/40—Information retrieval; Database structures therefor; File system structures therefor of multimedia data, e.g. slideshows comprising image and additional audio data
    • G06F16/43—Querying
    • G06F16/438—Presentation of query results
    • G06F16/4387—Presentation of query results by the use of playlists
    • G06F16/4393—Multimedia presentations, e.g. slide shows, multimedia albums
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
    • G06F16/51—Indexing; Data structures therefor; Storage structures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
    • G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
    • G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • G06F16/583—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
    • G06F16/5838—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using colour
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
    • G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • G06F16/5866—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/901—Indexing; Data structures therefor; Storage structures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/95—Retrieval from the web
    • G06F16/953—Querying, e.g. by the use of web search engines
    • G06F16/9535—Search customisation based on user profiles and personalisation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/95—Retrieval from the web
    • G06F16/953—Querying, e.g. by the use of web search engines
    • G06F16/9538—Presentation of query results
    • G06F17/3002—
    • G06F17/3028—
    • G06F17/30607—
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/30—Authentication, i.e. establishing the identity or authorisation of security principals
    • G06F21/31—User authentication
    • G06F21/36—User authentication by graphic or iconic representation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/60—Protecting data
    • G06F21/62—Protecting access to data via a platform, e.g. using keys or access control rules
    • G06F21/6218—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
    • G06F3/0482—Interaction with lists of selectable items, e.g. menus
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0484—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
    • G06F3/04842—Selection of displayed objects or displayed text elements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0484—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
    • G06F3/0486—Drag-and-drop
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0487—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
    • G06F3/0488—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures
    • G06F3/04883—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures for inputting data by handwriting, e.g. gesture or text
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/061—Improving I/O performance
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0638—Organizing or formatting or addressing of data
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/0671—In-line storage system
    • G06F3/0673—Single storage device
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/10—Text processing
    • G06F40/166—Editing, e.g. inserting or deleting
    • G06F40/186—Templates
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/30—Semantic analysis
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L63/00—Network architectures or network communication protocols for network security
    • H04L63/10—Network architectures or network communication protocols for network security for controlling access to devices or network resources
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L63/00—Network architectures or network communication protocols for network security
    • H04L63/10—Network architectures or network communication protocols for network security for controlling access to devices or network resources
    • H04L63/101—Access control lists [ACL]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/50—Network services
    • H04L67/535—Tracking the activity of the user
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2221/00—Indexing scheme relating to security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F2221/21—Indexing scheme relating to G06F21/00 and subgroups addressing additional information or applications relating to security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F2221/2113—Multi-level security, e.g. mandatory access control
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2221/00—Indexing scheme relating to security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F2221/21—Indexing scheme relating to G06F21/00 and subgroups addressing additional information or applications relating to security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F2221/2133—Verifying human interaction, e.g., Captcha
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2221/00—Indexing scheme relating to security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F2221/21—Indexing scheme relating to G06F21/00 and subgroups addressing additional information or applications relating to security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F2221/2147—Locking files
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2221/00—Indexing scheme relating to security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F2221/21—Indexing scheme relating to G06F21/00 and subgroups addressing additional information or applications relating to security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F2221/2149—Restricted operating environment
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L63/00—Network architectures or network communication protocols for network security
    • H04L63/08—Network architectures or network communication protocols for network security for authentication of entities
    • H04L63/083—Network architectures or network communication protocols for network security for authentication of entities using passwords

Definitions

  • a library of media objects may be organized according to a tree hierarchy (e.g., folders and sub-folders, or albums and sub-albums) where each node corresponds to a separate label. Membership of a media object at a node in the tree (e.g., folder or album) is maintained manually.
  • reclassifying media objects from one type of classification (e.g., location) to another (e.g., event) requires reclassification of every media object in the library.
  • Keyword based classifications may be better adapted to creating and maintain dynamic albums or folders.
  • keyword based classifications technique involve manual input of addition of every possible tag permutation.
  • tags may have to be added for just a single location (e.g., nature, park, tree, grass).
  • tags tend to be ambiguous (e.g., “Paris” could be a city name or a person name).
  • Classification based on direct attributes allows a user to create dynamic albums that are based on direct attributes associated to the constituent media object.
  • an album can include all photos from a certain timeframe (e.g., from Apr. 11, 2013 to May 5, 2014).
  • direct attribute systems offer poor user interface (UI) and impose significant limitations on searches conducted based on direct attributes.
  • Limited semantic tagging e.g., Facebook® tagging
  • tags do not distinguish between different types of relationships that may exist with respect to the media object. For example, a person who was tagged in a photo can appear in the photo, have an interest in the content of the photo, or is a creator of the photo.
  • Fully-automated media object indexing techniques are generally inaccurate. For example, image recognition systems have only a 70% success rate at identifying even a generic object (e.g., a shoe) depicted in a media object. These systems further are not able to determine relationships with respect to the media object (e.g., an owner, designer, and/or retailer for the shoe). By contrast, manual methods to index media objects tend to be tedious and error prone while offering little user incentive.
  • a system for semantic indexing by augmented object association includes processors configured to: receive a first digital object associated with a first set of semantic information; and associate the first digital object with a second digital object associated with a second set of semantic information.
  • a system for an object stamping user interface includes a digital platform configured to index digital objects of the digital platform to identify semantic information of each digital object, and associate a plurality of digital objects based on matching semantic information.
  • FIG. 1 is a network diagram illustrating a network environment various embodiments
  • FIG. 2 illustrates semantic indexing according to various embodiments
  • FIG. 3 illustrates a stamping user interface according to various embodiments
  • FIG. 4 is a flowchart illustrating a process for stamping according to various embodiments
  • FIG. 5 illustrates a process for adding an association to a stamp according to various embodiments
  • FIG. 6 illustrates an object selector according to various embodiments
  • FIG. 7A illustrates an association selector according to various embodiments
  • FIG. 7B illustrates single and multiple association selection according to various embodiments
  • FIG. 8 illustrates a visual interface according to various embodiments
  • FIG. 9 illustrates a visual browsing menu according to various embodiments
  • FIG. 10 illustrates selection criteria according to various embodiments
  • FIG. 11 illustrates a process for adding a new search selector according to various embodiments
  • FIG. 12 illustrates a facet navigation interface according to various embodiments
  • FIG. 13 illustrates a facet display section according to various embodiments
  • FIG. 14 illustrates facet representations according to various embodiments
  • FIG. 15 illustrates a limited discrete index according to various embodiments
  • FIG. 16 illustrates a simple derived index according to various embodiments
  • FIG. 17 illustrates a fuzzy derived index according to various embodiments
  • FIG. 18 illustrates a multi-mode control switch according to various embodiments
  • FIGS. 19A-C illustrate a multi-mode UI according to various embodiments
  • FIG. 20 illustrates data access computation according to various embodiments
  • FIG. 21 illustrates a process for enforcing access control according to various embodiments
  • FIG. 22 illustrates an automatic change to access control rule status according to various embodiments
  • FIG. 23 illustrates a manual change to access control rule status according to various embodiments
  • FIG. 24 illustrates central server based synchronization according to various embodiments
  • FIG. 25 illustrates peer-to-peer synchronization according to various embodiments
  • FIG. 26 illustrates hierarchical synchronization according to various embodiments
  • FIG. 27 illustrates access control rule cascade according to various embodiments
  • FIG. 28 illustrates a peer-to-peer browsing session according to various embodiments
  • FIG. 29 illustrates a process for initiating a peer-to-peer browsing session according to various embodiments
  • FIG. 30 illustrates a process for configuring a visual access code according to various embodiments
  • FIG. 31 illustrates an example lock code management interface for use with visual access codes
  • FIG. 32 illustrate an example flow for configuring visual access codes according to an exemplary embodiment
  • FIG. 33 illustrates an example process for mapping a user identifier to photo selection mapping according to various embodiments
  • FIG. 34 illustrates an example unique user hex digest, according to various embodiments
  • FIG. 35 illustrates an example registration process for assigning a visual access code according to an example embodiment
  • FIG. 36 illustrates an example process for encoding a visual access code with a password according to an example embodiment
  • FIG. 37 illustrates an example mobile interface in accordance with various embodiments
  • FIG. 38 illustrates an example implementation of entering a visual access code according to an example embodiment
  • FIGS. 39A-C illustrate an example implementations of the virtual input method on a website according to various embodiments
  • FIG. 40 illustrates an image blending process in accordance with various embodiments
  • FIG. 41 illustrates an example implementation of hotspot position shifting in accordance with various embodiments
  • FIG. 42 is a block diagram illustrating wired or wireless system according to various embodiments.
  • FIG. 1 is a network diagram illustrating a network environment 100 according to various embodiments.
  • a user device 110 communicates with a media platform 120 .
  • the user device 110 may be any device that is capable of communication with or causing communication with the media platform 120 through a wired or a wireless connection.
  • the user device 110 may be a wired or wireless communication device including, for example, but not limited to, a smartphone, a wearable device, a tablet personal computer (PC), a laptop, a desktop PC, a personal entertainment system, and an embedded processing system.
  • PC personal computer
  • FIG. 1 is a network diagram illustrating a network environment 100 according to various embodiments.
  • a user device 110 communicates with a media platform 120 .
  • the user device 110 may be any device that is capable of communication with or causing communication with the media platform 120 through a wired or a wireless connection.
  • the user device 110 may be a wired or wireless communication device including, for example, but not limited to, a smartphone, a wearable
  • the user device 110 may communicate with the media platform 120 via a communication network 130 .
  • the communication network 130 represents one or more wired and/or wireless connections.
  • the communication network 130 may include, for example, but not limited to, a wired and/or wireless local area network (LAN), a wired and/or wireless wide area network (WAN), and any combinations thereof.
  • LAN local area network
  • WAN wide area network
  • the media platform 120 may be communicatively coupled with a local data store 125 .
  • the media platform 120 may further communicate with a plurality of remote and/or third party data sources including, for example, but not limited to, a first data source 140 and a second data source 150 .
  • the media platform 120 can associate a media object with semantic information that includes, for example, but not limited to, attributes, relationships, and classifications.
  • the semantic information can be inherited from one or more other objects (i.e., including other media objects) that each provides an individual set of attributes, relationships, and/or classifications.
  • one media object can depict a smiling Bill Gates.
  • the media object can inherit all attributes of Bill Gates as a person and the relationships Bill Gates has with other people (e.g., age, an entrepreneur, an influencer, a billionaire, a philanthropist, a father, a family man, a techie, an American, etc.). Smiling implies that Bill Gates appeared in the photo and was in a good mood.
  • John Smith may be interested in the contents of the media object (e.g., the photo) depicting Bill Gates.
  • the media object can further inherit all attributes of John Smith as a person and the relationships John Smith has with other people although John Smith is not depicted in the photo.
  • a media object e.g., a video
  • the media object can depict a vehicle belonging to Alice after an accident.
  • the media object can inherit all attributes of Alice's car (e.g., make, model, year, mileage, and maintenance records), which can be determined via a third party source (e.g., Carfax®).
  • the media object can further inherit all attributes about Alice including, for example, but not limited to, Alice's driving records, professional activities, and biographic information.
  • the media object can also inherit all attributes about the particular accident (e.g., classification as a minor or a major accident).
  • the media platform 120 can allow the media object to be searched via any of corresponding semantic information.
  • the media object depicting Bill Gates may be found through a search for photos of 50-year old men who are smiling.
  • the video of Alice's car accident may be found through a search for videos of cars owned by women that are involved in accidents.
  • the media object may be any kind of computer stored file including, for example, but not limited to, a text and a multimedia (e.g., photograph, video) file.
  • FIG. 2 illustrates semantic indexing according to various embodiments.
  • a media object e.g., a photo of a vehicle
  • a media platform 120 can generate and add the automatic association 2 based on geolocation information included in the media object's meta-data.
  • the media object can inherit a plurality of other relationships including, for example, but not limited to, ownership (e.g., Bob Smith) and location (e.g., Jordan Middle School parking lot).
  • automatic associations can be added to a media object.
  • automatic associations can be generated based on a geolocation and/or timestamp.
  • Automatic associations can also be generated and added to the media object based on current events (e.g., fairs, holidays, private birthdays, etc.) and weather (e.g., rain, snow, storm).
  • current events e.g., fairs, holidays, private birthdays, etc.
  • weather e.g., rain, snow, storm.
  • the certainty of the automatic association is below a certain threshold, the automatic associations can be presented to a user for confirmation.
  • the media platform 120 can add new semantic information to a media object, which allows the media object to be searched based on the new semantic information. For example, if Bob Smith retires, then the media object depicting Bob Smith's vehicle is searchable as “photos of cars owned by retired people.”
  • a media object can further be searchable based on new semantic information that has been added to objects having existing associations with the media object. For example, if Bob Smith's son Charlie Smith is added as an object and Charlie Smith is a student at Jordan Middle School, then the media object depicting Bob Smith's vehicle is also searchable as “photos of cars owned by a student's parents.”
  • a system for semantic indexing includes a media platform.
  • the media platform may comprise one or more hardware processors configured to: receive a first media object associated with a first set of semantic information; and associate the first media object with a second media object associated with a second set of semantic information.
  • the first media object inherits the second set of semantic information associated with the second media object.
  • the first set and second set of semantic information can each includes at least one of attributes, relationships, and classifications.
  • the one or more hardware processors can be configured to automatically generate additional semantic information and associate the automatically generated semantic information with the first media object.
  • the one or more hardware processors can be configured to automatically generate the additional semantic information based at least in part one or more of a geolocation and a timestamp associated with the first media object.
  • the one or more hardware processors are configured to receive additional semantic information from a user and associate the additional semantic received from the user with the first media object.
  • the user can provide the additional semantic information at least in part by indicating an association between the first media object and a third media object associated with a third set of semantic information.
  • the user provides the additional semantic information at least in part by indicating an association between the first media object and one or more of an attribute, relationship, and classification.
  • the media platform 120 presents a user interface (UI) that allows the user to quickly attach semantic information to a media object.
  • UI user interface
  • the media platform 120 allows the user to assign semantic information to media content using the stamping UI 300 .
  • the stamping UI 300 allows the user assign semantic information to multiple media object using a single click.
  • the stamping UI 300 provides a stamp area 4 displaying the categories, sources, or values of semantic information to be added.
  • the user clicks on a target media object from a list 2 the stamp are 4 recommends a list of semantic information based on analysis of the target media object. Analysis of the target media object can include object recognition, metadata analysis, as well as, inspecting semantic information of other media items associated with the target media item.
  • the user has the option to edit the recommended list of semantic information in the stamp area 4 or can accept an entire set of recommendations. For example, a single click by the user can stamp the target media object with the set of semantic information shown in the stamp area 4 .
  • the stamp area 4 can include individual associations 6 , which may be added independent of other associations.
  • the stamp area 4 can further include association templates 8 .
  • a stamp association template 8 can include a group of semantic categories configured based on commonly used associations. Selecting a stamp association template 8 assigns a set of semantic information for each of the categories in the template. For example, a home stamp associate template can associate the target media object with other media objects associated with home, add semantic information for the geographic location of the home, add relationships to people that are also part of the home category, etc.
  • the stamp association template 8 can be pre-configured for repeated use for commonly used associations.
  • a system for semantic indexing can include a media platform with one or more processing device configured to: index media objects of the media platform to identify semantic information of each media object; and associate a plurality of media objects based on matching semantic information.
  • the processing device searches the index of media objects for semantic information in common with the stamp template and based on the search, the processing device sorts the corresponding media objects based on common semantic information; and present the sorted media objects for curation by a user.
  • the processing device generates a stamp template of the association comprising the common semantic information for applying the association to one or more media object.
  • curation can include applying the stamp template or one or more other associations.
  • the stamp template can be editable to modify semantic information of the association and include multiple associations.
  • the system includes an interface with stamp templates that allow a user to curate media objects of the media platform by applying an association from a selected stamp template to multiple media objects based on a single selection.
  • the processing device receives additional semantic information from user input to associate with one or more media object.
  • FIG. 4 is a flowchart illustrating a process 400 for stamping according to various embodiments.
  • the process 400 can be performed by the media platform 120 .
  • the process 400 is performed when a data collection mode is selected.
  • the media platform 120 can collect additional information for at least some specific associations in a stamp. That is, objects with common semantic information are identified and the common semantic information can be grouped as an association. The groupings of common semantic information can form a template. When a grouping of semantic information is attached via the stamping action and an actual media object is defined in the system. An association can also form a relationship between objects with common semantic information that allows modifications to propagate.
  • a screen display 8 appears in response to determining that the stamp includes a food item association.
  • the UI allows for collecting information about the food.
  • the screen display 8 is object template type specific. After the user activates a save feature, a new object is created (or stored) and the association is assigned to that object.
  • the user can click on a media object and other objects in the stamp area 4 are associated with the media object.
  • the two workflows are part of an overload approach of the media platform 120 .
  • the media platform 120 can collect information incrementally as specified by the user.
  • the user is incentivized to expend effort to input extra information because of an enhanced ability to search based on that information using a visual search.
  • the user may identify content of a media object, associate the semantic information with the media object, and select a type of the association. Once the user associates content of another media object with the target media object, the target media objects inherits the other associations of the contents of the other media object.
  • the user can efficiently identify and associate content with media object with high accuracy.
  • FIG. 4 illustrates a flowchart for an example data collection process for associating world objects with media content.
  • FIG. 5 illustrates a process 500 for adding an association to a stamp according to various embodiments.
  • the process 500 may be performed by the media platform 120 .
  • the user can examine media objects to be curated. From the stamp view, the user can see the associations to be added. At the stamp view, the user can stamp the media object by selecting (e.g., clicking, touching, etc.) a photo and all the associations in the stamp will be added to the photo. Alternately, the user can select to add new associations to the stamp.
  • the process advances to a present a screen display 4 where the menu allows the user to select from different processes to add new associations.
  • the user can add a new association by selecting object types via the screen display 6 .
  • the UI switches to screen display 8 (i.e., an object selection screen) that presents a searchable list of possible objects based on the selected object type. The user can search for an object to add.
  • the screen display 8 includes the stamp icon to provide for easy navigation.
  • an association selection screen display 10 Upon selecting a specific object, an association selection screen display 10 is presented to allow the user to further select an association for the object.
  • the user has the option to can cancel and return without completing the association operation.
  • the user can select one or more associations on an association selection screen display 10 to complete the stamping operation.
  • the UI provides additional confirmation controls (e.g., save, cancel, etc.) to complete the association, such as depicted on a screen display 12 .
  • FIG. 6 illustrates an object selector 600 according to various embodiments.
  • the user can select the object type and start a search for various objects to associate.
  • the UI 600 displays the association stamp that displays existing stamps to communicate to the users the objects are already loaded into the stamp.
  • the user can select to add the association to the stamp in multiple ways. For example, the user can select from a list of recent associations. In another example, the user can select from a previously configured association template. Each template is a group of associations input by the user. For example, a template can include “baby Jim playing”, “Project X”, or “expense receipts for project Y.” In some embodiments, the media platform 120 allows the user to define and edit a fixed number of templates. Having a limited number of configurable templates allows the user to easily access preconfigured templates without a template management system.
  • the user can select a world object type that allows the user to browse object types to select an association.
  • the user can select from common associations, such as general associations configured by an application author. For example, an application that employs thumbnails may use a temporary photo association.
  • FIG. 7A illustrates an association selector 700 according to an example embodiment.
  • FIG. 7B illustrates single and multiple association selection according to an example embodiment.
  • the media platform 120 enables the user add associations incrementally. Moreover, the media platform 120 allows the user to quickly group the artifacts with common parameters together for faster association.
  • the media platform 120 links the association process with the search process via the multi-mode user interface. Using the multi-mode user interface, the user can quickly switch between association and search and vice-versa. This quick switching allows the following scenarios:
  • the user While the user is searching, the user may determine that some semantic information is missing. Then the user can quickly switch to the stamping mode via the first level menu and start adding more associations to the media object.
  • the interface enables the user to add as much information as desired in an incremental fashion, which reduces the perceived amount of work and effort. Moreover, the user can immediately appreciate the benefits of the newly added information because the UI allows the user to start searching using the new added semantic information.
  • the media platform 120 provides a visual interface that allows the user to quickly glance at the criteria the presented media meet.
  • the selection criteria can be displayed in a specific area.
  • the interface can have a normal mode and a minimized mode to provide more space to the user.
  • FIG. 8 illustrates a visual interface 800 according to various embodiments. In minimized mode, the user is in a read-only mode and cannot interact with various selectors.
  • presenting the query as a group of selectors simplify the concept to the end users. Furthermore, the position of the selectors relative to each other is important and the user is able to reorder them.
  • FIG. 9 illustrates a visual browsing menu 900 according to various embodiments.
  • the media platform 120 when the user clicks the central menu button in the normal mode, the media platform 120 presents a first level menu view 2 .
  • This menu enables the user to browse the two level hierarchical facets classification.
  • a single click on the menu button presents the first level facet categories, which allows the user to navigate to a second level menu view 4 .
  • the user In the second level menu view 4 , the user is provided with information about various media objects. The user can determine whether multiple media objects from multiple years (e.g., 2008, 2010 and 2012 but not 2011) are present and filter to see the media objects from the month of March through August.
  • the interface displays the selectors used to pick the data.
  • FIG. 10 illustrates selection criteria 1000 according to various embodiments.
  • the selection criteria 1000 can be built by a user. As shown in FIG. 10 , there can be multiple selectors 2 for each facet. The selectors 2 can specify what facets values are used for searching.
  • the selection criteria 1000 can include a NOT criterion 4 and a MUST criterion 6 , both of which can be single value selectors.
  • the selection criteria 1000 can further include a map location criterion 8 and a value range criterion 10 , both of which can be an OR criterion (i.e., at least one of the map location criterion 8 and the value range criterion 10 has to be true).
  • OR criterion i.e., at least one of the map location criterion 8 and the value range criterion 10 has to be true.
  • the user can glance quickly and understand the criteria that is being used for the search.
  • FIG. 11 illustrates a process 1100 for adding a new search selector according to various embodiments.
  • the user clicks an open menu 2 , which leads the user to facets browsing mode 4 .
  • the user can drag that facet to the selector area. The dragging action is important because it allows the user to position the selector relative to the already existing ones.
  • a dialog box 6 is displayed to allow the user to edit the selections in more details.
  • the dialog box 6 may show an UI that differs from one facet type to another.
  • the purpose of displaying a large interface is to give the computer application designer more space to display various options to the user.
  • the large interface saves the user the frustration of having to deal with small spaces.
  • the user can specify AND, OR and MUST criteria to different facet values or range of values. Once the user clicks accept, the newly added selectors are presented in screen display 8 .
  • FIG. 12 illustrates a facet navigation interface 1200 according to various embodiments.
  • the user can navigate between facets by selecting one of first level facet categories.
  • the first level of facet categories can be divided into several (e.g., 5 ) main groups.
  • the groups are specific to media objects to be searched on, e.g., who, what, when, where, and how.
  • the second level facet categories can be displayed to simplify the user's search.
  • there can be multiple second level facet categories including, for example, but not limited to, specific dates, weather conditions, event types, and event names.
  • each top level category has a different color to highlight and help identify the category.
  • a second level facets category can include a single facet with unlimited values.
  • the facet navigation interface provides a way to quickly glance at the media objects' meta-data.
  • the facet navigation interface allows the user to explore the media collection based on the meta-information driven by curiosity. The user can be further encouraged to identify missing meta-information (e.g., the user can switch to the stamping mode and add the missing information).
  • the facet navigation interface provides a unified interface for the user to start requesting media that is not available in the media library. For instance, if a user discovers that the media library is missing a certain media object (e.g., a photo from the user's cousin's birthday last year), the user can use that interface to initiate a request for the missing media object.
  • FIG. 13 illustrates a facet display section 1300 according to various embodiments.
  • the facet display section 1300 displays the facets in a particular facet category whether or not the facets have corresponding values.
  • the user can become aware of what is missing about the collection of media objects the user is currently viewing. For example, the user can observes from the facet display section 1300 that neither the occupation facet nor the gender facet has been provided any values.
  • the system may display the facets in response to determining that there are no values associated with the facets. Presenting facets with empty values or no search hits communicates to the user that a background search has determined that the facet is not useful on the data set. Empty facets may also communicate to the user that semantic data is missing from one or more data objects or the data object is unavailable. Then the user can locate the media object to add missing the meta-data or acquire an unavailable media object.
  • the media platform 120 displays facets to the user in different ways based on a type associated with the facet's values.
  • displays facets based on a type associated with the facet's values improves communication to the user regarding the available types of data.
  • Discrete values are individual values. There are two types of discreet values: limited discreet values and unlimited discreet values. Limited discrete values (e.g., gender) are values having a limited number of possible values. Facets having limited discrete values are part of a second level facet category. By contrast, unlimited discrete values (e.g., humans, events) are displayed as an individual second level facet category.
  • Range values are values that can be grouped into a range. Some range values are a continuous range which includes an unlimited number of possible values (e.g., timestamp). Continuous range values are always displayed in a range or group of ranges. By contrast, an integer range is made up of discrete values (e.g., days of the week). Integer ranges can be displayed as groupings of discreet values. Map values can be displayed in a special map presentation where the map values are special values.
  • FIG. 14 illustrates facet representations according to various embodiments.
  • facet representations can include representation of limited discreet facets 12 , integer range facets 4 , and unlimited discreet facets 14 .
  • the unlimited discreet facet 14 can require an entire subcategory to display where other facet representations (e.g., the limited discreet facets 12 and the integer range facets 4 ) can be grouped together.
  • facets can be organized into a two level hierarchy.
  • the hierarchy is stored into a data structure (e.g., tree).
  • Each leaf node of the tree can point to an individual facet value's index.
  • the media platform 120 can handle different facet hierarchies based on the user. For example, the facet hierarchies may be established based on the user level of expertise or interests.
  • a hierarchy can include:
  • Appendix B further illustrates the hierarchical faceted search engine according to various embodiments.
  • a system for performing a hierarchical visual faceted search for one or more media objects includes a media platform.
  • the media platform may comprise one or more hardware processors configured to: provide, to a user device, a selector user interface (UI) adapted to receive a plurality of selection criteria; provide, to the user device, a first level menu comprising a plurality of first level selection criteria that includes a first selection criterion; receive, from the user device, an indication to add at least the first selection criterion to the selector UI; provide, to the user device, a second level menu comprising a plurality of second level selection criteria corresponding to the first selection criterion, wherein the plurality of second level selection criteria includes a second selection criterion; receive, from the user device, an indication to add at least the second selection criterion to the selector UI; add the first selection criterion and the second selection criterion to the selector UI; and execute a search to identify one or more
  • the first selection criterion can include an identity criterion, a location criterion, and the second selection criterion can include one of a name, relationship, interest, and profession of an individual associated with a media object.
  • the first selection criterion can include a time criterion and the second selection criterion comprises one of a date, event, event type, and weather conditions associated with a media object.
  • the first selection criterion comprises a location criterion and the second selection criterion can include one of map coordinates, location name, and location type associated with a media object.
  • the first selection criterion comprises a manner or means criterion
  • the second selection criterion can include one of a photographer, camera type, and media type associated with a media object.
  • the first selection criterion comprises an identification criterion
  • the second selection criterion can include one of an object and object type associated with a media object.
  • a faceted search engine may be deployed on the user device 110 .
  • the faceted search engine can rely on in-memory search indices that can be loaded on-demand based on the facets being searched.
  • the search indices are built on top of real objects attribute values.
  • a facet is a way to search for one or more media objects.
  • a facet can be associated with various facet values.
  • each facet corresponds to a specific field in the data being indexed. Fields have data types and expected values.
  • the faceted search engine can build an index of all the values in each filed.
  • the faceted search engine can very quickly execute any query.
  • conventional faced search engines depend on pre-calculated views of fixed queries and cannot handle dynamic complex queries.
  • a system performs a hybrid in-memory faceted search for one or more digital objects.
  • the system can include a media platform with one or more processing device configured to: store an index for a data set generated using one or more indexing processes, where the index includes a mapping of values to identifiers for each digital object in the data set; receive an update for the index; store the index update with a timestamp independent from the stored index; and in response to a request for the stored index, apply the index updates to the index.
  • the processing device determines a characteristic associated with the data set; and the index is processes based on the characteristic of the data set, where the index includes a mapping of values to identifiers for each digital object in the data set.
  • the data set can include discrete data and the index includes multiple arrays for each digital object including at least one of sorted values of the discrete data and identifiers.
  • the identifiers can be grouped in ordered groups.
  • the data set includes continuous data and each digital object is mapped to a unique timestamp.
  • the processing device can determine a certainty factor associated with each value for each digital object and rank the identifier based on the certainty factor.
  • the request for the stored index can be a search request with query criteria
  • the processing device can generate a selector object to match query criteria to identifiers of the index; calculate a count of the unmodified identifiers associate with each query criteria based on the selector object; and execute the search starting with the query criteria associated with the most unmodified identifiers of the query criteria.
  • the faceted search engine can use different types of indices depending on the type and nature of data being indexed. Each index type is built differently and has a specific purpose.
  • An index can be a basic index or a derived index. While each index type is implemented in a different way, all indices have the function of mapping real world values to certain rows of the media object they are indexing.
  • Basic indices are directly calculated from the world object's raw values. Different types of basic indices may be calculated depending on the nature of the raw value as each type of raw value is handled differently.
  • FIG. 15 illustrates a limited discrete index 1500 according to various embodiments.
  • the limited discrete index 1500 is a basic type of index that includes discrete an unlimited.
  • the limited discrete index 1500 is kept in memory (e.g., of the user device 110 ) at all times and is deployed whenever the limited discrete index 1500 is needed.
  • the limited discrete index 1500 can be implemented as two arrays. The first array can hold the values indexed in an ascending sorting order and the starting position into the second index.
  • the second array can hold the media row identifiers, which are grouped according to the raw values and are in a sorted order in each group.
  • storing the limited discrete index 1500 requires a small amount of memory.
  • a large discrete index includes very large number of discrete values.
  • a continuous index contains an unlimited number of possible values that have almost one-to-one mapping with values (e.g., timestamp). Almost every single photo, for example, can have a different timestamp.
  • Continuous range values can be handled by special index structure. For example, continuous indices may be handled using a regular B-Tree similar to the database indices.
  • a map index includes geolocation data.
  • Geolocation data can be three dimensional data (e.g., longitude, latitude, and altitude) that are handled as a whole.
  • map indices may be handled using a database engine (e.g., SQLite®).
  • Derived indices are based on other indices (e.g., basic or derived) and provide classifications and/or implications.
  • a derived index can be a simple derived index or a fuzzy derived index.
  • FIG. 16 illustrates a simple derived index 1600 according to various embodiments.
  • the index values of the simple derived index 1600 are based on values indexed by another index.
  • age groups may be indexed into a plurality of facet values including, for example, but not limited to, baby, toddle, kid, teenager, young adult, adult, middle age, and senior.
  • the actual indexed values are derived from age raw values, e.g., a toddler is a person whose age is between 2 and 5.
  • the simple derived index 1600 is built using a single array that holds the values and the corresponding raw value in the based index. As such, the simple derived index 1600 occupies very limited space in memory and can easily accommodate changes to the base index.
  • FIG. 17 illustrates a fuzzy derived index 1700 according to various embodiments.
  • the fuzzy derived index 1700 can include multiple indexed values that overlap with raw values.
  • each mapping has a certainty factor associated with it. For example, if age group has a fuzzy index, then a person who is 1.8 years old can be both a baby and a toddler. However, the person is more likely to be a toddler. As such, the certainty factor for toddler is 90% while the certainty factor for baby is 15%. The certainty factor is selected to be suitable for each value.
  • the benefit of such fuzzy indexing is to enable us to find the same information in different way and use the certainty to rank the search results.
  • the index can be created in memory by iterating through the raw data.
  • the index is kept in the most compact form in memory.
  • raw data is accessed differently in an orthogonal operation.
  • raw data may be iterated in batches in instances where the volume of raw data is large.
  • the process may include multiple iteration passes (e.g., one pass for each index) since the data is to be loaded in a sorted order.
  • the raw data can be loaded in to memory at once and sorted during the creation of the index.
  • the index is saved to a file in that form.
  • the individual arrays are saved in the same file in a specific order, e.g., the indexed data array before the row id array.
  • the index can also be updated including by adding or deleting values to and from the index.
  • the updates are applied to the index and the index saved to disk once the update is complete.
  • the updates are appended to an update file for that particular index.
  • the update file contains all the updates sorted by the timestamp.
  • the index is not uploaded for an update operation. Instead of loading an index to apply one or more updates, the faceted search engine stores the changes that are to be made to the index.
  • the index is loaded to memory from the disk, the stored updates are loaded and applied to the index, and the index is saved to memory and can be used in a search.
  • the update process reduces unnecessary calculations that from being performed each time the index is updated.
  • the update process preserves computational power for the index that is not being used between the updates.
  • the index is updated when the index is called (e.g., for performing a search).
  • the faceted search engine affords the user the power and control of an enterprise search interface but in an easy to understand way that does not have a steep learning curve.
  • the user selects one or more facet values and indicates if results must, can, or should not have the selected facet values. For example, suppose the user wants to find media objects taken during a holiday that is not Thanksgiving and depict a shoe, a dress, a bag, or sunglasses.
  • the user can define the following query:
  • a user can also specify a complex query for the user's son's photo around the house during autumn alone while my wife took the photo.
  • the query can be specified as follows:
  • the faceted search engine can create a selector object that operates on the facet index.
  • the selector's purpose is to match the criteria to the raw media object identifiers. It also can return the number of possible media object identifiers that match the given criteria, which in turn returns the possible matches.
  • the faceted search engine sorts the indices by the possible matches in an ascending order. As such, the index with the least number of matches is executed first, and the faceted search engine iterates over the list of indices and calculates the values that match the entire criteria.
  • each selector is independent and avoids race conditions.
  • the selectors are further executed in order from the strictest to the least strict (e.g., AND followed by NOT followed by OR).
  • the selector order provides the functionality to skip the execution of the lesser strict selectors if the result set will be empty.
  • Selection operations are performed to select matching row identifiers for a certain selection criteria.
  • Each index type implements the selector in a specific way that corresponds to that particular index type's structure.
  • Discrete value indices can be numeric or non-numeric.
  • a numeric discrete value index affords mathematical operations including, for example, but not limited to, range selection, greater than, and less than.
  • Derived value indices can also be either numeric or non-numeric, where numeric derived value indices are able to support mathematical operations.
  • Query operations may be performed in multiple embodiments.
  • compressed bit vector arrays are used to store the row identifiers (e.g., roaring array) and a binary bitmask is used to perform the operations that are suitable to such data structure.
  • the algorithms are applied such as:
  • the AND operation works on two arrays of row identifiers: rowIDs1 and rowID2. The results are included in resultingIDs.
  • the algorithm for an AND operation operates in linear time (i.e., O(n)) and includes:
  • the NOT operation is akin to a set subtraction operation. One set of row identifiers are subtracted from the result.
  • the algorithm for a NOT operation also operates in linear time (i.e., O(n)) and includes:
  • the OR operation works on two arrays of row identifiers: rowIDs1 and rowID2; the result is called resultingIDs. The results are included in resultingIDs.
  • the algorithm for an AND operation operates in linear time (i.e., O(n)) and includes:
  • the media platform 120 supports a multi-mode UI that is adaptable to various major activity types.
  • the UI can be in the visual search mode, the object association mode, or the object manipulation mode.
  • the UI can focus on the major activity without cluttering the screen with an endless set of options.
  • the commands are positioned in substantially the same relative place or region in the screen, allowing users access them by memory.
  • the UI can be controlled by a mode switching control that indicates to the user which mode the UI is in and allows the user to quickly switch to other modes.
  • a system for interacting with a media platform that includes a media platform with a processing device configured to: provide, to a user device, a user interface (UI) to control a multi-mode application of the media platform, where the user interface comprises multiple mode navigation regions and each mode navigation region is associated with a mode of the multi-mode application, where each mode navigation region comprises a specific set of functions for controlling the mode associated with the mode navigation region.
  • the UI is to present the navigation region for an active mode based on detecting device activity, where the UI includes a global navigation menu for switching to non-active modes and suppresses functions associated with the non-active modes.
  • each mode navigation region can include a specific set of functions for controlling the mode associated with the mode navigation region; and when the navigation region is presented, the specific set of functions are maintained in the mode navigation region.
  • the global navigation menu is controllable by a gesture input of the user.
  • the UI can include one or more expandable sub-mode navigation regions. The one or more expandable sub-mode navigation regions can be presented based on the detected device activity while the UI suppresses functions associated with the non-active sub-modes.
  • the UI can include an active mode indicator on the global navigation menu.
  • the processing device detects device activating by tracking a user's activity pattern to suggest a next mode by highlighting a shortcut on the global navigation menu.
  • the multi-mode application can include at least one of a visual search mode, object manipulation mode, or data entry mode.
  • FIG. 18 illustrates a multi-mode control switch 1800 according to various embodiments.
  • the multi-mode control switch 1800 can be displayed on the user device 110 (e.g., a smartphone).
  • the multi-mode control switch 1800 displays the current mode “Visual Search” as well as additional modes the user can switch to.
  • the user can use a swipe gesture (e.g., on a touch screen) to switch between different modes and/or reveal additional modes.
  • FIGS. 19A-C illustrate a multi-mode UI according to various embodiments.
  • the multi-mode UI can be used for associating semantic information to media objects and for searching for media objects.
  • the stamping mode provides an interface from determining associations for media objects.
  • the UI is constructed to allow efficient searching of the media objects that can be associated in a very effective way that allows the user to perform batch association.
  • the controls in the stamping UI are related to association functions.
  • To search the user can switch to the visual search mode (on the right side).
  • the multi-mode control switch as described in FIG. 18 , allows for quick navigation between modes. In search mode the visual search UI presents control options focused on searching the media library.
  • the multi-mode UI frees the user from guessing what actions are available for different activities (e.g., search, stamping, sharing, etc.).
  • the multi-mode UI efficiently categorizes and presents the actions which are related to each activity mode.
  • the UI commands are placed in the same location or region of the screen for a given mode.
  • the multi-mode UI reduces the cognitive requirements for individual users without reducing functionality of the application.
  • the media platform 120 controls access to stored data objects (e.g., media objects stored in the data store 125 ) in a manner that does not require user accounts.
  • access is controlled based on automatic or manual data object protection rules that are orthogonal to user account mechanisms.
  • Each data protection rule selects specific data objects to be protected and can be turned on and off to make the objects inaccessible or accessible respectively.
  • Data protection rules statuses are combinable to compute an effective data-masking layer. The data-masking layer determines if a given data object is accessible or not. Presenting data protection as simple data selection rules simplifies complex access control mechanisms.
  • FIG. 20 illustrates data access computation according to various embodiments.
  • the media platform 120 can calculate data object visibility by performing an effective data masking computation.
  • Access control rules are combinable to create a masking mechanism. Each rule identifies data objects that are to be inaccessible (e.g., hidden). When multiple rules are combined, the data objects that have not been obscured by any rule are left visible to the user.
  • the media platform 120 implements the masking mechanism by creating a lock count and attaches the lock count to each data object.
  • the rule identifies the data objects associated with the rule, determines a corresponding lock count incremented (e.g., by one), and restricts access to the associated data objects.
  • the system restricts access (e.g., hides the data objects from the search results, or prevents accessing the data object) to data objects that have a lock count greater than zero.
  • the media platform 120 can implement the masking mechanism by performing a check on whether a data object selected is associated with any active access control rules.
  • FIG. 21 illustrates a process 2100 for enforcing access control according to various embodiments.
  • a first UI is presented to challenge the user to re-establish the user identity before editing the rules (1).
  • a second UI is presented to enable the user to create a new access control rule or edit an existing one (2).
  • the second UI is a dynamic UI based on the available meta data to pick from and the relevant tags.
  • the second UI consults with two services. First, the second UI may consult with an ontology based tag search engine service to assist in the access control editor interface to facilitate the addition of relevant tags by the user. For example, the user may wish to pick tags for cities in Italy and the ontology based tag search engine helps the user find them efficiently.
  • the second UI may consult with an appropriate metadata recommendation engine, which indicates what kind of metadata is available for a particular object type. For example, videos may have duration as metadata while text documents may have a word or character count. The engine assists in narrowing down the selection to the user for easy editing.
  • the access control rule is packaged for efficient storage and transportation (3).
  • the rule is made ready for execution by the media platform 120 .
  • the access control rule is then stored (e.g., in a rules databased with existing rules).
  • the effect of the rule is pre-calculated for efficient enforcement at runtime and such effect is stored with each data object (4).
  • access control rules can be symbolic link access control rules, where the rules are a simple group of hand selected data objects.
  • symbolic links are used to identify files in a file system to be locked. As such, when a group of symbolic links are locked the actual files are also locked. Data control locks implemented using symbolic links are separate from organizational structure.
  • the access control rules can also be meta-data access control rules where the access control rules are based on meta-data instead of tags.
  • the access control rules can also be keyword based tags access control rules where the tags are keyword matching tags and not ontology based tags
  • the access control rules can also be ontology based tags access control rules.
  • Access control rule status can be changed either manually or automatically.
  • FIG. 22 illustrates an automatic change to access control rule status according to various embodiments.
  • an external system may be working in conjunction with the masking access control system.
  • the external system is responsible for controlling which access control rules are effective which access control rules are not.
  • an operating system can control the access control rules implemented in physical storage system controller (e.g., hard drive or solid state drive (SSD) controller).
  • the operating system can add an extra level of protection that works in conjunction with the underlying operating system level.
  • state 1 the external system sends a command to the described system where the command instructs to turn off one or more of the rules.
  • the system responds in state 2 by confirming the new status of the access control rule after changing that rule status.
  • FIG. 23 illustrates a manual change to access control rule status according to various embodiments.
  • the user controls the status of the access control rules.
  • the system presents a UI enabling the user to edit access control rules. For example, if the user is challenged for authentication with a login screen to confirm the user identity, the login screen can take a password form, a picture code form, a lock key form, or any other authentication form.
  • the UI can be presented that lists the access control rules and the status (e.g., enabled, disabled, active, deactivated, etc.) of each access control rule.
  • the user clicks on one of the rules the user is presented with a UI that allows the user to change the status (e.g., enabled, disabled, active, deactivated, etc.) of the rule.
  • a system for enforcing restrictive access control with respect to a set of media objects includes on a single device.
  • the single device may be configured to: determine, based at least in part on a first access control rule, to block access to at least a first media object included in the set of media objects; determine, based at least in part on a second access control rule, to block access to at least a second media object included in the set of media objects; and provide, to a user of the single device, at least a third media object included in the set of media objects but not the first media object and the second media object.
  • the device can be configured to provide the third media object but not the first media object and the second media object based at least in part on the lock count associated with each of the first media object, the second media object, and the third media object.
  • data may be dispersed across multiple independent systems including, for example, but not limited to, the user device 110 , the data store 125 , the first data source 140 , and the second data source 150 .
  • the media platform 120 can synchronize access control rules on separate systems on a separate and a higher priority synchronization channel than for data synchronization.
  • data objects metadata can also use a separate synchronization mechanism and/or channel, thereby allowing each system to enforce the rules independently of a centralized system.
  • the media platform 120 synchronizes access control rules, data objects, and metadata for data objects to enable each system to operate independently.
  • Multiple synchronization networks can operate independently.
  • the access control rules synchronization network can operate in a substantially real time fashion on a high priority level.
  • the metadata synchronization network can also work on a high priority level.
  • the data object synchronization network may be a third independent network.
  • the system can enforce restrictive access control with respect to a set of digital objects accessible by a first device and second device of a user.
  • the system includes the first device of the user configured to detect an update associated with a first system access control rule, wherein the first system access control rule is to block access to at least a first digital object included in the set of digital objects on the first device; determine, based at least in part on the update to the first system access control rule, to block access to at least a second digital object included in the set of digital objects on a second device; and provide, to the second device, the update associated with a first system access control rule to maintain restrictive access control over the set of digital objects on a second device.
  • the system for enforcing restrictive access control with respect to a set of media objects includes on multiple devices device for a single user.
  • a first device may be configured to: determine, based at least in part on a first access control rule, to block access to at least a first media object included in the set of media objects; determine, based at least in part on a second access control rule, to block access to at least a second media object included in the set of media objects; and provide, to a user of the first device, at least a third media object included in the set of media objects but not the first media object and the second media object.
  • the system can include a second device of the user and the first access control rule can include a universal rule applicable to the first device and the second device and the second access control rule can include a device specific rule applicable to the first device but not the second device.
  • the first device and the second device are configured to engage in a browsing session wherein a user of the second device browses the set of media objects via the first device.
  • the first device and the second device can be configured to conduct the browsing session based on a third access control rule that is applicable to the browsing session between the first device and the second device for the user.
  • the third access control rule can block access to the third media object included in the set of media objects.
  • the first device can be configured to provide to the user of the second device at least the second media object but not the first media object and the third media object.
  • synchronization can take place with a central server or cloud acting as a maestro. All changes are first transmitted to the central server or the cloud before the changes are propagated to other systems.
  • FIG. 24 illustrates central server based synchronization according to various embodiments.
  • a peer-to-peer paradigm is applied in synchronizing multiple systems.
  • peer-to-peer synchronization can use independent versioning to track the latest updates.
  • FIG. 25 illustrates peer-to-peer synchronization according to various embodiments.
  • some systems may act as a local synchronization server orchestrating the status between local devices.
  • the local synchronization server is responsible for communication with a centralized server.
  • a WiFi hotspot could host a server that orchestrates the synchronization between various systems connected to the hotspot as well and communicates with a central server.
  • FIG. 26 illustrates hierarchical synchronization according to various embodiments.
  • data object visibility is calculated by access control rule status cascade and effecting data masking computation.
  • the layers can include, for example, but not limited to, a universal layer (i.e., for the whole ecosystem), a system or device layer (i.e., for each individual device or system), an application layer (i.e., for systems implemented at a platform level), and a session layer (i.e., for peer-to-peer or temporary changes).
  • a universal layer i.e., for the whole ecosystem
  • a system or device layer i.e., for each individual device or system
  • an application layer i.e., for systems implemented at a platform level
  • a session layer i.e., for peer-to-peer or temporary changes.
  • access control rules can be turned on or off at each layer.
  • rule status is cascaded from the least specific (i.e., universal) layer to the most specific (i.e., session) layer.
  • the status of each rule is computed by allowing a rule status at a more general layer override the rule status at a more specific layer.
  • FIG. 27 illustrates access control rule cascade according to various embodiments.
  • the effective data masking computation is conducted in a similar manner as described above, but the process is repeated for each target session.
  • peer-to-peer browsing mode other systems are able to browse data objects stored in the host device, in an ad-hoc fashion, while maintaining access control rules.
  • system can create a browsing session and change the status of access control rules for a particular browsing session.
  • the session rules can be included in the computation of the rule status cascade, as described above.
  • peer-to-peer browsing converts the host device into an ad-hoc server for the purpose of data browsing.
  • FIG. 28 illustrates a peer-to-peer browsing session according to various embodiments.
  • FIG. 29 illustrates a process 2900 for initiating a peer-to-peer browsing session according to various embodiments.
  • the user For a user to initiate a peer-to-peer browsing session, the user is presented with the set of guests the user can invite to browse the user's own device (1). The user selects the desired guests then continues to examine the access control rules. The effective status of the access control rules for the new session is presented to the user allowing the user to change the effective status of each rule for that particular session (2). The user can active and deactivate each access control rule for that particular guest session (3). The user is presented with a UI allowing the user to see the currently active browsing sessions along with the guests participating in each session (4). The user can add guests, remove guests, and/or terminate the session via the UI.
  • the system can enforce restrictive access control for a user while browsing another user's device.
  • access may be controlled via a visual access code mechanism that makes it easier for the user to remember while providing enhanced security by increasing the possible combinations.
  • the visual access code mechanism is presented via a UI having two entry phases. The first entry phases requests the user to select one photo from a set of photos or images, which can be preconfigured by a system administrator.
  • the photo set can be the same for all users or be user-specific. Moreover, the photo set can be the same for all devices or be device-specific.
  • FIG. 30 illustrates a process 3000 for configuring a visual access code according to various embodiments.
  • the user can be presented with an image (e.g., photo) in step 1.
  • the user is required to select (e.g., by clicking, touching, gesturing, etc.) a subset of (e.g., four, five, etc.) hotspots from a group of possible (e.g., 16, 25, 36, etc.) hotspots.
  • the user can select a subset of 4 hotspots from 16 hotspots marked on the photo by touching the hotspots in any order.
  • the image can have an overlay or marking to make the hotspots visible and aid the user in selecting and recalling the selected hotspots.
  • the visual access code is composed of the photo index associated with the photo selected at step 1 and the subset of hotspots values selected at step 2.
  • the photo index and coordinates associated with the subset of hotspots can be stored as an encrypted digest.
  • a user's visual memory is employed to store and recall the registered image and the hotspots by using visual cues in the image.
  • Visual memory is a form of memory which preserves some characteristics of our senses pertaining to visual experience. Visual memory describes the relationship between perceptual processing and the encoding, storage and retrieval of the resulting neural representations. Visual memory occurs over time ranges from eye movements in order to visually navigate to a previously visited location. Visual access codes including a subset of hotspots on a registered image can stored longer and more readily recalled for providing authentication. Further, selecting trivial visual access codes is less likely than trivial alphanumeric passcodes (e.g., “1111,” “1234,” “password,” etc.) since available hotspots are different for each photo.
  • the user can place in memory visual information which resembles objects, places, animals or people as a mental image of the visual access code. The user can recall the visual access codes as patterns from long term visual memory using different areas of their prefrontal cortex and the anterior cingulate cortex
  • a system for visual access code process can include a first device configured to: present, to a user of the first device or service, a plurality of images; receive, from the user, a selection of a first of the plurality of images; receive, from the user, a selection of at least a first of a plurality of hotspots included in the first images; and generate a visual access code based at least in part on the selection of the first images and the first hotspot.
  • a registration process of the visual access process may include a first selection from a grid of images (e.g., photographs, pictures, complex shapes, images, etc.) and a second selection for a series of hotspot locations of the first selected image (e.g., pixel location, screen coordinates, overlay point, etc.)
  • the user may be presented with a grid of photographs depicting various landmarks, select a picture depicting a landmark from the grid. Then from the selected landmark picture, the user can select a series of hotspot locations on the selected picture.
  • the selected hotspot locations may be locations on the picture or image that correspond to different parts of a landmark, the background, border of the picture, etc.
  • the depicted features in the picture or image serve as visual clues to the user where the visual cues correspond to the selected hotspot locations. Accordingly, depicted features in the picture may be more readily stored in the user's memory than traditional alphanumerical combinations
  • each of the plurality of photographs can be associated with a corresponding index number.
  • each index number can include a globally unique photo identifier.
  • the first phase selection of the image from a grid of images can include additional pages of grids of multiple images. The user can scroll through multiple pages of grids of images to identify the registered image for the first phase image selection.
  • Each of the images in the multiple grids of images can include an index number based on the globally unique photo identifier of each image.
  • Each image can include a number of predefined hotspots for the user to select a subset of for the second phase set of hotspots.
  • each of the plurality of hotspots is associated with a corresponding hotspot identifier that can be cryptographically stored with the image index number.
  • each of the plurality of hotspots can be associated with a two dimensional coordinate of a corresponding pixel in the photograph.
  • the visual access code can be implemented by an authentication service, for example, on a destination device or service.
  • a website can replace a traditional alphanumeric login form with a visual access code two phase input method to authenticate the user.
  • the authentication service e.g., the mobile device or website
  • the authentication service can user various techniques for securely storing the user's visual access code to match subsequent entry of the visual access code to the stored registered access code. For example, after the user register's a visual access code, the authentication service can transform an identifier associated with the first phase and coordinates associated with the second phase into a text string and cryptographically store the string. Then, when the user re-visits the destination and enters the visual access code, the authentication service can decrypt the stored string to verify the user's authorization to access the destination.
  • the visual access code can be implemented by a client side visual authorization interface (VAI) that receives from the user a visual access code and outputs an alphanumeric password to various destinations.
  • VAI client side visual authorization interface
  • the VAI includes an algorithm that recreates the alphanumeric password based on the destination.
  • a user can access a website that uses a traditional alphanumeric login form with the VAI. To use the VAI with the destination, the user employs the VAI to set-up or registers the alphanumeric password.
  • the visual access code system can further include a system and method for entering visual access codes through on-screen virtual input mechanisms or visual authorization interface (VAI).
  • VAI acts as a client side visual password entry software that does not require support from other applications or websites.
  • the VAI present to the user a user interface that allows him to enter the password visually and then the software encodes such visual access code into regular alpha-numeric characters that are suitable for current websites and applications.
  • the system does not store the passwords anywhere and generates consistently them every time the user enters a visual access code.
  • a virtual keyboard may be VAI dedicated for entering visual access codes.
  • the VAI can perform client-side authentication for entry of visual passwords via the visual access code process.
  • the VAI presents the user with an interface to input the visual access code independent of the device hardware. That is, the VAI provides compatibility for secure authentication that does not require hardware, such as a fingerprint reader, and maintains integrity of the visual access codes independent of locally stored passwords.
  • the user may navigate to a reset my password form and launch the VAI to populate the alphanumeric password in the destination's password form.
  • the VAI will consistently regenerate the alphanumeric password for the destination based on the visual access code entered by the user.
  • the VAI generates a different alphanumeric password.
  • the VAI can authenticate a user using a visual access code compatible with the traditional destination login method. After the user uses the VAI to register with a destination, the destination stores the output of the VAI (e.g., an alphanumeric password).
  • the output of the VAI serves as a destination side authentication key while the visual access code is a client side authentication key.
  • the user can re-visits the destination, launch the VAI, enter the user's visual access code, and the VAI will output a passcode that matches the passcode previously stored by the destination.
  • the output of the VAI can be based a hex digest that uses a user's unique identifier, selection of a first image, a unique identifier of the first image, an image blending algorithm, selection of hotpots, shifting of hotspot coordinates, and/or one or more one-way cryptographic algorithms.
  • the user can be presented with the visual access code process for authenticating the user to the device or service. For example, the user may navigate to an access interface of the device or service, be presented with multiple photographs or image during a first phase of the visual access process. The user must recall the correct photograph previously selected during the registered process among multiple photographs presented. For example, the user may be presented with a grid of pictures depicting various famous landmarks. The user's registered image may be grouped with the same plurality of photographs from during the registration process or grouped with photographs that are different from the images presented during the registration process. The user first selects the registered image from the plurality of images. For example, the user selected image can be matched to the index number of the photograph's globally unique photo identifier.
  • FIG. 31 illustrates an example lock code management interface for use with visual access codes.
  • the lock code management interface 3100 allows the user to manage visual access codes, configure visual access code preferences, assign user profiles, etc.
  • the lock code management interface allows the user to configure different visual access codes based on an application category, such as media applications, financial applications, work applications, etc.
  • a master user can configure multiple visual access codes for different sub-users of a service or device. For example, a parent may configure guest visual access codes that allow children to access gaming applications. In another example, a spouse may configure a partner visual access code that allows the spouse's partner to access financial accounts but not social media or messaging accounts of the spouse.
  • FIG. 32 illustrates an example flow 3200 for configuring visual access codes according to an exemplary embodiment.
  • the process begins with gathering a unique identifier from the user.
  • the process 3200 uses a one-way cryptographic encoding to generate a consistent set of images for the user, as further described in reference to FIG. 33 .
  • the workflow 3200 proceeds to determine whether the user wants to set a master password, as further described in reference to FIG. 36 .
  • FIG. 33 illustrates an example process 3300 for mapping a user identifier to photo selection mapping according to various embodiments.
  • Process 3300 may begin with the user providing a unique identifier.
  • Process 3300 may determine a one-way cryptographic code and generate a unique user hex digest, as described in more detail in reference to FIGS. 35-36 .
  • unique user hex digest can be used to generate a unique list of photos and hotspots, as described in more detail in reference to FIGS. 40-41 .
  • process 3300 presents the list of photos to the user to register a passcode, as described in more detail in reference to FIGS. 37-39 .
  • the visual access code may be encoded using compact encoding.
  • each photo can have an index from 0 to 8 while each hotspot has an index from 0 to 15.
  • the correlation between the index that is assigned to the hotspot and the hotspot's position in the photo is photo dependent.
  • the index value associated with the hotspot is assigned randomly. Accordingly, random index assignments for the hotspot create secure access passcode.
  • the visual access code may be generated using positioned encoding based on each chosen hotspot's coordinates.
  • the coordinates of each hotspot may be fixed but those coordinates are not transportable from one photo to another.
  • Table 1 shows how a simple hotspot index encoding as described in the previous section is mapped to coordinate indices.
  • hotspot index 2 corresponds to values (140 and 59) in Photo 1 and (89 and 147) in Photo 2.
  • the value 2 is shared between photos but coordinate values for the same hotspot is not shared between photos.
  • the number of stored digits is also increased (e.g., 8 instead of 4 values).
  • positioned encoding generates even more possibilities and renders the corresponding visual access code even harder to break.
  • the visual access code can depend on the size and/or resolution of the photo. For instance, in a 500 ⁇ 500 pixel photo, each hotspot can generate a code from 0 to 499 in the horizontal axis and from 0 to 499 in the vertical axis. As such, 4 hotspots is equivalent to 8 digits, which gives 1.02432860e+17 possibilities. That number is further multiplied by the number of photos (e.g., 9), which yields ⁇ 9e+17 possibilities for a much stronger password than a conventional 8-character long alphanumeric case sensitive password with special characters (i.e., 2.02095455e+11 possibilities).
  • every photo can be associated with a globally unique identifier.
  • storing the passcode is dependent on the system or on the user. The user cannot select the same password for the two different systems. For example, a passcode of (0, 100, 101, 200, 201, 300, 301, 400, 401) that has been encoded using positioned coding corresponds to the first photo (i.e., photo 0).
  • the globally unique identifier could be a length value (e.g., a 16 characters long number) that is hard to guess.
  • FIG. 34 illustrates an example unique user hex digest, according to various embodiments.
  • the hex digest can include a photo selection index, a filter blending algorithm identifier, a final password mapping algorithm identifier, photo filter bitmap, and hotspot identifiers.
  • a unique user hex digest may begin with eight bytes designated for the photo selection index, followed by a one byte filter blending algorithm identifier, and an indicator for the final password mapping algorithm.
  • the photo filter bitmap may consist of 27 bytes.
  • the unique user hex digest can include 25 bytes for indicating hotspot shifting indices.
  • FIG. 35 illustrates an example registration process for assigning a visual access code according to an example embodiment.
  • the registration process 3500 may begin with a user navigating to a destination that requires authentication.
  • a virtual input (e.g., VAI) method may detect a traditional login form requiring a username and password. After the user types a username into a traditional login form, the registration process 3500 method may present a registration interface for assigning a visual access code to the destination using a hex digest. The user may proceed via the VAI with selecting a first image and series of hotspots to register a new visual access code for the destination, as described above.
  • VAI virtual input
  • the VAI can regenerate a password based on the hex digest to match a stored password with the destination.
  • the registration process 3500 may proceed to completing the traditional login form with the password based on the hex digest.
  • registration process 3500 can generate the alphanumeric password using one-way cryptographic encoding and seeding the password with a destination identifier.
  • a password seed is used to feed the one-way cryptographic algorithm prior to generating an alphanumeric password, as described in reference to FIG. 36 .
  • the seed has a direct one-to-one mapping between the user's selected hotspots and the seed.
  • the process generates a password seeded with the application/website destination name before the one-way cryptographic such that different application/websites destinations have different passwords even when the same visual passcode is used. Therefore, the user's system does not need to store the passwords for each site since the access code process consistently replicates generation of the password based on the user input.
  • the algorithm for setting up a visual access code can include
  • the VAI may determine the associated visual access code and present the user with the virtual input method to authenticate the user.
  • the system may populate the traditional login form with the assigned alphanumeric password to complete destination authentication.
  • FIG. 36 illustrates an example process 3600 for encoding a visual access code with a password according to an example embodiment.
  • the process 3600 may input photo SHA-512 has, sorted hotspot positions by the X axis, and an application/website destination identifier to generate a passcode seed.
  • the process 3600 uses the encoded passcode with a one-way cryptographic algorithm to generate the hex digest that is used to map an alphanumeric password.
  • the resulting password will generate a secure ASCII password that can include upper case and lower case English alphabet and numbers as well as special characters.
  • the algorithm can include:
  • FIG. 37 illustrates an example mobile interface in accordance with various embodiments.
  • the user begins by selecting a unique passphrase such as their name, a mother's maiden name, birthdate, or favorite location, etc.
  • Step two the system generates a set of images for the user to select a registration image.
  • the algorithm can include:
  • FIG. 38 illustrates an example implementation of entering a visual access code according to an example embodiment.
  • the user navigates to a destination website or application or login screen that requires authentication.
  • the VAI can detect the destination's authentication form and retrieves the visual access code associated with the destinations identifier.
  • the VAI presents the user with a set of images that includes the image previously registered by the user for the destination.
  • the VAI proceeds to step two to present the user hotspots of the registered image.
  • the registered image may be of a house in include 16 possible hotspots.
  • the visual input method proceeds to step three.
  • the user may identify four hotspots by touching different locations on the image that correspond to different parts of the house depicted that match the hotspots user selected during the registration process.
  • the virtual input method inputs the alphanumeric password stored with the visual access code into the destination's authentication form.
  • the virtual input method may present a confirmation message that the user has successfully input the visual access code. The user may proceed by clicking on the destination's authentication form to complete logging in without having to type an alphanumeric password.
  • FIGS. 39A-C illustrate an example implementations of the virtual input method on a website according to various embodiments.
  • FIG. 39A illustrates a first phase for the virtual input method that present the user with several images that include a previously registered image. The user may identify the previously registered image by clicking or touching the registered image. In response to the user selecting the image that matches the registered image, the user may be presented with the registered image and instructed to identify a series of hotspots the registered image
  • the user is presented a hotspot selection screen for the virtual input method.
  • the interface can present the user with several hotspots for the selected image.
  • the visual access code process may authenticate the user to the device or service. For example, the user can click or touch four locations on the picture that correspond to the hotspot locations the user selected during the registration process.
  • the visual access process determines if the user selected locations satisfy the corresponding index numbers stored during the registration process.
  • the visual access process determines if the user selected locations satisfy the two dimensional coordinate of a corresponding pixel in the photograph from the registration process.
  • the series of hotspots on the registered image may be identified in the same order as the hotspots were selected during the registration process.
  • the series of hotspots on the registered image may be identified in the any order to satisfy the second phase of the visual access code process for authenticating the user. Since permutations of locations and visual cues are greatly increased over traditional alphanumerical combinations, the user first recalling the correct image and then identifying the series of hotspot locations on the correct image may be sufficient for authenticating the user.
  • the user may be presented with a non-matching image and instructed to identify a series of hotspots the non-matching image.
  • the visual access code process can provide or suppress feedback to the user regarding the first selection of an image from the plurality of images. Accordingly, an unauthorized user may not be notified whether the first selection of an image or second identification of the series of hotspots failed to satisfy the visual access code process. Repeated attempts indicating a guessing of different combinations of images from the group of images and hotspot locations may be then be detected as a brute force attack.
  • the virtual input method can proceed to FIG. 39C to present the user with a confirmation and populate an alphanumeric final password in the destination's login form
  • the process can employ modified images that are unique for each user that appear visually indiscernible.
  • the process can include shifting center point of the original image to modify the coordinates of the hotspots and add a blended texture secure the image data
  • FIG. 40 illustrates an image blending process in accordance with various embodiments. Blending photos for each user provides different password for each user that are not detectable by simply looking observing a user's image selection.
  • the system combines a texture masking with an original photo via a blending algorithm creates a modified photo for generating secure visual access codes.
  • An example photo blending algorithm can include:
  • the system selects a blending algorithm and the photo filter bitmap based on data stored in the unique user hex digest.
  • the hex digest can include value to indicate a simple overlap blending algorithm for creating the modified photo.
  • Multiple blending and password mapping algorithms improves security of the access code.
  • FIG. 41 illustrates an example implementation of hotspot position shifting in accordance with various embodiments. By shifting the center point of the original photo coordinates, the location values of the hotspots the image can be changed to generate different passwords for each user using the same visual image and maintaining the visual appearance of the image
  • An example hotspot shifting algorithm can include:
  • the hex digest can provide a shift value for shifting the hotspot center and thereby differentiating the hotspot coordinates for the image for the user. For example, an original hotspot center with location coordinates 30, 50 when shifted based on a shift value of 9 in the unique user hex digest shifts the hotspot center to location coordinates 301, 49.
  • FIG. 42 is a block diagram illustrating wired or wireless system 550 according to various embodiments.
  • the system 550 may be used to implement the media platform 120 .
  • the system 550 can be a conventional personal computer, computer server, personal digital assistant, smart phone, tablet computer, or any other processor enabled device that is capable of wired or wireless data communication.
  • Other computer systems and/or architectures may be also used, as will be clear to those skilled in the art.
  • the system 550 preferably includes one or more processors, such as processor 560 .
  • Additional processors may be provided, such as an auxiliary processor to manage input/output, an auxiliary processor to perform floating point mathematical operations, a special-purpose microprocessor having an architecture suitable for fast execution of signal processing algorithms (e.g., digital signal processor), a slave processor subordinate to the main processing system (e.g., back-end processor), an additional microprocessor or controller for dual or multiple processor systems, or a coprocessor.
  • auxiliary processors may be discrete processors or may be integrated with the processor 560 .
  • the processor 560 is preferably connected to a communication bus 555 .
  • the communication bus 555 may include a data channel for facilitating information transfer between storage and other peripheral components of the system 550 .
  • the communication bus 555 further may provide a set of signals used for communication with the processor 560 , including a data bus, address bus, and control bus (not shown).
  • the communication bus 555 may comprise any standard or non-standard bus architecture such as, for example, bus architectures compliant with industry standard architecture (“ISA”), extended industry standard architecture (“EISA”), Micro Channel Architecture (“MCA”), peripheral component interconnect (“PCI”) local bus, or standards promulgated by the Institute of Electrical and Electronics Engineers (“IEEE”) including IEEE 488 general-purpose interface bus (“GPIB”), IEEE 696/S-100, and the like.
  • ISA industry standard architecture
  • EISA extended industry standard architecture
  • MCA Micro Channel Architecture
  • PCI peripheral component interconnect
  • IEEE Institute of Electrical and Electronics Engineers
  • IEEE Institute of Electrical and Electronics Engineers
  • IEEE Institute of Electrical and Electronics Engineers
  • IEEE Institute of Electrical and Electronics Engineers
  • GPIB general-purpose interface bus
  • IEEE 696/S-100 IEEE 696/S-100
  • the System 550 preferably includes a main memory 565 and may also include a secondary memory 570 .
  • the main memory 565 provides storage of instructions and data for programs executing on the processor 560 .
  • the main memory 565 is typically semiconductor-based memory such as dynamic random access memory (“DRAM”) and/or static random access memory (“SRAM”).
  • DRAM dynamic random access memory
  • SRAM static random access memory
  • Other semiconductor-based memory types include, for example, synchronous dynamic random access memory (“SDRAM”), Rambus dynamic random access memory (“RDRAM”), ferroelectric random access memory (“FRAM”), and the like, including read only memory (“ROM”).
  • SDRAM synchronous dynamic random access memory
  • RDRAM Rambus dynamic random access memory
  • FRAM ferroelectric random access memory
  • ROM read only memory
  • the secondary memory 570 may optionally include an internal memory 575 and/or a removable medium 580 , for example a floppy disk drive, a magnetic tape drive, a compact disc (“CD”) drive, a digital versatile disc (“DVD”) drive, etc.
  • the removable medium 580 is read from and/or written to in a well-known manner.
  • Removable storage medium 580 may be, for example, a floppy disk, magnetic tape, CD, DVD, SD card, etc.
  • the removable storage medium 580 is a non-transitory computer readable medium having stored thereon computer executable code (i.e., software) and/or data.
  • the computer software or data stored on the removable storage medium 580 is read into the system 550 for execution by the processor 560 .
  • the secondary memory 570 may include other similar means for allowing computer programs or other data or instructions to be loaded into the system 550 .
  • Such means may include, for example, an external storage medium 595 and a communication interface 590 .
  • external storage medium 595 may include an external hard disk drive or an external optical drive, or and external magneto-optical drive.
  • secondary memory 570 may include semiconductor-based memory such as programmable read-only memory (“PROM”), erasable programmable read-only memory (“EPROM”), electrically erasable read-only memory (“EEPROM”), or flash memory (block oriented memory similar to EEPROM). Also included are the removable medium 580 and a communication interface, which allow software and data to be transferred from an external storage medium 595 to the system 550 .
  • PROM programmable read-only memory
  • EPROM erasable programmable read-only memory
  • EEPROM electrically erasable read-only memory
  • flash memory block oriented memory similar to EEPROM
  • the System 550 may also include an input/output (“I/O”) interface 585 .
  • the I/O interface 585 facilitates input from and output to external devices.
  • the I/O interface 585 may receive input from a keyboard or mouse and may provide output to a display.
  • the I/O interface 585 is capable of facilitating input from and output to various alternative types of human interface and machine interface devices alike.
  • System 550 may also include a communication interface 590 .
  • the communication interface 590 allows software and data to be transferred between system 550 and external devices (e.g., printers, networks, information sources, etc.). For example, computer software or executable code may be transferred to system 550 from a network server via communication interface 590 .
  • Examples of communication interface 590 include a modem, a network interface card (“NIC”), a wireless data card, a communications port, a PCMCIA slot and card, an infrared interface, and an IEEE 1394 fire-wire, just to name a few.
  • Communication interface 590 preferably implements industry promulgated protocol standards, such as Ethernet IEEE 802 standards, Fiber Channel, digital subscriber line (“DSL”), asynchronous digital subscriber line (“ADSL”), frame relay, asynchronous transfer mode (“ATM”), integrated digital services network (“ISDN”), personal communications services (“PCS”), transmission control protocol/Internet protocol (“TCP/IP”), serial line Internet protocol/point to point protocol (“SLIP/PPP”), and so on, but may also implement customized or non-standard interface protocols as well.
  • industry promulgated protocol standards such as Ethernet IEEE 802 standards, Fiber Channel, digital subscriber line (“DSL”), asynchronous digital subscriber line (“ADSL”), frame relay, asynchronous transfer mode (“ATM”), integrated digital services network (“ISDN”), personal communications services (“PCS”), transmission control protocol/Internet protocol (“TCP/IP”), serial line Internet protocol/point to point protocol (“SLIP/PPP”), and so on, but may also implement customized or non-standard interface protocols as well.
  • the electrical communication signals 605 are preferably provided to communication interface 590 via a communication channel 600 .
  • the communication channel 600 may be a wired or wireless network, or any variety of other communication links.
  • Communication channel 600 carries the electrical communication signals 605 and can be implemented using a variety of wired or wireless communication means including wire or cable, fiber optics, conventional phone line, cellular phone link, wireless data communication link, radio frequency (“RF”) link, or infrared link, just to name a few.
  • RF radio frequency
  • Computer executable code i.e., computer programs or software
  • main memory 565 and/or the secondary memory 570 Computer programs can also be received via communication interface 590 and stored in the main memory 565 and/or the secondary memory 570 .
  • Such computer programs when executed, enable the system 550 to perform the various functions of the present invention as previously described.
  • computer readable medium is used to refer to any non-transitory computer readable storage media used to provide computer executable code (e.g., software and computer programs) to the system 550 .
  • Examples of the media include main memory 565 , secondary memory 570 (including internal memory 575 , removable medium 580 , and external storage medium 595 ), and any peripheral device communicatively coupled with communication interface 590 (including a network information server or other network device).
  • These non-transitory computer readable mediums are means for providing executable code, programming instructions, and software to the system 550 .
  • the software may be stored on a computer readable medium and loaded into the system 550 by way of removable medium 580 , I/O interface 585 , or communication interface 590 .
  • the software is loaded into the system 550 in the form of electrical communication signals 605 .
  • the software when executed by the processor 560 , preferably causes the processor 560 to perform the inventive features and functions previously described herein.
  • the system 550 also includes optional wireless communication components that facilitate wireless communication over a voice and over a data network.
  • the wireless communication components comprise an antenna system 610 , a radio system 615 and a baseband system 620 .
  • RF radio frequency
  • the antenna system 610 may comprise one or more antennae and one or more multiplexors (not shown) that perform a switching function to provide the antenna system 610 with transmit and receive signal paths.
  • received RF signals can be coupled from a multiplexor to a low noise amplifier (not shown) that amplifies the received RF signal and sends the amplified signal to the radio system 615 .
  • the radio system 615 may comprise one or more radios that are configured to communicate over various frequencies.
  • the radio system 615 may combine a demodulator (not shown) and modulator (not shown) in one integrated circuit (“IC”).
  • the demodulator and modulator can also be separate components. In the incoming path, the demodulator strips away the RF carrier signal leaving a baseband receive audio signal, which is sent from the radio system 615 to the baseband system 620 .
  • baseband system 620 decodes the signal and converts it to an analog signal. Then the signal is amplified and sent to a speaker.
  • the baseband system 620 also receives analog audio signals from a microphone. These analog audio signals are converted to digital signals and encoded by the baseband system 620 .
  • the baseband system 620 also codes the digital signals for transmission and generates a baseband transmit audio signal that is routed to the modulator portion of the radio system 615 .
  • the modulator mixes the baseband transmit audio signal with an RF carrier signal generating an RF transmit signal that is routed to the antenna system and may pass through a power amplifier (not shown).
  • the power amplifier amplifies the RF transmit signal and routes it to the antenna system 610 where the signal is switched to the antenna port for transmission.
  • the baseband system 620 is also communicatively coupled with the processor 560 .
  • the processor 560 has access to one or more data storage areas including, for example, but not limited to, the main memory 565 and the secondary memory 570 .
  • the processor 560 is preferably configured to execute instructions (i.e., computer programs or software) that can be stored in the main memory 565 or in the secondary memory 570 .
  • Computer programs can also be received from the baseband processor 610 and stored in the main memory 565 or in the secondary memory 570 , or executed upon receipt. Such computer programs, when executed, enable the system 550 to perform the various functions of the present invention as previously described.
  • the main memory 565 may include various software modules (not shown) that are executable by processor 560 .
  • ASICs application specific integrated circuits
  • FPGAs field programmable gate arrays
  • ASICs application specific integrated circuits
  • FPGAs field programmable gate arrays
  • DSP digital signal processor
  • a general-purpose processor can be a microprocessor, but in the alternative, the processor can be any processor, controller, microcontroller, or state machine.
  • a processor can also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.
  • a software module can reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium including a network storage medium.
  • An exemplary storage medium can be coupled to the processor such the processor can read information from, and write information to, the storage medium.
  • the storage medium can be integral to the processor.
  • the processor and the storage medium can also reside in an ASIC.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Databases & Information Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Software Systems (AREA)
  • Human Computer Interaction (AREA)
  • Computer Security & Cryptography (AREA)
  • Computer Hardware Design (AREA)
  • Library & Information Science (AREA)
  • Multimedia (AREA)
  • Computational Linguistics (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Fuzzy Systems (AREA)
  • Artificial Intelligence (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Bioethics (AREA)
  • Automation & Control Theory (AREA)
  • Probability & Statistics with Applications (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • User Interface Of Digital Computer (AREA)
  • Technology Law (AREA)
US15/410,658 2016-01-19 2017-01-19 Semantic indexing by augmented object association Abandoned US20170206225A1 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
US15/410,658 US20170206225A1 (en) 2016-01-19 2017-01-19 Semantic indexing by augmented object association
US15/952,533 US20180278600A1 (en) 2016-01-19 2018-04-13 Multi-factor masked access control system

Applications Claiming Priority (5)

Application Number Priority Date Filing Date Title
US201662280647P 2016-01-19 2016-01-19
US201662280674P 2016-01-19 2016-01-19
US201662280671P 2016-01-19 2016-01-19
US201662280653P 2016-01-19 2016-01-19
US15/410,658 US20170206225A1 (en) 2016-01-19 2017-01-19 Semantic indexing by augmented object association

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
US15/410,660 Continuation-In-Part US10621225B2 (en) 2016-01-19 2017-01-19 Hierarchical visual faceted search engine

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US15/410,654 Continuation-In-Part US10515111B2 (en) 2016-01-19 2017-01-19 Object stamping user interface

Publications (1)

Publication Number Publication Date
US20170206225A1 true US20170206225A1 (en) 2017-07-20

Family

ID=59314076

Family Applications (9)

Application Number Title Priority Date Filing Date
US15/410,679 Abandoned US20170208069A1 (en) 2016-01-19 2017-01-19 Masking restrictive access control in a networked environment
US15/410,668 Abandoned US20170205968A1 (en) 2016-01-19 2017-01-19 Multimode user interface
US15/410,671 Active US11093543B2 (en) 2016-01-19 2017-01-19 Masking restrictive access control system
US15/410,648 Active 2039-12-01 US11436274B2 (en) 2016-01-19 2017-01-19 Visual access code
US15/410,675 Expired - Fee Related US10614119B2 (en) 2016-01-19 2017-01-19 Masking restrictive access control for a user on multiple devices
US15/410,654 Expired - Fee Related US10515111B2 (en) 2016-01-19 2017-01-19 Object stamping user interface
US15/410,660 Expired - Fee Related US10621225B2 (en) 2016-01-19 2017-01-19 Hierarchical visual faceted search engine
US15/410,666 Expired - Fee Related US10747808B2 (en) 2016-01-19 2017-01-19 Hybrid in-memory faceted engine
US15/410,658 Abandoned US20170206225A1 (en) 2016-01-19 2017-01-19 Semantic indexing by augmented object association

Family Applications Before (8)

Application Number Title Priority Date Filing Date
US15/410,679 Abandoned US20170208069A1 (en) 2016-01-19 2017-01-19 Masking restrictive access control in a networked environment
US15/410,668 Abandoned US20170205968A1 (en) 2016-01-19 2017-01-19 Multimode user interface
US15/410,671 Active US11093543B2 (en) 2016-01-19 2017-01-19 Masking restrictive access control system
US15/410,648 Active 2039-12-01 US11436274B2 (en) 2016-01-19 2017-01-19 Visual access code
US15/410,675 Expired - Fee Related US10614119B2 (en) 2016-01-19 2017-01-19 Masking restrictive access control for a user on multiple devices
US15/410,654 Expired - Fee Related US10515111B2 (en) 2016-01-19 2017-01-19 Object stamping user interface
US15/410,660 Expired - Fee Related US10621225B2 (en) 2016-01-19 2017-01-19 Hierarchical visual faceted search engine
US15/410,666 Expired - Fee Related US10747808B2 (en) 2016-01-19 2017-01-19 Hybrid in-memory faceted engine

Country Status (5)

Country Link
US (9) US20170208069A1 (ja)
EP (1) EP3405903A4 (ja)
JP (1) JP7178904B2 (ja)
CN (1) CN108701194B (ja)
WO (1) WO2017127576A1 (ja)

Families Citing this family (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108701194B (zh) 2016-01-19 2022-06-24 雷韦兹公司 掩蔽限制访问控制系统
US10515063B1 (en) 2016-12-19 2019-12-24 Cboe Exchange, Inc. System and method for real-time data acquisition and display
US10289708B2 (en) * 2017-04-05 2019-05-14 Amadeus S.A.S. Management of unique identifiers in a database
US10333913B2 (en) * 2017-05-15 2019-06-25 International Business Machines Corporation Dynamic object passwords
US11055674B1 (en) * 2018-01-18 2021-07-06 Wells Fargo Bank, N.A. Systems and methods for check masking and interdiction
US10796016B2 (en) * 2018-03-28 2020-10-06 Visa International Service Association Untethered resource distribution and management
CN112689980A (zh) * 2018-04-13 2021-04-20 莱格维兹股份有限公司 单设备多因素认证系统
US11308059B2 (en) * 2018-06-12 2022-04-19 Chicago Mercantile Exchange Inc. Optimized data structure
US11176315B2 (en) * 2019-05-15 2021-11-16 Elsevier Inc. Comprehensive in-situ structured document annotations with simultaneous reinforcement and disambiguation
US11036724B2 (en) * 2019-09-04 2021-06-15 Microsoft Technology Licensing, Llc Interactive visual search engine
EP3907594A1 (en) * 2020-05-06 2021-11-10 Mastercard International Incorporated User input mechanism for reordering graphical elements
CN113268964B (zh) * 2020-07-23 2022-01-18 腾讯科技（深圳）有限公司 一种表格编辑方法、装置、设备及存储介质
US11507611B2 (en) * 2020-10-30 2022-11-22 Dell Products L.P. Personalizing unstructured data according to user permissions
CN112965987A (zh) * 2021-03-31 2021-06-15 华申数科(北京)信息科技有限责任公司 一种数字新治理业务的带权限高效模糊检索的方法及应用
US20240037102A1 (en) * 2022-08-01 2024-02-01 Motorola Solutions, Inc. Method and apparatus for securing databases

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090307261A1 (en) * 2008-06-05 2009-12-10 Apple Inc. Categorization of Digital Media Based on Media Characteristics
US7636733B1 (en) * 2003-10-03 2009-12-22 Adobe Systems Incorporated Time-based image management
US20100063961A1 (en) * 2008-09-05 2010-03-11 Fotonauts, Inc. Reverse Tagging of Images in System for Managing and Sharing Digital Images
US20150261789A1 (en) * 2012-03-26 2015-09-17 Amazon Technologies, Inc. Cloud-based photo management

Family Cites Families (155)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5315657A (en) 1990-09-28 1994-05-24 Digital Equipment Corporation Compound principals in access control lists
US5220604A (en) * 1990-09-28 1993-06-15 Digital Equipment Corporation Method for performing group exclusion in hierarchical group structures
US6947959B1 (en) 1992-10-01 2005-09-20 Quark, Inc. Digital media asset management system and process
US5560005A (en) * 1994-02-25 1996-09-24 Actamed Corp. Methods and systems for object-based relational distributed databases
US5983237A (en) 1996-03-29 1999-11-09 Virage, Inc. Visual dictionary
US6085191A (en) * 1997-10-31 2000-07-04 Sun Microsystems, Inc. System and method for providing database access control in a secure distributed network
US6158010A (en) 1998-10-28 2000-12-05 Crosslogix, Inc. System and method for maintaining security in a distributed computer network
US6630928B1 (en) 1999-10-01 2003-10-07 Hewlett-Packard Development Company, L.P. Method and apparatus for touch screen data entry
US6757680B1 (en) 2000-07-03 2004-06-29 International Business Machines Corporation System and method for inheriting access control rules
US9032097B2 (en) 2001-04-26 2015-05-12 Nokia Corporation Data communication with remote network node
US6804684B2 (en) 2001-05-07 2004-10-12 Eastman Kodak Company Method for associating semantic information with multiple images in an image database environment
US20030046586A1 (en) 2001-09-05 2003-03-06 Satyam Bheemarasetti Secure remote access to data between peers
US7325248B2 (en) 2001-11-19 2008-01-29 Stonesoft Corporation Personal firewall with location dependent functionality
WO2003058877A1 (en) 2001-12-28 2003-07-17 Woodstock Systems, Llc Personal digital servertm (pdstm)
US7076558B1 (en) 2002-02-27 2006-07-11 Microsoft Corporation User-centric consent management system and method
US20040044776A1 (en) 2002-03-22 2004-03-04 International Business Machines Corporation Peer to peer file sharing system using common protocols
US7043474B2 (en) 2002-04-15 2006-05-09 International Business Machines Corporation System and method for measuring image similarity based on semantic meaning
US6946715B2 (en) 2003-02-19 2005-09-20 Micron Technology, Inc. CMOS image sensor and method of fabrication
US7275063B2 (en) 2002-07-16 2007-09-25 Horn Bruce L Computer system for automatic organization, indexing and viewing of information from multiple sources
US20040174434A1 (en) 2002-12-18 2004-09-09 Walker Jay S. Systems and methods for suggesting meta-information to a camera user
US7644433B2 (en) 2002-12-23 2010-01-05 Authernative, Inc. Authentication system and method based upon random partial pattern recognition
KR100436667B1 (ko) * 2003-02-20 2004-06-22 브이알 포토 주식회사 다중 웹 서버에서의 다중 해상도 이미지 제공 어플리케이션 공급 서비스 방법 및 장치
US7421438B2 (en) 2004-04-29 2008-09-02 Microsoft Corporation Metadata editing control
US7900240B2 (en) 2003-05-28 2011-03-01 Citrix Systems, Inc. Multilayer access control security system
US7188254B2 (en) * 2003-08-20 2007-03-06 Microsoft Corporation Peer-to-peer authorization method
US7555497B2 (en) * 2003-08-21 2009-06-30 Microsoft Corporation Systems and methods for separating units of information manageable by a hardware/software interface system from their physical organization
US7436770B2 (en) 2004-01-21 2008-10-14 Alcatel Lucent Metering packet flows for limiting effects of denial of service attacks
JP4665406B2 (ja) 2004-02-23 2011-04-06 日本電気株式会社 アクセス制御管理方法、アクセス制御管理システムおよびアクセス制御管理機能付き端末装置
US7853663B2 (en) 2004-03-12 2010-12-14 Riip, Inc. Wireless management system for control of remote devices
US20120113273A1 (en) * 2004-11-29 2012-05-10 Ariel Inventions Llc System, Method, and Devices for Searching for a Digital Image over a Communication Network
US20060136246A1 (en) 2004-12-22 2006-06-22 Tu Edgar A Hierarchical program guide
US7802294B2 (en) * 2005-01-28 2010-09-21 Microsoft Corporation Controlling computer applications' access to data
US8561147B2 (en) 2005-04-19 2013-10-15 Lg Electronics Inc. Method and apparatus for controlling of remote access to a local network
US8365254B2 (en) * 2005-06-23 2013-01-29 Microsoft Corporation Unified authorization for heterogeneous applications
US20070002367A1 (en) 2005-06-29 2007-01-04 Eric Yuan Methods and apparatuses for selectively controlling a remote device
KR101054698B1 (ko) 2005-07-01 2011-08-08 인터내셔널 비지네스 머신즈 코포레이션 네트워크에 부착된 물리포트의 가상 인스턴스를 관리하는방법 및 시스템
CA2615659A1 (en) * 2005-07-22 2007-05-10 Yogesh Chunilal Rathod Universal knowledge management and desktop search system
US8832047B2 (en) * 2005-07-27 2014-09-09 Adobe Systems Incorporated Distributed document version control
US7831913B2 (en) 2005-07-29 2010-11-09 Microsoft Corporation Selection-based item tagging
AU2005100653A4 (en) 2005-08-12 2005-09-15 Agent Mobile Pty Ltd Mobile Device-Based End-User Filter
US8230491B2 (en) 2005-08-19 2012-07-24 Opnet Technologies, Inc. Automatic access to network devices using various authentication schemes
WO2007027153A1 (en) 2005-09-01 2007-03-08 Encentuate Pte Ltd Portable authentication and access control involving multiples identities
US7617541B2 (en) 2005-09-09 2009-11-10 Netapp, Inc. Method and/or system to authorize access to stored data
US7925666B1 (en) 2005-10-06 2011-04-12 Hewlett-Packard Development Company, L.P. System and method for managing the application of access control lists on network devices
US9141825B2 (en) 2005-11-18 2015-09-22 Qurio Holdings, Inc. System and method for controlling access to assets in a network-based media sharing system using tagging
US7945653B2 (en) * 2006-10-11 2011-05-17 Facebook, Inc. Tagging digital media
US8621549B2 (en) 2005-12-29 2013-12-31 Nextlabs, Inc. Enforcing control policies in an information management system
US8607149B2 (en) 2006-03-23 2013-12-10 International Business Machines Corporation Highlighting related user interface controls
US20070244896A1 (en) 2006-04-14 2007-10-18 Dell Products L.P. System and method for authenticating remote users
JP2007328623A (ja) 2006-06-08 2007-12-20 Fuji Xerox Co Ltd アクセス制御装置
US8732854B2 (en) 2006-11-01 2014-05-20 Time Warner Cable Enterprises Llc Methods and apparatus for premises content distribution
WO2008092263A1 (en) 2007-01-31 2008-08-07 Binary Monkeys, Inc. Method and apparatus for network authentication of human interaction and user identity
US8041643B2 (en) 2007-03-09 2011-10-18 At&T Intellectual Property I, L.P. System and method of providing media content
US8505071B2 (en) 2007-05-30 2013-08-06 Disney Enterprises, Inc. Preventing automated programs and unauthorized users in a network
US8239351B2 (en) 2007-06-07 2012-08-07 Apple Inc. Methods and systems for managing permissions data
US20080307362A1 (en) 2007-06-08 2008-12-11 Apple Inc. Desktop Filter
US8561148B2 (en) 2008-06-26 2013-10-15 Citrix Systems, Inc. Methods and systems for interactive evaluation using dynamically generated, interactive resultant sets of policies
US20110202982A1 (en) 2007-09-17 2011-08-18 Vidoop, Llc Methods And Systems For Management Of Image-Based Password Accounts
US8037425B2 (en) 2007-12-14 2011-10-11 Scenera Technologies, Llc Methods, systems, and computer readable media for controlling presentation and selection of objects that are digital images depicting subjects
US8176562B1 (en) 2007-12-21 2012-05-08 Symantec Corporation Privacy protection during remote administration
US20090198732A1 (en) 2008-01-31 2009-08-06 Realnetworks, Inc. Method and system for deep metadata population of media content
EP2255496B1 (en) 2008-03-14 2016-02-10 Telefonaktiebolaget L M Ericsson (publ) Method and apparatus for remote access to a local network
JP5043748B2 (ja) 2008-05-19 2012-10-10 キヤノン株式会社 コンテンツ管理装置、コンテンツ管理装置の制御方法、プログラム及び記録媒体
WO2010001406A1 (en) * 2008-07-04 2010-01-07 Yogesh Chunilal Bathod Methods and systems for brands social networks (bsn) platform
US8805110B2 (en) 2008-08-19 2014-08-12 Digimarc Corporation Methods and systems for content processing
US8589402B1 (en) 2008-08-21 2013-11-19 Adobe Systems Incorporated Generation of smart tags to locate elements of content
US20100107125A1 (en) 2008-10-24 2010-04-29 Microsoft Corporation Light Box for Organizing Digital Images
US20100115472A1 (en) * 2008-10-30 2010-05-06 Lee Kun-Bin Method of Facilitating Browsing and Management of Multimedia Files with Data Structure thereof
CN101431472B (zh) 2008-12-15 2011-05-25 中兴通讯股份有限公司 媒体访问控制条目的处理方法和装置
US20100186074A1 (en) * 2009-01-16 2010-07-22 Angelos Stavrou Authentication Using Graphical Passwords
CN102292722B (zh) 2009-01-21 2014-09-03 瑞典爱立信有限公司 基于多模元数据和结构化语义描述符来产生注释标签
US9351193B2 (en) 2009-01-28 2016-05-24 Headwater Partners I Llc Intermediate networking devices
US20120143931A1 (en) 2009-02-17 2012-06-07 Tagle Information Technology Inc. Context-aware folders
US20100287382A1 (en) 2009-05-07 2010-11-11 John Charles Gyorffy Two-factor graphical password for text password and encryption key generation
EP3260969B1 (en) 2009-09-22 2021-03-03 Apple Inc. Device, method, and graphical user interface for manipulating user interface objects
US8869296B2 (en) * 2009-11-06 2014-10-21 Verizon Patent And Licensing Inc. Access to user information
US8645287B2 (en) 2010-02-04 2014-02-04 Microsoft Corporation Image tagging based upon cross domain context
US8983210B2 (en) 2010-03-01 2015-03-17 Microsoft Corporation Social network system and method for identifying cluster image matches
US8266176B2 (en) 2010-03-12 2012-09-11 Hitachi, Ltd. Storage system and file access determination method of the same
EP2556625B1 (en) 2010-04-06 2020-08-19 Samsung Electronics Co., Ltd Method and apparatus for managing remote access authority in upnp remote access service
US8402516B2 (en) * 2010-05-06 2013-03-19 Jonathan Weizman Apparatus and method for establishing a peer-to-peer communication session with a host device
US8555365B2 (en) 2010-05-21 2013-10-08 Barracuda Networks, Inc. Directory authentication method for policy driven web filtering
US10382438B2 (en) 2010-05-27 2019-08-13 Nokia Technologies Oy Method and apparatus for expanded content tag sharing
US20110314071A1 (en) 2010-06-17 2011-12-22 Openwave Systems Inc. Metadata-based data access and control
US8774528B2 (en) 2010-09-24 2014-07-08 Kodak Alaris Inc. Method of selecting important digital images
US8935792B1 (en) 2010-10-05 2015-01-13 Mcafee, Inc. System, method, and computer program product for conditionally performing an action based on an attribute
US8893215B2 (en) 2010-10-29 2014-11-18 Nokia Corporation Method and apparatus for providing distributed policy management
WO2012061824A1 (en) * 2010-11-05 2012-05-10 Myspace, Inc. Image auto tagging method and application
EP2458548A1 (en) 2010-11-30 2012-05-30 France Telecom System and method for implementing dynamic access control rules to personal cloud information
US20120143894A1 (en) 2010-12-02 2012-06-07 Microsoft Corporation Acquisition of Item Counts from Hosted Web Services
US9811528B2 (en) 2010-12-09 2017-11-07 Yosef Benraz Systems and methods for providing continuing access to a remote computer program
US9892279B2 (en) 2010-12-22 2018-02-13 Koninklijke Philips N.V. Creating an access control policy based on consumer privacy preferences
US9218364B1 (en) * 2011-01-28 2015-12-22 Yahoo! Inc. Monitoring an any-image labeling engine
US8983985B2 (en) 2011-01-28 2015-03-17 International Business Machines Corporation Masking sensitive data of table columns retrieved from a database
US9588644B2 (en) 2011-03-17 2017-03-07 Red Hat, Inc. Time-based organization of desktop items
US8769668B2 (en) 2011-05-09 2014-07-01 Blackberry Limited Touchscreen password entry
US8918849B2 (en) 2011-05-12 2014-12-23 Konvax Corporation Secure user credential control
AU2011202415B1 (en) 2011-05-24 2012-04-12 Microsoft Technology Licensing, Llc Picture gesture authentication
AU2012272977A1 (en) 2011-06-20 2014-01-16 Tandemseven, Inc. System and method for building and managing user experience for computer software interfaces
JP5724716B2 (ja) 2011-07-25 2015-05-27 大日本印刷株式会社 携帯情報端末
US9659164B2 (en) 2011-08-02 2017-05-23 Qualcomm Incorporated Method and apparatus for using a multi-factor password or a dynamic password for enhanced security on a device
US20130097697A1 (en) 2011-10-14 2013-04-18 Microsoft Corporation Security Primitives Employing Hard Artificial Intelligence Problems
US9116893B2 (en) 2011-10-21 2015-08-25 Qualcomm Incorporated Network connected media gateway for communication networks
US8861347B2 (en) 2011-12-04 2014-10-14 Mellanox Technologies Ltd. Configurable access control lists using TCAM
AU2012347452A1 (en) * 2011-12-09 2014-06-26 Echarge2 Corporation Systems and methods for using cipher objects to protect data
US8813172B2 (en) * 2011-12-16 2014-08-19 Microsoft Corporation Protection of data in a mixed use device
AU2011265428B2 (en) 2011-12-21 2014-08-14 Canon Kabushiki Kaisha Method, apparatus and system for selecting a user interface object
US8942369B2 (en) 2012-02-28 2015-01-27 Teletech Holdings, Inc. Method for providing support services using multi-channel navigator and route sequences
US9002890B2 (en) * 2012-03-14 2015-04-07 International Business Machines Corporation Rule-based access control list management
US20130290818A1 (en) 2012-04-27 2013-10-31 Nokia Corporation Method and apparatus for switching between presentations of two media items
US8861804B1 (en) 2012-06-15 2014-10-14 Shutterfly, Inc. Assisted photo-tagging with facial recognition models
US20130346068A1 (en) 2012-06-25 2013-12-26 Apple Inc. Voice-Based Image Tagging and Searching
US9887992B1 (en) 2012-07-11 2018-02-06 Microstrategy Incorporated Sight codes for website authentication
KR101993241B1 (ko) 2012-08-06 2019-06-26 삼성전자주식회사 이미지에 대한 부가 정보 태깅 및 검색 방법과 시스템, 기기와 그 기록 매체
US20140047527A1 (en) 2012-08-07 2014-02-13 Timothy Ngo System and Method for Detecting and Preventing Automated Interaction Based on Detected Actions Performed by User to Solve a Proffered Puzzle
US20140047386A1 (en) 2012-08-13 2014-02-13 Digital Fridge Corporation Digital asset tagging
US9461876B2 (en) 2012-08-29 2016-10-04 Loci System and method for fuzzy concept mapping, voting ontology crowd sourcing, and technology prediction
US9471795B2 (en) 2012-10-11 2016-10-18 Automatic Labs, Inc. System and method providing permission based access to automotive computers
US9251321B2 (en) 2012-10-22 2016-02-02 Telefonaktiebolaget L M Ericsson (Publ) Methods and nodes for handling usage policy
US20140130182A1 (en) * 2012-11-02 2014-05-08 Genesismedia Llc Controlled Grant Of Access To Media Content
US10210341B2 (en) 2013-02-12 2019-02-19 Amazon Technologies, Inc. Delayed data access
US9378234B2 (en) 2013-03-11 2016-06-28 International Business Machines Corporation Management of updates in a database system
US9405771B2 (en) 2013-03-14 2016-08-02 Microsoft Technology Licensing, Llc Associating metadata with images in a personal image collection
US10229258B2 (en) 2013-03-27 2019-03-12 Samsung Electronics Co., Ltd. Method and device for providing security content
US9009806B2 (en) 2013-04-12 2015-04-14 Globoforce Limited System and method for mobile single sign-on integration
US9773023B2 (en) 2013-04-25 2017-09-26 Autodesk, Inc. Image selection using automatically generated semantic metadata
US20140343989A1 (en) 2013-05-16 2014-11-20 Phantom Technologies, Inc. Implicitly linking access policies using group names
KR102065418B1 (ko) * 2013-09-24 2020-01-13 엘지전자 주식회사 이동 단말기 및 그 제어 방법
US9058375B2 (en) 2013-10-09 2015-06-16 Smart Screen Networks, Inc. Systems and methods for adding descriptive metadata to digital content
WO2015122716A1 (en) 2014-02-14 2015-08-20 Samsung Electronics Co., Ltd. Electronic device, method for authenticating user, and computer readable recording medium
US20150339324A1 (en) 2014-05-20 2015-11-26 Road Warriors International, Inc. System and Method for Imagery Warehousing and Collaborative Search Processing
US20170249308A1 (en) 2014-06-04 2017-08-31 Grandios Technologies, Llc Image tagging
US9516028B1 (en) 2014-08-06 2016-12-06 Amazon Technologies, Inc. Hierarchical policy-based shared resource access control
US9471767B2 (en) 2014-08-22 2016-10-18 Oracle International Corporation CAPTCHA techniques utilizing traceable images
US11720663B2 (en) * 2014-09-15 2023-08-08 eData Platform, Corp. Login methodology
US10210246B2 (en) 2014-09-26 2019-02-19 Oracle International Corporation Techniques for similarity analysis and data enrichment using knowledge sources
US10318575B2 (en) 2014-11-14 2019-06-11 Zorroa Corporation Systems and methods of building and using an image catalog
US9942208B2 (en) 2014-11-14 2018-04-10 Microsoft Technology Licensing, Llc Updating stored encrypted data with enhanced security
US9785781B2 (en) 2014-12-08 2017-10-10 Dotalign, Inc. Method, apparatus, and computer-readable medium for data exchange
US9384337B1 (en) 2015-04-27 2016-07-05 Microsoft Technology Licensing, Llc Item sharing based on information boundary and access control list settings
US9959398B1 (en) 2015-04-30 2018-05-01 Ims Health Incorporated Dynamic user authentication and authorization
US10180921B2 (en) 2015-06-26 2019-01-15 International Business Machines Corporation Non-interfering transactions
US10742681B2 (en) 2015-08-21 2020-08-11 PushPull Technology Limited Data collaboration
AU2016324490A1 (en) 2015-09-18 2018-04-19 First Data Corporation System for validating a biometric input
WO2017070412A1 (en) 2015-10-23 2017-04-27 Oracle International Corporation Password-less authentication for access management
US20170177706A1 (en) 2015-12-16 2017-06-22 Quixey, Inc. Category-Based Search System and Method for Providing Application Related Search Results
US20170185670A1 (en) 2015-12-28 2017-06-29 Google Inc. Generating labels for images associated with a user
KR102465332B1 (ko) 2015-12-29 2022-11-11 에스케이플래닛 주식회사 사용자 장치, 그의 제어 방법 및 컴퓨터 프로그램이 기록된 기록매체
CN108701194B (zh) 2016-01-19 2022-06-24 雷韦兹公司 掩蔽限制访问控制系统
GB2547932B (en) 2016-03-03 2019-08-14 Arm Ip Ltd Time-limited access to configuration settings
WO2017152150A1 (en) 2016-03-04 2017-09-08 ShoCard, Inc. Method and system for authenticated login using static or dynamic codes
US20170323361A1 (en) 2016-05-06 2017-11-09 Oct8ne Inc. Rapid re-hosting of collaborative browsing sessions
EP3465478A1 (en) * 2016-06-02 2019-04-10 Kodak Alaris Inc. Method for providing one or more customized media centric products
US10929461B2 (en) 2016-07-25 2021-02-23 Evernote Corporation Automatic detection and transfer of relevant image data to content collections
US10536465B2 (en) 2017-01-18 2020-01-14 Microsoft Technology Licensing, Llc Security for accessing stored resources

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7636733B1 (en) * 2003-10-03 2009-12-22 Adobe Systems Incorporated Time-based image management
US20090307261A1 (en) * 2008-06-05 2009-12-10 Apple Inc. Categorization of Digital Media Based on Media Characteristics
US20100063961A1 (en) * 2008-09-05 2010-03-11 Fotonauts, Inc. Reverse Tagging of Images in System for Managing and Sharing Digital Images
US20150261789A1 (en) * 2012-03-26 2015-09-17 Amazon Technologies, Inc. Cloud-based photo management

Also Published As

Publication number Publication date
US11093543B2 (en) 2021-08-17
US10515111B2 (en) 2019-12-24
US11436274B2 (en) 2022-09-06
US10621225B2 (en) 2020-04-14
JP7178904B2 (ja) 2022-11-28
US10747808B2 (en) 2020-08-18
US20170205968A1 (en) 2017-07-20
CN108701194B (zh) 2022-06-24
EP3405903A4 (en) 2019-07-10
JP2019508826A (ja) 2019-03-28
US20170206197A1 (en) 2017-07-20
WO2017127576A1 (en) 2017-07-27
US20170206279A1 (en) 2017-07-20
US20170206346A1 (en) 2017-07-20
US10614119B2 (en) 2020-04-07
US20170208073A1 (en) 2017-07-20
US20170208069A1 (en) 2017-07-20
US20170208068A1 (en) 2017-07-20
CN108701194A (zh) 2018-10-23
US20170206226A1 (en) 2017-07-20
EP3405903A1 (en) 2018-11-28

Similar Documents

Publication Publication Date Title
US11436274B2 (en) Visual access code
US12032623B2 (en) Facial based image organization and retrieval method
US20180278600A1 (en) Multi-factor masked access control system
US10019136B1 (en) Image sharing device, apparatus, and method
US10027727B1 (en) Facial recognition device, apparatus, and method
US10713320B2 (en) De-duping events on online social networks
JP6303023B2 (ja) 一時的なイベンティングに関するシステム及び方法
US10027726B1 (en) Device, apparatus, and method for facial recognition
US20180365489A1 (en) Automatically organizing images
US9465890B1 (en) Method and system for managing and sharing geographically-linked content
US20240184833A1 (en) Advanced keyword search
US9542457B1 (en) Methods for displaying object history information
WO2019199460A1 (en) Single-device multi-factor authentication system
AU2012201659A1 (en) A computer implemented method, web server, client computing device and computer readable storage medium for selectively relating event data

Legal Events

Date Code Title Description
AS Assignment

Owner name: REGWEZ, INC., CALIFORNIA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:ELEISH, RAGY FAROUK;REEL/FRAME:041431/0867

Effective date: 20170119

STPP Information on status: patent application and granting procedure in general

Free format text: FINAL REJECTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: NON FINAL ACTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: FINAL REJECTION MAILED

STCV Information on status: appeal procedure

Free format text: NOTICE OF APPEAL FILED

STPP Information on status: patent application and granting procedure in general

Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION

STPP Information on status: patent application and granting procedure in general

Free format text: NON FINAL ACTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER

STPP Information on status: patent application and granting procedure in general

Free format text: FINAL REJECTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION

STPP Information on status: patent application and granting procedure in general

Free format text: NON FINAL ACTION MAILED

STCB Information on status: application discontinuation

Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION