KR102009629B1 - Event archiving, system and methods - Google Patents

Abstract

The method of searching for event information is represented. Monument objects can be recognized by the archiving engine. Based on the recognition, the archiving engine obtains information related to the monument object, perhaps one or more recognizable features, and uses the information to retrieve the event associated with the timeline with the corresponding tag. The archiving engine may then return the event information to the user as a result set.

Description

EVENT ARCHIVING, SYSTEM AND METHODS}

The field of the present invention is information archiving technology.

<Priority claim>

This application claims priority to US provisional application 61/651179, filed May 24, 2012. This and all other external materials discussed herein are incorporated herein by reference in their entirety.

Recently, Facebook ^® introduced the concept of a timeline, where user information is organized based on when the information enters Facebook or based on how the user links the information to the timeline. This arrangement of information can be useful to the user by allowing the user to review their time-related events in their life. Unfortunately, the information stored on the Facebook timeline is inaccessible without Facebook and lacks a mechanism to remind users of the information available.

There has been work aimed at archiving information for later retrieval. For example, US patent application 2004/0107270, filed on October 27, 2003, entitled “Method and System for Collaboration Recording,” describes a system in which collaboration sessions are archived. This approach is useful for collaboration in communication sessions. However, the Stephens system also does not provide memory queues to individual users.

Work has also progressed toward recognizable objects. For example, 7,016,532 filed on November 5, 2001, entitled "Image capture and identification system and process," and filed on May 20, 2004, titled "Image capture and identification system and process." 7,477,780 and 7,680,324 filed on August 15, 2005 entitled "Use of image-derived information as search criteria for internet and other search engines," and on January 26, 2005, "Data capture and identification system and U.S. Patent No. 7,565,008, filed under the title "process", describes various aspects of recognizing an object and returning content information based on the recognized object. However, these patents also do not provide insights that bind events of interest to an individual's timeline.

These and all other external materials discussed herein are incorporated herein by reference in their entirety. If the definition or use of a term in the cited reference does not match or contradict the definition of a term provided herein, the definition of the term provided herein applies, and the definition of the term in the reference does not apply.

Unless otherwise indicated in context, all ranges beginning here should be construed to include their endpoints, and open-ended ranges should be construed to include commercially viable values. Similarly, lists of all values should be considered to include intermediate values unless otherwise indicated in context.

Ideally, a person or other entity can have an monument that can be used to archive the event and also to recall the event. Thus, there is still a need for an archiving system that can trigger or restore memory to a user.

The subject matter of the present invention provides devices, systems, and methods that enable a person to track one or more events in their lives along a timeline and retrieve information related to the events through monuments. One aspect of the techniques of the present invention provides a method for retrieving event information. The method contemplated allows a user to access the event archiving engine through one or more interfaces (eg, mobile phones, APIs, browsers, social networks, search engines, etc.). The archiving engine is preferably configured to receive an event object, perhaps over a network, where the event object is an event in the life of an entity (eg, a person, animal, pet, building, vehicle, or other item). Contains data representing. For example, the method may include an archiving engine that receives a video of the concert captured by the user's mobile phone, which may be thought of as an event object. The archiving engine may also bind the event object to a timeline associated with the entity, which represents a plurality of event objects arranged over time. Another step of the method may include selecting one or more event objects associated with or associated with the monument object. The archiving engine may register a memo object with the event object by mapping one or more recognizable features of the memo object to the selected event object, possibly via intermediate indexing information. An archiving engine may search for the selected event by deriving one or more derived features from the digital representation of the monument object and using the derived features as a basis for searching for the selected event.

Various objects, features, aspects and advantages of the subject matter of the present invention will become more apparent from the following detailed description of the preferred embodiments, together with the features of the accompanying drawings in which like numerals represent like components.

1 is a schematic diagram of an event archiving ecosystem.
2 shows links between timelines, monuments, and event objects.
3 represents a possible way of retrieving event information based on the monument.

The following description is for a computer / server based event archiving system, but various alternative configurations are also considered suitable, and may include servers, interfaces, systems, databases, agents, peers, engines, controllers, or any other individual or integrated operation. It should be noted that computing devices including tangible computing devices may be employed. These terms include a processor configured to execute software instructions stored on a tangible, non-transitory computer readable storage medium (eg, hard drive, solid state drive, RAM, flash, ROM, etc.). It should be understood that it is considered to express something. The software instructions preferably constitute a computing device that provides the roles, responsibilities, or other functionality discussed below in connection with the disclosed device. In a particularly preferred embodiment, various servers, systems, databases, or interfaces are based on presumably HTTP, HTTPS, AES, public-private key exchanges, web services APIs, known financial transaction protocols, or other electronic information exchange methods. Exchange data using standard protocols or algorithms. The data exchange is preferably performed via a packet switching network, the Internet, a LAN, a WAN, a VPN, or some other type of packet switching network.

It is to be understood that the disclosed technology provides many beneficial technical effects, including searching for event information based on recognition signals sent to the archiving engine.

The following discussion provides many embodiments of the subject matter of the present invention. Although each embodiment represents a single combination of the components of the invention, it is contemplated that the subject matter of the invention encompasses all possible combinations of the components disclosed. Thus, if one embodiment includes components A, B and C and the second embodiment includes components B and D, the subject matter of the present invention is also deemed to include the other remaining combinations of A, B, C or D. do.

As used herein, and unless otherwise indicated in the context, the term “connected to” refers to a bidirectional direct connection (in which two connected components are in contact with each other) and at least one additional component between two components. Indirect connections). Thus, the terms "linked to" and "linked to" are used synonymously. In the context of a network, the terms "connected to" and "connected with" are also used euphemistically to mean "connected with".

1 shows an archiving system 100 that includes an event archiving engine 130 that is configured or programmed to store one or more event objects 145A through 145B according to a timeline. Examples of computing systems that may be suitably adapted to the subject matter of the present invention include search engines, social networking sites, online communities, CRM databases, source control engines, forums, product review sites, online game platforms, multiplayer online games, or Other computing systems for storing information.

The following discussion is described as the event archiving engine 130 providing access to an event-monument for fee serivce through the network 115, where the event archiving engine 130 is a platform as a service. operating as an as-a-Service (PaaS), Infrastructure-as-a-Service (IaaS), Software as a Service (Software-as-a-Service, SaaS), or any other type of platform. It can include one or more servers. However, it should be understood that event archiving engine 130 may represent other computing devices, including mobile phones, personal computers, tablets, or other devices. Thus, in some embodiments, a user can install an app on a smartphone, allowing the app to program the cell phone to operate as the event archiving engine 130.

In the example shown, the user of device 160A captures data of event 110 representing the event over time. For example, an individual may attend a wedding or sporting event and use the camera of the mobile phone to capture images or other content of the wedding or sporting event. The event 110 related content captured by the device 160A may be sent to the event archiving engine 130, perhaps over a network 115 (eg, network connection, Internet, LAN, WAN, VPN, PAN, etc.). May be submitted. In the example shown, device 160A is shown as a mobile phone. However, other types of devices may operate as device 160A, including tablets, computers, security cameras, or other sensor devices. In addition, the content representing event 110 may include a wide variety of data including image data, video data, audio data, speech data, motion data, acceleration data, temperature data, text data, biometric data, or other types of form data. Include the data form of the spectrum. Event archiving engine 130 receives content related to event 110 via service interface 135 (eg, cellular interface, GUI, API, HTTP server, etc.) and collectively as event objects 145. Materialize one or more event objects 145A-145B.

The event archiving engine 130 stores the events of a person's lifetime as an event object 145, which contains data representing the event or event 110. Each event object 145 may be considered an easy-to-manage object that is distinguished from other objects in the system. For example, event object 145 may be considered as image, video, or other type of data that may be captured with additional content, as mentioned above. The event object 145 may also include metadata (eg, attributes, properties, properties, etc.) that describe the nature of the event or event. In some embodiments, a user may select an image from a local computing device, perhaps device 160A, and also upload an image file to event archiving engine 130 via service interface 135 (eg, FTP). Can be. The user may then instruct the event archiving engine 130 to materialize the event object 145A from the image file and additionally provided content (eg, time stamps, comments, etc.). Providing the metadata allows the event archiving engine 130 to manage, analyze, explore, or otherwise manipulate the event object 145. In order to manage the event object 145 over time, the event object 145 allows the archiving engine to arrange the event object 145 based on time to construct a timeline represented by the timeline object 155. It may include one or more time stamps to allow.

Event object 145 includes a wide variety of data types. Preferred event objects include media objects or perhaps multimedia objects. Example data types that can be integrated into the event object 145 include audio data, video data, image data, motion data, acceleration data, temperature data, position data, time data, metadata, identification data, preference data, game data, Sensor data, security data, or some other type of data.

For greater clarity with respect to event object 145, event object 145 may be considered a representation of one or more events or events that may be "remembered" or "remembered." Example events that may be archived or explored may include parties, concerts, sporting events, news events, project tasks, accidents, or other types of events. In some embodiments, event object 145 may be automatically classified based on contextual ambient data (eg, sensor data) or based on user-provided data (eg, calendar).

As an example of contextual automatic creation of an event object 145, consider a scenario where a person attends a wedding. The event archiving engine 130 may obtain a data feed from the mobile phone of the person, wherein the data feed may include geo-location data, timestamp data, video data, and user ID (from a GPS sensor). For example, phone numbers, IP addresses, email addresses, user names, etc.) or other sensor feed data. The event archiving engine 130 receives the data feed with external information, possibly from other sources. For example, the external information may include a description in an individual's calendar with text in the form of “wedding attendance”. The data feed and external information can automatically trigger the specification of one or more event objects 145. The corresponding marriage event object may include a wedding time, a wedding venue, and one or more pieces of content commemorating the event. Accordingly, event archiving engine 130 may include one or more rule sets stored in memory that include contextual triggering criteria that cause event content to be materialized as event object 145.

The event object 145 may be stored in the event database 140. The event database 140 may be housed within the device 160A, within the event archiving engine 130, external to both devices, or located remotely via the network 115. In some embodiments, event database 140 may be part of a search engine, CRM system, or other data search system.

The event archiving engine 130 may manage the timeline as a separate object represented by a timeline object 155 in which the timeline includes a time-aligned event object 145. Thus, the timeline object 155 may represent interesting events that occur during a person's lifetime. Several points should be understood. First, the timeline can be segmented if necessary. Example segments may include segmenting the timeline based on life stage, year, month, day, or other range or criteria. Second, the timeline may be associated with any entity, perhaps including a pet, person, place, building, vehicle, company, animal, organization, group, family, school, yearbook, or other type of entity. Third, the timeline objects 155 include their own attributes and can be arranged, managed, analyzed or otherwise adjusted by them. For example, the timeline can be differentiated by location information or user identification information. In such cases, event archiving engine 130 may provide demographic analysis of all timeline objects 155 that may be associated with GPS coordinates or may generate a space-time line, where the timeline is a timestamp. Not only coordinates but also space or position coordinates.

Similar to the event object 145, the timeline object 155 is stored in the timeline database 150, where the timeline object 155 can be searched. When the event archiving engine 130 processes the event object 145, the event archiving engine 130 queries the timeline database 150 to query one or more timeline attributes (eg, user ID, geo-location, The desired timeline object 155 may be retrieved based on the timeline type).

As shown, the event object 145 may be associated with one or more monument objects 120, which may be leveraged by the user to retrieve information associated with the event object 145 or the event object 145. Event archiving engine 130 may select one or more event objects 145 to be linked or associated with monument object 120 according to different techniques. In some scenarios, the event object 145 is arbitrarily selected based on the geo-location attribute value, perhaps where the event object 145 falls within a time span on the timeline. In other embodiments, event object 145 may be selected according to preference, user selection, voting, ranking, or other technique.

 Monument object 120 may be a 2D object (e.g., image, photo, card, poster, etc.), 3D object (e.g., object, finger, vehicle, etc.) or 4D object (e.g., moving finger, time). A wide spectrum of objects, including changing objects, objects having multidimensional relevance, and the like. Preferably, monument object 120 may be recognized by event archiving engine 130 based on digital representation 123 of monument object 120. Digital representation 123 may include various forms of digital data, including image data, video data, audio data, sensor data, or other data formats. Example techniques that may be used to recognize objects that may be adapted to the subject matter of the present invention include those disclosed in US Pat. Nos. 7,016,532, 7,477,780, 7,680,324, and 7,565,008.

 The event archiving engine 130 registers with the selected event object 145 one or more sets of recognizable features 125 of the monument object 120. In the example shown, event object 145A is registered and later retrieved as event object 145B. Device 160B, which may be the same device as device 160A, captures an image of a wedding ring that is a monument object 120. The digital representation 123 includes an image and is analyzed by the event archiving engine 130, perhaps according to the data type, to produce a series of features 125. The series of features 125 becomes indexing information for the event object 145A and is registered with the monument object 120 in the timeline object 155. If the event archiving engine 130 later encounters at least a portion of the series of features 125, the event archiving engine 130 searches for the event object 145B to send the corresponding “remembered” information to the device 160B. Can be returned.

Consider another usecase in which the monument object 120 includes a Christmas card. The engine 130 may analyze the card's features to recognize and recognize a series of features 125 (e.g., size, shape, dimension, color, SIFT feature, barcode, encoded data, non-encoded data, description). , Fonts, color histograms, or other attributes). The feature may be registered with, for example, the event object 145 corresponding to the previous Christmas. A recognizable set of features 125 can be used to generate indexing information for the event object 145. In some embodiments, recognizable features or values are used as indexes, while in other embodiments, recognizable features are used as a basis for recognizing monument object 120 as a valid monument object. A valid monument object can be associated with indexing information to access the related event object 145 in the timeline object 155.

The event archiving engine 130 may search for one or more event objects 145 to derive one or more series of features 125 by analyzing the digital representation 123 of the monument object 120. Continuing the previous example, the person receiving the Christmas card can view the image of the card on his cell phone (eg, device 160B). The card has previously been “activated” so that the archiving engine 130 can recognize the card through a series of features 125, which features are derived that correspond to one or more recognizable features of the monument card object, perhaps within predicted reliability. It may include a feature. The archiving engine 130 finds the corresponding event object 145B using a series of features 125 or using other information related to the recognizable monument object. The event object 145 may be represented on the output device 160B (eg, mobile phone, browser, kiosk, appliance, television, etc.).

It should be understood that the series of features 125 is derived from the digital representation 123 of the monument object 120 rather than the event 110. This approach provides for registering a series of features 125 from monument object 120 into timeline object 155 along with event object 145A. When the archiving engine 130 encounters a similar feature again, it may search for the registered event object represented by the event object 145B. Thus, monument object 120 serves as the basis for "remembering" or "recalling" event 110.

Suitable techniques for retrieving or presenting an event object 145 that may be adapted to the subject matter of the present invention include Verizon® Viewdini ™, which allows for retrieving or browsing streamed content (URL news.cnet.com/ 8301-1023_3-57439056-93 / verizon-unveils-viewdini-video-service-for-mobile-devices /). Viewdini may be augmented to recognize the features of monument object 120 and to allow navigation of event object 145. Viewini can be adapted to present the timeline information to the user as a series of available streamed events. In addition, upon recognizing the monument object 120, Budini jumps to a particular event object 145 on the timeline, allowing the user to move their reference point on the timeline while moving to a relatively earlier or future time. Can be adapted to provide "time shifting" capability. For example, for time movement, the user may move the monument object 120 to various locations or orientations with respect to the field of view of the device 160B. Perhaps placing the monument object 120 to the right of the view field can cause the archiving engine 130 to advance rapidly through the event object 145 in the timeline object 155. Similarly, placing the monument object 120 near the left side of the view field can cause the archiving engine 130 to rewind through the event object 145 in the timeline object 155. Moreover, the user can jump from the timeline object 155 to another based on the content represented within the event object 145. For example, if two people are represented in a video event object, the interface may allow the user to jump to a timeline associated with some or all of the people in the video.

The archiving engine 130 obtains or receives content related to the event 110 that is used to materialize the event object 145 through many techniques. In some scenarios, event content may be pushed from capture device 160A to archiving engine 130. Alternatively, archiving engine 130 may draw event content from capture device 160A. In addition, the archiving engine 130 may automatically search for event content, perhaps via the network 115, periodically or based on other triggering conditions. This approach allows automatic construction of the timeline object 155 with minimal user interaction, if given appropriate authorization.

It should be understood that the event object 145 may be bound to one or more timeline objects 155. For example, a video of a birthday party may include some individual images. The archiving engine 130 can recognize individuals or other objects in the video or event content and can determine whether people have their own timeline. If so, and if properly applied, archiving engine 130 may bind event object 145A to the timeline of the recognized people.

Although the disclosed subject matter has been presented in connection with a central archiving engine, it is to be understood that the various roles or responsibilities of the engine may be distributed to a plurality of devices. For example, the mobile phone can derive a series of features 125 from monument object 120 while operating as a recognition engine. Engine 130 may also include a search engine, network infrastructure (e.g., switches, routers, etc.), devices 160A or 160B, social network services, cloud services, set-top boxes, game devices, or other devices. Can be distributed across a plurality of computing devices.

Consider a use-case where people attend a concert or other event. Users capture data associated with the concert via their smartphones. The captured information may include a record of the conversation, a video of the band, an image of a friend, or other event content. The captured data can be integrated into the event object 145 via the archiving engine 130. Archiving engine 130 assigns event content data to the user's timeline, perhaps based on a time stamp. The archiving engine 130 may also assign all or part of the event content data to the timeline of the venue, if location data is available. In some embodiments, the user may also capture an image of their concert ticket and register the ticket as monument object 120 for event 110. In the future, whenever the user comes back to the ticket, they can quickly recall event information (eg, conversations, videos, images, etc.).

Monument objects may be produced by monument services (eg, 3D printers, CNC machines, manufacturer websites, etc.) that are connected to, or own or operate, the archiving engine 130. The monument service may use the event data to construct or manufacture the monument object 120 based on the event object 145 in the timeline object 155. For example, perhaps for a child's birthday, a "trading card" may be made, each card forming a "one year review", corresponding to the event during that year's life of that child. Each trading card may be activated (eg, from the registered set of features 125 to the monument object 120) to allow people to acquire corresponding event data once they capture an image of the card. The card may include an image obtained from the corresponding event. Other examples may include making souvenirs, perhaps dolls, combat dolls, ornaments, plates, jewellery, game tokens, or other types of items. In some embodiments, multiple individuals may receive the same monument object 120, eg, the same ornament. In this case, event object 145 or timeline object 155 may be differentiated based on additional information available to the system, including user identification, location, device identification or other factors. Alternatively, each duplicate monument object may simply point to the same event object.

2 shows a relationship between event objects 245A through 245N collectively referred to as timeline object 255, monument object 223, and event object 245. Timeline object 255 represents an event that is aligned over time and associated with an entity (eg, person, vehicle, building, location, etc.). The timeline object 255 is also a separate manageable object within the ecosystem in which the timeline object 255 is embodied, moved, duplicated, branched, deleted, listed, or otherwise described. Can be. In addition, the timeline object 255 or other objects in the system may be stored on a non-volatile computer readable medium.

The timeline object 255 may include various features to help manage the timeline object 255. In some more preferred embodiments, timeline object 255 may include one or more timeline attributes that describe the nature of timeline object 255. Example attributes include the identifier of the timeline, the owner ID (e.g. username, key, token, etc.), the timeline type identifier (e.g., personal timeline, building timeline, location timeline, etc.), It can contain pointers or references to other timelines, user demographics, event object counts, time ranges, locations or locations, or other attributes. As the entities access or manage the timeline object 255, the entity may navigate the timeline object 255 by submitting one or more queries to the timeline database, which may be defined in terms of timeline attributes. have. For example, all timeline objects 255 associated with a series of geo-location coordinates and also with men aged between 18 and 34 may be searched.

The timeline object 255 may further include index information representing one or more series of recognizable features associated with the monument object 223. The index information may include various accepted recognition criteria defined as a function of recognition characteristics, where the criteria indicate a satisfaction condition or requirement that must be met to access the corresponding event object. Thus, timeline object 255 may be considered a mini or micro object recognition database that stores a set of recognition features pointing to event object 245. As illustrated, the indexing information may include scale invariant features (see, eg, US Pat. No. 6,711,293 to SIFT, Lowe et al.), Corner detection features (eg, FAST; see URL www.edwardrosten.com/work/fast.html). , Visual pattern recognition features (eg, ViPR® from Evolution Robotics, vSLAM, see US Pat. No. 7,015,831 to Karlsson, etc.), audio pattern recognition features, optical characteristic recognition features, or other recognition features. Each series of features may further include one or more pointers that reference corresponding event objects 245 in the timeline. In addition, the indexing information may include a condition that may be satisfied by the archiving engine to search for the corresponding event object.

The timeline object 255 includes a timeline in which events are aligned over time. In the illustrated show, the timeline represents the individual's timeline from birth to death, where certain events are already defined. Events in the timeline may be represented as actual data structures in the timeline or externally stored as event objects 245 as shown. When the event object 245 is outside of the timeline object 255, the timeline may consist of a link or other pointer to the event object 245. For example, the pointer represented by EO_LINK includes URLs, URIS, network addresses, digital object identifiers (DOIs), event GUIDs, index keys, or any other address that results in the location where the corresponding event object 245 is stored. can do.

In some embodiments, timeline object 255 may be embodied based on a template timeline, perhaps corresponding to the type of entity. In such a scenario, the timeline may be deduced and have a suitable description (eg, seconds, minutes, hours, days, weeks, months, years, etc.) with placeholders for the event objects. It should also be understood that timeline object 255 represents one possible structure for the timeline. Alternatives include generating an ordered list of links of event objects or an ordered list of pointers to event objects 245.

Event object 245A is represented as one possible structure for the event object. The event object 245A may include event attributes that describe the nature of the corresponding event. Similar to timeline attributes, event attributes allow various computing devices to retrieve event object 245 using event attribute-based queries. Example event attributes can include an event ID, event object ID, owner ID, event type ID, a pointer or link to a related timeline object, a pointer to another event object, a timestamp, a location stamp, or some other type of attribute. Can be. Event object 245 may also include event content or links to event content. For example, event object 245A may be embodied as actual content and may include image data, audio data, video data, links to external content, or some other type of content data.

3 represents a method 300 for searching for event information that allows an entity to register a series of recognition features of a monument object with event content. The entity can use the monument object as a basis for searching for event content. Monument objects may be at least two-dimensional (e.g., shadows, images, photographs, pictures, etc.), three-dimensional (e.g., battle figurines, faces, rings, moving videos, etc.), four-dimensional (e.g., moving combat dolls , Moving vehicles, etc.), or more real-world physical objects that may include more multidimensional associations.

Step 310 includes configuring the computing device to operate as an event archiving engine. Example computing systems may include servers, workstations, mobile phones, kiosks, appliances, vehicles, search engines, CRM systems, or other types of computing devices. The computing device may be configured by installing one or more sets of instructions in the memory of the computing device, where the instructions cause the processor of the computing device to execute the various steps of the method 300. For example, the mobile phone may be configured to operate as a personal event archiving engine by installing a suitably configured application on the mobile phone. In another embodiment, a user may contract with a remote server operating as a fee collection service that uses their personal computing device to archive and retrieve event information. One or more computing devices may also be programmed to access at least one of the following interfaces that support access to the services of the archiving engine, including application program interfaces, search engine interfaces, social network interfaces, memory access interfaces, mobile phone interfaces. , Browser interface, sensor interface, or other interface to access the engine.

Step 320 includes receiving an event object representing an event over time. Example events include parties, concerts, sporting events, accidents, medical emergencies, events within a virtual world, game events, or some other type of event. The event object may be obtained through a plurality of techniques. In some embodiments, an event object may be embodied from event content that reflects an event. In other embodiments, the event object may be retrieved from a database of event objects.

Consider the example where a person wishes to "recall" a sporting event. A person may be represented as having a plurality of choices for the event archiving engine to receive an event object. As discussed above, event objects may include media objects having various types of multi-media data. Example multimedia data may include audio data, video data, image data, motion data, acceleration data, temperature data, position data, time data, metadata, identification data, preference data, game data, security data, or some other type of data. Include. Step 321 may include pushing the event object to the event archiving engine. For example, when a person captures images, audio, or other data from a sporting event via a mobile phone, the mobile phone packages the event data into an event object to push the event object along with its associated event content data to the event archiving engine. Can be. Alternatively, step 323 may include the event archiving engine pulling the event object from the mobile phone or other computing device. Step 325 may include receiving the event object automatically based on a contextual query, possibly submitted to event data. As shown in step 327, it should be understood that event objects may be received over the network, as needed or desired.

Step 330 includes binding the event object to a timeline associated with the entity, where the timeline includes a plurality of event objects arranged over time. An entity may include a person, animal, company, organization, group, family, school, yearbook, building, vehicle, or other type of entity. Event objects may be automatically or manually bound to the timeline, perhaps through the user interface. Each timeline may include a time segment based on lifetime, life stage, year, month, week, day, time, and other temporal segments. Consider a scenario in which an individual tablet computer acts as a personal witness (eg, an event archiving engine) that automatically builds a witnessed timeline object on a daily basis. Over days, weeks, and months, the tablet captures images of personal events. The tablet can use the image to automatically construct a timeline by creating an event object and binding the image to the witnessed timeline object as an event object. As the event object arrives, the tablet may arrange the event objects in the timeline according to their occurrence, as shown in step 335. Thus, a list of events is constructed that are time-aligned, or perhaps spatially aligned. The engine may also bind the event object to a plurality of timelines based on the recognition of the object in the event content data of the event object. As an example, the event object may include images or photos of some friends. The event archiving engine operating as the recognition platform may recognize each friend's face and bind the event object having the image to the timeline of each recognized friend.

In preparation for registering an event object with a monument, step 340 may include selecting a selected event object from the event object bound to the timeline. It should be understood that the selected event object may be deductively independent of the timeline prior to selection. Thus, step 330 may occur after the selection process of step 340. The event object may be selected through various techniques. Typically the user will allow the event object to be specified based on event content data (eg, image, audio, video, temperature, biometrics, etc.). In addition, the user or other entity may retrieve the event object as presented in step 343. For example, a timeline may include many deductively bound event objects. Later, the user can search for event objects in the timeline that have a particular time stamp, content or other desired attributes.

It should also be considered that the selected event object may be selected based on a second, different timeline, as shown in step 341. Suppose two individuals have separate timelines and participate in a common event, such as a medical emergency. Two individuals can create an event object in their respective timeline. An individual may wish to generate an event that can be recalled based on the image of the wound that occurred, and may also select an event based on an event object that overlaps between the two timelines. This approach may be considered beneficial when constructing a medical timeline shared between the patient and the healthcare provider.

Some timelines may include many event objects having substantially similar timestamps associated with the event. For example, a birthday party can include hundreds of photos taken within a short span of time. In such a situation, step 345 may include randomly selecting the selected event object from a series of available event objects. In addition, the selected event object may be selected based on additional criteria that may further limit the set of available event objects. Perhaps the selected event object is selected from the event object in a particular time span in the timeline as presented in step 347. Returning to the example of a birthday, the user may specify a time span near a particular interesting sub-event (eg, blowing out candles, opening a gift, etc.). The time span can be defined based on the associated time (eg, plus or minus near some point in time) or based on an absolute time (eg, from start time to end time).

Step 360 may include registering indexed information in the timeline with the selected event object based on the series of recognizable features associated with the monument object. The set of recognizable features is considered one or more indices, whereby the selected event object can be retrieved from the timeline. A set of recognizable features can be derived by adapting one or more algorithms to the digital representation of the monument object. For image data in the digital representation, the example algorithm may include SIFT, FAST, ViPR, BRISK, OCR, or some other image processing algorithm. Video data can be similar image processing algorithms, vSLAM, video content analysis (see "Two novel motion-based algorithms for surveillance video analysis on embedded platforms prohibited" by Vijverberg et al., See URL dx.doi.org/10.1117/12.851371), or other May be processed by other video processing algorithms. Features from audio data in digital representations are based on automatic speech recognition (ASR), Hidden Markov models, neural networks, or other algorithms using dynamic time warping. Can be extracted. The set of recognizable features may include features that fit various forms of the digital representation of the monument object. A set of recognizable features may include audio data features, image data features, colors, shapes, dimensions of relevance, make, models, encoded data, non-encoded information, textures, or other derivations from digital representations. May contain other information. Thus, a set of recognizable features may be represented as a vector in which each member of the vector represents a different form (eg, image, audio, video, space, time, temperature, etc.). Each member of the vector can also contain a listing of related features. The set of recognizable features may further include one or more criteria that must be met with respect to the feature in order to determine a successful recognition “hit”. For example, if only image data is to be represented as a monument object and the series of features includes only SIFT features, then the recognition criteria may be such that the monument object is recognized so as to trigger a search of the corresponding event object. 75% may be required to be expressed. Recognition criteria may be automatically determined by the event archiving engine, the user, or other technology.

Step 365 may include indexing the selected event object according to a set of recognizable features. The selected event object may be indexed by storing the selected event object in the event database using one or more members of the set of recognizable features as an index. In addition, the timeline can be updated by a series of recognizable features as an index, the timeline itself acting as a small recognition database, at least at some level, archiving events while encountering similar recognizable features. You can search for event objects based on the engine. The selected event object can be integrated into the timeline or referenced by the timeline via a pointer or link (eg, network addresses, GUIDs, etc.).

Step 370 includes searching the selected event object from the timeline via indexing information that is at least partially obtained from at least one derived representation derived from the digital representation of the monument object, wherein the derived representation is a series of symbols. Corresponds to a recognizable feature within a recognizable feature of the. For example, after an entity wishes to recall or remember an event object, they can capture a new digital representation of the monument object to which the event object was registered. The archiving engine applies one or more algorithms (eg SIFT, BRISK, FAST, vSLAM, etc.) to the new digital representation to obtain one or more derived features. Similar to recognizable features, derived features can include audio data features, image data features, colors, shapes, dimensions of relevance, make, models, encoded data, non-encoded information, attributes, textures, Location, time, or other features that may be derived from the new digital representation. The engine then queries the timeline for indexed registered event objects based on similar recognizable features using one or more derived properties. In some embodiments, as presented at step 375, the timeline is contemplated to configure the event archiving engine as an event search engine, retrieving an event object having a registered recognizable feature corresponding to the derived feature. Can be. Thus, the event archiving engine may "recall" or "remember" the event object based on the monument. It should be understood that the indexing information used to navigate the event object is related to the monument object rather than the content of the event object itself.

Step 350 may optionally include producing a monument object based on the selected event object. In some embodiments, the event archiving engine may propose one or more recommendations for the presented monument object. For example, when the event archiving engine receives event content data, perhaps image data, the event archiving engine may select one or more images to construct a real world object. Perhaps the image data can be used to create photos, key chains, images, cards, figurines, devices, moving objects (eg, combat dolls, bubbleheads, etc.), or other physical items that can be sent to the user. This approach is considered beneficial because it allows entities to tailor or personalize their monuments to specified events to enhance memory.

It will be apparent to those skilled in the art that many more modifications than those already described are possible without departing from the concept of the invention. Accordingly, the subject matter of the present invention is therefore not limited except as by the scope of the appended claims. In addition, in interpreting both the specification and the claims, all terms should be interpreted in the broadest possible manner consistent with the context. In particular, the terms "comprises" and "comprising" refer to a component, component, or step in a non-exclusive manner, and the referenced component, component, or step refers to another component that is not explicitly referenced. It is to be construed as indicating that it is represented, utilized or combined with a component or a step. When a particular claim refers to at least one selected from the group consisting of A, B, C ... and N, the text is only from the group, not A plus N, B plus N or the like. Only one component should be construed as essential.

Claims (26)

A computer implemented method of retrieving item information,
Configuring the computing device to operate with an archiving engine;
Receiving, by the archiving engine, an item object representing an occurrence in time from a first user;
Binding, by the archiving engine, the item object to at least one timeline associated with an entity of the first user, wherein the timeline includes a plurality of item objects arranged over time. step;
Displaying an item object on a user device of a second user from item objects bound to the timeline;
Generating indexing information based on a series of recognizable features associated with the image object, wherein the series of recognizable features includes an information vector corresponding to a digital representation of the image object, wherein each of the vectors A member of may represent a different modality, comprising: (1) one or more related features for that form and (2) recognition criteria including a recognition threshold corresponding to the one or more related features. step;
Upon selection of at least one recognizable feature by the first user, registering, by the archiving engine, the generated indexing information with the selected item object in the timeline; And
Upon selection of the selected item object by the second user, searching, by the archiving engine, at least a portion of the information vector corresponding to the image object. The method of claim 1,
Configuring the device to operate with the archiving engine,
Accessing at least one of an application program interface, a search engine interface, a social network interface, a mobile phone interface, a browser interface, and a sensor interface. The method of claim 1,
Receiving the item object,
Pushing the item object, dragging the item object, automatically receiving the item object, and receiving the item object over a network. The method of claim 1,
And the item object comprises a media object. The method of claim 1,
And the item object comprises a multimedia object. The method of claim 1,
The item object may include audio data, video data, image data, motion data, acceleration data, temperature data, position data, time data, metadata, identification data, preference data, game data, sensor data, links to external content, At least one type of data of other types of content data and security data. The method of claim 1,
The entity comprises at least one of a person, animal, company, organization, group, family, school, yearbook, brand, building, and vehicle. The method of claim 1,
And the item object represents at least one of a party, concert, sporting event, catalog, convention, conference, memorial event, fashion show, and festival. The method of claim 1,
The timeline comprises at least one of a life stage, life time, year, month, week, and day. The method of claim 1,
Selection of the selected item object by the second user includes retrieving the item object. The method of claim 1,
The selection of the selected item object by the second user includes tapping on the selected item object. The method of claim 1,
The selection by the second user comprises selecting the selected item object from item objects within a specified time span on the timeline. The method of claim 1,
Registering the series of recognizable features comprises indexing the selected item object according to the series of recognizable features. The method of claim 1,
Searching for the selected item object comprises retrieving item objects having registered recognizable features corresponding to an associated feature. The method of claim 1,
The series of recognizable features includes at least one of audio data characteristics, image data characteristics, color, form, relevance dimension, make, model, encoded data, non-encoded information, attributes, and textures. How to. The method of claim 1,
The associated feature set includes at least one of audio data characteristics, image data characteristics, color, shape, dimension of relevance, make, model, encoded data, non-encoded information, attributes, and textures. The method of claim 1,
And the image object comprises at least substantially two-dimensional object. The method of claim 17,
And the image object comprises at least a three dimensional object. The method of claim 18,
And the image object comprises at least a four dimensional object. The method of claim 1,
And the series of recognizable features related to the image object include attributes based on past item objects associated with one or more timelines. The method of claim 1,
And the image object comprises one of an image, sound, brand, device, and video. The method of claim 1,
The selection of the selected item object by the second user includes selecting the selected item object based on a second, different timeline. The method of claim 1,
The step of binding the item object to the timeline comprises the step of binding the item object to a plurality of timelines. The method of claim 1,
Binding the item object includes recognizing objects in the item object. A computer readable medium comprising one or more instructions for retrieving item information, wherein the one or more instructions, when executed by a processor,
Configuring the computing device to operate as an archiving engine;
Receiving, by the archiving engine, an item object representing an event over time from a first user;
Binding, by the archiving engine, the item object to at least one timeline associated with an entity of the first user, wherein the timeline includes a plurality of item objects arranged over time. ;
Displaying an item object on a user device of a second user from item objects bound to the timeline;
Generating indexing information based on a set of recognizable features associated with an image object, the set of recognizable features comprising an information vector corresponding to a digital representation of the image object, wherein each of the vectors The member representing a different modality and comprising: (1) one or more related features for the modality and (2) recognition criteria comprising a recognition threshold corresponding to the one or more related features;
Upon selection of at least one recognizable feature by the first user, registering, by the archiving engine, the generated indexing information with the selected item object in the timeline; And
Upon selection of the selected item object by the second user, performing, by the archiving engine, searching for at least a portion of the information vector corresponding to the image object. A system for searching item information,
Receiving from the first user an item object representing an event over time;
Binding the item object to at least one timeline associated with an entity of the first user, wherein the timeline includes a plurality of item objects arranged over time;
Generating indexing information based on a set of recognizable features associated with an image object, the set of recognizable features comprising an information vector corresponding to a digital representation of the image object, wherein each of the vectors The member representing a different modality and comprising: (1) one or more related features for the modality and (2) recognition criteria comprising a recognition threshold corresponding to the one or more related features;
Upon selection of at least one recognizable feature by the first user, registering the generated indexing information with the selected item object in the timeline;
Displaying an item object on a user device of a second user from item objects bound to the timeline; And
Upon selection of the selected item object by the second user, searching for at least a portion of the information vector corresponding to the image object
An archiving engine configured for;
An item database operatively coupled to the archiving engine and configured to store one or more item objects; And
A timeline database operatively coupled to the archiving engine and configured to store the at least one timeline
System comprising.

Images