US20140031060A1 - Creating Context Slices of a Storyline from Mobile Device Data - Google Patents
Creating Context Slices of a Storyline from Mobile Device Data Download PDFInfo
- Publication number
- US20140031060A1 US20140031060A1 US13/950,150 US201313950150A US2014031060A1 US 20140031060 A1 US20140031060 A1 US 20140031060A1 US 201313950150 A US201313950150 A US 201313950150A US 2014031060 A1 US2014031060 A1 US 2014031060A1
- Authority
- US
- United States
- Prior art keywords
- context
- data
- instructions
- slices
- context data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04W—WIRELESS COMMUNICATION NETWORKS
- H04W4/00—Services specially adapted for wireless communication networks; Facilities therefor
- H04W4/02—Services making use of location information
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N5/00—Computing arrangements using knowledge-based models
- G06N5/04—Inference or reasoning models
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/955—Retrieval from the web using information identifiers, e.g. uniform resource locators [URL]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04W—WIRELESS COMMUNICATION NETWORKS
- H04W4/00—Services specially adapted for wireless communication networks; Facilities therefor
- H04W4/02—Services making use of location information
- H04W4/029—Location-based management or tracking services
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04W—WIRELESS COMMUNICATION NETWORKS
- H04W4/00—Services specially adapted for wireless communication networks; Facilities therefor
- H04W4/20—Services signaling; Auxiliary data signalling, i.e. transmitting data via a non-traffic channel
- H04W4/21—Services signaling; Auxiliary data signalling, i.e. transmitting data via a non-traffic channel for social networking applications
Definitions
- the described embodiments pertain to interpreting location data and other data about a person collected from mobile devices and internetworked services.
- GPS Global Positioning System
- Cell tower triangulation is imprecise, while also unavailable where there are no cell towers.
- WiFi triangulation is error-prone and also unavailable in the absence of nearby WiFi networks.
- sensor readings such as taking satellite or radio readings can cause a significant drain on the device's battery, so they must be done sparingly. Naturally, this increases the uncertainty in the data, with large gaps from one reading to the next.
- Embodiments of the invention include a method, a non-transitory computer readable storage medium and a system for creating context slices that together define a storyline.
- a context is a (possibly partial) specification of what a user was doing in the dimensions of time, place, and activity. Each of these dimensions may be defined specifically (e.g., location defined by latitude 47.60621, longitude ⁇ 122.332071) or very generally (e.g., the location “Seattle, Wash.”), or entirely unspecified (e.g., omitted or a default value). They may also be ascribed varying degrees of semantic meaning (e.g., “Seattle” contains more semantic information than “47.60621, ⁇ 122.332071”).
- a context can represent a stay in a certain location or travel from one place to another. Contexts may have probabilities associated with them. In some cases, contexts may be inferred from evidence rather than known with certainty. Thus, contexts can vary in their specificity, their semantic content, and their likelihood.
- a storyline is composed of a time-ordered sequence of contexts that partition a given span of time. Thus, a storyline is a single, internally-consistent accounting for what a user was doing at moments in time and where the user was doing it. In one embodiment, a storyline is created through a process of data collection, slicing and labeling.
- Raw context data can be obtained from a variety of observation sources, sensors, monitors, third party sources, and the like.
- Slicing is the process of refining the chaotic, multi-threaded collection of contexts produced by data collection into a single consistent storyline composed of a sequence of contexts representing homogeneous time intervals.
- a context refiner module groups the context data by combining the observations to create a coherent representation of a user's context, and may attach semantic content to groups or sequences of context data to form storylines.
- the slicing process is generally divided into three phases: preprocessing, segmentation, and reconciliation.
- Preprocessing involves filtering, smoothing, and interpolating data to reduce the effects of noise and uneven sampling rates during data collection.
- Segmentation involves determining distinct, contiguous series of contexts from the groomed sensor data representing different activities.
- Reconciliation resolves newly generated context slices with previously generated context slices.
- Labeling is the process of adding more specific and semantically meaningful data to the storyline produced by slicing. Labeling may add geography (such as the context's city or neighborhood), venue (public places, businesses, or personally significant places like “home”), and/or activity (such as “working”, “eating”, or “going to see a movie”).
- the labeling process is generally divided into three phases: label candidate search, label candidate ranking, and label application.
- label candidate search label sources are queried to discover possible labels based on the slice context.
- label candidate ranking the likelihood of each label candidate is evaluated based on the slice context and ranked according to likelihood.
- labels deemed sufficiently likely can be applied to label the slices.
- the storyline is stored.
- the storyline may be presented to the user or accessed for further processing by various mobile applications or web applications, for example.
- Embodiments of the computer-readable storage medium store computer-executable instructions for performing the steps described above.
- Embodiments of the system further comprise a processor for executing the computer-executable instructions.
- FIG. 1 is a diagram illustrating the relationship of sensor data, slices, and contexts, in accordance with an embodiment of the invention.
- FIG. 2 is a block diagram illustrating the system environment for creating and using storylines in accordance with an embodiment of the invention.
- FIG. 3 is a flow chart illustrating a storyline creation process in accordance with an embodiment of the invention.
- FIG. 4 is a flow chart illustrating a process of slicing in accordance with an embodiment of the invention.
- FIG. 5 is a flow chart illustrating a process of labeling in accordance with an embodiment of the invention.
- Embodiments of the invention enable the creation and labeling of context slices from observation data that together define a storyline of a user's movements.
- a context is a (possibly partial) specification of what a user was doing in the dimensions of time, place, and activity. Contexts can vary in their specificity, their semantic content, and their likelihood.
- a storyline is composed of a time-ordered sequence of contexts that partition a given span of time.
- a storyline is created through a process of data collection, slicing and labeling.
- Raw context data can be collected from a variety of observation sources with various error characteristics.
- Slicing refines the chaotic collection of contexts produced by data collection into a single consistent storyline composed of a sequence of contexts representing homogeneous time intervals. Labeling adds more specific and semantically meaningful data (e.g., geography, venue, activity) to the storyline produced by slicing. Accordingly, the storyline data is then available to offer a historical perspective to the user or for further processing in
- FIG. 2 is a diagram illustrating the relationship of sensor data, slices, and contexts, in accordance with an embodiment of the invention.
- FIG. 2 illustrates observations, which are the collections of sensor data from a mobile device.
- the observations include Global Positioning System readings, Cellular triangulation signals, and emails sent from the device.
- the points along the time axis at which these observations are collected are marked.
- the user's time can be divided into a sequence of slices.
- each slice has a type and a start and end time.
- the type is either “stay” or “travel” and the start and end times establish the limits or boundaries of the slice.
- the slice also has a centroid location
- the slice also has a speed of the travel.
- the metadata may describe the dimensions of time, place, and activity of the user at various levels of generality.
- the first slice is associated with a place dimension of the context at various levels of generality/specificity: at a city level, a street address level, and a venue level.
- storyline data One use of storyline data is to offer a historical perspective to the user, who may peruse the storyline to view his/her previous activities. Another use of storyline data is for further processing into interesting aggregations, e.g., showing how often certain activities were performed within some time period, in accordance with another embodiment of the invention. This information can be useful for increasing self-awareness or goal tracking purposes, e.g., seeing how much time was spent exercising or eating out at restaurants per month.
- FIG. 2 is a block diagram illustrating a system environment 100 for creating and using storylines, in accordance with an embodiment of the invention.
- the system environment 100 includes one or more raw context collectors 110 , a context refiner module 120 , a storyline storage 130 , and a storyline retrieval module 140 .
- the entities of the system environment may be contained on a single processing node, such as a user's mobile device, and in another embodiment, the entities of the system environment may be divided between the user's mobile device and a centralized processing node to decrease the processing requirements of the mobile device.
- An example distributed embodiment is one where the raw context collectors 111 are located on the user's mobile device and data is sent to a central device of context refinement, storage, and retrieval.
- the raw context collectors 110 collect raw context data from observation sources, sensors, monitors, third party sources, and the like.
- a raw context represents a single observation and so is generally very specific, often carries little semantic information on its own, and is of high probability. Naturally, different observation sources may have greater degrees of noise and uncertainty or may inherently include semantic information.
- the raw context collectors 110 include a location module 111 , an activity module 112 , and a social module 113 , but different and/or other context collectors 110 may be included in other embodiments.
- the context collectors include sensors for measuring device orientation (e.g., a compass), magnetic fields, user heart rate, and user stress level, as well as modules for audio and visual sampling of the user's environment.
- Examples of location module 111 include a GPS receiver, and a Wi-Fi receiver that enable the location module 111 to determine an absolute or relative location of the user's mobile device, within a margin of error.
- Examples of the activity module 112 include, a monitor of key presses that determines when a user is typing or otherwise interacting with the user's mobile device, and an accelerometer that measures the acceleration forces on the mobile device to determine movement and movement patterns of the mobile device.
- Examples of social module 113 include a FACEBOOK friend graph, PINTEREST pins, FOURSQUARE check ins, and other social media data that identify a user's social acquaintances, activities, and locations.
- the context refiner module 120 receives the raw context data from the raw context collectors 110 .
- the context refiner module 120 groups the context data by combining the observations to create a more coherent representation of the user's context.
- the context refiner module 120 may also attach semantic content to groups or sequences of context data to form storylines.
- the context refiner module 120 includes a plurality of context refiner sub-modules (not shown); one for each type of context data received from the raw context collectors 110 .
- Each context refiner sub-module groups context data by combining the observations from the corresponding raw context collector 110 into slices in order to create a more coherent representation of the user's context indicated by the specific type of context data the sub-module is operating on.
- the context refiner module 120 includes an additional refiner sub-module (not shown) that analyzes the multiple streams of contexts generated by the other context refiner sub-modules to detect overlapping slices and generate combined slices containing context information from the corresponding overlapping slices.
- the storyline storage 130 receives the storylines formed by the context refiner module 120 and stores them. Then, a storyline retrieval module 140 can access the stored storylines from storage 130 .
- Examples of storyline retrieval modules 140 include mobile applications and web applications that use the storylines stored in storage 130 .
- An example of an application that uses a storyline is a mobile phone application that displays the user's history, showing the user the places the user has stayed and the travel between the stays. The process of grouping context data and attaching semantic content to form storylines will be described in detail in the section below.
- Embodiments of the invention divide the process of storyline creation into three phases: data collection, slicing, and labeling. These phases are illustrated in the flow chart of FIG. 3 , and described in detail in this section.
- Data collection 301 involves accessing the various sources of information and observations of user behavior, optionally transporting their data to servers for analysis and storage (to offload CPU load and reduce battery usage), and translating the data into a collection of raw contexts for additional analysis.
- These observations may come from a variety of sources, including but not limited to the following:
- Data collection 301 may be run as a constant on-going process, with different techniques appropriate to different sources. Alternatively, data collection 301 may be run periodically at an interval appropriate for the application.
- Slicing 302 is the process of refining the chaotic, multi-threaded collection of contexts produced by data collection into a single consistent storyline composed of a sequence of contexts representing homogeneous time intervals. These homogeneous time intervals generally represent either a stay at one place or a process of travel from one place to another.
- place information may be refined, in that each stay context defines an area that includes most of the individual points observed during that time.
- Travel contexts will generally have a start and end point, with some definition of the route between them (e.g., waypoints).
- travel slices may be further annotated with mode of conveyance (such as driving, walking, flying, or riding a bus), and travel slices may be segmented into distinct slices for different modes of conveyance. For example, a trip involving a drive to the airport, a flight to another city, and a taxi to a hotel might be separated into three distinct travel slices.
- no additional semantic meaning or activity information is added during slicing 302 .
- Other types of data can be used to produce other types of slices, such as slices representing a period of consistent activity.
- an activity like “shopping” may be represented by a longer-duration slice that overlaps with multiple location slices representing stays at different retail businesses as well as travel between those businesses.
- an activity like “sleeping” may span only part of a stay at home.
- a biometric slice such as “high caloric burn” may cover part of a visit to the park as well as time at the gym.
- Embodiments of the invention divide the process of slicing 302 into three phases: preprocessing 410 , segmentation 420 , and reconciliation 430 . Each of these phases is described in detail in this section, with reference to the flow chart illustrated in FIG. 4 .
- the steps of FIG. 4 are illustrated from the perspective of the context refiner module 120 performing the method. However, some or all of the steps may be performed by other entities and/or components. In addition, some embodiments may perform the steps in parallel, perform the steps in different orders, or perform different steps.
- preprocessing 3021 involves a combination of filtering 3022 , smoothing 3023 , and interpolation 3024 .
- Filtering 3022 Filters on raw context data eliminate from consideration raw context data that are deemed more inaccurate than some desirable threshold.
- the value of the filter threshold can be sensor-specific, due to different sensor error characteristics. For example, a GPS device's data uncertainty is calculated by physical factors related to the timing of signals received from the device's acquired satellites, so it can report a reliable estimate of sensor inaccuracy. In contrast, reported uncertainty is less reliable with location technology based on cell tower or Wi-Fi triangulation, which lack the measurement precision necessary to account for fluctuations in wireless signal strength, therefore the threshold for filtering 3022 those contexts may be higher. When using any location technology, the amount of filtering 3022 will depend on its expected error characteristics, and the error characteristics are expected to vary between sources of data.
- default threshold values for filters may be set system-wide, set per sensor type, or based on user preferences. In addition to filtering 3022 by location technology, physically unlikely readings (e.g., traveling at higher speeds than possible) may also be filtered.
- Smoothing 3023 It is also helpful to later slicing phases for context grooming to smooth sequences of contexts from the same sensor when each individual context is noisy. Noise characteristics are hardware dependent, so the smoothing 3023 of each sensor should be parameterized to limit the noise expected from that sensor. For example, a certain accelerometer may generate noisy contexts at a high sampling rate, characterized by large magnitude swings in all axes. One way to smooth such data is to compute an average of the magnitude values over a time window and then output the smoothed magnitude values at a less frequent sampling rate. Smoothing 3023 is also used when different sensors conflict.
- the degree of smoothing 3023 will depend on the variability in data noise from each particular location technology.
- Interpolation 3024 Uneven sampling rates can also be due to power conservation, where a device chooses to go into a low-power, low sampling rate state, or is forced to by a governing operating system such as a mobile phone OS. It is common for sensors to be configured to increase sampling when the environment is changing, and to decrease it when the environment (from the sensor's perspective) is static. As slicing 302 occurs over a finite window of time, a decreased sampling rate could lead to relevant context data falling outside the window. Therefore, it is desirable in some cases to interpolate less frequent context data to ensure that the later phases of slicing 302 have sufficient data to analyze. Interpolation 3024 generates virtual context data between sensed context data.
- sensor data payload may include an indication that there has been an interruption in contexts since the last time a sensor generated context. This may be the default behavior whenever a sensor is (re)started for data collection by the controlling data collection process. In addition, if there is an exceptionally long gap between context data from sensors, it may indicate an interruption even if the sensors fail to set the flag and would be treated as such.
- Segmentation 3025 involves determining distinct, contiguous series of slices from the groomed sensor data representing different activities. For example, the simple day of a user who is an office worker could be segmented into a stay slice located in the morning at her domicile, then a commute to work travel slice, a stay slice at an office, then a commute back home travel slice, followed by a stay slice in the evening at her domicile.
- k-means clustering can be applied to find clusters of contexts (by location, or a distance function combining location and time).
- Stay slices can be distinguished from travel slices by the dispersion of location context and/or velocity data. Because k-means has fundamental limitations, other more sophisticated clustering algorithms can be used additionally or alternatively to extract slices.
- segmentation 3025 can also be performed by applying time-series analysis algorithms, using the variance of a sliding window of input contexts to detect inflection points in the distribution.
- the algorithm divides the two subsequences into slices that can then be classified as a stay or travel. For example, a stay is distinguishable from a travel by the low amount of variance in each individual input context in the stay sequence to its centroid, the geographic average location.
- This meta-segmenter can pick and choose slices output with the highest associated probability among all constituent segmentation algorithms.
- Segmentation 3025 can also be followed by filter and merge steps that smooth the output slices.
- Filters can remove short slices with more uncertainty associated with the contexts included therein, e.g., those with few actual sensor observations, and merge adjacent segments that are likely to be the same activity.
- the thresholds on minimum required observation uncertainty or distance from adjacent segments for filtering and merging can be parameterized to control the false positive rate (groups of raw context data that should not have been segmented) compared to the false negative rate (groups of raw context data that should have been segmented but were not).
- the final phase of slicing 302 deals with resolving newly generated slices with existing contexts generated from a previous slicing 302 run. While this reconciliation 3026 is optional—if it were computationally feasible to run slicing 302 on an entire raw context set, the brand new contexts could simply replace the older ones—in some cases reconciliation 3026 provides desirable qualities for the purpose of presentation. For example, it is desirable not to change contexts and slices in a user's history that have been previously displayed to the user, unless new data is in significant conflict, because the instability in data shown to the user would appear inconsistent.
- One way to limit the scope of changes between new and preexisting slices is to specify a time window within which preexisting data may be changed or replaced. Any data outside the window (i.e., older than a certain age), would be left unchanged in later slicing 302 runs. Contexts from newer slices are then integrated into the eligible preexisting slices by comparing type (stay or travel) and time spans. If a new slice is of the same type and begins and ends at approximately the same time as an existing slice, it could retain the same metadata of the existing slice, including any identifier labels (ids) and contexts.
- ids identifier labels
- the process can prefer the new slice except when there has been manual intervention, for example when a user has already interacted with the existing slice or confirmed it in some way using a user interface.
- the last slice is most likely to have changed due to new data, and could have its ending time extended if it aligns with a new slice starting at a time near its own start time, or completely replaced if the type changed (if a previously presumed stay were actually the beginning of a travel slice, for instance).
- Labeling 303 is the process of adding more specific and semantically meaningful data to the slices produced by slicing 302 . In one embodiment, some or all of these labels are themselves contexts associated with the slices. In particular, labeling 303 adds geography (such as a slice's city or neighborhood), venue (public places, businesses, or personally significant places like “home”), and activity (such as “working”, “eating”, or “going to see a movie”).
- geography such as a slice's city or neighborhood
- venue public places, businesses, or personally significant places like “home”
- activity such as “working”, “eating”, or “going to see a movie”.
- the process of labeling 303 may suggest a revision in slicing 302 , such as when the labeling 303 process determines that a user was eating and then seeing a movie at the theater next door, while the slicing 302 phase represented both activities as a single slice, prompting the single slice to be split into two successive slices, taking place at distinct venues.
- a slice can be labeled using identifiers from predefined data sets, such as public venue records, or automatically generated, for example using a reverse geocoding system that converts latitude and longitude coordinates into an approximate street address.
- the labeling 303 process uses these data sources to apply probable labels to each slice. Some labels are exclusive while others may coexist alongside one another.
- Example data sources for the labeling 303 process include:
- the labeling 303 process can be broken into three phases: label candidate search 3031 , label candidate ranking 3032 , and label application 3033 , illustrated in FIG. 5 .
- the label sources are first queried to discover possible labels based on the slice's existing contexts.
- the following provides examples of how various label types can be searched.
- Venues and places are found based on the slice's location, which by definition is a consistent estimate of a user's location over a period of time. However, there is a degree of uncertainty when using the associated slice location. Essentially, raw sensors providing locations are imprecise.
- the label candidate search 3031 phase does not rely on the exact location represented by the slice, but instead expands the search within some radius calculated as an estimate of the uncertainty. For example, if a slice's location was calculated using Wi-Fi triangulation, the triangulation error is often in the tens to low hundreds of meters, so the search process may query for venues and places centered at the slice location within two hundred meters.
- Events and appointments can be found based on the slice's location and time boundaries. An event at a venue would be matched against the overlapping time boundaries between the event and the slice. Appointments are also matched against location and time. Because event and appointment time boundaries are imprecise, and slice time boundaries may be imperfect, the slice's time boundaries do not need to exactly match those of an event or appointment. Similarly, the slice location does not need to be an exact match either.
- the label candidate search 3031 finds possible events and appointments within the likely uncertainty radius of the slice location.
- the label candidate search 3031 process can bring up associated activity labels.
- the slice context can be compared to similar slices in the past if the user had previously labeled activities manually. For example, if a user previously labeled an activity at the same venue or a venue in the same category as the slice that has not yet been labeled with an activity, that activity would be considered as a candidate for labeling the slice.
- the likelihood of each one given the contexts already associated with the slice is evaluated.
- the likelihood of each label is computed and the labels are ranked.
- slices are constrained to only having one label of some types (e.g., venue labels), and the top-ranked label meeting the minimum likelihood threshold is applied to the slice. For other label types, multiple labels can be valid for a single slice, and all labels meeting the minimum threshold of likeliness are applied.
- Likelihoods are calculated by scoring a candidate label given the contexts already associated with a slice.
- a model is defined to be an algorithm for computing likelihoods given slice context. Models treat the various aspects of a slice context as features. Some example features include:
- models can use other sources of information to inform the likelihood estimate.
- Some example information sources include:
- One or more models can be run to process a slice's context(s) into labels, which are applied in label application 3033 .
- multiple models can be represented by a single meta-model that runs the appropriate features through its composing models. Once the meta-model outputs probabilities, labels deemed sufficiently likely are applied to the slice. In one embodiment, labels that are not deemed to be sufficiently likely can still be surfaced as options to the user should he/she wish to alter the label set by adding new labels, with the label candidate ranking 3032 intact. In such cases, it is not necessary for the same meta-model to produce the label candidate ranking 3032 —different meta-models can rank labels differently to produce whatever is specified by the system design of a particular embodiment.
- automatic label application 3033 does not require that every label is ranked solely by likelihood. For example, when users interact with the label candidates (e.g., to manually apply labels), it can be desirable to present candidate labels in different orders to make finding desired labels easier. For example, an alphabetical order or a hybrid order that incorporates both likelihoods and lexicographical positions can be used. Once labels are applied to slices, the additional contextual data is presentable to the user and available for further processing in a variety of forms.
- the new storyline data is presentable to the user and available for further processing in a variety of forms.
- one use of storyline data is to offer a historical perspective to the user, who may peruse the storyline to view his/her previous activities.
- Another use of storyline data is for further processing into interesting aggregations, e.g., showing how often certain activities were performed within some time period, or for processing by other software applications.
- a computer is adapted to execute computer program modules for providing the functionality described herein.
- module refers to computer program logic utilized to provide the specified functionality.
- a module can be implemented in hardware, firmware, and/or software.
- program modules are stored on a storage device, loaded into memory, and executed by a processor.
- Embodiments of the physical components described herein can include other and/or different modules than the ones described here.
- the functionality attributed to the modules can be performed by other or different modules in other embodiments.
- this description occasionally omits the term “module” for purposes of clarity and convenience.
- the present invention also relates to an apparatus for performing the operations herein.
- This apparatus may be specially constructed for the required purposes, or it may comprise a general-purpose computer selectively activated or reconfigured by a computer program stored on a computer readable medium that can be accessed by the computer.
- a computer program may be stored in a computer readable storage medium, such as, but is not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, application specific integrated circuits (ASICs), or any type of computer-readable storage medium suitable for storing electronic instructions, and each coupled to a computer system bus.
- the computers referred to in the specification may include a single processor or may be architectures employing multiple processor designs for increased computing capability.
- any reference to “one embodiment” or “an embodiment” means that a particular element, feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment.
- the appearances of the phrase “in one embodiment” in various places in the specification are not necessarily all referring to the same embodiment.
- the terms “comprises,” “comprising,” “includes,” “including,” “has,” “having” or any other variation thereof, are intended to cover a non-exclusive inclusion.
- a process, method, article, or apparatus that comprises a list of elements is not necessarily limited to only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus.
- “or” refers to an inclusive or and not to an exclusive or. For example, a condition A or B is satisfied by any one of the following: A is true (or present) and B is false (or not present), A is false (or not present) and B is true (or present), and both A and B are true (or present).
Abstract
Embodiments create and label context slices from observation data that together define a storyline of a user's movements. A context is a (possibly partial) specification of what a user was doing in the dimensions of time, place, and activity. Contexts can vary in their specificity, their semantic content, and their likelihood. A storyline is composed of a time-ordered sequence of contexts that partition a given span of time. A storyline is created through a process of data collection, slicing and labeling. Raw context data can be collected from a variety of observation sources with various error characteristics. Slicing refines the chaotic collection of contexts produced by data collection into a single consistent storyline composed of a sequence of contexts representing homogeneous time intervals. Labeling adds more specific and semantically meaningful data (e.g., geography, venue, activity) to the storyline produced by slicing.
Description
- This application claims the benefit of U.S. Provisional Application No. 61/675,732, filed Jul. 25, 2012, entitled “Creating a Storyline from Mobile Device Data,” and U.S. Provisional Application No. 61/675,733, filed Jul. 25, 2012, entitled “Recommendation Agents Using Routine, Personality, Learning, and Sharing,” both of which are incorporated herein by reference in their entirety.
- 1. Technical Field
- The described embodiments pertain to interpreting location data and other data about a person collected from mobile devices and internetworked services.
- 2. Description of Related Art
- Many mobile devices now have the capability of recording location and other information. Having a complete record of when and where the user goes is useful for a variety of applications, including recommendation systems, lifelogging, and goal tracking. However, there are a number of obstacles to building useful applications on top of the kinds of data streams currently available.
- First, these data streams are noisy, imprecise, and sometimes unavailable. Global Positioning System (GPS) technology, for example, can be confused by surrounding buildings or other features and is not available indoors. Cell tower triangulation is imprecise, while also unavailable where there are no cell towers. WiFi triangulation is error-prone and also unavailable in the absence of nearby WiFi networks.
- Second, sensor readings such as taking satellite or radio readings can cause a significant drain on the device's battery, so they must be done sparingly. Naturally, this increases the uncertainty in the data, with large gaps from one reading to the next.
- Third, location data in the form of latitude/longitude readings has no semantic content and is useless for many applications. Because of the noisy nature of the data, translating lat/long into addresses or named locations is difficult, especially in the dense urban areas where most users live.
- Embodiments of the invention include a method, a non-transitory computer readable storage medium and a system for creating context slices that together define a storyline. A context is a (possibly partial) specification of what a user was doing in the dimensions of time, place, and activity. Each of these dimensions may be defined specifically (e.g., location defined by latitude 47.60621, longitude −122.332071) or very generally (e.g., the location “Seattle, Wash.”), or entirely unspecified (e.g., omitted or a default value). They may also be ascribed varying degrees of semantic meaning (e.g., “Seattle” contains more semantic information than “47.60621, −122.332071”). A context can represent a stay in a certain location or travel from one place to another. Contexts may have probabilities associated with them. In some cases, contexts may be inferred from evidence rather than known with certainty. Thus, contexts can vary in their specificity, their semantic content, and their likelihood. A storyline is composed of a time-ordered sequence of contexts that partition a given span of time. Thus, a storyline is a single, internally-consistent accounting for what a user was doing at moments in time and where the user was doing it. In one embodiment, a storyline is created through a process of data collection, slicing and labeling.
- Data collection is performed by context collectors collecting raw context data. Raw context data can be obtained from a variety of observation sources, sensors, monitors, third party sources, and the like.
- Slicing is the process of refining the chaotic, multi-threaded collection of contexts produced by data collection into a single consistent storyline composed of a sequence of contexts representing homogeneous time intervals. A context refiner module groups the context data by combining the observations to create a coherent representation of a user's context, and may attach semantic content to groups or sequences of context data to form storylines. In one embodiment, the slicing process is generally divided into three phases: preprocessing, segmentation, and reconciliation. Preprocessing involves filtering, smoothing, and interpolating data to reduce the effects of noise and uneven sampling rates during data collection. Segmentation involves determining distinct, contiguous series of contexts from the groomed sensor data representing different activities. Reconciliation resolves newly generated context slices with previously generated context slices.
- Labeling is the process of adding more specific and semantically meaningful data to the storyline produced by slicing. Labeling may add geography (such as the context's city or neighborhood), venue (public places, businesses, or personally significant places like “home”), and/or activity (such as “working”, “eating”, or “going to see a movie”). In one embodiment, the labeling process is generally divided into three phases: label candidate search, label candidate ranking, and label application. In label candidate search, label sources are queried to discover possible labels based on the slice context. In label candidate ranking, the likelihood of each label candidate is evaluated based on the slice context and ranked according to likelihood. In label application, labels deemed sufficiently likely can be applied to label the slices.
- In one embodiment, after the data collection, slicing, and labeling process has created a storyline, the storyline is stored. The storyline may be presented to the user or accessed for further processing by various mobile applications or web applications, for example.
- Embodiments of the computer-readable storage medium store computer-executable instructions for performing the steps described above. Embodiments of the system further comprise a processor for executing the computer-executable instructions.
- The features and advantages described in the specification are not all inclusive and, in particular, many additional features and advantages will be apparent to one of ordinary skill in the art in view of the drawings, specification, and claims. Moreover, it should be noted that the language used in the specification has been principally selected for readability and instructional purposes, and may not have been selected to delineate or circumscribe the inventive subject matter.
-
FIG. 1 is a diagram illustrating the relationship of sensor data, slices, and contexts, in accordance with an embodiment of the invention. -
FIG. 2 is a block diagram illustrating the system environment for creating and using storylines in accordance with an embodiment of the invention. -
FIG. 3 is a flow chart illustrating a storyline creation process in accordance with an embodiment of the invention. -
FIG. 4 is a flow chart illustrating a process of slicing in accordance with an embodiment of the invention. -
FIG. 5 is a flow chart illustrating a process of labeling in accordance with an embodiment of the invention. - The figures depict embodiments of the present invention for purposes of illustration only. One skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles of the invention described herein.
- Embodiments of the invention enable the creation and labeling of context slices from observation data that together define a storyline of a user's movements. A context is a (possibly partial) specification of what a user was doing in the dimensions of time, place, and activity. Contexts can vary in their specificity, their semantic content, and their likelihood. A storyline is composed of a time-ordered sequence of contexts that partition a given span of time. A storyline is created through a process of data collection, slicing and labeling. Raw context data can be collected from a variety of observation sources with various error characteristics. Slicing refines the chaotic collection of contexts produced by data collection into a single consistent storyline composed of a sequence of contexts representing homogeneous time intervals. Labeling adds more specific and semantically meaningful data (e.g., geography, venue, activity) to the storyline produced by slicing. Accordingly, the storyline data is then available to offer a historical perspective to the user or for further processing in other applications.
-
FIG. 2 is a diagram illustrating the relationship of sensor data, slices, and contexts, in accordance with an embodiment of the invention.FIG. 2 illustrates observations, which are the collections of sensor data from a mobile device. In this example, the observations include Global Positioning System readings, Cellular triangulation signals, and emails sent from the device. The points along the time axis at which these observations are collected are marked. As a result of these observations, the user's time can be divided into a sequence of slices. In this example, each slice has a type and a start and end time. In this example, the type is either “stay” or “travel” and the start and end times establish the limits or boundaries of the slice. For “stay” type slices, the slice also has a centroid location, and for “travel” type slices, the slice also has a speed of the travel. In this example, based on the division of observations into time slices, a variety of metadata representing contexts have been attached to the slices. The metadata may describe the dimensions of time, place, and activity of the user at various levels of generality. For example, the first slice is associated with a place dimension of the context at various levels of generality/specificity: at a city level, a street address level, and a venue level. - One use of storyline data is to offer a historical perspective to the user, who may peruse the storyline to view his/her previous activities. Another use of storyline data is for further processing into interesting aggregations, e.g., showing how often certain activities were performed within some time period, in accordance with another embodiment of the invention. This information can be useful for increasing self-awareness or goal tracking purposes, e.g., seeing how much time was spent exercising or eating out at restaurants per month.
-
FIG. 2 is a block diagram illustrating a system environment 100 for creating and using storylines, in accordance with an embodiment of the invention. The system environment 100 includes one or more raw context collectors 110, a context refiner module 120, a storyline storage 130, and a storyline retrieval module 140. In one embodiment, the entities of the system environment may be contained on a single processing node, such as a user's mobile device, and in another embodiment, the entities of the system environment may be divided between the user's mobile device and a centralized processing node to decrease the processing requirements of the mobile device. An example distributed embodiment is one where the raw context collectors 111 are located on the user's mobile device and data is sent to a central device of context refinement, storage, and retrieval. - The raw context collectors 110 collect raw context data from observation sources, sensors, monitors, third party sources, and the like. A raw context represents a single observation and so is generally very specific, often carries little semantic information on its own, and is of high probability. Naturally, different observation sources may have greater degrees of noise and uncertainty or may inherently include semantic information. In the example illustrated in
FIG. 1 , the raw context collectors 110 include a location module 111, an activity module 112, and a social module 113, but different and/or other context collectors 110 may be included in other embodiments. For example, in various embodiments, the context collectors include sensors for measuring device orientation (e.g., a compass), magnetic fields, user heart rate, and user stress level, as well as modules for audio and visual sampling of the user's environment. - Examples of location module 111 include a GPS receiver, and a Wi-Fi receiver that enable the location module 111 to determine an absolute or relative location of the user's mobile device, within a margin of error. Examples of the activity module 112 include, a monitor of key presses that determines when a user is typing or otherwise interacting with the user's mobile device, and an accelerometer that measures the acceleration forces on the mobile device to determine movement and movement patterns of the mobile device. Examples of social module 113 include a FACEBOOK friend graph, PINTEREST pins, FOURSQUARE check ins, and other social media data that identify a user's social acquaintances, activities, and locations.
- The context refiner module 120 receives the raw context data from the raw context collectors 110. The context refiner module 120 groups the context data by combining the observations to create a more coherent representation of the user's context. The context refiner module 120 may also attach semantic content to groups or sequences of context data to form storylines. In some embodiments, the context refiner module 120 includes a plurality of context refiner sub-modules (not shown); one for each type of context data received from the raw context collectors 110. Each context refiner sub-module groups context data by combining the observations from the corresponding raw context collector 110 into slices in order to create a more coherent representation of the user's context indicated by the specific type of context data the sub-module is operating on. In one such embodiment, the context refiner module 120 includes an additional refiner sub-module (not shown) that analyzes the multiple streams of contexts generated by the other context refiner sub-modules to detect overlapping slices and generate combined slices containing context information from the corresponding overlapping slices.
- The storyline storage 130 receives the storylines formed by the context refiner module 120 and stores them. Then, a storyline retrieval module 140 can access the stored storylines from storage 130. Examples of storyline retrieval modules 140 include mobile applications and web applications that use the storylines stored in storage 130. An example of an application that uses a storyline is a mobile phone application that displays the user's history, showing the user the places the user has stayed and the travel between the stays. The process of grouping context data and attaching semantic content to form storylines will be described in detail in the section below.
- Embodiments of the invention divide the process of storyline creation into three phases: data collection, slicing, and labeling. These phases are illustrated in the flow chart of
FIG. 3 , and described in detail in this section. -
Data Collection 301 -
Data collection 301 involves accessing the various sources of information and observations of user behavior, optionally transporting their data to servers for analysis and storage (to offload CPU load and reduce battery usage), and translating the data into a collection of raw contexts for additional analysis. These observations may come from a variety of sources, including but not limited to the following: -
- Location technologies (e.g., GPS, cell tower triangulation, Wi-Fi location), typically embedded in mobile devices like smartphones. The location technologies may be included, for example, in the location module 111 illustrated in
FIG. 2 . - Activity data such as accelerometer data from mobile devices and device interaction notices (e.g. taps, key presses, power on). The activity data may be collected, for example, by the activity module 112 illustrated in
FIG. 2 . - Ambient data from the user's environment (e.g., sound, temperature, or light intensity).
- Biometric data such as the user's skin temperature, galvanic skin response, heat flux, and heart rate. This data may be used to calculate caloric burn, stress level, sleep quality, and other indicators of the user's physical state.
- Social data from any networks or applications the user may use, such as FACEBOOK, TWITTER, FOURSQUARE, FLICKR, or PINTEREST, as well as personal communication applications like email and text messaging. The social data may be collected, for example, by a social module 113 illustrated in
FIG. 2 . In one embodiment, the social data is made available to the social module 113 through application programming interfaces (APIs). - Schedule data, such as from the user's calendar application.
- Explicit annotation created by the user to inform the system of a location (e.g., a “check in” at a baseball stadium) or activity (e.g., marking the time boundaries of a jog to track fitness goals).
- Location technologies (e.g., GPS, cell tower triangulation, Wi-Fi location), typically embedded in mobile devices like smartphones. The location technologies may be included, for example, in the location module 111 illustrated in
-
Data collection 301 may be run as a constant on-going process, with different techniques appropriate to different sources. Alternatively,data collection 301 may be run periodically at an interval appropriate for the application. - Slicing 302
- One challenge with
data collection 301 described above is that multiple sources of observation data may result in a collection of contexts that contain conflicting information. For example, an observation from the user's calendar may place him at a meeting downtown, while GPS observations may show him to be at the beach. Resolving these inconsistencies is key to the interpretation of the data. Slicing 302 is the process of refining the chaotic, multi-threaded collection of contexts produced by data collection into a single consistent storyline composed of a sequence of contexts representing homogeneous time intervals. These homogeneous time intervals generally represent either a stay at one place or a process of travel from one place to another. In one embodiment, place information may be refined, in that each stay context defines an area that includes most of the individual points observed during that time. Travel contexts will generally have a start and end point, with some definition of the route between them (e.g., waypoints). In one embodiment, travel slices may be further annotated with mode of conveyance (such as driving, walking, flying, or riding a bus), and travel slices may be segmented into distinct slices for different modes of conveyance. For example, a trip involving a drive to the airport, a flight to another city, and a taxi to a hotel might be separated into three distinct travel slices. In another embodiment, no additional semantic meaning or activity information is added during slicing 302. Other types of data can be used to produce other types of slices, such as slices representing a period of consistent activity. For example, an activity like “shopping” may be represented by a longer-duration slice that overlaps with multiple location slices representing stays at different retail businesses as well as travel between those businesses. Conversely, an activity like “sleeping” may span only part of a stay at home. As another example, a biometric slice such as “high caloric burn” may cover part of a visit to the park as well as time at the gym. - Embodiments of the invention divide the process of slicing 302 into three phases: preprocessing 410, segmentation 420, and reconciliation 430. Each of these phases is described in detail in this section, with reference to the flow chart illustrated in
FIG. 4 . The steps ofFIG. 4 are illustrated from the perspective of the context refiner module 120 performing the method. However, some or all of the steps may be performed by other entities and/or components. In addition, some embodiments may perform the steps in parallel, perform the steps in different orders, or perform different steps. -
Preprocessing 3021 - Since raw context data input into the slicing 302 process come from a variety of sources with varying degrees of inaccuracy, the raw context data are systematically groomed into a suitable form for later steps to minimize the amount of error in the output. In one embodiment, preprocessing 3021 involves a combination of
filtering 3022, smoothing 3023, andinterpolation 3024. -
Filtering 3022. Filters on raw context data eliminate from consideration raw context data that are deemed more inaccurate than some desirable threshold. The value of the filter threshold can be sensor-specific, due to different sensor error characteristics. For example, a GPS device's data uncertainty is calculated by physical factors related to the timing of signals received from the device's acquired satellites, so it can report a reliable estimate of sensor inaccuracy. In contrast, reported uncertainty is less reliable with location technology based on cell tower or Wi-Fi triangulation, which lack the measurement precision necessary to account for fluctuations in wireless signal strength, therefore the threshold for filtering 3022 those contexts may be higher. When using any location technology, the amount of filtering 3022 will depend on its expected error characteristics, and the error characteristics are expected to vary between sources of data. Optionally, default threshold values for filters may be set system-wide, set per sensor type, or based on user preferences. In addition to filtering 3022 by location technology, physically unlikely readings (e.g., traveling at higher speeds than possible) may also be filtered. -
Smoothing 3023. It is also helpful to later slicing phases for context grooming to smooth sequences of contexts from the same sensor when each individual context is noisy. Noise characteristics are hardware dependent, so the smoothing 3023 of each sensor should be parameterized to limit the noise expected from that sensor. For example, a certain accelerometer may generate noisy contexts at a high sampling rate, characterized by large magnitude swings in all axes. One way to smooth such data is to compute an average of the magnitude values over a time window and then output the smoothed magnitude values at a less frequent sampling rate. Smoothing 3023 is also used when different sensors conflict. For example, if there is minimal change in values across a series of accelerometer readings, it indicates that a device was immobile, which could contradict a series of location readings that would otherwise suggest the device was wandering due to inaccurate location technology. In general, the degree of smoothing 3023 will depend on the variability in data noise from each particular location technology. -
Interpolation 3024. Uneven sampling rates can also be due to power conservation, where a device chooses to go into a low-power, low sampling rate state, or is forced to by a governing operating system such as a mobile phone OS. It is common for sensors to be configured to increase sampling when the environment is changing, and to decrease it when the environment (from the sensor's perspective) is static. As slicing 302 occurs over a finite window of time, a decreased sampling rate could lead to relevant context data falling outside the window. Therefore, it is desirable in some cases to interpolate less frequent context data to ensure that the later phases of slicing 302 have sufficient data to analyze.Interpolation 3024 generates virtual context data between sensed context data. For example, when there is a gap between two location contexts, a number of interpolated context data points may be generated that correspond to locations between the two endpoints.Interpolation 3024 runs into the risk of adding contexts that should not exist. For example, if a sensor is not functional and therefore not reporting, a gap in contexts should not be interpolated. To preventinvalid interpolation 3024, sensor data payload may include an indication that there has been an interruption in contexts since the last time a sensor generated context. This may be the default behavior whenever a sensor is (re)started for data collection by the controlling data collection process. In addition, if there is an exceptionally long gap between context data from sensors, it may indicate an interruption even if the sensors fail to set the flag and would be treated as such. -
Segmentation 3025 -
Segmentation 3025 involves determining distinct, contiguous series of slices from the groomed sensor data representing different activities. For example, the simple day of a user who is an office worker could be segmented into a stay slice located in the morning at her domicile, then a commute to work travel slice, a stay slice at an office, then a commute back home travel slice, followed by a stay slice in the evening at her domicile. - There are a variety of algorithms to segment the input raw context data into stays, travels, and gaps. For example, k-means clustering can be applied to find clusters of contexts (by location, or a distance function combining location and time). Stay slices can be distinguished from travel slices by the dispersion of location context and/or velocity data. Because k-means has fundamental limitations, other more sophisticated clustering algorithms can be used additionally or alternatively to extract slices.
- Besides clustering,
segmentation 3025 can also be performed by applying time-series analysis algorithms, using the variance of a sliding window of input contexts to detect inflection points in the distribution. When the variation across a subsequence of input context data differs from a subsequence before it, the algorithm divides the two subsequences into slices that can then be classified as a stay or travel. For example, a stay is distinguishable from a travel by the low amount of variance in each individual input context in the stay sequence to its centroid, the geographic average location. - Because there are a variety of algorithms that can be applied to
segmentation 3025, each with different features and limitations, it is also possible to combine their resulting outputs with a meta-segmenter. This meta-segmenter can pick and choose slices output with the highest associated probability among all constituent segmentation algorithms. -
Segmentation 3025 can also be followed by filter and merge steps that smooth the output slices. Filters can remove short slices with more uncertainty associated with the contexts included therein, e.g., those with few actual sensor observations, and merge adjacent segments that are likely to be the same activity. The thresholds on minimum required observation uncertainty or distance from adjacent segments for filtering and merging can be parameterized to control the false positive rate (groups of raw context data that should not have been segmented) compared to the false negative rate (groups of raw context data that should have been segmented but were not). -
Reconciliation 3026 - In one embodiment, the final phase of slicing 302 deals with resolving newly generated slices with existing contexts generated from a
previous slicing 302 run. While thisreconciliation 3026 is optional—if it were computationally feasible to run slicing 302 on an entire raw context set, the brand new contexts could simply replace the older ones—in somecases reconciliation 3026 provides desirable qualities for the purpose of presentation. For example, it is desirable not to change contexts and slices in a user's history that have been previously displayed to the user, unless new data is in significant conflict, because the instability in data shown to the user would appear inconsistent. Instability is even less desirable in cases when the user has performed some operation on a previous context or slice, such as manually labeling or otherwise attaching metadata to it, that thesubsequent slicing 302 run would overwrite. As such, there are rules governing when new slices and contexts can replace preexisting data in a user's history. - One way to limit the scope of changes between new and preexisting slices is to specify a time window within which preexisting data may be changed or replaced. Any data outside the window (i.e., older than a certain age), would be left unchanged in later slicing 302 runs. Contexts from newer slices are then integrated into the eligible preexisting slices by comparing type (stay or travel) and time spans. If a new slice is of the same type and begins and ends at approximately the same time as an existing slice, it could retain the same metadata of the existing slice, including any identifier labels (ids) and contexts. When a new slice and old slice overlap in time but conflict in type, the process can prefer the new slice except when there has been manual intervention, for example when a user has already interacted with the existing slice or confirmed it in some way using a user interface. Finally, the last slice is most likely to have changed due to new data, and could have its ending time extended if it aligns with a new slice starting at a time near its own start time, or completely replaced if the type changed (if a previously presumed stay were actually the beginning of a travel slice, for instance).
-
Labeling 303 - Labeling 303 is the process of adding more specific and semantically meaningful data to the slices produced by slicing 302. In one embodiment, some or all of these labels are themselves contexts associated with the slices. In particular, labeling 303 adds geography (such as a slice's city or neighborhood), venue (public places, businesses, or personally significant places like “home”), and activity (such as “working”, “eating”, or “going to see a movie”). Note that the process of
labeling 303 may suggest a revision in slicing 302, such as when thelabeling 303 process determines that a user was eating and then seeing a movie at the theater next door, while the slicing 302 phase represented both activities as a single slice, prompting the single slice to be split into two successive slices, taking place at distinct venues. - A slice can be labeled using identifiers from predefined data sets, such as public venue records, or automatically generated, for example using a reverse geocoding system that converts latitude and longitude coordinates into an approximate street address. The
labeling 303 process uses these data sources to apply probable labels to each slice. Some labels are exclusive while others may coexist alongside one another. Example data sources for thelabeling 303 process include: -
- Public venue database—a set of geographically annotated public venue names, such as businesses, public spaces, or landmarks. The venue data should be able to be queried geographically (e.g., to find likely venues within some specified distance of the slice's observed location observations), therefore it should have a location represented either as a single point (latitude, longitude, altitude) or as a set of points that defines or approximates its shape. The venue may also contain a unique identifier, which is useful, for example, to use to associate the venue with manually entered observations from the user. In addition to location and name, the data entry for the venue may contain other metadata such as address, business hours, categories describing the type of venue, and reviews useful to present back to the user. Because the set of public venues changes over time, this database may be configured to receive updates whenever available.
- User-specified database of places—a set of manually or automatically generated locations considered private to the user, identified by location and optionally by name and other metadata the user chooses to attach. The purpose of this database is to provide labels for slices that cannot be associated with public venues due to gaps in coverage. For example, many homes are not public venues and therefore would not be found in any public venue database, so a user may need to manually label his/her home. Labels such as “home” and “work” can also be automatically inferred.
- A set of additional labels associated with certain venue metadata such as a venue category. These labels could include descriptions of activities commonly applicable to the venue category (e.g., “jogging” at public parks or “dining out” at restaurants). These labels may be either predefined or automatically extracted, e.g., by analyzing the texts of some corpora such as online reviews. As with venue or place, the user can manually apply an activity label to a slice, or the
labeling 303 process can infer it based on a model of likelihood given the input context. - Public and user-specific calendar data—listings of public events and private appointments that can then be applied to matching, consistent slices.
- A database to store user corrections to automatically applied labels that were made by the system in error. This database has multiple uses. First, in the case of
continuous slicing 302 andlabeling 303, the correct label can be used duringreconciliation 3026 to prevent incorrect labels from being reapplied. Second, the presence of the correction indicates with high confidence what the correct description for the slice is, and can influence futureautomated labeling 303 decisions for similar slices. The user corrections may be stored, for example, in storyline storage 130 or similar data store accessible by the context refiner module 120.
- Conceptually, it is possible to view the
labeling 303 process as a collection of subprocesses responsible for outputting one type of label at a time. Labels of different types can then be run simultaneously on slices, or chained when one type of label is triggered by the presence of another (i.e., activities that are category- or venue-specific are only considered when a precedinglabeling 303 subprocess applies a corresponding category or venue label, respectively, to the slice). In general, thelabeling 303 process can be broken into three phases:label candidate search 3031,label candidate ranking 3032, andlabel application 3033, illustrated inFIG. 5 . -
Label Candidate Search 3031 - In the
label candidate search 3031 phase, the label sources are first queried to discover possible labels based on the slice's existing contexts. The following provides examples of how various label types can be searched. - Venues and places are found based on the slice's location, which by definition is a consistent estimate of a user's location over a period of time. However, there is a degree of uncertainty when using the associated slice location. Essentially, raw sensors providing locations are imprecise. The
label candidate search 3031 phase does not rely on the exact location represented by the slice, but instead expands the search within some radius calculated as an estimate of the uncertainty. For example, if a slice's location was calculated using Wi-Fi triangulation, the triangulation error is often in the tens to low hundreds of meters, so the search process may query for venues and places centered at the slice location within two hundred meters. - Events and appointments can be found based on the slice's location and time boundaries. An event at a venue would be matched against the overlapping time boundaries between the event and the slice. Appointments are also matched against location and time. Because event and appointment time boundaries are imprecise, and slice time boundaries may be imperfect, the slice's time boundaries do not need to exactly match those of an event or appointment. Similarly, the slice location does not need to be an exact match either. The
label candidate search 3031 finds possible events and appointments within the likely uncertainty radius of the slice location. - Several methods may also be used to find candidate activities. For example, based on the category and/or venue labels already applied to the slice, the
label candidate search 3031 process can bring up associated activity labels. As another example, the slice context can be compared to similar slices in the past if the user had previously labeled activities manually. For example, if a user previously labeled an activity at the same venue or a venue in the same category as the slice that has not yet been labeled with an activity, that activity would be considered as a candidate for labeling the slice. -
Label Candidate Ranking 3032 - Once a set of label candidates of a given type are found, the likelihood of each one given the contexts already associated with the slice is evaluated. In one embodiment, the likelihood of each label is computed and the labels are ranked. There may also be a threshold for likelihoods, such that if no label is deemed likely enough, none is applied to the slice at all—this avoids the case of having a label (e.g., an incomplete label) applied inappropriately. In one implementation, slices are constrained to only having one label of some types (e.g., venue labels), and the top-ranked label meeting the minimum likelihood threshold is applied to the slice. For other label types, multiple labels can be valid for a single slice, and all labels meeting the minimum threshold of likeliness are applied.
- Likelihoods are calculated by scoring a candidate label given the contexts already associated with a slice. A model is defined to be an algorithm for computing likelihoods given slice context. Models treat the various aspects of a slice context as features. Some example features include:
-
- Slice location—while the
label candidate search 3031 also uses location to discover the set of eligible labels to apply, a ranking model can further determine the likelihood of a label given its distance to the slice location, or relative likelihood between several candidates (e.g., a venue that is significantly farther from the slice location would to considered less likely, but two venues that differ only a small amount in distance from the slice location may be considered equally likely given the location context, all else being equal). - The particular user whose slice is being labeled—since users have individual differences, a model may use specific algorithms tailored to each to achieve the best labeling accuracy. One example of how an individual user could affect
label candidate ranking 3032 is for the ranking process to use the user's accumulated history of slices and labels, some of which the user may have explicitly confirmed to be accurate. Labels that occurred more often in the user's history may be considered more likely when labeling new contexts. - The beginning and end stay times—since different labels are more likely to occur at different times (e.g., restaurants at meal times, rock concerts in the evenings), and events, appointments and activities last for different lengths of time (e.g., movies between 1.5-3 hours), the likelihood of a particular label can depend on the corresponding day and time range.
- Slice location—while the
- Besides the context provided by the slice, models can use other sources of information to inform the likelihood estimate. Some example information sources include:
-
- Venue hours of operation—used to reduce the likelihood that a venue be applied to a slice when it's known to be closed during some significant portion of the slice's time boundaries.
- Venue popularity—e.g., relative popularity over all time compared to other venues, or historic popularity at the time of day, day of week, etc, which can indicate the likelihood that the label is applicable given the slice's time boundaries. If the duration of the slice is known, it can also be compared to the distribution of stay durations at the venue to determine whether the length of time spent in one place is consistent with other visits to the candidate venue.
- Category popularity—used when data is scarce about specific venues in the candidate list. This can also be relative to time of day, day of week, etc., and can also include typical stay durations so that the slice's time boundaries factor into the likelihood calculation.
- Routine—considering how often in the past the user has similar slices with the candidate label, can determine whether a certain label is more likely (if there are many such instances) or less likely (if there are few or no instances). Routine is not limited to only considering a specific user's historical patterns. Users might be clustered into cohort groups, or even aggregated into a global routine model, depending on data scarcity due to limited interactions with the system or the area where the slice occurs.
- Social interest—some users are more likely to visit a venue if their friends recommended it, if they have been there before, or were labeled as being there during an overlapping time period by the
labeling 303 process. Some of this information is available through existing social network APIs, for example recommendations may be based off of a friend “liking” the venue on FACEBOOK. Information about a user's friends visits to a venue can also come from a friend “checking in” or retrieved from the contextual history storage 130 (in embodiments where the storyline storage is centralized).
-
Label Application 3033 - One or more models can be run to process a slice's context(s) into labels, which are applied in
label application 3033. Conceptually, multiple models can be represented by a single meta-model that runs the appropriate features through its composing models. Once the meta-model outputs probabilities, labels deemed sufficiently likely are applied to the slice. In one embodiment, labels that are not deemed to be sufficiently likely can still be surfaced as options to the user should he/she wish to alter the label set by adding new labels, with thelabel candidate ranking 3032 intact. In such cases, it is not necessary for the same meta-model to produce thelabel candidate ranking 3032—different meta-models can rank labels differently to produce whatever is specified by the system design of a particular embodiment. - In one embodiment,
automatic label application 3033 does not require that every label is ranked solely by likelihood. For example, when users interact with the label candidates (e.g., to manually apply labels), it can be desirable to present candidate labels in different orders to make finding desired labels easier. For example, an alphabetical order or a hybrid order that incorporates both likelihoods and lexicographical positions can be used. Once labels are applied to slices, the additional contextual data is presentable to the user and available for further processing in a variety of forms. - Once labels are applied to slices, the new storyline data is presentable to the user and available for further processing in a variety of forms. As described above, one use of storyline data is to offer a historical perspective to the user, who may peruse the storyline to view his/her previous activities. Another use of storyline data is for further processing into interesting aggregations, e.g., showing how often certain activities were performed within some time period, or for processing by other software applications.
- A computer is adapted to execute computer program modules for providing the functionality described herein. As used herein, the term “module” refers to computer program logic utilized to provide the specified functionality. Thus, a module can be implemented in hardware, firmware, and/or software. In one embodiment, program modules are stored on a storage device, loaded into memory, and executed by a processor.
- Embodiments of the physical components described herein can include other and/or different modules than the ones described here. In addition, the functionality attributed to the modules can be performed by other or different modules in other embodiments. Moreover, this description occasionally omits the term “module” for purposes of clarity and convenience.
- Some portions of the above description describe the embodiments in terms of algorithmic processes or operations. These algorithmic descriptions and representations are commonly used by those skilled in the data processing arts to convey the substance of their work effectively to others skilled in the art. These operations, while described functionally, computationally, or logically, are understood to be implemented by computer programs comprising instructions for execution by a processor or equivalent electrical circuits, microcode, or the like. Furthermore, it has also proven convenient at times, to refer to these arrangements of functional operations as modules, without loss of generality. The described operations and their associated modules may be embodied in software, firmware, hardware, or any combinations thereof.
- The present invention also relates to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, or it may comprise a general-purpose computer selectively activated or reconfigured by a computer program stored on a computer readable medium that can be accessed by the computer. Such a computer program may be stored in a computer readable storage medium, such as, but is not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, application specific integrated circuits (ASICs), or any type of computer-readable storage medium suitable for storing electronic instructions, and each coupled to a computer system bus. Furthermore, the computers referred to in the specification may include a single processor or may be architectures employing multiple processor designs for increased computing capability.
- As used herein any reference to “one embodiment” or “an embodiment” means that a particular element, feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. The appearances of the phrase “in one embodiment” in various places in the specification are not necessarily all referring to the same embodiment.
- As used herein, the terms “comprises,” “comprising,” “includes,” “including,” “has,” “having” or any other variation thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, article, or apparatus that comprises a list of elements is not necessarily limited to only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Further, unless expressly stated to the contrary, “or” refers to an inclusive or and not to an exclusive or. For example, a condition A or B is satisfied by any one of the following: A is true (or present) and B is false (or not present), A is false (or not present) and B is true (or present), and both A and B are true (or present).
- In addition, use of the “a” or “an” are employed to describe elements and components of the embodiments herein. This is done merely for convenience and to give a general sense of the disclosure. This description should be read to include one or at least one and the singular also includes the plural unless it is obvious that it is meant otherwise.
- Upon reading this disclosure, those of skill in the art will appreciate still additional alternative structural and functional designs. Thus, while particular embodiments and applications have been illustrated and described, it is to be understood that the present invention is not limited to the precise construction and components disclosed herein and that various modifications, changes and variations which will be apparent to those skilled in the art may be made in the arrangement, operation and details of the method and apparatus disclosed herein without departing from the spirit and scope as defined in the appended claims.
Claims (26)
1. A method of creating context slices of a storyline of a user's movements, the method comprising:
collecting context data from a plurality of observation sources in a span of time;
grooming the collected context data to reduce errors; and
segmenting the span of time into a sequence of distinct context slices based on the groomed context data, each distinct context slice associated with one selected from a group consisting of a stay in a location, travel from one location to another, and a gap between contexts.
2. The method of claim 1 , wherein the plurality of observation sources include a location module, an activity module, and a social module.
3. The method of claim 1 , wherein grooming further comprises filtering collected context data to eliminate context data that are deemed more inaccurate than a desirable threshold.
4. The method of claim 3 , wherein the desirable threshold is sensor-specific and depends on expected error characteristics of the respective observation source.
5. The method of claim 1 , wherein grooming further comprises smoothing sequences of context data from an observation source to limit noise.
6. The method of claim 1 , wherein grooming further comprises smoothing a sequence of context data from an observation source responsive to conflicting context data from another observation source.
7. The method of claim 1 , wherein grooming further comprises interpolating context data between sensed context data.
8. The method of claim 1 , wherein segmenting comprises performing clustering to distinguish a stay in a location from travel from one location to another.
9. The method of claim 1 , wherein segmenting comprises performing time-series analysis to detect inflection points in a distribution of context data to determine how to segment a span of time into separate context slices.
10. The method of claim 1 , further comprising removing a short context slice by merging it with an adjacent context slice that is likely to correspond to a same activity as the short context slice.
11. The method of claim 1 , further comprising reconciling the sequence of distinct context slices with preexisting context slices by comparing type and length.
12. The method of claim 1 , further comprising reconciling location data from a plurality of sensors into a single sequence of context slices.
13. The method of claim 12 , further comprising varying a sampling frequency of the plurality of sensors in order to lower energy requirements of collecting context data.
14. A non-transitory computer-readable storage medium having computer program instructions embodied therein for creating context slices of a storyline of a user's movements, the computer program instructions comprising instructions for:
collecting context data from a plurality of observation sources in a span of time;
grooming the collected context data to reduce errors; and
segmenting the span of time into a sequence of distinct context slices based on the groomed context data, each distinct context slice associated with one selected from a group consisting of a stay in a location, travel from one location to another, and a gap between contexts.
15. The computer-readable storage medium of claim 14 , wherein the plurality of observation sources include a location module, an activity module, and a social module.
16. The computer-readable storage medium of claim 14 , wherein the instructions for grooming further comprise instructions for filtering collected context data to eliminate context data that are deemed more inaccurate than a desirable threshold.
17. The computer-readable storage medium of claim 16 , wherein the desirable threshold is sensor-specific and depends on expected error characteristics of the respective observation source.
18. The computer-readable storage medium of claim 14 , wherein the instructions for grooming further comprise instructions for smoothing sequences of context data from an observation source to limit noise.
19. The computer-readable storage medium of claim 14 , wherein the instructions for grooming further comprise instructions for smoothing a sequence of context data from an observation source responsive to conflicting context data from another observation source.
20. The computer-readable storage medium of claim 14 , wherein the instructions for grooming further comprise instructions for interpolating context data between sensed context data.
21. The computer-readable storage medium of claim 14 , wherein the instructions for segmenting comprise instructions for performing clustering to distinguish a stay in a location from travel from one location to another.
22. The computer-readable storage medium of claim 14 , wherein the instructions for segmenting comprise instructions for performing time-series analysis to detect inflection points in a distribution of context data to determine how to segment a span of time into separate context slices.
23. The computer-readable storage medium of claim 14 , the instructions further comprising instructions for removing a short context slice by merging it with an adjacent context slice that is likely to correspond to a same activity as the short context slice.
24. The computer-readable storage medium of claim 14 , the instructions further comprising instructions for reconciling the sequence of distinct context slices with preexisting context slices by comparing type and length.
25. The computer-readable storage medium of claim 14 , the instructions further comprising instructions for reconciling location data from a plurality of sensors into a single sequence of context slices.
26. The computer-readable storage medium of claim 25 , the instructions further comprising instructions for varying a sampling frequency of the plurality of sensors in order to lower energy requirements of collecting context data.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US13/950,150 US20140031060A1 (en) | 2012-07-25 | 2013-07-24 | Creating Context Slices of a Storyline from Mobile Device Data |
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201261675732P | 2012-07-25 | 2012-07-25 | |
US201261675733P | 2012-07-25 | 2012-07-25 | |
US13/950,150 US20140031060A1 (en) | 2012-07-25 | 2013-07-24 | Creating Context Slices of a Storyline from Mobile Device Data |
Publications (1)
Publication Number | Publication Date |
---|---|
US20140031060A1 true US20140031060A1 (en) | 2014-01-30 |
Family
ID=49995366
Family Applications (7)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/950,117 Active US9020864B2 (en) | 2012-07-25 | 2013-07-24 | Recommendation agent using a personality model determined from mobile device data |
US13/950,150 Abandoned US20140031060A1 (en) | 2012-07-25 | 2013-07-24 | Creating Context Slices of a Storyline from Mobile Device Data |
US13/950,177 Abandoned US20140032358A1 (en) | 2012-07-25 | 2013-07-24 | Sharing Recommendation Agents |
US13/950,105 Active US9179250B2 (en) | 2012-07-25 | 2013-07-24 | Recommendation agent using a routine model determined from mobile device data |
US13/950,169 Active US8838436B2 (en) | 2012-07-25 | 2013-07-24 | Labeling context slices to produce a storyline from mobile device data |
US13/950,224 Active US8892480B2 (en) | 2012-07-25 | 2013-07-24 | Contextual information provider |
US14/629,456 Abandoned US20150170042A1 (en) | 2012-07-25 | 2015-02-23 | Recommendation agent using a personality model determined from mobile device data |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/950,117 Active US9020864B2 (en) | 2012-07-25 | 2013-07-24 | Recommendation agent using a personality model determined from mobile device data |
Family Applications After (5)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/950,177 Abandoned US20140032358A1 (en) | 2012-07-25 | 2013-07-24 | Sharing Recommendation Agents |
US13/950,105 Active US9179250B2 (en) | 2012-07-25 | 2013-07-24 | Recommendation agent using a routine model determined from mobile device data |
US13/950,169 Active US8838436B2 (en) | 2012-07-25 | 2013-07-24 | Labeling context slices to produce a storyline from mobile device data |
US13/950,224 Active US8892480B2 (en) | 2012-07-25 | 2013-07-24 | Contextual information provider |
US14/629,456 Abandoned US20150170042A1 (en) | 2012-07-25 | 2015-02-23 | Recommendation agent using a personality model determined from mobile device data |
Country Status (4)
Country | Link |
---|---|
US (7) | US9020864B2 (en) |
EP (1) | EP2877935A4 (en) |
KR (1) | KR101512278B1 (en) |
WO (1) | WO2014018687A1 (en) |
Cited By (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20140101104A1 (en) * | 2012-09-26 | 2014-04-10 | Huawei Technologies Co., Ltd. | Method for generating terminal log and terminal |
US20140136186A1 (en) * | 2012-11-15 | 2014-05-15 | Consorzio Nazionale Interuniversitario Per Le Telecomunicazioni | Method and system for generating an alternative audible, visual and/or textual data based upon an original audible, visual and/or textual data |
US20150356421A1 (en) * | 2014-06-05 | 2015-12-10 | Mitsubishi Electric Research Laboratories, Inc. | Method for Learning Exemplars for Anomaly Detection |
WO2016057531A1 (en) * | 2014-10-08 | 2016-04-14 | Analog Devices, Inc. | Configurable pre-processing array |
CN106528679A (en) * | 2016-10-24 | 2017-03-22 | 天津大学 | Time series analysis method based on multilinear autoregression model |
CN107004170A (en) * | 2014-12-05 | 2017-08-01 | 微软技术许可有限责任公司 | To be used to the service content of exceptional cast customization |
CN111406423A (en) * | 2017-11-30 | 2020-07-10 | 瑞典爱立信有限公司 | Core network allocation processing |
US11205103B2 (en) | 2016-12-09 | 2021-12-21 | The Research Foundation for the State University | Semisupervised autoencoder for sentiment analysis |
US11250630B2 (en) | 2014-11-18 | 2022-02-15 | Hallmark Cards, Incorporated | Immersive story creation |
Families Citing this family (242)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8677377B2 (en) | 2005-09-08 | 2014-03-18 | Apple Inc. | Method and apparatus for building an intelligent automated assistant |
US9318108B2 (en) | 2010-01-18 | 2016-04-19 | Apple Inc. | Intelligent automated assistant |
US8977255B2 (en) | 2007-04-03 | 2015-03-10 | Apple Inc. | Method and system for operating a multi-function portable electronic device using voice-activation |
US8126881B1 (en) | 2007-12-12 | 2012-02-28 | Vast.com, Inc. | Predictive conversion systems and methods |
US10002189B2 (en) | 2007-12-20 | 2018-06-19 | Apple Inc. | Method and apparatus for searching using an active ontology |
US9330720B2 (en) | 2008-01-03 | 2016-05-03 | Apple Inc. | Methods and apparatus for altering audio output signals |
US20100030549A1 (en) | 2008-07-31 | 2010-02-04 | Lee Michael M | Mobile device having human language translation capability with positional feedback |
US8676904B2 (en) | 2008-10-02 | 2014-03-18 | Apple Inc. | Electronic devices with voice command and contextual data processing capabilities |
US20120311585A1 (en) | 2011-06-03 | 2012-12-06 | Apple Inc. | Organizing task items that represent tasks to perform |
US10276170B2 (en) | 2010-01-18 | 2019-04-30 | Apple Inc. | Intelligent automated assistant |
US8682667B2 (en) | 2010-02-25 | 2014-03-25 | Apple Inc. | User profiling for selecting user specific voice input processing information |
US9262612B2 (en) | 2011-03-21 | 2016-02-16 | Apple Inc. | Device access using voice authentication |
US10057736B2 (en) | 2011-06-03 | 2018-08-21 | Apple Inc. | Active transport based notifications |
JP5562362B2 (en) * | 2012-02-01 | 2014-07-30 | ビッグローブ株式会社 | RECOMMENDED INFORMATION PROVIDING DEVICE, MOBILE TERMINAL, RECOMMENDED INFORMATION PROVIDING METHOD, RECOMMENDED INFORMATION PROVIDING SUPPORT METHOD, AND PROGRAM |
US10134385B2 (en) | 2012-03-02 | 2018-11-20 | Apple Inc. | Systems and methods for name pronunciation |
US9202047B2 (en) | 2012-05-14 | 2015-12-01 | Qualcomm Incorporated | System, apparatus, and method for adaptive observation of mobile device behavior |
US9609456B2 (en) * | 2012-05-14 | 2017-03-28 | Qualcomm Incorporated | Methods, devices, and systems for communicating behavioral analysis information |
US9690635B2 (en) | 2012-05-14 | 2017-06-27 | Qualcomm Incorporated | Communicating behavior information in a mobile computing device |
US10417037B2 (en) | 2012-05-15 | 2019-09-17 | Apple Inc. | Systems and methods for integrating third party services with a digital assistant |
US9020864B2 (en) | 2012-07-25 | 2015-04-28 | Aro, Inc. | Recommendation agent using a personality model determined from mobile device data |
US9747440B2 (en) | 2012-08-15 | 2017-08-29 | Qualcomm Incorporated | On-line behavioral analysis engine in mobile device with multiple analyzer model providers |
US9319897B2 (en) | 2012-08-15 | 2016-04-19 | Qualcomm Incorporated | Secure behavior analysis over trusted execution environment |
US9495537B2 (en) | 2012-08-15 | 2016-11-15 | Qualcomm Incorporated | Adaptive observation of behavioral features on a mobile device |
CN103685208B (en) * | 2012-09-25 | 2017-07-14 | 华为技术有限公司 | User data mask method, terminal device and server |
US20140108307A1 (en) * | 2012-10-12 | 2014-04-17 | Wipro Limited | Methods and systems for providing personalized and context-aware suggestions |
JP6814236B2 (en) * | 2012-11-30 | 2021-01-13 | パナソニック インテレクチュアル プロパティ コーポレーション オブ アメリカPanasonic Intellectual Property Corporation of America | Information processing method |
US9686023B2 (en) | 2013-01-02 | 2017-06-20 | Qualcomm Incorporated | Methods and systems of dynamically generating and using device-specific and device-state-specific classifier models for the efficient classification of mobile device behaviors |
US10089582B2 (en) | 2013-01-02 | 2018-10-02 | Qualcomm Incorporated | Using normalized confidence values for classifying mobile device behaviors |
US9684870B2 (en) | 2013-01-02 | 2017-06-20 | Qualcomm Incorporated | Methods and systems of using boosted decision stumps and joint feature selection and culling algorithms for the efficient classification of mobile device behaviors |
RU2661773C2 (en) * | 2013-01-03 | 2018-07-19 | Синарра Системз Пте. Лтд. | Location and time-aware systems and methods for mobile user context detection |
US9742559B2 (en) | 2013-01-22 | 2017-08-22 | Qualcomm Incorporated | Inter-module authentication for securing application execution integrity within a computing device |
BR112015018905B1 (en) | 2013-02-07 | 2022-02-22 | Apple Inc | Voice activation feature operation method, computer readable storage media and electronic device |
US9491187B2 (en) | 2013-02-15 | 2016-11-08 | Qualcomm Incorporated | APIs for obtaining device-specific behavior classifier models from the cloud |
US10007946B1 (en) | 2013-03-07 | 2018-06-26 | Vast.com, Inc. | Systems, methods, and devices for measuring similarity of and generating recommendations for unique items |
US9465873B1 (en) | 2013-03-07 | 2016-10-11 | Vast.com, Inc. | Systems, methods, and devices for identifying and presenting identifications of significant attributes of unique items |
US9104718B1 (en) | 2013-03-07 | 2015-08-11 | Vast.com, Inc. | Systems, methods, and devices for measuring similarity of and generating recommendations for unique items |
US9235804B1 (en) * | 2013-03-12 | 2016-01-12 | Google Inc. | System and method for selecting and serving content items based on sensor data from mobile devices |
US9830635B1 (en) | 2013-03-13 | 2017-11-28 | Vast.com, Inc. | Systems, methods, and devices for determining and displaying market relative position of unique items |
US10652394B2 (en) | 2013-03-14 | 2020-05-12 | Apple Inc. | System and method for processing voicemail |
US10748529B1 (en) | 2013-03-15 | 2020-08-18 | Apple Inc. | Voice activated device for use with a voice-based digital assistant |
JP2014207110A (en) * | 2013-04-12 | 2014-10-30 | 株式会社日立ハイテクノロジーズ | Observation apparatus and observation method |
WO2014197335A1 (en) | 2013-06-08 | 2014-12-11 | Apple Inc. | Interpreting and acting upon commands that involve sharing information with remote devices |
US10176167B2 (en) | 2013-06-09 | 2019-01-08 | Apple Inc. | System and method for inferring user intent from speech inputs |
CN105264524B (en) | 2013-06-09 | 2019-08-02 | 苹果公司 | For realizing the equipment, method and graphic user interface of the session continuity of two or more examples across digital assistants |
US9467411B2 (en) | 2013-07-31 | 2016-10-11 | International Business Machines Corporation | Identifying content in an incoming message on a social network |
US9405964B1 (en) * | 2013-09-09 | 2016-08-02 | Amazon Technologies, Inc. | Processes for generating content sharing recommendations based on image content analysis |
US9338242B1 (en) | 2013-09-09 | 2016-05-10 | Amazon Technologies, Inc. | Processes for generating content sharing recommendations |
US9531823B1 (en) | 2013-09-09 | 2016-12-27 | Amazon Technologies, Inc. | Processes for generating content sharing recommendations based on user feedback data |
US10049413B2 (en) | 2013-09-20 | 2018-08-14 | Vulcan Technologies Llc | Automatically creating a hierarchical storyline from mobile device data |
KR102268052B1 (en) * | 2013-11-11 | 2021-06-22 | 삼성전자주식회사 | Display apparatus, server apparatus and user interface screen providing method thereof |
US20150142824A1 (en) * | 2013-11-21 | 2015-05-21 | At&T Mobility Ii Llc | Situational Content Based on Context |
US9760756B2 (en) | 2013-12-05 | 2017-09-12 | Aro, Inc. | Venue identification using sensor fingerprints |
US9541652B2 (en) | 2013-12-06 | 2017-01-10 | Aro, Inc. | Accurate mobile context detection at low sensor cost |
US10296160B2 (en) | 2013-12-06 | 2019-05-21 | Apple Inc. | Method for extracting salient dialog usage from live data |
US10127596B1 (en) | 2013-12-10 | 2018-11-13 | Vast.com, Inc. | Systems, methods, and devices for generating recommendations of unique items |
US10078691B2 (en) * | 2013-12-27 | 2018-09-18 | Futurewei Technologies, Inc. | System and method for biometrics-based music recommendation |
US10089310B2 (en) | 2014-01-14 | 2018-10-02 | Microsoft Technology Licensing, Llc | Complementary and shadow calendars |
US11030708B2 (en) | 2014-02-28 | 2021-06-08 | Christine E. Akutagawa | Method of and device for implementing contagious illness analysis and tracking |
US9704205B2 (en) | 2014-02-28 | 2017-07-11 | Christine E. Akutagawa | Device for implementing body fluid analysis and social networking event planning |
US9959508B2 (en) * | 2014-03-20 | 2018-05-01 | CloudMade, Inc. | Systems and methods for providing information for predicting desired information and taking actions related to user needs in a mobile device |
US9710546B2 (en) | 2014-03-28 | 2017-07-18 | Microsoft Technology Licensing, Llc | Explicit signals personalized search |
KR102216049B1 (en) * | 2014-04-21 | 2021-02-15 | 삼성전자주식회사 | System and method for semantic labeling |
US11074293B2 (en) | 2014-04-22 | 2021-07-27 | Microsoft Technology Licensing, Llc | Generating probabilistic transition data |
US10110677B2 (en) * | 2014-05-06 | 2018-10-23 | Avago Technologies General Ip (Singapore) Pte. Ltd. | Context-aware decision making |
US11120408B2 (en) | 2014-05-06 | 2021-09-14 | Microsoft Technology Licensing, Llc | Scheduling conflict notification |
EP3149728B1 (en) | 2014-05-30 | 2019-01-16 | Apple Inc. | Multi-command single utterance input method |
US10170123B2 (en) | 2014-05-30 | 2019-01-01 | Apple Inc. | Intelligent assistant for home automation |
US9430463B2 (en) | 2014-05-30 | 2016-08-30 | Apple Inc. | Exemplar-based natural language processing |
US9633004B2 (en) | 2014-05-30 | 2017-04-25 | Apple Inc. | Better resolution when referencing to concepts |
US9715875B2 (en) | 2014-05-30 | 2017-07-25 | Apple Inc. | Reducing the need for manual start/end-pointing and trigger phrases |
US9338493B2 (en) | 2014-06-30 | 2016-05-10 | Apple Inc. | Intelligent automated assistant for TV user interactions |
WO2016020906A2 (en) * | 2014-08-08 | 2016-02-11 | Samsung Electronics Co., Ltd. | Electronic system with custom notification mechanism and method of operation thereof |
US10887376B2 (en) | 2014-08-08 | 2021-01-05 | Samsung Electronics Co., Ltd. | Electronic system with custom notification mechanism and method of operation thereof |
US9818400B2 (en) | 2014-09-11 | 2017-11-14 | Apple Inc. | Method and apparatus for discovering trending terms in speech requests |
US10074360B2 (en) | 2014-09-30 | 2018-09-11 | Apple Inc. | Providing an indication of the suitability of speech recognition |
US9668121B2 (en) | 2014-09-30 | 2017-05-30 | Apple Inc. | Social reminders |
US10127911B2 (en) | 2014-09-30 | 2018-11-13 | Apple Inc. | Speaker identification and unsupervised speaker adaptation techniques |
US10192583B2 (en) | 2014-10-10 | 2019-01-29 | Samsung Electronics Co., Ltd. | Video editing using contextual data and content discovery using clusters |
CN104360875B (en) * | 2014-10-22 | 2017-09-12 | 小米科技有限责任公司 | Privacy mode starts method and apparatus |
US20160165391A1 (en) * | 2014-11-12 | 2016-06-09 | Homeaway, Inc. | System and processes to facilitate context-based messaging |
AU2015353736A1 (en) * | 2014-11-25 | 2017-06-15 | Fynd Technologies, Inc. | Geolocation bracelet, systems, and methods |
US10459925B2 (en) * | 2014-12-08 | 2019-10-29 | Iprova Sarl | Computer-enabled method of assisting to generate an innovation |
US9904932B2 (en) * | 2014-12-29 | 2018-02-27 | Google Llc | Analyzing semantic places and related data from a plurality of location data reports |
US10242107B2 (en) | 2015-01-11 | 2019-03-26 | Microsoft Technology Licensing, Llc | Extraction of quantitative data from online content |
US10176457B2 (en) | 2015-02-05 | 2019-01-08 | Sap Se | System and method automatically learning and optimizing sequence order |
US9554356B2 (en) | 2015-02-19 | 2017-01-24 | Microsoft Technology Licensing, Llc | Personalized reminders |
US10135937B2 (en) | 2015-02-19 | 2018-11-20 | Microsoft Technology Licensing, Llc | Personalized notifications |
US10152299B2 (en) | 2015-03-06 | 2018-12-11 | Apple Inc. | Reducing response latency of intelligent automated assistants |
US10567477B2 (en) | 2015-03-08 | 2020-02-18 | Apple Inc. | Virtual assistant continuity |
US9886953B2 (en) | 2015-03-08 | 2018-02-06 | Apple Inc. | Virtual assistant activation |
US9721566B2 (en) | 2015-03-08 | 2017-08-01 | Apple Inc. | Competing devices responding to voice triggers |
US10462211B2 (en) | 2015-03-09 | 2019-10-29 | International Business Machines Corporation | System and method for providing more appropriate question/answer responses based upon profiles |
JP2016170667A (en) * | 2015-03-13 | 2016-09-23 | ソニー株式会社 | Information processing device, information processing method and program |
US10185973B2 (en) * | 2015-04-07 | 2019-01-22 | Microsoft Technology Licensing, Llc | Inferring venue visits using semantic information |
US10460227B2 (en) | 2015-05-15 | 2019-10-29 | Apple Inc. | Virtual assistant in a communication session |
US10083688B2 (en) | 2015-05-27 | 2018-09-25 | Apple Inc. | Device voice control for selecting a displayed affordance |
US10200824B2 (en) | 2015-05-27 | 2019-02-05 | Apple Inc. | Systems and methods for proactively identifying and surfacing relevant content on a touch-sensitive device |
US9578173B2 (en) | 2015-06-05 | 2017-02-21 | Apple Inc. | Virtual assistant aided communication with 3rd party service in a communication session |
US11025565B2 (en) | 2015-06-07 | 2021-06-01 | Apple Inc. | Personalized prediction of responses for instant messaging |
US9939923B2 (en) | 2015-06-19 | 2018-04-10 | Microsoft Technology Licensing, Llc | Selecting events based on user input and current context |
US20160378747A1 (en) | 2015-06-29 | 2016-12-29 | Apple Inc. | Virtual assistant for media playback |
US10803391B2 (en) * | 2015-07-29 | 2020-10-13 | Google Llc | Modeling personal entities on a mobile device using embeddings |
KR101656785B1 (en) * | 2015-08-28 | 2016-09-12 | 주식회사 코노랩스 | Method, system and non-transitory computer-readable recording medium for providing notification on schedule |
US10747498B2 (en) | 2015-09-08 | 2020-08-18 | Apple Inc. | Zero latency digital assistant |
US10331312B2 (en) | 2015-09-08 | 2019-06-25 | Apple Inc. | Intelligent automated assistant in a media environment |
US10740384B2 (en) | 2015-09-08 | 2020-08-11 | Apple Inc. | Intelligent automated assistant for media search and playback |
US10671428B2 (en) | 2015-09-08 | 2020-06-02 | Apple Inc. | Distributed personal assistant |
US10691736B2 (en) | 2015-09-25 | 2020-06-23 | International Business Machines Corporation | Contextualized analytics platform |
US10445655B2 (en) * | 2015-09-29 | 2019-10-15 | Cognitive Scale, Inc. | Cognitive learning framework |
US20170091660A1 (en) * | 2015-09-29 | 2017-03-30 | Cognitive Scale, Inc. | Method for Cognitive Learning |
US10445647B2 (en) * | 2015-09-29 | 2019-10-15 | Cognitive Scale, Inc. | Method for performing a cognitive machine learning operation |
US10445656B2 (en) * | 2015-09-29 | 2019-10-15 | Cognitive Scale, Inc. | Cognitive machine learning system |
US10445646B2 (en) * | 2015-09-29 | 2019-10-15 | Cognitive Scale, Inc. | Method for performing a cognitive learning operation via a cognitive learning framework |
US10726340B2 (en) | 2015-09-29 | 2020-07-28 | Cognitive Scale, Inc. | Cognitive learning system having a knowledge model |
US10691473B2 (en) | 2015-11-06 | 2020-06-23 | Apple Inc. | Intelligent automated assistant in a messaging environment |
US10956666B2 (en) | 2015-11-09 | 2021-03-23 | Apple Inc. | Unconventional virtual assistant interactions |
US10049668B2 (en) | 2015-12-02 | 2018-08-14 | Apple Inc. | Applying neural network language models to weighted finite state transducers for automatic speech recognition |
US20170161272A1 (en) * | 2015-12-08 | 2017-06-08 | International Business Machines Corporation | Social media search assist |
US9796388B2 (en) | 2015-12-17 | 2017-10-24 | Ford Global Technologies, Llc | Vehicle mode determination |
CN106897313B (en) * | 2015-12-21 | 2020-10-27 | 中国联合网络通信集团有限公司 | Mass user service preference evaluation method and device |
US10990714B2 (en) * | 2015-12-22 | 2021-04-27 | Bwxt Mpower, Inc. | Apparatus and method for safety analysis evaluation with data-driven workflow |
US10223066B2 (en) | 2015-12-23 | 2019-03-05 | Apple Inc. | Proactive assistance based on dialog communication between devices |
US20170185917A1 (en) * | 2015-12-29 | 2017-06-29 | Cognitive Scale, Inc. | Method for Monitoring Interactions to Build a Cognitive Profile |
US20170185916A1 (en) * | 2015-12-29 | 2017-06-29 | Cognitive Scale, Inc. | Cognitive Profile Builder |
US20170221125A1 (en) | 2016-02-03 | 2017-08-03 | International Business Machines Corporation | Matching customer and product behavioral traits |
US9813875B2 (en) * | 2016-03-31 | 2017-11-07 | Intel Corporation | Ad-hoc community context awareness for mobile device |
US10178171B2 (en) | 2016-04-21 | 2019-01-08 | Samsung Electronics Company, Ltd. | Content management system for distribution of content |
US10225369B2 (en) | 2016-06-02 | 2019-03-05 | At&T Intellectual Property I, L.P. | Method and apparatus for providing a recommended action for a venue via a network |
US11227589B2 (en) | 2016-06-06 | 2022-01-18 | Apple Inc. | Intelligent list reading |
US10049663B2 (en) | 2016-06-08 | 2018-08-14 | Apple, Inc. | Intelligent automated assistant for media exploration |
US10586535B2 (en) | 2016-06-10 | 2020-03-10 | Apple Inc. | Intelligent digital assistant in a multi-tasking environment |
DK201670540A1 (en) | 2016-06-11 | 2018-01-08 | Apple Inc | Application integration with a digital assistant |
DK179415B1 (en) | 2016-06-11 | 2018-06-14 | Apple Inc | Intelligent device arbitration and control |
WO2018016843A1 (en) * | 2016-07-20 | 2018-01-25 | 안강석 | User-personalised value information search, social network configuration system, and method therefor |
US10460255B2 (en) | 2016-07-29 | 2019-10-29 | Splunk Inc. | Machine learning in edge analytics |
US11087236B2 (en) | 2016-07-29 | 2021-08-10 | Splunk Inc. | Transmitting machine learning models to edge devices for edge analytics |
US10536351B2 (en) | 2016-07-29 | 2020-01-14 | Splunk Inc. | Analytics for edge devices |
US11455545B2 (en) * | 2016-08-10 | 2022-09-27 | Palo Alto Research Center Incorporated | Computer-implemented system and method for building context models in real time |
US10474753B2 (en) | 2016-09-07 | 2019-11-12 | Apple Inc. | Language identification using recurrent neural networks |
US11068791B2 (en) * | 2016-09-14 | 2021-07-20 | International Business Machines Corporation | Providing recommendations utilizing a user profile |
US10043516B2 (en) | 2016-09-23 | 2018-08-07 | Apple Inc. | Intelligent automated assistant |
US10726466B2 (en) | 2016-11-03 | 2020-07-28 | International Business Machines Corporation | System and method for recommending products to bridge gaps between desired and actual personal branding |
US10957306B2 (en) * | 2016-11-16 | 2021-03-23 | International Business Machines Corporation | Predicting personality traits based on text-speech hybrid data |
US11281993B2 (en) | 2016-12-05 | 2022-03-22 | Apple Inc. | Model and ensemble compression for metric learning |
US11062225B2 (en) * | 2016-12-09 | 2021-07-13 | Adobe Inc. | Techniques for providing sequential recommendations to users |
US11409463B2 (en) | 2016-12-28 | 2022-08-09 | Microsoft Technology Licensing, Llc | Systems and methods for contextual memory capture and recall |
US11204787B2 (en) | 2017-01-09 | 2021-12-21 | Apple Inc. | Application integration with a digital assistant |
US10909371B2 (en) | 2017-01-19 | 2021-02-02 | Samsung Electronics Co., Ltd. | System and method for contextual driven intelligence |
US10515355B2 (en) | 2017-01-19 | 2019-12-24 | Mastercard International Incorporated | Systems and methods for collecting device data from digital wallet authentications |
EP3559804B1 (en) | 2017-01-19 | 2024-03-06 | Samsung Electronics Co., Ltd. | Vision intelligence management for electronic devices |
CN108446281B (en) * | 2017-02-13 | 2021-03-12 | 北京嘀嘀无限科技发展有限公司 | Method, device and storage medium for determining user intimacy |
US10356074B2 (en) * | 2017-02-15 | 2019-07-16 | Bank Of America Corporation | Computing system for resource management based on resource attributes and predicting user actions |
US10476813B2 (en) | 2017-02-15 | 2019-11-12 | Bank Of America Corporation | Computerized system for identifying and redistributing complementary resources |
US10990912B2 (en) | 2017-02-15 | 2021-04-27 | Bank Of America Corporation | System for identification and integration of like resources and configuring resources for common use |
US10298706B2 (en) * | 2017-02-20 | 2019-05-21 | At&T Intellectual Property I, L.P. | Social media and location-based informed entertainment recommendations |
WO2018164435A1 (en) | 2017-03-08 | 2018-09-13 | Samsung Electronics Co., Ltd. | Electronic apparatus, method for controlling the same, and non-transitory computer readable recording medium |
DK201770383A1 (en) | 2017-05-09 | 2018-12-14 | Apple Inc. | User interface for correcting recognition errors |
US10417266B2 (en) | 2017-05-09 | 2019-09-17 | Apple Inc. | Context-aware ranking of intelligent response suggestions |
US10726832B2 (en) | 2017-05-11 | 2020-07-28 | Apple Inc. | Maintaining privacy of personal information |
DK201770439A1 (en) | 2017-05-11 | 2018-12-13 | Apple Inc. | Offline personal assistant |
US10395654B2 (en) | 2017-05-11 | 2019-08-27 | Apple Inc. | Text normalization based on a data-driven learning network |
DK180048B1 (en) | 2017-05-11 | 2020-02-04 | Apple Inc. | MAINTAINING THE DATA PROTECTION OF PERSONAL INFORMATION |
DK201770429A1 (en) | 2017-05-12 | 2018-12-14 | Apple Inc. | Low-latency intelligent automated assistant |
DK179496B1 (en) | 2017-05-12 | 2019-01-15 | Apple Inc. | USER-SPECIFIC Acoustic Models |
US11301477B2 (en) | 2017-05-12 | 2022-04-12 | Apple Inc. | Feedback analysis of a digital assistant |
DK179745B1 (en) | 2017-05-12 | 2019-05-01 | Apple Inc. | SYNCHRONIZATION AND TASK DELEGATION OF A DIGITAL ASSISTANT |
DK201770432A1 (en) | 2017-05-15 | 2018-12-21 | Apple Inc. | Hierarchical belief states for digital assistants |
DK201770431A1 (en) | 2017-05-15 | 2018-12-20 | Apple Inc. | Optimizing dialogue policy decisions for digital assistants using implicit feedback |
US20180336892A1 (en) | 2017-05-16 | 2018-11-22 | Apple Inc. | Detecting a trigger of a digital assistant |
DK179549B1 (en) | 2017-05-16 | 2019-02-12 | Apple Inc. | Far-field extension for digital assistant services |
US10303715B2 (en) | 2017-05-16 | 2019-05-28 | Apple Inc. | Intelligent automated assistant for media exploration |
US10403278B2 (en) | 2017-05-16 | 2019-09-03 | Apple Inc. | Methods and systems for phonetic matching in digital assistant services |
US10311144B2 (en) | 2017-05-16 | 2019-06-04 | Apple Inc. | Emoji word sense disambiguation |
US10657328B2 (en) | 2017-06-02 | 2020-05-19 | Apple Inc. | Multi-task recurrent neural network architecture for efficient morphology handling in neural language modeling |
US11188809B2 (en) * | 2017-06-27 | 2021-11-30 | International Business Machines Corporation | Optimizing personality traits of virtual agents |
US11373229B2 (en) | 2017-07-13 | 2022-06-28 | The Toronto-Dominion Bank | Contextually-aware recommendation and translation engine |
US11067409B2 (en) | 2017-07-14 | 2021-07-20 | Allstate Insurance Company | Distributed data processing systems for processing remotely captured sensor data |
US10445429B2 (en) | 2017-09-21 | 2019-10-15 | Apple Inc. | Natural language understanding using vocabularies with compressed serialized tries |
US10755051B2 (en) | 2017-09-29 | 2020-08-25 | Apple Inc. | Rule-based natural language processing |
US10460748B2 (en) | 2017-10-04 | 2019-10-29 | The Toronto-Dominion Bank | Conversational interface determining lexical personality score for response generation with synonym replacement |
US10339931B2 (en) | 2017-10-04 | 2019-07-02 | The Toronto-Dominion Bank | Persona-based conversational interface personalization using social network preferences |
US10713320B2 (en) * | 2017-10-09 | 2020-07-14 | Facebook, Inc. | De-duping events on online social networks |
US10268704B1 (en) | 2017-10-12 | 2019-04-23 | Vast.com, Inc. | Partitioned distributed database systems, devices, and methods |
US11036938B2 (en) * | 2017-10-20 | 2021-06-15 | ConceptDrop Inc. | Machine learning system for optimizing projects |
US10636424B2 (en) | 2017-11-30 | 2020-04-28 | Apple Inc. | Multi-turn canned dialog |
CN109960539A (en) * | 2017-12-21 | 2019-07-02 | 广东欧珀移动通信有限公司 | Application program preloads method, apparatus, storage medium and mobile terminal |
US10733982B2 (en) | 2018-01-08 | 2020-08-04 | Apple Inc. | Multi-directional dialog |
US10733375B2 (en) | 2018-01-31 | 2020-08-04 | Apple Inc. | Knowledge-based framework for improving natural language understanding |
US10789959B2 (en) | 2018-03-02 | 2020-09-29 | Apple Inc. | Training speaker recognition models for digital assistants |
US10592604B2 (en) | 2018-03-12 | 2020-03-17 | Apple Inc. | Inverse text normalization for automatic speech recognition |
US10818288B2 (en) | 2018-03-26 | 2020-10-27 | Apple Inc. | Natural assistant interaction |
US10909331B2 (en) | 2018-03-30 | 2021-02-02 | Apple Inc. | Implicit identification of translation payload with neural machine translation |
US10928918B2 (en) | 2018-05-07 | 2021-02-23 | Apple Inc. | Raise to speak |
US11145294B2 (en) | 2018-05-07 | 2021-10-12 | Apple Inc. | Intelligent automated assistant for delivering content from user experiences |
EP3776436A1 (en) * | 2018-05-07 | 2021-02-17 | Google LLC | Personalized match score for places |
US10984780B2 (en) | 2018-05-21 | 2021-04-20 | Apple Inc. | Global semantic word embeddings using bi-directional recurrent neural networks |
DK201870355A1 (en) | 2018-06-01 | 2019-12-16 | Apple Inc. | Virtual assistant operation in multi-device environments |
DK179822B1 (en) | 2018-06-01 | 2019-07-12 | Apple Inc. | Voice interaction at a primary device to access call functionality of a companion device |
US10892996B2 (en) | 2018-06-01 | 2021-01-12 | Apple Inc. | Variable latency device coordination |
US11386266B2 (en) | 2018-06-01 | 2022-07-12 | Apple Inc. | Text correction |
DK180639B1 (en) | 2018-06-01 | 2021-11-04 | Apple Inc | DISABILITY OF ATTENTION-ATTENTIVE VIRTUAL ASSISTANT |
US10496705B1 (en) | 2018-06-03 | 2019-12-03 | Apple Inc. | Accelerated task performance |
US11176626B1 (en) * | 2018-06-20 | 2021-11-16 | Grubhub Holdings, Inc. | Personalizing food discovery and search based on inferred taste preference |
CA3109115A1 (en) * | 2018-08-06 | 2020-02-13 | Olive Seed Industries, Llc | Methods and systems for personalizing visitor experience at a venue |
CN115550304B (en) * | 2018-08-22 | 2023-10-10 | 谷歌有限责任公司 | Method, apparatus and storage medium for determining a set of active instances for a group of users |
US11010561B2 (en) | 2018-09-27 | 2021-05-18 | Apple Inc. | Sentiment prediction from textual data |
US11462215B2 (en) | 2018-09-28 | 2022-10-04 | Apple Inc. | Multi-modal inputs for voice commands |
US11170166B2 (en) | 2018-09-28 | 2021-11-09 | Apple Inc. | Neural typographical error modeling via generative adversarial networks |
US10839159B2 (en) | 2018-09-28 | 2020-11-17 | Apple Inc. | Named entity normalization in a spoken dialog system |
CN111209245B (en) * | 2018-11-21 | 2021-11-16 | 上海寒武纪信息科技有限公司 | Data processing device, method and related product |
US11475898B2 (en) | 2018-10-26 | 2022-10-18 | Apple Inc. | Low-latency multi-speaker speech recognition |
US11638059B2 (en) | 2019-01-04 | 2023-04-25 | Apple Inc. | Content playback on multiple devices |
US11436657B2 (en) * | 2019-03-01 | 2022-09-06 | Shopify Inc. | Self-healing recommendation engine |
US11348573B2 (en) | 2019-03-18 | 2022-05-31 | Apple Inc. | Multimodality in digital assistant systems |
US11423908B2 (en) | 2019-05-06 | 2022-08-23 | Apple Inc. | Interpreting spoken requests |
DK201970509A1 (en) | 2019-05-06 | 2021-01-15 | Apple Inc | Spoken notifications |
US11307752B2 (en) | 2019-05-06 | 2022-04-19 | Apple Inc. | User configurable task triggers |
US11475884B2 (en) | 2019-05-06 | 2022-10-18 | Apple Inc. | Reducing digital assistant latency when a language is incorrectly determined |
US11140099B2 (en) | 2019-05-21 | 2021-10-05 | Apple Inc. | Providing message response suggestions |
DK180129B1 (en) | 2019-05-31 | 2020-06-02 | Apple Inc. | User activity shortcut suggestions |
US11496600B2 (en) | 2019-05-31 | 2022-11-08 | Apple Inc. | Remote execution of machine-learned models |
US11289073B2 (en) | 2019-05-31 | 2022-03-29 | Apple Inc. | Device text to speech |
DK201970511A1 (en) | 2019-05-31 | 2021-02-15 | Apple Inc | Voice identification in digital assistant systems |
US11360641B2 (en) | 2019-06-01 | 2022-06-14 | Apple Inc. | Increasing the relevance of new available information |
US11468890B2 (en) | 2019-06-01 | 2022-10-11 | Apple Inc. | Methods and user interfaces for voice-based control of electronic devices |
CN110276017A (en) * | 2019-06-28 | 2019-09-24 | 百度在线网络技术(北京)有限公司 | A kind of data analysing method and device |
CN110428298A (en) * | 2019-07-15 | 2019-11-08 | 阿里巴巴集团控股有限公司 | A kind of shop recommended method, device and equipment |
US11544761B1 (en) | 2019-08-29 | 2023-01-03 | Inmar Clearing, Inc. | Food product recommendation system and related methods |
US11488406B2 (en) | 2019-09-25 | 2022-11-01 | Apple Inc. | Text detection using global geometry estimators |
US20210158424A1 (en) * | 2019-11-27 | 2021-05-27 | L'oreal | Techniques for improving product recommendations using personality traits |
US20210174216A1 (en) * | 2019-12-04 | 2021-06-10 | International Business Machines Corporation | Signaling concept drift during knowledge base population |
US20210304043A1 (en) * | 2020-03-26 | 2021-09-30 | International Business Machines Corporation | Evaluating a recommender system for data processing |
US11768945B2 (en) * | 2020-04-07 | 2023-09-26 | Allstate Insurance Company | Machine learning system for determining a security vulnerability in computer software |
US11061543B1 (en) | 2020-05-11 | 2021-07-13 | Apple Inc. | Providing relevant data items based on context |
US11043220B1 (en) | 2020-05-11 | 2021-06-22 | Apple Inc. | Digital assistant hardware abstraction |
US11755276B2 (en) | 2020-05-12 | 2023-09-12 | Apple Inc. | Reducing description length based on confidence |
US20210374122A1 (en) * | 2020-05-27 | 2021-12-02 | Koninklijke Philips N.V. | Method and systems for cleaning and enriching data from a real-time locating system |
US11490204B2 (en) | 2020-07-20 | 2022-11-01 | Apple Inc. | Multi-device audio adjustment coordination |
US11438683B2 (en) | 2020-07-21 | 2022-09-06 | Apple Inc. | User identification using headphones |
US11494675B2 (en) * | 2020-08-03 | 2022-11-08 | Kpn Innovations, Llc. | Method and system for data classification to generate a second alimentary provider |
US20230252388A1 (en) * | 2022-02-04 | 2023-08-10 | Workday, Inc. | Computerized systems and methods for intelligent listening and survey distribution |
US20240070746A1 (en) * | 2022-08-30 | 2024-02-29 | Maplebear Inc. (Dba Instacart) | Machine learning prediction of user responses to recommendations selected without contextual relevance |
Citations (15)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20030018430A1 (en) * | 2001-04-23 | 2003-01-23 | Quentin Ladetto | Pedestrian navigation method and apparatus operative in a dead reckoning mode |
US20050186965A1 (en) * | 2002-05-27 | 2005-08-25 | John Pagonis | Method of enabling a wireless information device to access location data |
US20090124241A1 (en) * | 2007-11-14 | 2009-05-14 | Qualcomm Incorporated | Method and system for user profile match indication in a mobile environment |
US20090231198A1 (en) * | 2008-03-17 | 2009-09-17 | Disney Enterprises, Inc. | Active player tracking |
US20100082301A1 (en) * | 2008-09-30 | 2010-04-01 | Sense Netwoks, Inc. | Event Identification In Sensor Analytics |
US20100286899A1 (en) * | 2007-08-16 | 2010-11-11 | Google Inc. | Combining Road and Vehicle Traffic Information |
US20100332124A1 (en) * | 2009-06-30 | 2010-12-30 | Apple Inc. | Analyzing and Consolidating Track File Data |
US20110087688A1 (en) * | 2009-10-10 | 2011-04-14 | Mitel Networks Corporation | System and method for creation and management of location information |
US20110154119A1 (en) * | 2009-12-23 | 2011-06-23 | Jia Wang | Device and Method for Detecting and Diagnosing Correlated Network Anomalies |
US20120088525A1 (en) * | 2010-10-12 | 2012-04-12 | Kddi Corporation | Estimation of significant places visited by mobile-terminal user based on communications log to base stations |
US20120213404A1 (en) * | 2011-02-18 | 2012-08-23 | Google Inc. | Automatic event recognition and cross-user photo clustering |
US20120253548A1 (en) * | 2011-03-31 | 2012-10-04 | United Parcel Service Of America, Inc. | Systems and methods for segmenting operational data |
US20130022282A1 (en) * | 2011-07-19 | 2013-01-24 | Fuji Xerox Co., Ltd. | Methods for clustering collections of geo-tagged photographs |
US20130058542A1 (en) * | 2011-09-07 | 2013-03-07 | Peter O. Stubler | Event classification method using lit candle detection |
US20130184007A1 (en) * | 2010-09-23 | 2013-07-18 | Nokia | Methods and apparatuses for context determination |
Family Cites Families (46)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7251637B1 (en) | 1993-09-20 | 2007-07-31 | Fair Isaac Corporation | Context vector generation and retrieval |
US20060020614A1 (en) | 1997-08-08 | 2006-01-26 | Kolawa Adam K | Method and apparatus for automated selection, organization, and recommendation of items based on user preference topography |
US6581058B1 (en) | 1998-05-22 | 2003-06-17 | Microsoft Corporation | Scalable system for clustering of large databases having mixed data attributes |
US6466232B1 (en) | 1998-12-18 | 2002-10-15 | Tangis Corporation | Method and system for controlling presentation of information to a user based on the user's condition |
US20020049816A1 (en) | 2000-03-24 | 2002-04-25 | Costin William Gilmore | System and method for raising funds and establishing user affinity over a distributed network |
US6389308B1 (en) | 2000-05-30 | 2002-05-14 | Vladimir Shusterman | System and device for multi-scale analysis and representation of electrocardiographic data |
US6655963B1 (en) | 2000-07-31 | 2003-12-02 | Microsoft Corporation | Methods and apparatus for predicting and selectively collecting preferences based on personality diagnosis |
US20020052934A1 (en) | 2000-08-28 | 2002-05-02 | Doherty Michael Emmett | Personalized agent for website direction |
US6912386B1 (en) * | 2001-11-13 | 2005-06-28 | Nokia Corporation | Method for controlling operation of a mobile device by detecting usage situations |
AU2002366902A1 (en) * | 2001-12-21 | 2003-07-09 | Nokia Corporation | Location-based novelty index value and recommendation system and method |
US7203909B1 (en) | 2002-04-04 | 2007-04-10 | Microsoft Corporation | System and methods for constructing personalized context-sensitive portal pages or views by analyzing patterns of users' information access activities |
US6946715B2 (en) | 2003-02-19 | 2005-09-20 | Micron Technology, Inc. | CMOS image sensor and method of fabrication |
AU2002353389A1 (en) | 2002-12-20 | 2004-07-14 | Nokia Corporation | Method and device for organizing user provided information with meta-information |
US20040162830A1 (en) | 2003-02-18 | 2004-08-19 | Sanika Shirwadkar | Method and system for searching location based information on a mobile device |
US7526458B2 (en) | 2003-11-28 | 2009-04-28 | Manyworlds, Inc. | Adaptive recommendations systems |
US7707039B2 (en) * | 2004-02-15 | 2010-04-27 | Exbiblio B.V. | Automatic modification of web pages |
US20060053097A1 (en) | 2004-04-01 | 2006-03-09 | King Martin T | Searching and accessing documents on private networks for use with captures from rendered documents |
JP4816207B2 (en) | 2005-04-01 | 2011-11-16 | ソニー株式会社 | Information processing system and method |
US7925995B2 (en) | 2005-06-30 | 2011-04-12 | Microsoft Corporation | Integration of location logs, GPS signals, and spatial resources for identifying user activities, goals, and context |
US7853485B2 (en) | 2005-11-22 | 2010-12-14 | Nec Laboratories America, Inc. | Methods and systems for utilizing content, dynamic patterns, and/or relational information for data analysis |
US7739280B2 (en) * | 2006-03-06 | 2010-06-15 | Veveo, Inc. | Methods and systems for selecting and presenting content based on user preference information extracted from an aggregate preference signature |
US7849027B2 (en) | 2006-10-18 | 2010-12-07 | Yahoo! Inc. | Automated clustering of records, biased by supervised classification processing |
US8310542B2 (en) * | 2007-11-28 | 2012-11-13 | Fuji Xerox Co., Ltd. | Segmenting time based on the geographic distribution of activity in sensor data |
CN102037481A (en) | 2008-03-19 | 2011-04-27 | 苹果核网络股份有限公司 | Method and apparatus for detecting patterns of behavior |
US8224714B2 (en) | 2008-05-01 | 2012-07-17 | Microsoft Corporation | Peer to peer network personal assistant |
US10163113B2 (en) * | 2008-05-27 | 2018-12-25 | Qualcomm Incorporated | Methods and apparatus for generating user profile based on periodic location fixes |
US9547710B2 (en) | 2008-08-05 | 2017-01-17 | Vmware, Inc. | Methods for the cyclical pattern determination of time-series data using a clustering approach |
US9245000B2 (en) | 2008-08-05 | 2016-01-26 | Vmware, Inc. | Methods for the cyclical pattern determination of time-series data using a clustering approach |
US10373079B2 (en) | 2008-12-18 | 2019-08-06 | Oracle International Corporation | Method and apparatus for generating recommendations from descriptive information |
US20100228590A1 (en) * | 2009-03-03 | 2010-09-09 | International Business Machines Corporation | Context-aware electronic social networking |
US10013489B2 (en) | 2009-08-12 | 2018-07-03 | Oath Inc. | System and method for providing recommendations |
GB2487027A (en) * | 2009-10-23 | 2012-07-04 | Cadio Inc | Analyzing consumer behavior using electronically-captured consumer location data |
US8612134B2 (en) * | 2010-02-23 | 2013-12-17 | Microsoft Corporation | Mining correlation between locations using location history |
US8266243B1 (en) | 2010-03-30 | 2012-09-11 | Amazon Technologies, Inc. | Feedback mechanisms providing contextual information |
US8606636B1 (en) * | 2010-07-14 | 2013-12-10 | Amazon Technologies, Inc. | Recommendations based on environmental variables |
WO2012016057A1 (en) | 2010-07-29 | 2012-02-02 | True Fit Corporation | Enabling proxy shopping |
BR112013005412A2 (en) * | 2010-09-17 | 2016-06-07 | Nokia Technologies Oy | apparatus method, computer readable storage medium that has one or more sequences of one or more instructions, computer program product |
US20120117006A1 (en) | 2010-11-04 | 2012-05-10 | Nokia Corporation | Method and apparatus for building a user behavior model |
US8412665B2 (en) * | 2010-11-17 | 2013-04-02 | Microsoft Corporation | Action prediction and identification temporal user behavior |
US9122693B2 (en) * | 2010-11-30 | 2015-09-01 | Nokia Technologies Oy | Method and apparatus for determining contextually relevant geographical locations |
JP5221630B2 (en) * | 2010-12-07 | 2013-06-26 | 楽天株式会社 | Server, information management method, information management program, and computer-readable recording medium for recording the program |
US8682956B2 (en) | 2011-06-09 | 2014-03-25 | Gm Global Technology Operations, Inc | Systems and methods for determining recommended media content for exchange between vehicles |
US8788307B2 (en) * | 2011-09-02 | 2014-07-22 | Woofound, Inc. | System for using personality trait identification to match consumers with businesses |
CA2879619A1 (en) * | 2012-07-20 | 2014-01-23 | Intertrust Technologies Corporation | Information targeting systems and methods |
US9020864B2 (en) | 2012-07-25 | 2015-04-28 | Aro, Inc. | Recommendation agent using a personality model determined from mobile device data |
US8655307B1 (en) | 2012-10-26 | 2014-02-18 | Lookout, Inc. | System and method for developing, updating, and using user device behavioral context models to modify user, device, and application state, settings and behavior for enhanced user security |
-
2013
- 2013-07-24 US US13/950,117 patent/US9020864B2/en active Active
- 2013-07-24 US US13/950,150 patent/US20140031060A1/en not_active Abandoned
- 2013-07-24 US US13/950,177 patent/US20140032358A1/en not_active Abandoned
- 2013-07-24 US US13/950,105 patent/US9179250B2/en active Active
- 2013-07-24 EP EP13822842.4A patent/EP2877935A4/en not_active Withdrawn
- 2013-07-24 US US13/950,169 patent/US8838436B2/en active Active
- 2013-07-24 WO PCT/US2013/051909 patent/WO2014018687A1/en active Application Filing
- 2013-07-24 KR KR1020147008609A patent/KR101512278B1/en active IP Right Grant
- 2013-07-24 US US13/950,224 patent/US8892480B2/en active Active
-
2015
- 2015-02-23 US US14/629,456 patent/US20150170042A1/en not_active Abandoned
Patent Citations (15)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20030018430A1 (en) * | 2001-04-23 | 2003-01-23 | Quentin Ladetto | Pedestrian navigation method and apparatus operative in a dead reckoning mode |
US20050186965A1 (en) * | 2002-05-27 | 2005-08-25 | John Pagonis | Method of enabling a wireless information device to access location data |
US20100286899A1 (en) * | 2007-08-16 | 2010-11-11 | Google Inc. | Combining Road and Vehicle Traffic Information |
US20090124241A1 (en) * | 2007-11-14 | 2009-05-14 | Qualcomm Incorporated | Method and system for user profile match indication in a mobile environment |
US20090231198A1 (en) * | 2008-03-17 | 2009-09-17 | Disney Enterprises, Inc. | Active player tracking |
US20100082301A1 (en) * | 2008-09-30 | 2010-04-01 | Sense Netwoks, Inc. | Event Identification In Sensor Analytics |
US20100332124A1 (en) * | 2009-06-30 | 2010-12-30 | Apple Inc. | Analyzing and Consolidating Track File Data |
US20110087688A1 (en) * | 2009-10-10 | 2011-04-14 | Mitel Networks Corporation | System and method for creation and management of location information |
US20110154119A1 (en) * | 2009-12-23 | 2011-06-23 | Jia Wang | Device and Method for Detecting and Diagnosing Correlated Network Anomalies |
US20130184007A1 (en) * | 2010-09-23 | 2013-07-18 | Nokia | Methods and apparatuses for context determination |
US20120088525A1 (en) * | 2010-10-12 | 2012-04-12 | Kddi Corporation | Estimation of significant places visited by mobile-terminal user based on communications log to base stations |
US20120213404A1 (en) * | 2011-02-18 | 2012-08-23 | Google Inc. | Automatic event recognition and cross-user photo clustering |
US20120253548A1 (en) * | 2011-03-31 | 2012-10-04 | United Parcel Service Of America, Inc. | Systems and methods for segmenting operational data |
US20130022282A1 (en) * | 2011-07-19 | 2013-01-24 | Fuji Xerox Co., Ltd. | Methods for clustering collections of geo-tagged photographs |
US20130058542A1 (en) * | 2011-09-07 | 2013-03-07 | Peter O. Stubler | Event classification method using lit candle detection |
Cited By (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20140101104A1 (en) * | 2012-09-26 | 2014-04-10 | Huawei Technologies Co., Ltd. | Method for generating terminal log and terminal |
US20200364204A1 (en) * | 2012-09-26 | 2020-11-19 | Huawei Technologies Co., Ltd. | Method for generating terminal log and terminal |
US20140136186A1 (en) * | 2012-11-15 | 2014-05-15 | Consorzio Nazionale Interuniversitario Per Le Telecomunicazioni | Method and system for generating an alternative audible, visual and/or textual data based upon an original audible, visual and/or textual data |
US20150356421A1 (en) * | 2014-06-05 | 2015-12-10 | Mitsubishi Electric Research Laboratories, Inc. | Method for Learning Exemplars for Anomaly Detection |
US9779361B2 (en) * | 2014-06-05 | 2017-10-03 | Mitsubishi Electric Research Laboratories, Inc. | Method for learning exemplars for anomaly detection |
WO2016057531A1 (en) * | 2014-10-08 | 2016-04-14 | Analog Devices, Inc. | Configurable pre-processing array |
US11250630B2 (en) | 2014-11-18 | 2022-02-15 | Hallmark Cards, Incorporated | Immersive story creation |
CN107004170A (en) * | 2014-12-05 | 2017-08-01 | 微软技术许可有限责任公司 | To be used to the service content of exceptional cast customization |
CN106528679A (en) * | 2016-10-24 | 2017-03-22 | 天津大学 | Time series analysis method based on multilinear autoregression model |
US11205103B2 (en) | 2016-12-09 | 2021-12-21 | The Research Foundation for the State University | Semisupervised autoencoder for sentiment analysis |
CN111406423A (en) * | 2017-11-30 | 2020-07-10 | 瑞典爱立信有限公司 | Core network allocation processing |
Also Published As
Publication number | Publication date |
---|---|
US9020864B2 (en) | 2015-04-28 |
US9179250B2 (en) | 2015-11-03 |
US20150170042A1 (en) | 2015-06-18 |
US8838436B2 (en) | 2014-09-16 |
US20140032572A1 (en) | 2014-01-30 |
US8892480B2 (en) | 2014-11-18 |
US20140032452A1 (en) | 2014-01-30 |
US20140032453A1 (en) | 2014-01-30 |
KR101512278B1 (en) | 2015-04-17 |
US20140032208A1 (en) | 2014-01-30 |
EP2877935A1 (en) | 2015-06-03 |
WO2014018687A1 (en) | 2014-01-30 |
KR20140071398A (en) | 2014-06-11 |
EP2877935A4 (en) | 2016-01-20 |
US20140032358A1 (en) | 2014-01-30 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US8838436B2 (en) | Labeling context slices to produce a storyline from mobile device data | |
US20150118667A1 (en) | Automated lifelogging to identify, monitor, and help achieve user goals | |
US10049413B2 (en) | Automatically creating a hierarchical storyline from mobile device data | |
US20220277248A1 (en) | User objective assistance technologies | |
US10748121B2 (en) | Enriching calendar events with additional relevant information | |
Yuan et al. | Measuring similarity of mobile phone user trajectories–a Spatio-temporal Edit Distance method | |
US8510247B1 (en) | Recommendation of media content items based on geolocation and venue | |
US20180285827A1 (en) | Distinguishing events of users for efficient service content distribution | |
Ojagh et al. | A location-based orientation-aware recommender system using IoT smart devices and Social Networks | |
US10185973B2 (en) | Inferring venue visits using semantic information | |
US20140057659A1 (en) | Inferring user interests | |
US20120143859A1 (en) | Real-time personalized recommendation of location-related entities | |
CN106878939A (en) | The method and system of accessible location is marked based on associated person information | |
US9047315B1 (en) | Geotagged hashtags | |
US20170116285A1 (en) | Semantic Location Layer For User-Related Activity | |
US10713286B2 (en) | Storage of point of interest data on a user device for offline use | |
Terroso-Sáenz et al. | Human mobility prediction based on social media with complex event processing | |
Li et al. | On route planning by inferring visiting time, modeling user preferences, and mining representative trip patterns | |
Castellanos et al. | Context-aware and Ontology-based Recommender System for E-tourism | |
Li et al. | Smartphone based lifelog with meaningful place detection | |
US11337036B1 (en) | Providing events on a map using an activity ontology for event suggestion | |
Alghamdi | E-Tourism: context-aware points of interest finder and trip designer | |
Vallejos et al. | A Lightweight Approach for Building User Mobility Profiles | |
Yang et al. | Sasll: A system annotating semantic label of location |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: ARO, INC., WASHINGTON Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:BENSLEY, JEREMY;EUSTICE, KEVIN FRANCIS;LIU, ALAN LINCHUAN;SIGNING DATES FROM 20130723 TO 20130724;REEL/FRAME:030871/0389 |
|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |
|
AS | Assignment |
Owner name: VULCAN TECHNOLOGIES LLC, WASHINGTON Free format text: TRANSFER STATEMENT;ASSIGNOR:ARO, INC.;REEL/FRAME:045661/0606 Effective date: 20171220 |