EP3074888B1 - Contextual information lookup and navigation - Google Patents

Contextual information lookup and navigation Download PDF

Info

Publication number
EP3074888B1
EP3074888B1 EP14812062.9A EP14812062A EP3074888B1 EP 3074888 B1 EP3074888 B1 EP 3074888B1 EP 14812062 A EP14812062 A EP 14812062A EP 3074888 B1 EP3074888 B1 EP 3074888B1
Authority
EP
European Patent Office
Prior art keywords
selection
user
contextual search
text
initial content
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
EP14812062.9A
Other languages
German (de)
English (en)
French (fr)
Other versions
EP3074888A1 (en
Inventor
Pradeep CHILAKAMARRI
Bernhard S.J. KOHLMEIER
Kristen M. Saad
Constance Gwendolyn Duong
Lin Ma
Annie Y. Bai
Lorrissa REYES
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Microsoft Technology Licensing LLC
Original Assignee
Microsoft Technology Licensing LLC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Microsoft Technology Licensing LLC filed Critical Microsoft Technology Licensing LLC
Publication of EP3074888A1 publication Critical patent/EP3074888A1/en
Application granted granted Critical
Publication of EP3074888B1 publication Critical patent/EP3074888B1/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/95—Retrieval from the web
    • G06F16/953—Querying, e.g. by the use of web search engines
    • G06F16/9535—Search customisation based on user profiles and personalisation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G06F16/2457—Query processing with adaptation to user needs
    • G06F16/24575—Query processing with adaptation to user needs using context
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/95—Retrieval from the web
    • G06F16/951—Indexing; Web crawling techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/95—Retrieval from the web
    • G06F16/953—Querying, e.g. by the use of web search engines
    • G06F16/9532—Query formulation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/95—Retrieval from the web
    • G06F16/953—Querying, e.g. by the use of web search engines
    • G06F16/9538—Presentation of query results
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/95—Retrieval from the web
    • G06F16/954—Navigation, e.g. using categorised browsing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/451—Execution arrangements for user interfaces
    • G06F9/453—Help systems

Definitions

  • a user will often leave the application and go to a web browser to perform a search and review the results.
  • the user may perform the search on a second device, for example, the application that the user is using to consume or create content may be on one device and then the user may access a browser on a second device (e.g., a mobile browser on a phablet or phone) to perform a search on the term or topic.
  • a second device e.g., a mobile browser on a phablet or phone
  • the separate search may be distracting to a user's workflow and even unproductive for many users, for example when a user leaves an application to enter terms for a search query.
  • the context of the content the user is working with may not be easily conveyed to a search engine since it is up to the user to provide key terms to indicate the context of the search. This can result in the search returning unrelated results and/or requiring multiple attempts by the user to narrow or modify a search query in order to achieve a relevant result.
  • Document WO 2007/064639 discloses search engine methods for generating highly personalized and relevant search results based on the context of a user's search constraint and user characteristics, whereby upon receipt of a user's search constraint, the method determines all semantic variations for each word within the user search constraint. Additionally, for each determined unique word and topic within the user search constraint, possible contexts are determined. A matrix of feasible context scenarios is established. Each context scenario is then ranked to determine the most likely context scenario for which the user search constraint relates based on user characteristics. The weighting used to rank the contexts is based on previous user searches and knowledge of their interests. Search results associated with the highest ranking context are provided to the user, along with topics associated with lower ranked contexts.
  • Document US 2008/133504 discloses a method for searching for information using an electronic device, such as a CE device, that can be connected to a network, whereby such searching for information involves determining a context for a search for information, forming a search query based on the context of search for information, and performing a contextual search based on the search query.
  • Performing the contextual search further includes performing a query refinement.
  • Document D3 US 2006/026013 discloses a method for implementing searches using contextual information associated with a page or other document that a user is viewing when a query is entered, whereby the page or document includes a contextual search interface that has an associated context vector representing content of the page or document.
  • the context vector is also provided to the query processor and used in responding to the query.
  • Context vectors may also be used in other ways, such as selecting additional content that may be of interest to a user.
  • a user who would like additional information about an item or topic in content within an application may indicate interest in certain text of the content.
  • the portion of the text indicated by the user, along with additional text around the portion of the text selected by the user, can be used to determine the intended items or topics for exploration.
  • One or more queries can be generated from the determined intended items or topics.
  • relevant results can be presented to the user based on the determined intended items or topics.
  • the contextual search service includes functionality and logic for producing contextually relevant information including results that are related through context and not just from a conventional search.
  • the portion of the text indicated by the user, along with additional text around the portion of the text selected by the user, is sent to the contextual search service, which performs a determination as to the intended item or topic for search.
  • the contextual search service can provide one or more proposed terms found in the text selected by the user (as well as in the additional text) for the selection along with relevant results obtained from one or more queries generated based on the one or more proposed terms.
  • the proposed terms for the selection can be indicated as identified terms in the application using a visual motif so that the user is made aware of what the application understands the user's intent to be.
  • a hierarchical interface can be presented within the original application context for displaying and navigating the results that are organized, for example, by source.
  • a preview of the results, grouped into modules according to source can be displayed non-obtrusively along with the user's original application surface. This may involve rendering a pane in a same window or display screen of the user's device.
  • an interactive, expanded view containing the search results may replace the preview interface enabling a deeper exploration of the results.
  • a user may select links or elements shown in the interface to conduct a deeper exploration of the results while remaining within the expanded view. Navigation through the levels of information can be carried out while within the application.
  • the hierarchical interface may be presented on a different device in response to receiving the results (or by command of the user).
  • Embodiments described herein enable a user seamlessly to retrieve additional information about content displayed in an application without having to leave the context (and even interface) of the application.
  • the application may be a program whose purpose includes the reading, writing, or arranging of text which is conducive to searching.
  • Example applications include word processing applications such as MICROSOFT WORD; email applications; layout applications; and note-taking applications such as MICROSOFT ONENOTE, EVERNOTE, and GOOGLE KEEP.
  • a "reader” refers to a software application and/or electronic device designed primarily for the purpose of reading electronic books (ebooks) and/or other electronic content.
  • Example software ebook readers include Google Reader, Apple iBooks, Acrobat eBook Reader, Amazon Kindle Reader, and Microsoft Reader applications.
  • Example hardware ebook readers include the Amazon Kindle and Barnes and Noble Nook.
  • An ebook is a text and/or image based publication in digital form that can be read on an ebook reader.
  • One step towards facilitating minimizing the distraction of switching context between an application in which a user is creating or consuming content and a web search is to provide a user experience that displays search results beside document content.
  • implementations of the techniques described herein are suitable for scenarios in which results are surfaced in a device that is different than the one in which the user is consuming or creating content.
  • a web search interface in a task pane can provide a user with the ability to conduct a search on a topic, term, or other item while remaining in the document.
  • a web search interface requires a user to determine the terms of a search query and leaves the user with the task of engaging with the search interface and crafting the appropriate query to obtain contextually relevant information.
  • a query can be generated for the user in response to a user indicating interest in certain text from within a document.
  • Embodiments are described herein that enable a user to indicate interest in certain text from within the document (without requiring the entering of search terms into a search field) and receive contextually relevant results based on a query generated by the application (either directly or via communicating with a contextual search service).
  • the user may be informed of the text that the application believes the user has selected for searching (e.g., a selection of text that is shaped into a potentially more contextually relevant search text) and, in some cases, the user may also be informed of the contextual terms used by the application to craft the search.
  • Further embodiments allow the user to preview and explore the search results for additional information by displaying interactive interfaces which do not require the user to switch away from the existing application.
  • search pane being provided separately from the application (and even on a separate device) so long as the search results can be generated without requiring entry of a search query directly by a user.
  • Figure 1A illustrates an operating environment in which certain implementations of a contextual information lookup and navigation experience may be carried out
  • Figure 1B illustrates an example process flow for contextual information lookup and navigation.
  • a user interface (UI) 100 of an application 105 in which a user 110 is consuming or creating content can be rendered on a display 111 of a user device 112.
  • the application 105 may run directly on the user device 112 or via a browser running on the user device 112.
  • the user device 112 may be, but is not limited to, a personal computer, a laptop computer, a desktop computer, a tablet computer, a reader, a mobile device, a personal digital assistant, a smart phone, a gaming device or console, a wearable computer with an optical head-mounted display, computer watch, or a smart television, of which computing system 800, discussed below with respect to Figure 8 , is representative.
  • the UI 100 can be influenced by a contextual search component 115 of the application 105.
  • the contextual search component 115 may be in the form of one or more software modules of the application 105.
  • the contextual search component 115 may provide instructions to carry out one or more of the methods for facilitating contextual information lookup and navigation described herein. Some aspects of the contextual information look up and navigation are performed on the user device 112 and rendered for display in the user interface 100, while other aspects may be performed, at least in part, by a contextual search service 120.
  • a separate device 118 may be, but is not limited to a personal computer, a laptop computer, a desktop computer, a tablet computer, a reader, a mobile device, a personal digital assistant, a smart phone, a gaming device or console, a wearable computer with an optical head-mounted display, computer watch, or a smart television, of which computing system 800, discussed below with respect to Figure 8 , is representative.
  • Various types of physical or virtual computing systems may be used to implement the contextual search service 120, such as server computers, desktop computers, laptop computers, tablet computers, smart phones, or any other suitable computing appliance.
  • server computer 122 any of a variety of servers may be used including, but not limited to, application servers, database servers, mail servers, rack servers, blade servers, tower servers, or any other type server, variation of server, or combination thereof.
  • the contextual search component 115 may interact with the contextual search service 120 over a network.
  • the network can include, but is not limited to, a cellular network (e.g., wireless phone), a point-to-point dial up connection, a satellite network, the Internet, a local area network (LAN), a wide area network (WAN), a Wi-Fi network, an ad hoc network or a combination thereof.
  • a cellular network e.g., wireless phone
  • LAN local area network
  • WAN wide area network
  • Wi-Fi network e.g., Wi-Fi network
  • ad hoc network a combination thereof.
  • Such networks are widely used to connect various types of network elements, such as hubs, bridges, routers, switches, servers, and gateways.
  • the network may include one or more connected networks (e.g., a multi-network environment) including public networks, such as the Internet, and/or private networks such as a secure enterprise private network. Access to the network may be provided via one or more wired
  • the contextual search component 115 facilitates the interaction between a user 110 interacting with the UI 100 and the contextual search service 120 using an application programming interface (API) of the contextual search service 120.
  • API application programming interface
  • An API is generally a set of programming instructions and standards for enabling two or more applications to communicate with each other and is commonly implemented as a set of Hypertext Transfer Protocol (HTTP) request messages and a specified format or structure for response messages according to a REST (Representational state transfer) or SOAP (Simple Object Access Protocol) architecture.
  • HTTP Hypertext Transfer Protocol
  • REST Real state transfer
  • SOAP Simple Object Access Protocol
  • the contextual search component 115 may facilitate a call (or invocation) of a contextual search service 120 using the API of the contextual search service 120.
  • the contextual search service can then perform a determination of the user's intended text selection based on information provided by the contextual search component 115 and construct one or more queries for requesting a search from one or more search services 130.
  • the search services 130 may include web search engines such as, but not limited to, Microsoft Bing and Google, and intranet search such as FAST Search.
  • the results of the determination and search can be returned to the user device 112 (e.g., for the contextual search component 115 ).
  • the contextual search component 115 can then surface elements in the UI 100 based on the information received from the contextual search service 120.
  • At least some of the results of the determination and search may be sent to a separate device 118, which may include one or more components related to the application 105 and/or the contextual search component 115.
  • a user 110 may be using multiple devices either simultaneously or at different times.
  • the application 105 and/or contextual search component 115 may be running on both devices and at least some of the information received from the contextual search service 120 can be received on the device in which the user 110 is not directly consuming the content (via the application 105 ).
  • a navigable user interface may be provided on the separate device (e.g., device 118 ) in which the relevant results may be selectively surfaced.
  • content in the form of a document may be displayed ( 151 ) through the UI 100.
  • the contextual information lookup process flow may begin with a user interacting with an application surface containing text (or other readily web-searchable content). For example, a user can indicate an initial selection of text for contextual search ( 152 ). Interaction by the user with the UI 100 indicating the initial text selection may take myriad forms.
  • the input indicating text selection can include, but is not limited to, a verbal selection (of one or more words or phrases), contact or contact-less gestural selection, touch selection (finger or stylus), swipe selection, cursor selection, encircling using a stylus/pen, or any other available technique that can be detected by the user device 112.
  • contextual information lookup may initiate without an active selection by a user.
  • the user may also, for instance, utilize a device which is capable of detecting eye movements, and the device might detect that the user's eye lingers on the text for a length of time, indicating the user's interest in selecting the text for contextual searching.
  • a computing device capable of detecting voice commands might recognize a spoken command to initially select text for contextual information lookup.
  • many other user interface elements as diverse as drop-down menus, buttons, or right-click context menus, may signify that the user has set an initial text selection. Further, it can be understood that an initial text selection may involve some or all of the text available on the document, page, or window.
  • Figures 2A-2D illustrate example interactions indicating an initial selection of text for contextual search.
  • a word (or phrase) 201 in which text is depicted, the user may select a word (or phrase) 201.
  • the selection of the word (or phrase) may be a swipe gesture 202 on a touch enabled display screen such as illustrated in Figure 2A .
  • Other gestures such as insertion point, tap, double tap, and pinch could be used.
  • non-touch selection of a word is possible.
  • a cursor 203 may be used to indicate, for example, via a mouse click, a point on the surface of the user interface 200.
  • the cursor 203 may be placed within a term without highlighting a word or words.
  • a similar selection may be conducted by touch (e.g., using a finger or pen/stylus).
  • a user may select more than a single word using any of the methods of user interaction described above.
  • multiple words may be "marked” by the user using interface techniques such as illustrated in Figure 2C , where a cursor 204 is shown selecting multiple words 205 of a sentence.
  • the user is not limited to selecting a particular amount of text.
  • an initial selection may include a contiguous series of words (a phrase).
  • a phrase contiguous series of words
  • multiple, non-contiguous words or phrases may be selected by highlighting, circling or underlining with a digital stylus.
  • Multiple words or phrases of interest also may be prioritized by the user. For example, one word or phrase may be marked as the primary text selection of interest, and other related words may be marked as supporting words or phrases which are of secondary, but related interest.
  • the input for initial text selection may be discerned from passive, rather than active, interactions by the user. For example, while the user is scrolling through the text rendered by an application, a paragraph on which the user lingers for a significant time might constitute an initial text selection.
  • the computing device allows the user's eye movements to be tracked, words or phrases on which the user's eye lingers may form the input for initial text selection.
  • the entire document, window, or page may be considered to be selected based on a passive interaction.
  • a user may not include an indication of selected text ( 152 ), but still indicate that a contextual search mode is desired.
  • the contextual search may treat the indication of contextual search mode (either from an implied indication or an explicit action by a user) as being a selection of the entire page or document (or at least a viewable portion of the page or document) and present keywords or phrases that may be of interest to the user, such as illustrated in Figure 2D .
  • a user may then select one or more of the highlighted (or otherwise indicated) terms 250 to initiate the contextual search service (and receive the contextual search results ( 156 )).
  • the interaction indicating an initial selection of text may also involve selection of a contextual search mode (or specific contextual search command) from a menu or particular interaction with the user interface (which may include an object oriented user interface (OOUI) and/or graphical user interface (GUI) element).
  • the selection of the contextual search mode may occur before or after text is selected by the user. For example, as illustrated in Figure 3 , selection of a word 201 (or a "right click" or other action) may cause a menu 300 to surface with command options that may be available for a user to perform with respect to the selected text.
  • a user may select a contextual search command element 301 from the menu 300 in order to invoke the contextual search.
  • the selected portion (even if only a point such as illustrated in Figure 2B or the entire page, window, or document when no specific selection is indicated) and additional information ( 153 ) is sent to a contextual search service 120.
  • the additional information can be referred to as selection context.
  • Selection context refers to supplementary information not specifically part of the user's initial text selection.
  • the selection context is utilized by the service to improve the relevance or clarity of the contextual search results by enhancing a traditional web search with additional information.
  • the additional information which collectively forms the selection context will vary by embodiment and scenario, but in some embodiments will include such information as the text surrounding the selection (which can also be referred to as an expanded portion of text, for example, a certain number of symbols or characters before and/or after the selection), the type of application in which the text is displayed, or the type of device on which the application runs.
  • the selection context may include relevant and related prior searches the user has performed through the service or on the device generally.
  • Device type and user history may also be included as selection context. For example, previous selections by the user may be used as part of the selection context. In addition, or alternatively, previous selections by other users (and statistical probabilities based thereon) may be used as part of the selection context.
  • Some embodiments may also send the type of user, e.g., whether the user is working from inside a corporate network or outside. In the former case, the user might find work-related results more relevant, while in the latter, more home or consumer-related results may be useful.
  • availability of computing resources may also be part of the context, enabling searching of additional resources like corporate intranets.
  • the contextual search service can process the initial text selection and selection context by refining the initial text selection into a second, "proposed" text selection based on the selection context and generating queries to Internet and/or intranet search services using the proposed text selection.
  • the second text selection may be an expansion or a reduction of the initial text selection.
  • the contextual search service will possess processing logic to intelligently refine the initial text selection into a more relevant search.
  • the service may exist at one or many different layers in the computing device architecture, including on the local device, on a shared resource inside a private network, or on the internet itself.
  • the service may request information from a single search engine, or it may consolidate the results of many search engines.
  • the service may also search information available on local network resources or intranets.
  • the word "service" itself may denote one or more services at different physical locations in communication with one another.
  • a communications link with a contextual search service may be accessible to an application in any number of ways implemented by the underlying device operating system, or by programming frameworks layered atop it, including but not limited to an API.
  • the contextual search service processes the initial text selection and selection context and returns a second (or more), proposed text selection(s) in response (e.g., the identified text selection(s)) ( 154 ).
  • the identified text selection(s) can be displayed ( 155 ) in the UI 100 (and/or on separate device 118 ) so that the user can see what was understood by the system to be selected and adjust the selection if needed.
  • relevant results can be received ( 156 ) from the contextual search service.
  • the identified text selection(s) can be indicated in the application using various visual motifs such as highlighting, underlining, bolding, animation, flashing text, font size, font style, and font color so that the user is made aware of what the application understands the user's intent to be.
  • the identification can also include an audio indication.
  • the user may refine or otherwise adjust the identified selection. For example, the user may select or deselect terms from the identified selection ( 157 ). The refinement may be sent to the service ( 158 ) to generate new contextual results, which then returns the refined contextual results ( 159 ). The original contextual results and/or the refined contextual results may or may not be immediately displayed to the user ( 160 ).
  • the UI 100 and contextual search component 106 ) can select the results that are displayed as well as present the results in a particular manner ( 160 ).
  • the identified text selection(s) can include terms and phrases that the contextual search found relevant to the search and these supporting/contextual terms may be displayed to the user, such as illustrated in Figure 4 .
  • an interface showing the percentage weight of the different words or phrases may be displayed. Many such visual signals indicating the factors used by the service to refine the search from the initial selection text are possible.
  • the user may adjust the proposed second selection by invoking a command to modify the proposed selection, for example, by de-selecting words not considered to be relevant to the search. That is, according to an implementation, a user may refine the supporting/contextual terms ( 161 ) in order to further refine the contextual results ( 162 ). The service may then generate further refined contextual results ( 163 ), which can be displayed to the user ( 164 ).
  • the initial text selection of a single word 201 in the user interface 200 has been refined by the addition of words to form a phrase 400.
  • the user may have selected Eleanor and the contextual search service determined that the intended selection was "Eleanor Roosevelt".
  • text 401 which was determined by the service to be particularly relevant to the search (from the additional text sent to the service as part of the selection context) can be displayed to the user to denote to the user additional factors which the service considered.
  • underline is shown in this example, it should be noted that a many types of visual signals or elements may be used to depict the proposed/identified text selection (and/or contextual terms).
  • the user's initial text selection may be denoted by highlighting of one color
  • the additional text presented in the UI in response to receiving the identified text selection(s) may be denoted by highlighting of at least a second color (e.g., there may be additional colors or distinctions provided depending on relevancy to the original selection, number of times the additional text was used in a query, and the like). For example, words of varying importance "levels" may be highlighted in different colors. In addition, contextual terms may be highlighted to let the user understand why a particular result is returned.
  • Processing may begin with the receipt of relevant results.
  • the results can be analyzed and grouped for displaying to a user 110.
  • a hierarchical presentation of the relevant results may be available such that a user can delve deeper into a topic or result while within the UI 100.
  • the results may first be displayed as preview results ( 160 ) and then in response to receiving an indication of a result selection ( 165 ) from within the UI 100, a next level of information may be displayed ( 166 ).
  • a next level of information may be displayed ( 166 ).
  • context refinement ( 161 ) and/or selection refinement ( 157 ) may be performed, resulting in modifications to the displayed results.
  • a hierarchical interface can be presented within the original application context for displaying relevant results.
  • the user interface (e.g., UI 100 ) can enable "disambiguation" so that a user may perfect an incorrect understanding of intent.
  • the identified selection (and contextual results) may be wrong ( i.e., not related to the intended selection).
  • the user may clarify the intent through one or more available mechanisms. For example, a list of possible suggestions may be presented when the system (or service) determines that the probability of a correct result is below a particular threshold.
  • the user may still be given the opportunity to correct or adjust the selection even when the results with a higher than threshold likelihood of being correct are provided.
  • a user may override the identified selection as well as clarify or correct the contextual entity or articles (e.g., terms) selected by the service as relevant to context.
  • the user interface may include interactive views of the relevant results, grouped into modules.
  • an "interactive" view refers to a view where the user may select a relevant result by clicking with the mouse, using a touch gesture, or any other user selection indicia known in the art.
  • An interactive view also allows the user to scroll content, select and copy text, or select links within content to additional content.
  • a user may navigate through multiple levels to achieve an in-depth exploration of a subject.
  • an interactive view may be rendered using a standardized web markup language such as HTML or JSON, but other rendering methods are possible.
  • Figures 5A-5B illustrate an example preview contextual result interface.
  • Figures 6A-6D illustrate example expanded contextual result interface views and result groupings that may be generated for the user interface.
  • a preview pane 500 can be displayed non-obtrusively atop the existing application surface 501, only partly obscuring the content displayed in the application surface 501.
  • a quick summary can be provided that may include a title 502 (as provided by the identified text 503 ), an image (still or moving) 504 (if available) and summary text 505 (if available).
  • a preview of each module of the relevant results is also included in the example preview pane 500.
  • the relevant results 510 can be grouped by the application into modules 520.
  • the relevant results 510 are grouped by source.
  • Source in this context, may mean a network location, website, type of application, type of result (such as an image) or other logical method of grouping results.
  • sources might be the Wikipedia online encyclopedia; a local network source, such as an internal web server and/or social graph, privately available to the users in a company; a particular news website; image files from a photo-sharing website; structured data from a database; or private files on the user's drives or personal cloud storage.
  • modular groupings may be displayed differently based on contextual information about the user. For example, a user at home may receive consumer or entertainment-oriented information sources. The same user might receive different groupings (and, as noted above, different results) when at work. Many such forms of groupings are possible.
  • Figure 5B illustrates one entry mechanism to an expanded view of relevant results.
  • a user may select an expansion command 525 in order to view the expanded view. Selection via a touch gesture is depicted in the embodiment in Figure 5B , but other methods of selecting a command commonly known in the art are possible.
  • the preview pane 500 may be interactive. If a user selects one of the modules or one of the results in a module, additional information may be surfaced within the preview pane 500 or the selection of an element in the preview pane may cause the expanded view to launch.
  • Figure 6A illustrates an example user interface in which the interactive expanded view 600 is displayed atop the existing application surface 501, replacing the preview pane 500 while maintaining the original application context.
  • the results 601 available from a given source 602 may be viewed in detail (e.g., the results of Source 1 602-1 ). In some cases, a user may navigate between results grouped in pages according to source. When the user selects a different source 602, the interactive expanded view 600 can surface results from the selected source (e.g., Source 2 602-2 ) as shown in Figure 6B .
  • the results for each source may be arranged in a manner suitable for its content.
  • one source may be suitable for a mixed content arrangement such as shown in Figure 6A
  • another source such as shown in Figure 6B may depict content in a columnar, article-style layout (which can be suitable for a source such as Wikipedia).
  • Other layouts may be suitable for content from different sources.
  • a source e.g., Source 3 602-3
  • Source 3 602-3 that generates primarily images 610
  • a thumbnail layout as illustrated in Figure 6C may be used.
  • a view for source of web results may use a layout with tabular previews.
  • some or all of the information may be provided to a separate device and/or saved for later.
  • scrolling and/or zooming may be utilized to access the available results within the interactive expanded view.
  • the interactive expanded view 600 may be similar to the preview pane 500 in that a preview of the results from each source may be displayed.
  • Figure 6D illustrates an example where all results may be provided in the interactive expanded view 650. From the interactive expanded view, the user may further explore the results of a particular source by selecting a link, title, source name, or other entry point.
  • the results shown in the interactive expanded view 600, 650 can include links. If a user selects to follow a link, the link can be launched in a browser window separate or within the interactive expanded view 600, 650.
  • Figure 7 illustrates an example hierarchy of relevant results based on source.
  • a top node may be the overview node 700.
  • the sources e.g., source 1 701, source 2 702, and source 3 703
  • the sources are known as well as any preview results passed through to the overview from the source level nodes.
  • a source node may include overlapping results with another source node (e.g., result 704 may be available through source 2 702 and source 3 703 ), for example, where one of the sources is defined as any images and another source includes an image in its results.
  • results that are received will necessarily be included in the hierarchy of relevant results navigable through the interactive expanded view. For example, when the results are organized, an analysis of the results with respect to one or more aspects related to context may be performed. Results deemed to not be relevant may be omitted (e.g., as an omitted result 705 ). In some cases, context may refine the presentation of source results. For example, an article in one language may be different than that written in a second language and the selection between presenting results from the two sources can be made based on the location of the user and/or previous selection history.
  • FIG. 8 shows a block diagram illustrating components of a computing device used in some embodiments (e.g., user device 112 ).
  • System 800 may be implemented within a single computing device or distributed across multiple computing devices or sub-systems that cooperate in executing program instructions.
  • System 800 can be used to implement myriad computing devices, including but not limited to a personal computer, a tablet computer, a reader, a mobile device, a personal digital assistant, a wearable computer, a smartphone, a laptop computer (notebook or netbook), a gaming device or console, a desktop computer, or a smart television. Accordingly, more or fewer elements described with respect to system 800 may be incorporated to implement a particular computing device.
  • System 800 includes a processor 805 which processes data according to the instructions of one or more application programs 810 interacting with the device operating system (OS) 815.
  • processors 805 include general purpose central processing units, application specific processors, and logic devices, as well as any other type of processing device, combinations, or variations thereof.
  • the application programs 810, OS 815 and other software may be loaded into and stored in a storage system 820.
  • Device operating systems 815 generally control and coordinate the functions of the various components in the computing device, providing an easier way for applications to connect with lower level interfaces like the networking interface.
  • Non-limiting examples of operating systems include WINDOWS from Microsoft Corp., IOS from Apple, Inc., ANDROID OS from Google, Inc., WINDOWS RT from Microsoft, and the Ubuntu variety of the Linux OS from Canonical.
  • OS 815 may be implemented both natively on the computing device and on software virtualization layers running atop the native Device OS.
  • Virtualized OS layers while not depicted in Figure 8 , can be thought of as additional, nested groupings within the OS 815 space, each containing an OS, application programs, and APIs.
  • Storage system 820 may comprise any computer readable storage media readable by the processor 805 and capable of storing software (e.g., application programs 810 and OS 815 ).
  • Storage system 820 may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. Examples of storage media include random access memory, read only memory, magnetic disks, optical disks, flash memory, virtual memory and non-virtual memory, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other suitable storage media. In no case is the storage medium a propagated signal. In addition to storage media, in some implementations storage system 820 may also include communication media over which software may be communicated internally or externally. Storage system 820 may be implemented as a single storage device but may also be implemented across multiple storage devices or sub-systems co-located or distributed relative to each other. Storage system 820 may comprise additional elements, such as a controller, capable of communicating with processor 805.
  • storage media include random access memory, read only memory, magnetic disks, optical disks, flash memory, virtual memory and non-virtual memory, magnetic cassettes, magnetic tape, magnetic disk storage
  • Software may be implemented in program instructions and among other functions may, when executed by system 800 in general or processor 805 in particular, direct system 800 or processor 805 to operate as described herein.
  • Software may include additional processes, programs, or components, such as operating system software or other application software.
  • Software may also comprise firmware or some other form of machine-readable processing instructions executable by processor 805.
  • software may, when loaded into processor 805 and executed, transform computing system 800 overall from a general-purpose computing system into a special-purpose computing system customized to facilitate the contextual information lookup and navigation process flow as described herein for each implementation.
  • encoding software on storage system 820 may transform the physical structure of storage system 820. The specific transformation of the physical structure may depend on various factors in different implementations of this description. Examples of such factors may include, but are not limited to the technology used to implement the storage media of storage system 820 and whether the computer-storage media are characterized as primary or secondary storage.
  • the computer-storage media are implemented as semiconductor-based memory
  • software may transform the physical state of the semiconductor memory when the program is encoded therein, such as by transforming the state of transistors, capacitors, or other discrete circuit elements constituting the semiconductor memory.
  • a similar transformation may occur with respect to magnetic or optical media.
  • Other transformations of physical media are possible without departing from the scope of the present description, with the foregoing examples provided only to facilitate this discussion.
  • system 800 may be included in a system-on-a-chip (SoC) device. These elements may include, but are not limited to, the processor 805, a communications interface 835, an audio interface 840, a video interface 845, and even elements of the storage system 820.
  • SoC system-on-a-chip
  • Communications interface 835 may include communications connections and devices that allow for communication with other computing systems over one or more communication networks (not shown). Examples of connections and devices that together allow for inter-system communication may include network interface cards, antennas, power amplifiers, RF circuitry, transceivers, and other communication circuitry. The connections and devices may communicate over communication media (such as metal, glass, air, or any other suitable communication media) to exchange communications with other computing systems or networks of systems. Transmissions to and from the communications interface are controlled by the OS 815, which informs applications and APIs of communications events when necessary.
  • Interface devices 850 may include input devices such as a mouse 851, track pad, keyboard 852, microphone 853, a touch device 854 for receiving a touch gesture from a user, a motion input device 855 for detecting non-touch gestures and other motions by a user, and other types of input devices and their associated processing elements capable of receiving user input.
  • input devices such as a mouse 851, track pad, keyboard 852, microphone 853, a touch device 854 for receiving a touch gesture from a user, a motion input device 855 for detecting non-touch gestures and other motions by a user, and other types of input devices and their associated processing elements capable of receiving user input.
  • the interface devices 850 may also include output devices such as display screens 856, speakers 857, haptic devices for tactile feedback, and other types of output devices.
  • the input and output devices may be combined in a single device, such as a touchscreen display which both depicts images and receives touch gesture input from the user.
  • Visual output may be depicted on the display 851 in myriad ways, presenting graphical user interface elements, text, images, video, notifications, virtual buttons, virtual keyboards, or any other type of information capable of being depicted in visual form. Other kinds of user interface are possible.
  • User interface 850 may also include associated user interface software executed by the OS 815 in support of the various user input and output devices. Such software assists the OS in communicating user interface hardware events to application programs 810 using defined mechanisms.
  • computing system 800 is generally intended to represent a computing system with which software is deployed and executed in order to implement an application with the contextual information lookup and navigation process flow as described herein. However, computing system 800 may also represent any computing system on which software may be staged and from where software may be distributed, transported, downloaded, or otherwise provided to yet another computing system for deployment and execution, or yet additional distribution.
  • Figure 9 illustrates an application environment 900 in which an application with the proposed improvements may be implemented utilizing the principles depicted in system 800 ( Figure 8 ) and discussed above.
  • Figure 9 shows various application platforms 910, 920, 930, and 940, each of which is capable of communicating with service platforms 970 and 980 over communications network 901 to perform the contextual search and return the relevant results.
  • the application platforms 910, 920, 930, and 940 may be any computing apparatus, device, system, or collection thereof employing a computing architecture suitable for implementing the application ( 911, 921, 931, 941 ) on that platform.
  • the described interfaces and process flow may be implemented within applications designed to view and manipulate textual content.
  • the functionality of detecting text selections and non-intrusively rendering relevant results according to the described methods may be implemented by the OS or by layered components accessible to the applications via API.
  • Application 911 may be considered a full or "native" version that is locally installed and executed. In some cases, application 911 may operate in a hybrid manner whereby a portion of the application is locally installed and executed and other portions are executed remotely and then streamed to application platform 910 for local rendering.
  • Non-limiting examples of application 911 include word processors such as Microsoft® Word; email applications; electronic readers like Microsoft® Reader or Amazon Kindle® Reader; and note-taking applications such as Microsoft OneNote®, Evernote®, or Google KeepTM.
  • Browser-based application 921 implemented on application platform 920, may be considered a browser-based version that is executed wholly or partly in the context of a browser application 922. In this model, all or part of the programming instructions are executed remotely and the browser 922 renders the result to the user's device through a visual expression language such as HTML.
  • a non-limiting example of a browser-based application 921 is the Microsoft® Word Web App Service available through Microsoft SkyDrive®. Examples of the browser application 922 include Google ChromeTM, Microsoft Internet ExplorerTM, and Mozilla FirefoxTM.
  • Application 931 may be considered a mobile application version that is locally installed and executed on a mobile device. In some cases, application 931 may operate in a hybrid manner whereby a portion of the application is locally installed and executed and other portions are executed remotely and then streamed to application platform 930 for local rendering.
  • mobile applications 931 include QuickOffice® HD on the Google AndroidTM and Apple IOSTM devices.
  • Application 941 implemented on application platform 940, may be considered a browser-based version that is executed wholly or partly in the context of a mobile browser application 942.
  • all or part of the programming instructions are executed remotely and the mobile browser 942 renders the result to the user's device through a visual expression language such as HTML.
  • Non-limiting examples of a mobile browser-based application 941 include mobile-device-enhanced views of content through Microsoft SkyDrive®, Google DriveTM.
  • Examples of the mobile browser application 942 include Google ChromeTM and Mozilla FirefoxTM.
  • the application platforms 910, 920, 930, and 940 may communicate with service platforms 970 and 980 connected by network 901.
  • Service platforms may deliver a variety of services useful to the application platforms and applications capable of performing contextual information lookup and navigation as described herein.
  • service platform 970 may deliver the contextual search service 971 which enables the processing and contextual shaping of search text described above.
  • the contextual search service may also retrieve, sort, and group relevant results from search engines for presentation by the interface described above.
  • Service 971 may also host remote programming instructions and render their results to applications or browsers on any of the application platforms.
  • the contextual search service 971 may be implemented using one or more physical and/or virtual servers communicating over a network.
  • service platform 980 may deliver storage provider service 981, which enables non-local storage of files or other data which can be utilized by applications 911, 921, 931, and 941, and by contextual search service 971.
  • storage provider service 981 might be a cloud storage provider, a database server, or a local area network file server.
  • the contextual search service may contain functionality for searching these storage providers for content and presenting the results in a grouped manner as described herein.
  • Non-limiting examples of storage provider services include Microsoft SkyDrive®, Google DriveTM, DropBoxTM, BoxTM, and Microsoft® SQL Server.
  • service platform 990 may deliver search engine service 991, which enables relevant results to be located and returned to applications 911, 921, 931, and 941, and to the contextual search service 971.
  • search engine service 991 might be an internet search engine such as Bing.com or Google.com, or a search engine available on a private network such as Microsoft® Search Server Express.
  • any reference in this specification to "one embodiment,” “an embodiment,” “example embodiment,” etc. means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the invention.
  • the appearances of such phrases in various places in the specification are not necessarily all referring to the same embodiment.
  • any elements or limitations of any invention or embodiment thereof disclosed herein can be combined with any and/or all other elements or limitations (individually or in any combination) or any other invention or embodiment thereof disclosed herein, and all such combinations are contemplated with the scope of the invention without limitation thereto.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Software Systems (AREA)
  • Human Computer Interaction (AREA)
  • Mathematical Physics (AREA)
  • Computational Linguistics (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Remote Sensing (AREA)
  • User Interface Of Digital Computer (AREA)
EP14812062.9A 2013-11-27 2014-11-20 Contextual information lookup and navigation Active EP3074888B1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US14/091,847 US9754034B2 (en) 2013-11-27 2013-11-27 Contextual information lookup and navigation
PCT/US2014/066487 WO2015080924A1 (en) 2013-11-27 2014-11-20 Contextual information lookup and navigation

Publications (2)

Publication Number Publication Date
EP3074888A1 EP3074888A1 (en) 2016-10-05
EP3074888B1 true EP3074888B1 (en) 2020-02-19

Family

ID=52023652

Family Applications (1)

Application Number Title Priority Date Filing Date
EP14812062.9A Active EP3074888B1 (en) 2013-11-27 2014-11-20 Contextual information lookup and navigation

Country Status (5)

Country Link
US (1) US9754034B2 (ko)
EP (1) EP3074888B1 (ko)
KR (1) KR102310648B1 (ko)
CN (1) CN105793844A (ko)
WO (1) WO2015080924A1 (ko)

Families Citing this family (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9292495B2 (en) * 2014-02-20 2016-03-22 International Business Machines Corporation Contextual updating of existing documents using natural language processing
US10628848B2 (en) * 2014-05-15 2020-04-21 Oath Inc. Entity sponsorship within a modular search object framework
US10534502B1 (en) * 2015-02-18 2020-01-14 David Graham Boyers Methods and graphical user interfaces for positioning the cursor and selecting text on computing devices with touch-sensitive displays
US10592541B2 (en) * 2015-05-29 2020-03-17 Intel Corporation Technologies for dynamic automated content discovery
CN114143093B (zh) 2016-09-29 2024-06-14 艾普拉控股有限公司 存储和检索设备的网络上下文
US10606924B2 (en) * 2016-11-18 2020-03-31 Microsoft Technology Licensing, Llc Contextual file manager
US10783320B2 (en) 2017-05-16 2020-09-22 Apple Inc. Device, method, and graphical user interface for editing screenshot images
CN107590123B (zh) * 2017-08-07 2022-07-05 大众问问(北京)信息科技有限公司 车载中地点上下文指代消解方法及装置
US11320983B1 (en) * 2018-04-25 2022-05-03 David Graham Boyers Methods and graphical user interfaces for positioning a selection, selecting, and editing, on a computing device running applications under a touch-based operating system
US11442972B2 (en) * 2018-12-07 2022-09-13 Comcast Cable Communications, Llc Methods and systems for modifying a search result
US10891024B2 (en) * 2019-01-30 2021-01-12 Ncr Corporation Hierarchical user interface
CN110209939B (zh) * 2019-05-31 2021-10-12 腾讯科技（深圳）有限公司 推荐信息的获取方法、装置、电子设备及可读存储介质
US11347756B2 (en) 2019-08-26 2022-05-31 Microsoft Technology Licensing, Llc Deep command search within and across applications
US11900046B2 (en) * 2020-08-07 2024-02-13 Microsoft Technology Licensing, Llc Intelligent feature identification and presentation
US11741852B2 (en) * 2020-11-24 2023-08-29 Kyndryl, Inc. Smart reading assistant

Family Cites Families (23)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2002095595A1 (fr) * 2001-05-18 2002-11-28 Sony Corporation Procede pour fournir des informations, systeme pour fournir des informations et appareil serveur d'informations
GB0322877D0 (en) * 2003-09-30 2003-10-29 British Telecomm Search system and method
CN100357942C (zh) * 2004-03-31 2007-12-26 宇龙计算机通信科技(深圳)有限公司 一种移动互联网智能信息搜索引擎的搜索方法
US7958115B2 (en) 2004-07-29 2011-06-07 Yahoo! Inc. Search systems and methods using in-line contextual queries
WO2006011819A1 (en) * 2004-07-30 2006-02-02 Eurekster, Inc. Adaptive search engine
US20070208751A1 (en) * 2005-11-22 2007-09-06 David Cowan Personalized content control
US9165039B2 (en) 2005-11-29 2015-10-20 Kang Jo Mgmt, Limited Liability Company Methods and systems for providing personalized contextual search results
US8935269B2 (en) 2006-12-04 2015-01-13 Samsung Electronics Co., Ltd. Method and apparatus for contextual search and query refinement on consumer electronics devices
US20080155426A1 (en) * 2006-12-21 2008-06-26 Microsoft Corporation Visualization and navigation of search results
US20100241663A1 (en) * 2008-02-07 2010-09-23 Microsoft Corporation Providing content items selected based on context
EP2304667A4 (en) * 2008-06-03 2011-08-10 Just Parts Online Inc SYSTEM AND METHOD FOR LISTING POSTS ONLINE
US20100241507A1 (en) * 2008-07-02 2010-09-23 Michael Joseph Quinn System and method for searching, advertising, producing and displaying geographic territory-specific content in inter-operable co-located user-interface components
US8365059B2 (en) 2009-11-03 2013-01-29 Oto Technologies, Llc E-reader semantic text manipulation
US9208435B2 (en) * 2010-05-10 2015-12-08 Oracle Otc Subsidiary Llc Dynamic creation of topical keyword taxonomies
US20120124126A1 (en) * 2010-11-17 2012-05-17 Microsoft Corporation Contextual and task focused computing
US20120151310A1 (en) 2010-12-13 2012-06-14 El-Kalliny Ahmed M Method and system for identifying and delivering contextually-relevant information to end users of a data network
US9116995B2 (en) * 2011-03-30 2015-08-25 Vcvc Iii Llc Cluster-based identification of news stories
US20130007004A1 (en) * 2011-06-30 2013-01-03 Landon Ip, Inc. Method and apparatus for creating a search index for a composite document and searching same
CN102402589A (zh) * 2011-10-26 2012-04-04 北京百度网讯科技有限公司 一种提供与搜索请求相关的参考搜索信息的方法与设备
US20130124496A1 (en) * 2011-11-11 2013-05-16 Microsoft Corporation Contextual promotion of alternative search results
CN103136237B (zh) * 2011-11-28 2017-11-14 深圳市世纪光速信息技术有限公司 一种基于多数据源的信息搜索方法和系统
US10984337B2 (en) * 2012-02-29 2021-04-20 Microsoft Technology Licensing, Llc Context-based search query formation
CN103020224B (zh) * 2012-12-12 2019-01-15 百度在线网络技术（北京）有限公司 一种智能搜索方法及装置

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
None *

Also Published As

Publication number Publication date
US9754034B2 (en) 2017-09-05
WO2015080924A1 (en) 2015-06-04
US20150149429A1 (en) 2015-05-28
CN105793844A (zh) 2016-07-20
KR102310648B1 (ko) 2021-10-07
KR20160089379A (ko) 2016-07-27
EP3074888A1 (en) 2016-10-05

Similar Documents

Publication Publication Date Title
EP3074888B1 (en) Contextual information lookup and navigation
US10394420B2 (en) Computer-implemented method of generating a content recommendation interface
EP3198481B1 (en) Productivity tools for content authoring
US10248305B2 (en) Manipulating documents in touch screen file management applications
US20150169285A1 (en) Intent-based user experience
RU2645276C2 (ru) Создание вариаций при преобразовании данных в потребляемый контент
US9576049B2 (en) Semantic searching using zoom operations
US9342233B1 (en) Dynamic dictionary based on context
US10210146B2 (en) Productivity tools for content authoring
US9495454B2 (en) User apparatus, system and method for dynamically reclassifying and retrieving target information object
WO2016196697A1 (en) Graph-driven authoring in productivity tools
US9940396B1 (en) Mining potential user actions from a web page
TW201606631A (zh) 利用關聯指示器及浮動選單列的關聯選單
US11157679B2 (en) Semantic navigation of content documents
US20180330004A1 (en) Method and system for controlling presentation of web resources in a browser window
US20140164366A1 (en) Flat book to rich book conversion in e-readers
US10204156B2 (en) Displaying graphical representations of query suggestions
US20220334686A1 (en) Intuitive display of intelligent snap assist recommendations
US20130179832A1 (en) Method and apparatus for displaying suggestions to a user of a software application
CN109313662B (zh) 到本机应用体验中的对网页的解构和呈现
US8413062B1 (en) Method and system for accessing interface design elements via a wireframe mock-up
US11625148B2 (en) Intelligent snap assist recommendation model
US20190220537A1 (en) Context-sensitive methods of surfacing comprehensive knowledge in and between applications

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

17P Request for examination filed

Effective date: 20160527

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

DAX Request for extension of the european patent (deleted)
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

17Q First examination report despatched

Effective date: 20190207

REG Reference to a national code

Ref country code: DE

Ref legal event code: R079

Ref document number: 602014061283

Country of ref document: DE

Free format text: PREVIOUS MAIN CLASS: G06F0017300000

Ipc: G06F0009451000

RIC1 Information provided on ipc code assigned before grant

Ipc: G06F 9/451 20180101AFI20190712BHEP

Ipc: G06F 16/9532 20190101ALI20190712BHEP

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTG Intention to grant announced

Effective date: 20190927

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602014061283

Country of ref document: DE

REG Reference to a national code

Ref country code: AT

Ref legal event code: REF

Ref document number: 1235735

Country of ref document: AT

Kind code of ref document: T

Effective date: 20200315

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20200219

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200219

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200219

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200519

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG4D

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200219

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200219

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200520

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200619

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200519

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200219

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200219

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200712

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200219

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200219

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200219

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200219

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200219

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200219

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200219

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200219

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 1235735

Country of ref document: AT

Kind code of ref document: T

Effective date: 20200219

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602014061283

Country of ref document: DE

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

26N No opposition filed

Effective date: 20201120

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200219

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200219

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200219

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200219

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200219

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20201120

REG Reference to a national code

Ref country code: BE

Ref legal event code: MM

Effective date: 20201130

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20201130

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20201130

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20201120

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200219

Ref country code: MT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200219

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200219

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200219

Ref country code: AL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200219

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20201130

P01 Opt-out of the competence of the unified patent court (upc) registered

Effective date: 20230505

P01 Opt-out of the competence of the unified patent court (upc) registered

Effective date: 20230523

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GB

Payment date: 20231019

Year of fee payment: 10

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20231019

Year of fee payment: 10

Ref country code: DE

Payment date: 20231019

Year of fee payment: 10