JP2017201510A - Context recognition application - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a method and a system of a context recognition application for determining a plurality of resources related to context of a user and displaying them by using a single user interface.SOLUTION: A context recognition application includes: a step 605 of obtaining context data describing context of a user of an electronic device; a step 610 of transmitting the context data to a backend device; a step 615 of receiving from the backend device a command to dynamically generate a single user interface including interactive content from a plurality of related resources related to the context of the user; a step 620 of dynamically generating the single user interface; and a step of displaying the single user interface to the user of the electronic device.SELECTED DRAWING: Figure 6A

Description

  Embodiments discussed in this disclosure relate to context-aware applications.

  With the widespread use of mobile electronic devices, many users rely on their electronic devices for various purposes. Since many electronic devices are mobile, the context in which such electronic devices can be used can vary greatly.

  The subject matter claimed herein is not limited to embodiments that solve the above disadvantages or that operate only in environments such as those described above. Rather, this background section is provided merely to illustrate one exemplary technical area in which embodiments of the disclosure described herein may be implemented.

  In one or more embodiments, the present disclosure may include a method that may include obtaining context data describing a context of a user of an electronic device and transmitting the context data to a back-end device. . A method receives instructions from the backend device, the instructions instructing the electronic device to dynamically generate a single user interface that includes interactive content from a plurality of related resources. May further be included. The plurality of related resources may be determined by the back-end device as related to each other and the user's context. Features of the interactive dynamic content may be extracted by the back-end device. The method may further comprise dynamically generating the single user interface and displaying the single user interface to the user of the electronic device.

  The objects and advantages of the embodiments will be understood and at least achieved using the elements, features and combinations particularly pointed out in the claims.

  Both the foregoing general description and the following detailed description are exemplary and explanatory and do not limit the scope of the invention.

Exemplary embodiments are described and explained with additional specificity and detail through the use of the accompanying drawings in which:
FIG. 2 is an illustration of an example system for a context-aware application. 1 is a block diagram of an exemplary electronic device. 3 is an exemplary user interface for a context aware application. Figure 3 is a further exemplary user interface for a context aware application. Figure 3 is a further exemplary user interface for a context aware application. Figure 3 is a further exemplary user interface for a context aware application. Figure 3 is a further exemplary user interface for a context aware application. 2 is a flowchart of an exemplary method for utilizing a context aware application. 2 is a flowchart of an exemplary method for utilizing a context aware application. 2 is a flowchart of an exemplary method of a backend device associated with a context awareness application. 2 is a flowchart of an exemplary method of a backend device associated with a context awareness application.

  The present disclosure relates to a context-aware application in which a user's context can be considered in providing a single view to the user to interact with various resources associated with a particular context. For example, if the user's physical location is determined to be a railway station, the context-aware application knows the location of the train coming to the station (first resource set), so the user can purchase a train ticket Options of (second resource) and a menu of a neighboring restaurant (third resource set) can be provided to the user. Context-aware applications can provide a user with various resource features related to the user's context in a single view. From a single view, the user may interact with a context aware application to interact with various resources.

  As used herein, the term “context” refers to a physical location (eg, grocery store, bus stop, latitude / longitude, etc.) or physical, even if the user is not present at the physical location. It may represent any task or feature (eg, ordering from a restaurant menu, ordering a ticket at a railway station, checking a bus schedule at a bus stop, etc.) associated with a specific location.

  Embodiments of the present disclosure are described below with reference to the accompanying drawings.

  FIG. 1 is a diagram of an example system 100 for context-aware applications in accordance with one or more embodiments of the present disclosure. The system 100 may include an electronic device 110 and a back-end device 120 that can communicate via a network 130.

  The electronic device 110 may obtain context information regarding the context of the user of the electronic device 110. For example, a position sensor in the electronic device 110 may detect the physical position of the electronic device 110. Such position sensors may include a global positioning system (GPS) sensor or any other position sensor.

  As another example of obtaining context information, the electronic device 110 may receive or connect to data from a beacon 140 adjacent to the electronic device 110. The beacon 140 may include any device that is configured to communicate with the other device to identify the context of the other device (eg, electronic device 110). The beacon 140 enables detection of the physical location of the electronic device 110, tracking of the physical location of the electronic device 110, and / or triggering events based on location in the electronic device 110. The beacon 140 may operate using any beacon protocol such as iBeacon (TM), Eddystone (TM), Seeknfind (TM), and the like. Additionally or alternatively, the beacon 140 may operate using near field communication (NFC) technology to communicate with the electronic device 110. As a further example, the beacon 140 is detected to detect frequency radiation (eg, 2.4 gigahertz (GHz), 5 GHz, 800-900 megahertz (MHz), 1700-1800 MHz, etc.) used by the electronic device. It may operate using technology and may communicate with the detected electronic device to achieve position detection.

  Further examples of obtaining context information may include electronic device 110 scanning a quick response (QR) code 150 to obtain context information. The electronic device 110 may utilize a camera or other optical sensor to detect or scan the QR code 150. The QR code 150 may provide context information to the electronic device 110. For example, QR code 150 may identify the physical location of electronic device 110 and / or provide access to activities associated with the physical location.

  After obtaining the context information, the electronic device 110 may send the context information to the backend device 120. The back-end device 120 may provide instructions to the electronic device 110 and provide a user interface that may include interactive content from a plurality of resources 160 associated with each other and associated with the user's context of the electronic device 110. Instruct to generate dynamically. Such a user interface may be a single view that incorporates features and / or content from each of a plurality of resources. After receiving the instructions, the electronic device 110 may generate a user interface and display the user interface to a user of the electronic device 110. The user interface may allow a user of electronic device 110 to interact with multiple resources from a single interface.

  In some embodiments, the backend device 120 may communicate with multiple resources 160. A resource may comprise any device, component, system, or combination thereof that is configured to provide information or services to a user of electronic device 110. For example, the resource may include a web service such as an online ticket ordering system, a restaurant website, a restaurant online menu, an online map or map service, or any other website hosting information on the Internet. good. As another example, the resources may be lighting, outlets, thermostats, surveillance cameras, buses, cars, motorcycles, appliances (eg, washing machines, dryers, refrigerators, furnaces, etc.), trains, point-of-sale terminals, or the Internet Any device connected to the Internet may be included, such as any other device that can receive or transmit information over the Internet. As shown in FIG. 1, the back-end device 120 communicates with an outlet 161, a first web server 162, a second web site 3, a camera 164, a bus 165, a dryer 166, a train 167, and a point-of-sale terminal 168. good. Each of these may be one of the resources 160.

  In some embodiments, the back end device 120 may prepare a database to implement providing instructions to the electronic device 110. In the above and other embodiments, database construction may include a training phase and a recognition phase. The training phase may include generating a class in which a resource 160 that includes a basic application programming interface (API) model for a given class can be classified and can be mapped to a newly discovered resource. There may be a step of specifying the characteristics of the model. The recognition phase may include connecting to the new resource 160 to classify the new resource 160 and identify the characteristics of the new resource 160. Additionally or alternatively, the recognition phase may include identifying and storing relationships between related resources 160. Additionally or alternatively, the recognition phase may include identifying a context to which a group of related resources may be applicable and storing a relationship between the context and the related resource. In the above and other embodiments, after a plurality of classes and a basic API model for the classes are established, the classes and basic API models may be stored in a database.

  In the training phase, the backend device 120 may generate multiple classes into which resources can be classified. A class may have one or more resources that have a set of features or characteristics common to a category or group of resources. The classes may be hierarchical, for example, the first class may have multiple subclasses. In some embodiments, a basic API model of an individual resource of a class may be generated, having features and / or characteristics that are likely to be common to other resources belonging to the class. For example, the basic API model of the “Restaurant” class may have a “menu” feature, an “order” feature, a “reservation” feature, and a “contact” feature. Other examples of classes include any web service (eg, commercial, library, transport service, etc.), or “public transport vehicle” class (eg, bus, rail, etc.) or “contracted transport vehicle” class (eg, It may include any device connected to the Internet, such as a taxi or UBER® driver. In the above and other embodiments, the class may be generated automatically or may be generated by a user specifying a particular class of features and / or other features.

  In some embodiments, a basic API model with various features may be generated for comparison purposes. For example, a user of backend device 120, such as an administrator, may manually generate a basic API model for various classes of resources. For example, a user can use an API document from one or more resources to generate a basic API model, thereby creating a basic for a restaurant with “menu”, “order”, and “reservation” features. An API model may be generated. In the above and other embodiments, the basic API model and associated features may be stored in a database.

  In the above and other embodiments, the back-end device 120 is one of the resources 160, such as a first web server 162 that may host a web service that includes a portal for purchasing rail routes and rail tickets. It may be connected to one. The backend device 120 may connect to the resource 160 by a web scraping process, web crawling process, web indexing, or other similar process. As another example, the back-end device 120 may look for an open portal to the device, such as an open portal for accessing the train 167.

  After connecting to the resource, the backend device 120 may generate an API model for the resource. For example, when an API document (for example, an API description manual document such as an extensible markup language (XML) document, a schema, a PDF (portable document format) document, etc.) is available, the back-end device 120 may May be used to identify various features of the resource. In the above and other embodiments, API documents may be parsed using a natural language processing (NLP) process to identify and identify various characteristics of the resource. Additionally or alternatively, the API document may be processed manually depending on the user and the various features identified. As another example, a vendor, owner, or other entity that holds a resource may provide instructions regarding the characteristics of a particular resource. In the above and other embodiments, resources and associated features may be stored in a database.

  In some embodiments, the back-end device 120 may classify resources within a given class based on a comparison of resources according to one or more basic API models of classes stored in a database. In the above and other embodiments, the comparison may be automatic (eg, using NLP processing) or may be performed manually (eg, by the user or by crowdsourcing the comparison). The backend device 120 may classify resources in the class that the resource maps closest to on the basic API model. For example, if the first web server 162 operates a restaurant web service that includes a “menu” feature and an “order” feature, the resources include a “menu” feature, an “order” feature, and a “reservation” feature. May be mapped closest to the basic API model of the “restaurant” class. In the above and other embodiments, for newly classified resources, a list of resources, and their proposed classifications, may be provided to the user of backend device 120 for viewing and / or authorization. Additionally or alternatively, a list of resources that did not fit in the classification may be provided in the list for the user to manually assign a class and to generate a new class of resources. In some embodiments, the classification for a given resource may be provided by the vendor, owner, or other entity that holds the resource.

  In some embodiments, the features identified in the API model of the newly discovered resource (eg, first web server 162) may be mapped to the resource features in the database. For example, a “menu browsing” feature in a resource may be mapped to a “menu” feature in a basic API model. Such mapping may be based on an automatic comparison of “menu browsing” and “menu” features, or may be based on user input mapping “menu browsing” features to “menu” features. For example, the mapping may be crowdsourced so that a list of features and possible mappings can be provided to the user for mapping. Alternatively, a list of features and a suggestion mapping may be provided to the user for confirmation. In the above and other embodiments, the mapping may identify features of other resources that perform similar or identical functions.

  The recognition phase of preparing the database may include identifying and storing relationships between related resources 160. Identification of related resources may include a vendor, owner, or other entity that holds resources that provide a number of resources that may be related. For example, a restaurant with delivery options may have ordering system resources from the restaurant and resources of the delivery driver or vehicle's physical location when the food is delivered. As another example, a city may identify bus scheduling resources, bus ticket ordering resources, and physical locations of buses as related resources. Additionally or alternatively, identifying related resources may include a user identifying or suggesting relationships between resources. For example, a user at a mall may identify the merchant's ordering system and inventory resources as being associated with an interactive mall map resource. As another example, a user may identify neighboring restaurant resources as being associated with a bus stop map resource. In some embodiments, the electronic device 110 may prompt the user of the electronic device 110 if neighboring resources may be related to the resource that the user is accessing or browsing. In the above and other embodiments, the relationships between resources may be stored in a database.

  In some embodiments, the recognizing stage of preparing the database may include identifying contexts where a group of related resources may be applicable. For example, when a user is physically close to a physical restaurant that has one or more resources associated with a restaurant, the resource associated with the restaurant and any resource associated with the restaurant are: You may have a relationship with the context. As another example, when a user scans a QR code, there may be associated resources (eg, a bus stop with a QR code may be associated with bus schedule and ticket purchase resources, and taxi service resources). The relationship between a context and a resource can be based on the vendor, owner, or other entity that holds the resource that provides the context with which the resource can be related, or has a user that identifies the context with which the resource can be related. You can do it. For example, the user of electronic device 110 may be prompted to indicate whether one or more neighboring resources are related to the user's location. For example, a dialog box or prompt may be provided to the user of the electronic device 110 to confirm that neighboring resources are related to the user's location. In the above and other embodiments, the relationship between a context and a resource or group of related resources may be stored in a database.

  In some embodiments, the back-end device 120 may combine features from two or more related resources in a model graphical user interface that may have a single view that includes features from at least two different resources. . For example, the interface may have restaurant “menu” and “reservation” features, as well as associated “movie list” resources for nearby theaters. In the above and other embodiments, features from various related resources may be mapped to a single view of visual elements, such as windows, buttons, tags, frames, etc. In the above and other embodiments, the model graphical user interface may be stored in a database. In addition or alternatively, the mapping to a specific function or visual element may be stored in a database, and the electronic device 110 may dynamically generate a user interface based on the visual element and / or the model graphical user interface. You may have. In some embodiments, the dynamically generated user interface may be displayed in a common format and using a common visual scheme. For example, a common set of fonts, colors, buttons, tags, frames, etc. may be used even if the corresponding resource can use a different format by definition.

  Modifications, additions, or omissions may be made to FIG. 1 without departing from the scope of the present disclosure. For example, the system 100 may have more or fewer elements than those shown and described in this disclosure. For example, the system 100 may include any number or type or various resources 160 in addition to those shown. As another example, system 100 may include any number or type or various components that provide context in addition to beacon 140 and QR code 150. As an additional example, optional resource 160 may communicate with backend device 120 via network 130. As a further example, any of the resources 160 may communicate directly with the electronic device 110 in addition to or instead of the back-end device 120.

  FIG. 2 is a diagram of an example electronic device 200 in accordance with one or more embodiments of the present disclosure. For example, the electronic device 110 and / or the back-end device 120 of FIG. 1 may be implemented as the electronic device 200. As another example, one or more of the resources 160 may be implemented as the electronic device 200.

  The electronic device 200 may include a processor 210, a memory 220, a data storage device 230, a communication component 240, a position sensor 250, a scanning device 260, and a display 270. The processor 210, memory 220, data storage device 230, communication component 240, position sensor 250, scanning device 260, and / or display 270 may all be communicatively coupled such that each of the components can communicate with each other. The electronic device 200 may perform or control execution of any of the operations described in this disclosure.

  In general, the processor 210 may comprise any suitable special purpose or general purpose computer, computing entity, or various computer hardware or software modules that execute instructions stored on any suitable computer readable medium. It may be implemented using a processing device that may be configured to perform. For example, the processor 210 may interpret and / or execute a microprocessor, microcontroller, digital signal processor (DSP), application specific integrated circuit (ASIC), field programmable gate array (FPGA) or program instructions and / or data. Any other digital or analog circuitry configured to process the may be included. Although a single processor is shown in FIG. 2, processor 210 may be any number of processors configured to perform or control the execution of any number of steps described in this disclosure individually or jointly. You may have.

  In some embodiments, processor 210 may interpret and / or execute program instructions and / or process data stored in memory 220, data store 230 or memory 220 and data store 230. In some embodiments, processor 210 may fetch program instructions from data storage device 230 and load the program instructions into memory 220. After the program instructions are loaded into the memory 220, the processor 210 may execute the program instructions.

  Memory 220 and data storage device 230 may include computer-readable storage media that carry or store computer-executable instructions or data structures. Such computer-readable media can include any available media that can be accessed by a general purpose or special purpose computer such as processor 210. By way of example, such computer readable storage media include RAM, ROM, EEPROM, CD-ROM or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, flash memory devices (eg, solid state memory devices). Including tangible or non-transitory computer readable storage media or other storage accessible to a general purpose or special purpose computer used to convey or store the desired program code means in the form of computer-executable instructions or data structures It can have a medium. Combinations of the above can also be included within the scope of computer-readable storage media. Computer-executable instructions may include, for example, instructions and data configured to cause processor 210 to perform a particular process or group of processes or to control execution.

  Communication component 240 may be any device, system, component, or device configured to enable or assist communication between device 200 and another device (eg, by communicating via network 130 of FIG. 1). You may have a set of components. For example, the communication component 240 may be a modem, a network card (wireless or wired), an infrared communication device, an optical communication device, a wireless communication device (such as an antenna), and / or a (Bluetooth) device, an 802.6 device. It may include a chipset (e.g., MAN (Metropolitan Area Network)), WiFi device, WiMax device, cellular communication facility, etc.) and / or the like. Communication component 240 may be any network (including network 130) such as a cellular network, a WiFi network, a MAN, etc., and / or any other described in this disclosure including remote devices, to name a few examples. Can exchange data with other devices. Combinations of any of the above can also be included within the scope of communication component 240.

  The position sensor 250 may comprise any device, system, component, or collection of components that is configured to determine the physical position of the electronic device 200 and to implement the determination. For example, the position sensor 250 may include a global positioning service (GPS) sensor, a proximity sensor, an acceleration sensor, a motion sensor, a gyroscope, a compass, and the like. Additionally or alternatively, the position sensor 250 may include a software routine or set of instructions that utilizes the communication component 240 or other components of the electronic device 200 to determine neighboring physical components and estimate the physical position. For example, the position sensor 250 may determine three closest cellular telephone towers and estimate the physical location, or determine the WiFi hotspot to which the communication component 240 is connected and estimate the physical location. good. In some embodiments, the position sensor 250 may be configured to communicate with a beacon (such as the beacon 140 of FIG. 1) to detect the position of the electronic device 200. Combinations of any of the above can also be included within the scope of the position sensor 250.

  Scanning device 260 may comprise any device, system, component, or collection of components (eg, a camera) that is configured to scan an image. For example, the scanning device 260 may include, for example, optical elements such as lenses, filters, holograms, splitters, and the like, and image sensors that can record images. Such an image sensor may include any device that converts an image represented by incident light into an electronic signal. An image sensor may include a plurality of pixel elements that may be arranged in a pixel array (eg, a grid of pixel elements). For example, the image sensor may comprise a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) sensor. Various other components may also be included in the scanning device 260. Pixel arrays can have aspect ratios of 1: 1, 4: 3, 5: 4, 3: 2, 16: 9, 10: 7, 6: 5, 9: 4, 17: 6, etc. or any other ratio It may have a two-dimensional array. The image sensor may be optically aligned with one or more optical elements, such as lenses, that focus on the pixel array or process the light. In some embodiments, the scanning device 260 may be configured to scan among QR codes or other visual elements that convey information. Such information may convey the location, resources, context, etc. of the electronic device 200. Combinations of any of the above can also be included within the scope of the scanning device 260.

  Display 270 may include any component, device, system, or combination thereof that is configured to visually display information. The display 270 is a cathode ray tube (CRT), liquid crystal display (LCD), thin film transistor LCD, plasma display, one or more light emitting diodes (LED), one or more organic LEDs, one or more bulbs, a dot matrix display, a vacuum Fluorescent displays, twisted nematic effect LCDs, super twisted nematic effect LCDs, electronic paper (eg, electrophoretic displays, electrowetting displays, electrofluidic displays, interferometric modulator displays), and the like. Display 270 may display information in any format or structure. For example, the display 270 may be implemented as a monitor, a screen, an optical panel, or the like. Display 270 may be implemented as a stand-alone display or as a subpart or component of another device or system. Display 270 may have any secondary characteristics such as a three-dimensional display, touch screen control, directional display control (eg, viewing angle variation of display 270), and so on. Combinations of any of the above can also be included within the scope of display 270.

  Changes, additions, or omissions may be made to FIG. 2 without departing from the scope of the present disclosure. For example, the apparatus 200 may have more or fewer elements than those shown and described in this disclosure. For example, device 200 may weigh memory 220, data storage device 230, position sensor 250, scanning device 260, and / or display 270, or any combination thereof.

  FIG. 3A is an exemplary user interface 300a of a context-aware application according to one or more embodiments of the present disclosure. For example, user interface 300a may be displayed on a display (eg, display 270 of FIG. 2) and / or dynamically generated after the user scans a QR code at a bus stop.

  The user interface 300a may display information related to the bus schedule resource based on the user context, that is, the user is at a specific bus stop. For example, the user interface 300a may display bus routes that use a particular bus stop and may identify when the next bus is scheduled to arrive for each of those routes. In some embodiments, the time may be based on additional resources of the GPS locator associated with each of the next buses arriving at the bus stop.

  In some embodiments, a user viewing the graphical user interface 300a may interact with the graphical user interface 300a to request additional information or interact with resources presented by the graphical user interface 300a. You can do it. For example, the user may select one of the bus routes (eg, route 337) and be provided with additional information or resources associated with the resource (eg, the user is shown in FIG. 3B). Interface 300b may be presented).

  Further, as shown in FIG. 3A, the user interface 300a may include references or links to other neighborhood services, such as neighborhood theaters, neighborhood restaurants, and neighborhood libraries. Additional services and associated resources interact with the user interface 300a to select one or more of the neighboring services (or more specifically, references or links to the neighboring services included in the user interface 300a). By selecting one of them).

  In some embodiments, the user's context may be sent to the backend device when the user changes the context, and the user interface 300a may be updated or changed based on the user's context change. For example, if the user is on a bus, the context and the service displayed to the user will be the context of the user on the bus or (if the user enters the destination or frequently gets off at the same bus stop) The shift may be based on the user's destination context from the bus. As another example, if the user has picked up a taxi, the context may be updated periodically or constantly based on the taxi's location as the taxi travels through different locations.

  In some embodiments, additional services may be suggested or displayed to the user based on information other than the context information of the electronic device. For example, identifying information from the electronic device and / or time of day may form the basis for selecting and / or displaying various services. In some embodiments, the service selected by the user and the time history may be stored on the backend device and used to select and / or display various services. For example, if a particular user orders a takeout on the evening most of the way home, the displayed service will allow the user to order a takeout when the user's context indicates that they are coming home soon or on the way home May have restaurant service.

  In some embodiments, the back-end device may monitor and maintain a history of selections that are generally made by any user. Thus, for example, when displaying services, the services are ordered based on the most popular services in a given context or generally the services that are most often used or at a particular time of day. Good.

  FIG. 3B is a further exemplary user interface 300b of a context-aware application according to one or more embodiments of the present disclosure. For example, the user interface 300b may show an example of what the user can be presented after selecting the bus route 337 from the user interface 300a of FIG. 3A.

  The user interface 300b may include a map of the arrival time of the next bus on the 337 bus route and the location of the bus. The arrival time, the next three bus stops, etc. may be based on a scheduled bus service resource, and the map may be derived from a map resource based on the Internet. The location of the bus on the map may be based on bus positioning resources.

  Further, the user interface 300b may have additional information from additional related resources. For example, a back-end device (eg, back-end device 120 of FIG. 1) may associate Uber® resources and taxi service resources with bus route resources. Related resources (eg, Uber® resources and taxi service resources) may be included in the user interface 300b. For example, the cost difference for using such resources and the difference in arrival time at the destination may be included in the user interface 300b. Additionally or alternatively, the user may be presented with an option to purchase a ticket / service associated with the associated resource, such as requesting a Uber® pickup, rather than waiting for a bus.

  In the above and other embodiments, the user interface 300b may present some or all of the information associated with various resources in a single view and using a common vision scheme. For example, a traditional taxi service web page may use a single font, but information related to taxi service resources is displayed in a common visual manner in a common view and with bus information related to bus routes. (Eg, taxi service information may be displayed in a different font from traditional taxi service web pages and in the same font as bus information).

  FIG. 4A is a further exemplary user interface 400a of a context awareness application, according to one or more embodiments of the present disclosure. The user interface 400a may display features associated with resources presented to the user when the user arrives at the restaurant or when the user selects a restaurant.

  The user interface 400a may present a user with a plurality of options including options for browsing a restaurant menu, ordering a takeout or delivery from the restaurant, or reserving a seat for a selected number of people. The user interface 400a may also provide information related to how long the waiting time for a selected number of people is. For example, a restaurant may include resources regarding the number of vacant tables, the number of people already waiting for a seat, and the estimated time that the seat will be available. As another example, a restaurant may include resources for browsing menus and ordering takeaway food.

  FIG. 4B is a further exemplary user interface 400b of a context-aware application according to one or more embodiments of the present disclosure. FIG. 4B may show an example of a user interface 400b that a user may be presented after making a reservation.

  User interface 400b may indicate how the user is contracted to inform the user that his table is ready. Additionally or alternatively, the user interface 400b may present the user with an option to select a menu or to examine the menu in detail before taking a seat. In the above and other embodiments, user interface 400b may include an option to purchase items from a menu while waiting for the user to sit down. For example, the user may interact with the user interface 400b and order appetizers from a menu. The requested order may be sent to a resource provider (e.g., a restaurant) so that when the user is seated, an appetizer can be prepared for the user without placing an order at the seat.

  FIG. 5 is a further exemplary user interface 500 of a context aware application, according to one or more embodiments of the present disclosure. The user interface 500 may show the characteristics of resources that may be presented to the user when approaching or in the vicinity of (eg, within a predetermined distance from) a beacon at a railway station.

  The user interface 500 may provide the user with various features from related resources associated with the railway station. For example, the user interface 500 may provide the user with an option to select a destination such as a museum. Additionally or alternatively, the user interface 500 may provide details regarding what trains leave the railway station at a given time, such as the next 20 minutes. The user interface 500 may additionally or alternatively provide the user with a map of where to go to take particular trains and where those trains run. The map may include real time data of where the train is on the map based on the train location resources. User interface 500 may additionally or alternatively include neighborhood services such as restaurants or libraries. In the above and other embodiments, the user interface 500 may have multiple language options for resources. For example, the traveler may be able to access the resource set in the user's native language. Such language options may include resource providers (eg, railway stations), back-end devices or end-user devices (eg, back-end devices that perform machine translation of resources from one language to another), or It may be provided by a third party service (eg, by a web-based translation service).

  Changes, additions, or omissions may be made to any of FIGS. 3A, 3B, 4A, 4B, and / or 5 without departing from the scope of the present disclosure. For example, the user interfaces 300a, 300b, 400a, 400b, and / or 500 may have more or fewer elements than those shown and described in this disclosure. For example, any number of resources or resource associations may be displayed on the user interfaces 300a, 300b, 400a, 400b, and / or 500. The user interfaces 300a, 300b, 400a, 400b, and / or 500 serve to show various combinations of related resources and how such resources can be presented to the user. Any combination of resources is encompassed within the scope of this disclosure.

  6A and 6B are flowcharts of an exemplary method 600 for utilizing a context-aware application according to one or more embodiments of the present disclosure. Method 600 may be performed by any suitable system, device, or apparatus. For example, the system 100 and / or the electronic device 110 of FIG. 1 and / or the electronic device 200 of FIG. 2 may perform one or more of the steps associated with the method 600. Although shown as separate blocks, depending on the desired implementation, steps and processes associated with one or more of the blocks of method 600 are divided into further blocks, combined into fewer blocks, or removed. May be.

  At block 605, context data describing the context of the user of the electronic device (eg, electronic device 110 of FIG. 1) may be obtained. For example, a user of an electronic device may receive a QR code (eg, QR code 150 in FIG. 1) at a specific physical location (eg, a bus stop) or associated with a specific service (eg, in a restaurant advertisement). The electronic device 110 may be used for scanning. As another example, a user of an electronic device may approach or come within a certain distance range of a beacon that can communicate with the electronic device (eg, beacon 140 of FIG. 1), and the electronic device obtains context data. You may use communication with a beacon. As an additional example, an electronic device position sensing device, such as a GPS sensor, may detect a user's physical location.

  At block 610, the context data may be sent to the backend device (eg, backend device 120 of FIG. 1). For example, the electronic device may send context data to the backend device over a network (eg, network 130 of FIG. 1).

  At block 612, the electronic device may send information to the personal learning agent. For example, a personal learning agent may act on a back-end device to track user interaction with an electronic device and / or with one or more services. A personal learning agent may track the time of day during which a particular action occurs and / or the context in which a particular action occurs. The personal learning agent may receive some or all of such data at the backend device to be used to personalize and / or prioritize the services displayed to the user. In some embodiments, the personal learning agent may use a machine learning method, such as subject learning, to determine which resources to display. In the above and other embodiments, the personal learning agent may be located on an electronic device.

  At block 615, an interaction may be received from the backend device to dynamically generate a single user interface having interactive content from a plurality of related resources. For example, the electronic device may receive instructions that map features from multiple resources to visual elements of the user interface. As another example, the electronic device may receive instructions for generating a user interface having data derived from one or more of the plurality of resources. In the above and other embodiments, the instructions may have a model user interface.

  At block 620, a single user interface may be dynamically generated. For example, the electronic device may use instructions received from the backend device to dynamically generate a single user interface. In the above and other embodiments, a single user interface may have features from multiple resources. Additionally or alternatively, features may be mapped to one or more visual elements of the interface. Examples of such user interfaces are shown in FIGS. 3A, 3B, 4A, 4B, and 5.

  At block 625, features extracted by the backend device may be mapped to individual elements that are generated as part of a single user interface. For example, if the back-end device has extracted “menu” and “order” features of a restaurant resource, these features may be mapped to individual visual elements of a single user interface. In some embodiments, the mapping of block 625 may be part of the dynamic generation performed at block 620.

  At block 630, interactive content from each of a plurality of related resources may be incorporated into a common format and a common visual scheme. In the above and other embodiments, in generating the single user interface, the electronic device may incorporate at least one feature from each of the resources. Common formats and common visual schemes may include, for example, common fonts, common color sets, and the like. In some embodiments, the incorporation of block 630 may be part of the dynamic generation performed at block 620.

  At block 635 (of FIG. 6B), a single user interface may be displayed to the user of the electronic device. For example, the electronic device may output a single user interface on a display associated with the electronic device or portion thereof.

  At block 640, a user interaction with a single user interface element may be received to generate interaction data. For example, a user of an electronic device may click, touch, or select one of the elements of a single user interface that is mapped to one of the resource characteristics (eg, the user Click the “Menu” button corresponding to the “Menu” feature of the restaurant resource). User interaction with the electronic device may be captured as interaction data.

  At block 645, the interaction data may be sent to the backend device to be communicated to the resource associated with the element. For example, after capturing the interaction data, the electronic device may send the interaction data to the backend device (eg, if the user touches a “menu” button, the interaction is sent to the backend device. good). The backend device may communicate the interaction to the resource associated with the element. For example, the backend device may send a “menu” request to the resource.

  Accordingly, the method 600 may utilize a context aware application. Changes, additions, or omissions may be made to method 600 without departing from the scope of the present disclosure. For example, one or more of the blocks 605, 610, 615, 620, 625, 630, 640, 645 may be omitted. For example, the steps of method 600 may be performed in a different order. In addition or alternatively, two or more steps may be performed simultaneously. Furthermore, the schematic steps and operations are provided as examples, and some steps and operations are optional, combined with fewer steps and operations, or additional steps and operations without departing from the essence of the disclosed embodiments. It may be extended to operation.

  7A and 7B are flowcharts of an exemplary method 700 of a back-end device associated with a context-aware application according to one or more embodiments of the present disclosure. Method 700 may be performed by any suitable system, device, or apparatus. For example, the system 100 and / or the back-end device 120 of FIG. 1 and / or the electronic device 200 of FIG. 2 may perform one or more of the steps associated with the method 700. Although shown as separate blocks, depending on the desired implementation, steps and processes associated with one or more of the blocks of method 700 are divided into additional blocks, combined into fewer blocks, or removed. Good.

  At block 705, the connection may be created by multiple resources (eg, resource 160 of FIG. 1). For example, the back-end device (for example, the back-end device 120 in FIG. 1) may be connected to a plurality of resources. Connections may be created simultaneously or individually or in any combination thereof. For example, the back-end device may connect to the first resource and then connect simultaneously with the second and third resources.

  At block 710, an API model may be built for each resource. For example, an API document (eg, XML schema) for a particular resource may be parsed by the backend device. As another example, resources may be manually parsed to generate an API model. Additionally or alternatively, the resource provider may identify or provide a resource API model.

  At block 715, each resource may be classified into a class. For example, the backend device may compare the API model and / or other characteristics of a given resource with one or more different classes of one or more basic API models. Certain resources may be classified into classes that the API model most closely matches. Additionally or alternatively, the user of the backend device may identify a specific class to which the resource can belong. For example, the back-end device may provide recommendations for a given resource, and the user of the back-end device may select the resource as a new class of basic model.

  At block 720, features may be extracted from each of the resources. For example, the backend device may extract one or more features from the first resource and may extract one or more different or similar features from the second resource. Such extraction may be performed by parsing or identifying features in the API model built for each of the resources. As another example, a user of a backend device may extract or identify one or more features associated with a resource from an API model.

  At block 725, each feature and class of the resource may be stored in a database. For example, the entries may be generated by the backend device in a database of specific resources that may have specific resource characteristics and / or classes. In addition or alternatively, the database may store the address or location of the resource (eg, an Internet Protocol (IP) address or web address).

  At block 730, a determination may be made that two or more of the resources are relevant. For example, a resource provider may identify other resources that may be associated with the resource (eg, a Uber® resource provider may identify a resource as being associated with a bus and / or taxi service resource. ). Additionally or alternatively, a user of the backend device or another electronic device may identify a relationship between two or more resources. For example, a user may be queried as to whether a resource near where the user is physically located is related to another resource that the user is accessing. In the above and other embodiments, the user of the back end device may identify or confirm a relationship between two or more resources.

  At block 735, the relationship between two or more resources may be stored in a database. For example, the backend device may store the relationship as an entry associated with a resource in the database. For example, if a bus schedule resource is associated with a bus location resource and a Uber® resource, each of these three resources indicates, identifies and / or describes a relationship with the other two resources. May have entries in the database.

  At block 740, a model interface that combines at least one resource from each of the related resources may be generated. For example, the back-end device may generate a model interface that can be transmitted to the electronic device, such as a model layout when generating a user interface. Additionally or alternatively, the model interface may include instructions that map specific features of the basic model API of the class to given visual elements or features of the model interface. For example, in a restaurant class for a resource, the model interface may map a “menu” feature to a particular visual element, and other resources in the restaurant class that have a “menu” feature may also be mapped to that particular visual element. You can do it.

  At block 745, the model interface may be stored in a database. For example, the backend device may store the model interface in a database.

  At block 750 (as shown in FIG. 7B), a relationship between the model interface and the context of each of the resources having features in the model interface may be established. For example, in a restaurant class of resources, any resource that has a “menu” feature may be associated with a model interface that has that feature of context at the physical location of the restaurant. As another example, in a model interface that has a bus schedule on the model interface, a relationship between a particular context (eg, at a bus stop or on a bus) and the model interface may be established. In addition or alternatively, a relationship between the context and a particular resource may be created. The relationship may be stored in a database. For example, a relationship between a context (eg, on a bus) and a given resource (eg, bus location and / or bus schedule) may be determined.

  At block 755, context data may be received from an electronic device (eg, electronic device 110 of FIG. 1). For example, the electronic device may send context data such as a QR code, physical location (eg, latitude and longitude, or street address) to the back-end device over the network.

  At block 760, resources associated with the context may be updated. For example, the backend device may retrieve the model user interface associated with the context and may connect resources to the features in the model interface to update any data associated with the features. Additionally or alternatively, the backend device may retrieve the resource associated with the context and connect to the resource to update any data associated with the resource characteristics. In the above and other embodiments, the backend device may store the updated data in a database. For example, if the context data is that the user of the electronic device is at a restaurant, the back-end device may connect to the restaurant's “menu” resource to update any changes to the “menu” resource. . In some embodiments, the back-end device may update resources based on data received at the personal learning agent from the electronic device. Additionally or alternatively, the backend device may change the order of resources based on such data. In the above and other embodiments, the back-end device may further update resources based on personalized learning data from multiple users.

  At block 765, information in the database regarding the resource may be sent to the electronic device. For example, instructions for dynamically generating a single user interface that may include one or more of the resource characteristics and associated data from a database may be sent to the electronic device. Additionally or alternatively, the back-end device may send a model interface from the database having a mapping of resource features associated with the context mapped to the visual elements of the model interface. For example, the back-end device may send an updated “menu” to a user at a restaurant for inclusion in a single user interface.

  At block 770, user interaction data (eg, user interaction data generated at block 640 of FIG. 6) may be received from the electronic device. For example, an optional user touch or selection to order an item from a menu may be received from the electronic device by the backend device.

  At block 775, the interaction data may be communicated to the appropriate resource. For example, if the interaction data includes a user selection of an option to order items from a menu, the backend device may communicate a request to order to a restaurant resource.

  Accordingly, the method 700 may be performed by a backend device in conjunction with a context awareness application. Changes, additions, or omissions may be made to method 700 without departing from the scope of the present disclosure. For example, the steps of method 600 may be performed in a different order. In addition or alternatively, two or more steps may be performed simultaneously. Furthermore, the schematic steps and operations are provided as examples, and some steps and operations are optional, combined with fewer steps and operations, or additional steps and operations without departing from the essence of the disclosed embodiments. May be extended to work.

  As used in this disclosure, the term “module” or “component” refers to a specific hardware implementation configured to perform the actions of the module or component, and / or general purpose hardware of a computing system (eg, a computer-readable medium). Software objects or software routines that may be stored in and / or executed by a processing device, etc.). In some embodiments, different components, modules, engines, and services than described in this disclosure may be implemented as objects or processes that execute on the computing system (eg, as separate threads). Although some of the systems and methods described in this disclosure have been generally described as being implemented in software (stored on and / or executed by general purpose hardware), It is possible and possible to implement a combination of dedicated hardware. In this description, a “computer entity” may be a computing system or a module or combination of modules that executes on a computing system as defined earlier in this disclosure.

  Terms used in this disclosure and specifically in the appended claims (eg, the body of the appended claims) are generally considered to be open terms (eg, the term “including”). ”Should be interpreted as“ including but not limited to ”, the term“ having ”should be interpreted as“ having at least ”, and the term“ includes ”should include“ Is not to be construed, and the term “containing” is to be construed as “including but not limited to”).

  Further, where a specific number of introduced claims are intended to be quoted, such intentions are explicitly indicated in the claims, and if such citations do not exist, such intentions are present. do not do. For example, as an aid to understanding, the following appended claims may include the use of the prefix “at least one” and “one or more” to introduce claim citations. However, the use of such terms does not mean that the indefinite article "a, an" can be used even when the same claim includes the indefinite article such as the prefix "one or more" or "at least one" and "a or an". Is not to be construed as indicating that any particular claim, including such introduced claim citations, is limited to embodiments that include only one such citation ( For example, “a” and / or “an” should be taken to mean “at least one” or “one or more”. The same is true for the use of definite articles used to introduce claim citations.

  Further, if a particular number of introduced claim citations are explicitly cited, those skilled in the art will recognize that such citations should at least be construed to mean the cited numbers. Wax (e.g., "two citations" are left intact, meaning at least two citations, or two or more citations, unless otherwise changed). Further, in examples where a convention similar to “at least one of A, B, C, etc.” or “one or more of A, B, C, etc.” is used, such a configuration is typically , A only, B only, C only, A and B together, A and C together, B and C together, or A, B, C together, and the like.

  In addition, any disjunctive word or phrase that represents two or more alternative terms may be either one of the terms, one of the terms, or both, whether in the description, the claims, or the drawings. It should be understood to encompass the possibility of including the terms For example, the phrase “A or B” should be understood to include the possibilities of “A” or “B” or “A and B”.

  All examples and conditional statements in this disclosure are for educational purposes only to help readers understand the principles of this disclosure and the concepts devised by the inventor and to promote technology. Should not be considered as being limited to the specifically described examples and conditions. Although embodiments of the present disclosure have been described in detail, various changes, substitutions and modifications can be made without departing from the spirit and scope of the present disclosure.

In addition to the above embodiment, the following additional notes are disclosed.
(Supplementary Note 1) Obtaining context data describing a context of a user of an electronic device;
Transmitting the context data to a back-end device;
Receiving instructions from the back-end device, the instructions instructing the electronic device to dynamically generate a single user interface including interactive content from a plurality of related resources; Related resources are determined to be related to each other and to the context of the user by the back-end device, and features of the interactive dynamic content are extracted by the back-end device;
Dynamically generating the single user interface;
Displaying the single user interface to the user of the electronic device;
Having a method.
(Supplementary Note 2) The step of obtaining the context data includes: position information from a position detection component of the electronic device, beacon data from an electronic beacon proximate to the electronic device, or a quick response (QR) scanned by the electronic device. The method of claim 1, comprising obtaining at least one of the codes.
(Supplementary note 3) The method according to supplementary note 1, wherein the plurality of related resources are read from two or more third organizations.
(Supplementary note 4) The method according to supplementary note 1, wherein the plurality of related resources include one web service and one device connected to the Internet.
(Supplementary note 5) The method according to supplementary note 4, wherein the one apparatus includes one of a vehicle, a point-of-sale terminal, an appliance, a surveillance camera, an electrical outlet, a thermostat, or a lighting.
(Supplementary Note 6) The step of dynamically generating the single user interface includes:
Mapping each of the features extracted by the backend device to individual elements generated as part of the single user interface;
Incorporating the interactive content from each of the plurality of related resources into a common format and common visual scheme of the single user interface;
The method according to appendix 1, wherein:
(Supplementary note 7) Receiving the user interaction with an element of the single user interface to generate interaction data;
Transmitting the interaction data to the back-end device to be communicated to one of the plurality of related resources associated with the element of the single user interface;
The method according to appendix 1, further comprising:
(Supplementary Note 8) The back-end device determines that the plurality of related resources are related to each other and the context of the user;
Connect to the plurality of resources;
Building an application interface (API) model for each of the plurality of resources;
Classifying each of the plurality of resources into a class;
Extracting features from each of the plurality of resources;
Storing the feature and the class for each of the plurality of resources in a database;
Determining that two or more of the plurality of resources are related by a relationship;
Storing the relationship between the two or more of the plurality of resources in the database;
Generating a model interface that combines at least one of the features from each of the plurality of resources associated with each other;
Storing the model interface in the database;
Associating the model interface with the context of each of the plurality of resources including features in the model interface;
To extract the features of the interactive dynamic content,
The method according to appendix 1.
(Supplementary Note 9) A non-transitory computer-readable medium having instructions, wherein the instructions are configured to cause the processor to perform one or more steps or to control execution when executed by a processor, the steps Is
Obtaining context data describing the context of the user of the electronic device;
Transmitting the context data to a back-end device;
Receiving instructions from the back-end device, the instructions instructing the electronic device to dynamically generate a single user interface including interactive content from a plurality of related resources; Related resources are determined to be related to each other and to the context of the user by the back-end device, and features of the interactive dynamic content are extracted by the back-end device;
Dynamically generating the single user interface;
Displaying the single user interface to the user of the electronic device;
A non-transitory computer readable medium having:
(Supplementary Note 10) The step of obtaining the context data includes: position information from a position detection component of the electronic device, beacon data from an electronic beacon proximate to the electronic device, or a quick response (QR) scanned by the electronic device. The non-transitory computer readable medium of claim 9, comprising obtaining at least one of the codes.
(Supplementary note 11) The non-transitory computer-readable medium according to supplementary note 9, wherein the plurality of related resources are read from two or more third organizations.
(Supplementary note 12) The non-transitory computer-readable medium according to Supplementary note 9, wherein the plurality of related resources include one web service and one device connected to the Internet.
(Supplementary note 13) The non-transitory computer-readable medium according to Supplementary note 9, wherein the one device includes one of a vehicle, a point-of-sale terminal, an appliance, a surveillance camera, an electrical outlet, a thermostat, or a light. .
(Supplementary Note 14) The step of dynamically generating the single user interface includes:
Mapping each of the features extracted by the backend device to individual elements generated as part of the single user interface;
Incorporating the interactive content from each of the plurality of related resources into a common format and common visual scheme of the single user interface;
The non-transitory computer readable medium of claim 9, comprising:
(Supplementary Note 15)
Receiving the user interaction with an element of the single user interface to generate interaction data;
Transmitting the interaction data to the back-end device to be communicated to one of the plurality of related resources associated with the element of the single user interface;
The non-transitory computer-readable medium according to claim 9, further comprising:
(Supplementary Note 16) The back-end device determines that the plurality of related resources are related to each other and the context of the user;
Connect to the plurality of resources;
Building an application interface (API) model for each of the plurality of resources;
Classifying each of the plurality of resources into a class;
Extracting features from each of the plurality of resources;
Storing the feature and the class for each of the plurality of resources in a database;
Determining that two or more of the plurality of resources are related by a relationship;
Storing the relationship between the two or more of the plurality of resources in the database;
Generating a model interface that combines at least one of the features from each of the plurality of resources associated with each other;
Storing the model interface in the database;
Associating the model interface with the context of each of the plurality of resources including features in the model interface;
To extract the features of the interactive dynamic content,
The non-transitory computer-readable medium according to appendix 9.
(Supplementary Note 17) Back-end device,
An electronic device;
The electronic device comprises:
Display,
A processor;
A computer readable medium having instructions that, when executed by the processor, cause the processor to perform or control execution, the process comprising:
Obtaining context data describing a context of a user of the electronic device;
Transmitting the context data to the backend device;
Receiving instructions from the back-end device, the instructions instructing the electronic device to dynamically generate a single user interface including interactive content from a plurality of related resources; Related resources are determined by the backend device to be related to each other and to the user's context, and features of the interactive dynamic content are extracted by the backend device;
Dynamically generating the single user interface;
Displaying the single user interface on the display of the electronic device;
A computer-readable medium comprising:
Having a system.
(Supplementary note 18) The back-end device determines that the plurality of related resources are related to each other and to the context of the user;
Connect to the plurality of resources;
Building an application interface (API) model for each of the plurality of resources;
Classifying each of the plurality of resources into a class;
Extracting features from each of the plurality of resources;
Storing the feature and the class for each of the plurality of resources in a database;
Determining that two or more of the plurality of resources are related;
Storing a relationship between the two or more of the plurality of resources in the database;
Generating a model interface that combines at least one of the features from each of the plurality of resources associated with each other;
Storing the model interface in the database;
Associating the model interface with the context of each of the plurality of resources including features in the model interface;
To extract the features of the interactive dynamic content,
The system according to appendix 17.
(Supplementary note 19) The system according to supplementary note 17, wherein the electronic device further includes at least one of a sensor that detects a physical position of the electronic device and a scanner that scans a quick response (QR) code.
(Supplementary note 20) The system according to supplementary note 17, wherein the plurality of related resources are read from two or more third organizations.

100 system 110 electronic device 120 back-end device 130 network 140 beacon 150 QR code 160 resource

Claims (20)

Obtaining context data describing the context of the user of the electronic device;
Transmitting the context data to a back-end device;
Receiving instructions from the back-end device, the instructions instructing the electronic device to dynamically generate a single user interface including interactive content from a plurality of related resources; Related resources are determined to be related to each other and to the context of the user by the back-end device, and features of the interactive dynamic content are extracted by the back-end device;
Dynamically generating the single user interface;
Displaying the single user interface to the user of the electronic device;
Having a method.   Obtaining the context data comprises: location information from a location sensing component of the electronic device, beacon data from an electronic beacon proximate to the electronic device, or a quick response (QR) code scanned by the electronic device. The method of claim 1, comprising obtaining at least one.   The method of claim 1, wherein the plurality of related resources are read from two or more third institutions.   The method of claim 1, wherein the plurality of related resources comprises a web service and a device connected to the Internet.   The method of claim 4, wherein the one device comprises one of a vehicle, a point-of-sale terminal, an appliance, a surveillance camera, an electrical outlet, a thermostat, or a light. Dynamically generating the single user interface comprises:
Mapping each of the features extracted by the backend device to individual elements generated as part of the single user interface;
Incorporating the interactive content from each of the plurality of related resources into a common format and common visual scheme of the single user interface;
The method of claim 1, comprising: Receiving the user interaction with an element of the single user interface to generate interaction data;
Transmitting the interaction data to the back-end device to be communicated to one of the plurality of related resources associated with the element of the single user interface;
The method of claim 1 further comprising: The backend device determines that the plurality of related resources are related to each other and the context of the user;
Connect to the plurality of resources;
Building an application interface (API) model for each of the plurality of resources;
Classifying each of the plurality of resources into a class;
Extracting features from each of the plurality of resources;
Storing the feature and the class for each of the plurality of resources in a database;
Determining that two or more of the plurality of resources are related by a relationship;
Storing the relationship between the two or more of the plurality of resources in the database;
Generating a model interface that combines at least one of the features from each of the plurality of resources associated with each other;
Storing the model interface in the database;
Associating the model interface with the context of each of the plurality of resources including features in the model interface;
To extract the features of the interactive dynamic content,
The method of claim 1. A non-transitory computer readable medium having instructions configured to cause the processor to perform or control execution of one or more steps when executed by a processor, the steps comprising:
Obtaining context data describing the context of the user of the electronic device;
Transmitting the context data to a back-end device;
Receiving instructions from the back-end device, the instructions instructing the electronic device to dynamically generate a single user interface including interactive content from a plurality of related resources; Related resources are determined to be related to each other and to the context of the user by the back-end device, and features of the interactive dynamic content are extracted by the back-end device;
Dynamically generating the single user interface;
Displaying the single user interface to the user of the electronic device;
A non-transitory computer readable medium having:   Obtaining the context data comprises: location information from a location sensing component of the electronic device, beacon data from an electronic beacon proximate to the electronic device, or a quick response (QR) code scanned by the electronic device. The non-transitory computer readable medium of claim 9, comprising obtaining at least one.   The non-transitory computer-readable medium of claim 9, wherein the plurality of related resources are read from two or more third parties.   The non-transitory computer-readable medium of claim 9, wherein the plurality of related resources comprises a web service and a device connected to the Internet.   The non-transitory computer readable medium of claim 9, wherein the one device comprises one of a vehicle, a point of sale terminal, an appliance, a surveillance camera, an electrical outlet, a thermostat, or a light. Dynamically generating the single user interface comprises:
Mapping each of the features extracted by the backend device to individual elements generated as part of the single user interface;
Incorporating the interactive content from each of the plurality of related resources into a common format and common visual scheme of the single user interface;
The non-transitory computer readable medium of claim 9, comprising: The process includes
Receiving the user interaction with an element of the single user interface to generate interaction data;
Transmitting the interaction data to the back-end device to be communicated to one of the plurality of related resources associated with the element of the single user interface;
10. The non-transitory computer readable medium of claim 9, further comprising: The backend device determines that the plurality of related resources are related to each other and the context of the user;
Connect to the plurality of resources;
Building an application interface (API) model for each of the plurality of resources;
Classifying each of the plurality of resources into a class;
Extracting features from each of the plurality of resources;
Storing the feature and the class for each of the plurality of resources in a database;
Determining that two or more of the plurality of resources are related by a relationship;
Storing the relationship between the two or more of the plurality of resources in the database;
Generating a model interface that combines at least one of the features from each of the plurality of resources associated with each other;
Storing the model interface in the database;
Associating the model interface with the context of each of the plurality of resources including features in the model interface;
To extract the features of the interactive dynamic content,
The non-transitory computer readable medium of claim 9. A back-end device;
An electronic device;
The electronic device comprises:
Display,
A processor;
A computer readable medium having instructions that, when executed by the processor, cause the processor to perform or control execution, the process comprising:
Obtaining context data describing a context of a user of the electronic device;
Transmitting the context data to the backend device;
Receiving instructions from the back-end device, the instructions instructing the electronic device to dynamically generate a single user interface including interactive content from a plurality of related resources; Related resources are determined by the backend device to be related to each other and to the user's context, and features of the interactive dynamic content are extracted by the backend device;
Dynamically generating the single user interface;
Displaying the single user interface on the display of the electronic device;
A computer-readable medium comprising:
Having a system. The backend device determines that the plurality of related resources are related to each other and the context of the user;
Connect to the plurality of resources;
Building an application interface (API) model for each of the plurality of resources;
Classifying each of the plurality of resources into a class;
Extracting features from each of the plurality of resources;
Storing the feature and the class for each of the plurality of resources in a database;
Determining that two or more of the plurality of resources are related;
Storing a relationship between the two or more of the plurality of resources in the database;
Generating a model interface that combines at least one of the features from each of the plurality of resources associated with each other;
Storing the model interface in the database;
Associating the model interface with the context of each of the plurality of resources including features in the model interface;
To extract the features of the interactive dynamic content,
The system of claim 17.   The system of claim 17, wherein the electronic device further comprises at least one of a sensor that detects a physical position of the electronic device and a scanner that scans a quick response (QR) code.   The system of claim 17, wherein the plurality of related resources are read from two or more third institutions.

Images