WO2022051734A1

WO2022051734A1 - Service actions triggered by multiple applications

Info

Publication number: WO2022051734A1
Application number: PCT/US2021/056050
Authority: WO
Inventors: Xiaofeng Li; Yiwei ZHAO
Original assignee: Innopeak Technology, Inc.
Priority date: 2021-10-21
Filing date: 2021-10-21
Publication date: 2022-03-10

Abstract

A system may include a first user device operating a system service framework, multiple sources processor, and a first source app. The first user device may be configured to obtain, via the system service framework, a first system message from a first source service of the first source app, wherein the first system message includes a service invocation, and to obtain a second system message from a second source service of a second source app, wherein the second system message comprises one or more second data objects. The first user device may further be configured to parse, via the multiple source processor, the first system message and the second system message for data objects, and compose a composite system message, wherein the composite system message includes the service invocation from the one or more data objects and the data structure from the one or more second data objects.

Description

SERVICE ACTIONS TRIGGERED BY MULTIPLE APPLICATIONS

COPYRIGHT STATEMENT

[0001] A portion of the disclosure of this patent document contains material that is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever.

FIELD

[0002] The present disclosure relates, in general, to methods, systems, and apparatuses for implementing a scheme for triggering service actions in the context of mobile devices.

BACKGROUND

[0003] As capabilities and functions of mobile devices continue to expand, the ways in which users invoke functions has also expanded. Virtual assistants have become a ubiquitous tool, capable of executing user- specified functions or commands, responding to user queries, or otherwise performing functions in response to user inputs. A plethora of applications (also referred to as "apps") have functionality to act as a source of data and/or services invoked by another app or user, as well as providing desired functionalities that may invoke resources from another app and/or input from user. Some apps can perform actions automatically, such as a camera app recognizing text or other objects of an image.

[0004] Typically, however, functionality is limited to services and/or resources recognized by an app, and functions outside of this scope typically require user intervention to determine how a desired action is handled. As a result, a request or desired action that may seem intuitive to a user requesting the action may require a sequence of cumbersome steps to accomplish, or in some cases, may not be executed. [0005] Thus, methods, systems, and apparatuses for service actions triggered by multiple applications are provided.

SUMMARY

[0006] Novel tools and techniques for service actions triggered by multiple applications are provided.

[0007] A method may include obtaining, via a system service framework, a first system message from a first source service of a first source app, wherein the first system message comprises one or more first data objects generated by the first source service, wherein the one or more first data objects includes a service invocation. The method continues by obtaining, via the system service framework, a second system message from a second source service of a second source app, wherein the second system message comprises one or more second data objects generated by the second source service. The method further includes parsing, via a multiple source processor of the system service framework, the first system message and the second system message, wherein parsing the first system message further comprises determining a service identification of the service invocation, and wherein parsing the second system message further comprises determining a data structure of the one or more second data objects, wherein the data structure is an input parameter accepted by sink services associated with the service identification. The method may continue by composing, via the multiple source processor, a composite system message, wherein the composite system message includes the service invocation from the one or more data objects and the data structure from the one or more second data objects. The method may further include determining, via the system service framework, whether any sink services match the service identification determined from the service invocation, and in response to determining there is a matching sink service that matches the service identification, invoking, via the system service framework, the matching sink service, wherein invoking the matching sink service further includes launching a sink app providing the matching sink service, and providing the sink app with the composite intent.

[0008] An apparatus may include a non-transitory computer readable medium in communication with the processor, the non-transitory computer readable medium having encoded thereon a set of instructions executable by the processor to perform various functions. The set of instructions may be executed by the processor to obtain, via a system service framework, a first system message from a first source service of a first source app, wherein the first system message comprises one or more first data objects generated by the first source service, wherein the one or more first data objects includes a service invocation, and obtain, via the system service framework, a second system message from a second source service of a second source app, wherein the second system message comprises one or more second data objects generated by the second source service. The instructions may further be executable by the processor to parse, via a multiple source processor of the system service framework, the first system message and the second system message, wherein parsing the first system message further comprises determining a service identification of the service invocation, and wherein parsing the second system message further comprises determining a data structure of the one or more second data objects, wherein the data structure is an input parameter accepted by sink services associated with the service identification. The instructions may be executed by the processor to compose, via the multiple source processor, a composite system message, wherein the composite system message includes the service invocation from the one or more data objects and the data structure from the one or more second data objects. The instructions may further be executed by the processor to determine, via the system service framework, whether any sink services match the service identification determined from the service invocation, and in response to determining there is a matching sink service that matches the service identification, invoke, via the system service framework, the matching sink service, wherein invoking the matching sink service further includes launching a sink app providing the matching sink service, and providing the sink app with the composite intent. [0009] A system may include one or more user devices, the one or more user devices comprising a first user device, a system service framework operating on one of the one or more user devices, the system service framework further comprising a multiple sources processor configured to parse system messages from one or more source services and compose new system messages from data objects parsed from the system messages, one or more source apps operating on one of the one or more user devices, and a sink app operating on one of the one or more user devices. The first user device may further include a processor, and a non-transitory computer readable medium in communication with the processor, the non-transitory computer readable medium having encoded thereon a set of instructions executable by the processor to obtain, via the system service framework, a first system message from a first source service of a first source app of the one or more source apps, wherein the first system message comprises one or more first data objects generated by the first source service, wherein the one or more first data objects includes a service invocation, and obtain, via the system service framework, a second system message from a second source service of a second source app of the one or more source services, wherein the second system message comprises one or more second data objects generated by the second source service. The instructions may further be executed by the processor to parse, via the multiple source processor, the first system message and the second system message, wherein parsing the first system message further comprises determining a service identification of the service invocation, and wherein parsing the second system message further comprises determining a data structure of the one or more second data objects, wherein the data structure is an input parameter accepted by sink services associated with the service identification, and compose, via the multiple source processor, a composite system message, wherein the composite system message includes the service invocation from the one or more data objects and the data structure from the one or more second data objects. The instructions may further be executed by the processor to determine, via the system service framework, whether any sink services match the service identification determined from the service invocation, and in response to determining there is a matching sink service that matches the service identification, invoke, via the system service framework, the matching sink service, wherein invoking the matching sink service further includes launching a sink app providing the matching sink service, and providing the sink app with the composite intent.

[0010] These illustrative embodiments are mentioned not to limit or define the disclosure, but to provide examples to aid understanding thereof. Additional embodiments are discussed in the Detailed Description, and further description is provided therein.

BRIEF DESCRIPTION OF THE DRAWINGS

[0011] A further understanding of the nature and advantages of particular embodiments may be realized by reference to the remaining portions of the specification and the drawings, in which like reference numerals are used to refer to similar components. In some instances, a sub-label is associated with a reference numeral to denote one of multiple similar components. When reference is made to a reference numeral without specification to an existing sub-label, it is intended to refer to all such multiple similar components.

[0012] Fig. 1 is a schematic block diagram of a system for service actions triggered by multiple applications, in accordance with various embodiments;

[0013] Fig. 2A is a sequence diagram of a system for service actions triggered by multiple applications, in accordance with various embodiments;

[0014] Fig. 2B is a sequence diagram of a system for service actions triggered by multiple applications with a query interface, in accordance with various embodiments;

[0015] Fig. 3 is a schematic diagram of a system in which service actions are triggered by two applications, in accordance with various embodiments;

[0016] Fig. 4 is a sequence diagram of the system in which service actions are triggered by two applications, in accordance with various embodiments;

[0017] Fig. 5A is a flow diagram of a method for implementing service actions triggered by multiple applications, in accordance with various embodiments;

[0018] Fig. 5B is a flow diagram of a method for implementing service actions triggered by multiple applications, in accordance with various embodiments;

[0019] Fig. 6 is a flow diagram of a method for resolving multiple sink services for providing a service action triggered by multiple applications, in accordance with various embodiments;

[0020] Fig. 7 is a schematic block diagram of a computer system for providing service actions triggered by multiple applications, in accordance with various embodiments.

DETAILED DESCRIPTION OF CERTAIN EMBODIMENTS

[0021] Various embodiments provide tools and techniques for service actions triggered by multiple applications. In some embodiments, a method may include. [0022] In some examples, a method for triggering service actions by multiple applications is provided. A method may include obtaining, via a system service framework, a first system message from a first source service of a first source app, wherein the first system message comprises one or more first data objects generated by the first source service, wherein the one or more first data objects includes a service invocation. The method continues by obtaining, via the system service framework, a second system message from a second source service of a second source app, wherein the second system message comprises one or more second data objects generated by the second source service. The method further includes parsing, via a multiple source processor of the system service framework, the first system message and the second system message, wherein parsing the first system message further comprises determining a service identification of the service invocation, and wherein parsing the second system message further comprises determining a data structure of the one or more second data objects, wherein the data structure is an input parameter accepted by sink services associated with the service identification. The method may continue by composing, via the multiple source processor, a composite system message, wherein the composite system message includes the service invocation from the one or more data objects and the data structure from the one or more second data objects. The method may further include determining, via the system service framework, whether any sink services match the service identification determined from the service invocation, and in response to determining there is a matching sink service that matches the service identification, invoking, via the system service framework, the matching sink service, wherein invoking the matching sink service further includes launching a sink app providing the matching sink service, and providing the sink app with the composite intent.

[0023] In some examples, the first system message and second system message may be obtained by the system service framework during a sliding window. In some examples, the second system message may obtained during a time window that includes at least one of a duration of time before and a duration of time after the service invocation of the first system message is received.

[0024] In further examples, the method may further include registering, by the sink app, the service identification of the matching sink service and one or more input parameters accepted by the matching sink service, the one or more input parameters including at least one required input parameter, and in response to receiving the service invocation of the first system message, determining whether the one or more first data objects includes the at least one required input parameter. The second system message may be obtained in response to determining that the one or more first data objects does not include the at least one required input parameter.

[0025] In yet further examples, the method may include determining, via the system service framework, one or more required input parameters of the sink service associated with the service identification, determining, via the multiple sources processor, whether all of the one or more required input parameters have been received, wherein determining whether all of the one or more input parameters have been received further comprises determining whether the one or more first data objects and one or more second data objects includes data structures corresponding to each of the one or more required input parameters, and in response to determining that one or more of the one or more required input parameters are missing, invoking, via the multiple sources processor, a query interface of a top app, wherein the top app is the second source app, wherein invoking the query interface further comprises querying the top app for data structures corresponding to one or more missing input parameters of the one or more required input parameters. In some examples, the method may further include determining, via the multiple sources processor, whether the one or more missing input parameters are returned from the top app via the query interface. In response to determining that the one or more missing parameters are not returned from the top app, the method may continue by invoking a computer vision (CV) algorithm. Invoking the computer vision algorithm may further include capturing, via the CV algorithm, a screenshot of a user device currently used by the user; identifying, via the CV algorithm, objects in the screenshot, generating, via the CV algorithm, one or more third data objects corresponding to the objects identified in the screen shot, and obtaining, via the multiple sources processor, the one or more third data objects. The method may continue by determining, via the multiple sources processor, whether the one or more third data objects includes each of the one or more missing parameters.

[0026] In some examples, the method may further include, in response to determining that there is more than one matching sink service that matches the service identification, presenting a user with a list of two or more sink apps, each of the two or more sink apps respectively providing a respective matching sink service of the more than one matching sink service, and prompting, via the system service framework, the user to select one of the two or more sink apps to provide the matching sink service.

[0027] In some embodiments, an apparatus triggering service actions by multiple applications is provided. The apparatus may include a processor, and a non-transitory computer readable medium in communication with the processor, the non-transitory computer readable medium having encoded thereon a set of instructions executable by the processor to perform various functions. The set of instructions may be executed by the processor to obtain, via a system service framework, a first system message from a first source service of a first source app, wherein the first system message comprises one or more first data objects generated by the first source service, wherein the one or more first data objects includes a service invocation, and obtain, via the system service framework, a second system message from a second source service of a second source app, wherein the second system message comprises one or more second data objects generated by the second source service. The instructions may further be executable by the processor to parse, via a multiple source processor of the system service framework, the first system message and the second system message, wherein parsing the first system message further comprises determining a service identification of the service invocation, and wherein parsing the second system message further comprises determining a data structure of the one or more second data objects, wherein the data structure is an input parameter accepted by sink services associated with the service identification. The instructions may be executed by the processor to compose, via the multiple source processor, a composite system message, wherein the composite system message includes the service invocation from the one or more data objects and the data structure from the one or more second data objects. The instructions may further be executed by the processor to determine, via the system service framework, whether any sink services match the service identification determined from the service invocation, and in response to determining there is a matching sink service that matches the service identification, invoke, via the system service framework, the matching sink service, wherein invoking the matching sink service further includes launching a sink app providing the matching sink service, and providing the sink app with the composite intent. [0028] In some examples, the first system message and second system message are respectively a first input intent and second input intent. In further examples, the one or more first data objects and the one or more second data objects each comprise at least one of a service invocation or data structure. In some examples, the first system message and second system message are obtained by the system service framework during a sliding window.

[0029] In further examples, the set of instructions may further be executable by the processor to register, via the sink app, the service identification of the matching sink service and one or more input parameters accepted by the matching sink service with the system service framework, wherein the one or more input parameters includes at least one required input parameter, and in response to receiving the service invocation of the first system message, determine whether the one or more first data objects includes the at least one required input parameter. The second system message may thus be obtained in response to determining that the one or more first data objects does not include the at least one required input parameter.

[0030] In further examples, the set of instructions may further be executable by the processor to determine, via the system service framework, one or more required input parameters of the sink service associated with the service identification, determine, via the multiple sources processor, whether all of the one or more required input parameters have been received, wherein determining whether all of the one or more input parameters have been received further comprises determining whether the one or more first data objects and one or more second data objects includes data structures corresponding to each of the one or more required input parameters, and in response to determining that one or more of the one or more required input parameters are missing, invoke, via the multiple sources processor, a query interface of a top app, wherein the top app is the second source app, wherein invoking the query interface further comprises querying the top app for data structures corresponding to one or more missing input parameters of the one or more required input parameters. In some examples, the set of instructions may further be executable by the processor to determine, via the multiple sources processor, whether the one or more missing input parameters are returned from the top app via the query interface, and in response to determining that the one or more missing parameters are not returned from the top app, invoke a computer vision (CV) algorithm. Invoking the computer vision algorithm may further include capturing, via the CV algorithm, a screenshot of a user device currently used by the user, identifying, via the CV algorithm, objects in the screenshot, generating, via the CV algorithm, one or more third data objects corresponding to the objects identified in the screen shot, and obtaining, via the multiple sources processor, the one or more third data objects. The instructions may further be executed by the processor to determine, via the multiple sources processor, whether the one or more third data objects includes each of the one or more missing parameters.

[0031] In yet further examples, the set of instructions may further be executable by the processor to, in response to determining that there is more than one matching sink service that matches the service identification, present a user with a list of two or more sink apps, each of the two or more sink apps respectively providing a respective matching sink service of the more than one matching sink service, and prompt, via the system service framework, the user to select one of the two or more sink apps to provide the matching sink service.

[0032] In in further embodiments, a system for triggering service actions by multiple applications is provided. The system may include an one or more user devices, the one or more user devices comprising a first user device, a system service framework operating on one of the one or more user devices, the system service framework further comprising a multiple sources processor configured to parse system messages from one or more source services and compose new system messages from data objects parsed from the system messages, one or more source apps operating on one of the one or more user devices, and a sink app operating on one of the one or more user devices. The first user device may further include a processor, and a non-transitory computer readable medium in communication with the processor, the non-transitory computer readable medium having encoded thereon a set of instructions executable by the processor to obtain, via the system service framework, a first system message from a first source service of a first source app of the one or more source apps, wherein the first system message comprises one or more first data objects generated by the first source service, wherein the one or more first data objects includes a service invocation, and obtain, via the system service framework, a second system message from a second source service of a second source app of the one or more source services, wherein the second system message comprises one or more second data objects generated by the second source service. The instructions may further be executed by the processor to parse, via the multiple source processor, the first system message and the second system message, wherein parsing the first system message further comprises determining a service identification of the service invocation, and wherein parsing the second system message further comprises determining a data structure of the one or more second data objects, wherein the data structure is an input parameter accepted by sink services associated with the service identification, and compose, via the multiple source processor, a composite system message, wherein the composite system message includes the service invocation from the one or more data objects and the data structure from the one or more second data objects. The instructions may further be executed by the processor to determine, via the system service framework, whether any sink services match the service identification determined from the service invocation, and in response to determining there is a matching sink service that matches the service identification, invoke, via the system service framework, the matching sink service, wherein invoking the matching sink service further includes launching a sink app providing the matching sink service, and providing the sink app with the composite intent.

[0033] In some examples, the set of instructions may further be executable by the processor to register, via the sink app, the service identification of the matching sink service and one or more input parameters accepted by the matching sink service with the system service framework, wherein the one or more input parameters includes at least one required input parameter, and in response to receiving the service invocation of the first system message, determine whether the one or more first data objects includes the at least one required input parameter. The second system message may be obtained in response to determining that the one or more first data objects does not include the at least one required input parameter.

[0034] In some examples, the set of instructions may further be executable by the processor to determine, via the system service framework, one or more required input parameters of the sink service associated with the service identification, determine, via the multiple sources processor, whether all of the one or more required input parameters have been received, wherein determining whether all of the one or more input parameters have been received further comprises determining whether the one or more first data objects and one or more second data objects includes data structures corresponding to each of the one or more required input parameters, and in response to determining that one or more of the one or more required input parameters are missing, invoke, via the multiple sources processor, a query interface of the second source app. Invoking the query interface may further include querying the top app for data structures corresponding to one or more missing input parameters of the one or more required input parameters. In some examples, the instructions may further be executed by the processor to determine, via the multiple sources processor, whether the one or more missing input parameters are returned from the top app via the query interface. In response to determining that the one or more missing parameters are not returned from the top app, the instructions may be executed by the processor to invoke a computer vision (CV) algorithm, wherein invoking the computer vision algorithm further includes capturing, via the CV algorithm, a screenshot of a user device currently used by the user, identifying, via the CV algorithm, objects in the screenshot, generating, via the CV algorithm, one or more third data objects corresponding to the objects identified in the screen shot, and obtaining, via the multiple sources processor, the one or more third data objects. The instructions may further be executed by the processor to determine, via the multiple sources processor, whether the one or more third data objects includes each of the one or more missing parameters.

[0035] In yet further examples, the first user device may be configured to run the first source app and system service framework, and a second user device of the one or more user devices may be configured to run the second source app.

[0036] In the following description, for the purposes of explanation, numerous details are set forth to provide a thorough understanding of the described embodiments. It will be apparent to one skilled in the art, however, that other embodiments may be practiced without some of these details. In other instances, structures and devices are shown in block diagram form. Several embodiments are described herein, and while various features are ascribed to different embodiments, it should be appreciated that the features described with respect to one embodiment may be incorporated with other embodiments as well. By the same token, however, no single feature or features of any described embodiment should be considered essential to every embodiment of the invention, as other embodiments of the invention may omit such features. [0037] Unless otherwise indicated, all numbers used herein to express quantities, dimensions, and so forth used should be understood as being modified in all instances by the term "about." In this application, the use of the singular includes the plural unless specifically stated otherwise, and use of the terms "and" and "or" means "and/or" unless otherwise indicated. Moreover, the use of the term "including," as well as other forms, such as "includes" and "included," should be considered non-exclusive. Also, terms such as "element" or "component" encompass both elements and components comprising one unit and elements and components that comprise more than one unit, unless specifically stated otherwise.

[0038] The various embodiments include, without limitation, methods, systems, apparatuses, and/or software products. Merely by way of example, a method might comprise one or more procedures, any or all of which may be executed by a computer system. Correspondingly, an embodiment might provide a computer system configured with instructions to perform one or more procedures in accordance with methods provided by various other embodiments. Similarly, a computer program might comprise a set of instructions that are executable by a computer system (and/or a processor therein) to perform such operations. In many cases, such software programs are encoded on physical, tangible, and/or non-transitory computer readable media (such as, to name but a few examples, optical media, magnetic media, and/or the like).

[0039] Various embodiments described herein, embodying software products and computer-performed methods, represent tangible, concrete improvements to existing technological areas, including, without limitation, the invocation of sink services to provide service actions. Specifically, implementations of various embodiments provide for a way to trigger service actions using multiple applications. Conventional system service frameworks (e.g., Android® framework, iOS® framework), actions are not able to be triggered using multiple source service applications. Service actions are provided by an application providing a sink service (referred to herein as a "sink app"). An application providing a source service (referred to herein as a "source app") may invoke a service and/or provide a data structure to the system service framework, for example, by sending an intent. The system service framework may then determine an intent handler (e.g., a sink app for handling the intent) for the service invocation and/or data structure by searching an intent filter for matches to the intent. When a match is found, the system starts the matching activity, and passes the intent to the sink service. Any further resources required by the sink service are provided by the user or must be obtained by the sink service from another application having resource. Thus, service actions may be invoked from a single app, but is unable to be triggered using inputs from multiple applications to trigger the service action.

[0040] To the extent any abstract concepts are present in the various embodiments, those concepts can be implemented as described herein by devices, software, systems, and methods that involve novel functionality (e.g., steps or operations), such as the triggering of a service action by multiple applications, and more specifically composing intents based on inputs from the multiple applications.

[0041] Fig. 1 is a schematic block diagram of a system 100 for service actions triggered by multiple applications. The system 100 includes one or more source apps 105a- 105n (collectively "source apps 105"), including a first source app 105a through an nth source app 105n, a system service framework 110, multiple sources processor (MSP) 115, and sink app 120. It should be noted that the various components of the system 100 are schematically illustrated in Fig. 1, and that modifications to the various components and other arrangements of system 100 may be possible and in accordance with the various embodiments.

[0042] The one or more source apps 105a- 105n may be coupled to the system service framework 110. The system service framework 110 may further comprise the MSP 115. The system service framework 110 may further be coupled to the sink app 120.

[0043] In various embodiments, the one or more source apps 105a- 105n may include apps that provide a source service. Apps may include applications configured to run on mobile operating systems, such as, without limitation, Android®, iOS®, iPad OS®, and Chrome® OS. In some embodiments, the apps may include apps configured to run on traditional operating systems such as, without limitation, Windows®, MacOS®, Linux® among others. An app may provide various services, which may be divided into two kinds of services: source services and sink services. Thus, source apps 105 may be apps which provide source services. However, a given app may be a source app 105 in one context, and a sink app 120 in another context in which the app is providing a sink service. Thus, an app may include one or more services, which may further include one or more source services, one or more sink services, or both source and sink services (e.g., one or more source services and one or more sink services).

[0044] Source services may refer to services that generate information (e.g., data structures and/or service invocations). In some examples, source services may generate information that is processed by the system service framework 110 to be mapped to and feed to a sink service. In some embodiments, sink services may run in another application (e.g., a sink app 120), separate from the application providing the source service (e.g., a source app 105). Sink services, therefore, refer to services that receive and process information from source services. In some examples, source apps 105 may be configured to run multiple source services. Thus, each of the one or more source apps 105a- 105n may in turn be configured to run one or more source services. Each source service may be configured to generate information. In various embodiments, the information generated by the one or more source services of the source apps 105 may be provided to the system service framework 110.

[0045] In some embodiments, service invocations may include, for example, an indication of a function requested by the source service and/or an indication of a function provided by a sink service. In some examples, the indication of the function may be a service ID. Data structures may include data generated by the source service, as well as metadata associated with the data. In some examples, metadata may include an indication of data type using schemas as defined, for example, on https://schema.org. In some examples, service invocations may be generated along with data structures by the source service. In such instances, the data structure may be provided as an input parameter for the invoked service ID, and passed to the sink service. In some examples, a data structure may be generated without a service invocation. For example, a camera app may have multiple source services. Every data type generated by the camera app may be considered a respective source service. For example, data types of the camera app may include , "name card," "flower," "people," etc. Thus, when the camera app scans an object, it may generate data structures corresponding to one or more source services respectively.

[0046] As previously described, the system service framework 110 may include, for example, Android® framework, or other mobile OS framework. Traditionally, a single source service may generate information that is sent to the system service framework 110 to be handled. The information may include a service invocation or other data structure, which may then be processed by the system service framework to identify an appropriate sink service. Due to this arrangement, the system service framework is limited to information generated by the single source service from which to determine an appropriate sink service to handle the requested action, or determine that the requested action cannot be completed.

[0047] Therefore, to allow further functionality to handle information from multiple source services, the system service framework 110 may include MSP 115. In some embodiments, MSP 115 may be implemented, for example, in the Activity Manager Service (AMS), within the Android® framework. In some examples, the MSP may be configured to receive information from one or more source services 105a- 105n. The information received may include one or more data structures and/or a service invocation. The MSP 115 may be configured to compose, based on the information from the one or more source services, a data object called an intent. It is to be understood, as used herein, "intent" refers to the messaging object in the context of a system service framework 110, such as the Android® framework. Thus, the MSP 115 may thus be configured to parse an intent (referred to as an "input intent"), which includes all of the information generated by one or more source services 105a- 105n, to generate a system message (e.g., a newly composed intent, referred to as a "composite intent") for processing by an appropriate sink service. In some examples, an input intent may include information generated by a respective, individual source service of the one or more source services 105a- 105n. In further examples, the input intent may include information generated by multiple source services of the one or more source services 105a- 105n, which may include multiple source services. Thus, the MSP 115 may be configured to accept one or more input intents from the one or more source services 105a- 105n.

[0048] In some embodiments, the system 100 may implement a unified definition for service invocation and data structure. In some embodiments, every "service invocation" may be configured to specify a "service ID," which may have a specific function definition. If the function requires or otherwise accepts (e.g., optional) input parameters, the parameters may be defined as data structures. Thus, the required and/or accepted input parameters of a function may further be specified as data structures the function requires. In some embodiments, this may include indication of required and/or accepted data structures by data type, as previously described.

[0049] In various embodiments, a source app 105 and/or sink app 120 may be configured to declare what services are supported via one or more service IDs. Thus, each source app 105 and/or sink app 120 may declare the services it supports, and further whether the service is a source or a sink service. In some examples, the source app 105 and/or sink app 120 may declare input parameters as data structures that each service may generate (source) or accept (sink). In further examples, the source app 105 and/or sink app 120 may be configured to further indicate whether an input parameter is optional or required. In some embodiments, the source app 105 and/or sink app 120 may declare what data structures it can generate or accept separate from a service and/or service invocation. Specifically, the data structure may be declared by an application (e.g., source app 105 and/or sink app 120) without being associated with a service and/or service invocation.

[0050] In some embodiments, an application (e.g., source app 105 and/or sink app 120) may be configured to declare the service invocation and data structure in a standard format. Acceptable standard formats may include, without limitation, an extensible markup language (XML) or JavaScript Object Notation (JSON). In some embodiments, all sink services of an application may be combined into a single file as the service manifest. In some embodiments, a listing of all sink services of an application (e.g., the service manifest) may be registered to the system service framework 110 when the app is installed or loaded at runtime. In some examples, the source services that generate only data structures, and not service invocations, may be implemented as “broadcast” messages. Thus, by transmitting the broadcast message, the data structure-only source service may register with the system.

[0051] While source services are running, service invocations and data structures generated by the source service may be encapsulated into a system message (e.g., an "intent" object in Android®), and sent to the system services framework 110. Thus, the intent may be passed to the system service framework 110, for example, the AMS of the Android® framework. As previously described, the AMS may further include the MSP 115, which may be configured to process the intent message. Specifically, the MSP 115 may be configured to parse the intent to identify the service invocation(s) and data structure(s), and further compose a new system message (e.g., intent), and invoke the appropriate sink service to handle the composed intent. In some embodiments, the intent composed by the MSP 115 may be referred to as a composite intent, composed of the service invocation and data structures from an intent generated by the one or more source services.

[0052] In addition to declaring source and/or sink services, an app (source app 105 and/or sink app 120) may be configured to be implemented with a query interface. In turn, the MSP 115 may be configured to check a current status of an app through the query interface. The query interface may, therefore, comprise an interface through which the MSP 115 may query what object (e.g., a service ID and/or data structure) the app is currently processing. In some examples, the query interface may resemble the following: onQueryCurrentObjectWorkingOn() - return the “Data Structure” to describe the object the app is processing; formatted in XML or JSON.

[0053] In one example, a user may be editing a document in a first source app. The user may then say to a voice assistant app (e.g., second source app), "send the document to John by email." At this time, the system service framework 110 receives the user's input through the Voice Assistant app, and determines a service action that a user intends to be performed (also referred to as the "user's intention"), but must further determine what document should be sent. Thus, the voice input from the user may be packaged into an intent including a service invocation, the service ID identifying the action of sending an email, with input parameters corresponding to data structures for a recipient email address (e.g., John's email address), and a data structure for an attachment (e.g., the open document). According to various embodiments, the MSP 115 may concurrently invoke the query interface to ask current Top App (i.e., the app that user is using), what object the app is processing. Through the use of the query interface, the MSP 115 may determine the data structure of the document currently opened in the Top App and/or first source app, thus identifying the document being referred to by the user. The MSP 115 may then compose a new system message (e.g., a composite intent), which includes the service invocation and all relevant data structures as determined from the user input and through the query interface. The system service framework 110 may be configured to determine a matching sink service for handling the composite intent, and the composite intent may be provided to the sink service (e.g., a sink app associated with the identified sink service) for further handling.

[0054] In further embodiments, the MSP 115 may be configured to capture a screenshot of what is displayed on the user's screen (e.g., on the Top App), and use a computer vision (CV) algorithm to recognize objects currently on the user's screen. In one example, a user may be browsing pictures. While the user is viewing a picture of a watch, the user may ask a voice assistant app "what is the price of the watch?" At this time, the system service framework 110 receives the user's input through the Voice Assistant app, and may determine the user's intention, and corresponding intent generated by the respective source app (e.g., the Voice Assistant app). However, the system service framework 110 must still resolve the object to which the user is referring. For example, the Voice Assistant app may generate an intent having a service invocation, the service invocation having a service ID corresponding to a price check for an item, and further including a data structure corresponding to the item (in this case a watch). According to various embodiments, the MSP 115 may be configured to screenshot of the user's current screen, and use a CV algorithm to recognize the watch being displayed on the screen and the data structure corresponding to the watch displayed, concurrently and/or in response to receiving the intent from the source app. The MSP 115 may then compose a new system message (e.g., a composite intent), which includes the service invocation and all relevant data structures as determined from the user input and the CV algorithm. The system service framework 110 may be configured to determine a matching sink service for handling the composite intent, and the composite intent may be provided to the sink service (e.g., a sink app associated with the identified sink service) for further handling.

[0055] In various embodiments, an intent matching algorithm may be employed to determine how the system service framework 110, and specifically the MSP 115, handles an intent received from the one or more source apps 105a- 105n. In some embodiments, the MSP 115 may be configured to compare a service invocation (service ID) and data structure between the declarations of various source services and the declarations of various sink services. [0056] Based on the matching algorithm, the MSP 115 may be configured to take actions based on the results of the comparison. For example, in some embodiments, the matching algorithm may follow the table below:

Table 1: Matching Algorithm for Sink Service Invocation

[0057] In yet further embodiments, source services and sink services may be configured to run on different devices. Specifically, in some examples, the query interface may be a remote procedure call (RPC). The MSP 115 may thus obtain the result (e.g., identification of a data object) from the RPC. In some examples, a screenshot may be taken across devices, and a CV algorithm may run on separate devices. Thus, in some embodiments, the MSP 115 may obtain a necessary data structure and/or service invocation from one or more remote devices.

[0058] Implementations of the above embodiments enable a user device to directly trigger a service action using inputs from multiple services. For example, as discussed in previous embodiments, when the user points the camera to certain object and captures the object via a camera app, or a CV app identifies an object in a picture, and the user inputs their voice (or text) simultaneously to indicate a user intention, the MSP 115 may be able to receive one or more input intents from the one or more source apps 105. The MSP 115 may then parse the one or more input intents, and generate a new composite intent based on the service invocation and data structures in the one or more intents. For example, the user can point the camera to a business card containing an email address, and say to a voice assistant app "send an email." The MSP 115 may generate a composite intent, indicating the service invocation (service ID) and corresponding data structures to the service invocation. Thus, the email application may be launched with the automatically recognized email address on the business card added to a recipient field of the email message.

[0059] Accordingly, in various embodiments, the system service framework 110 may be configured to support multiple applications to trigger a service action through a semantic mapping technique. For example, a sink application may be configured to declare that it "accepts business card," "send email," etc. The camera may recognize a business card and generate "business card" information. When the user scans the business card and says "send an email" at the same time, the system service framework 110 may check which application is mapped to both "accept business card" and "send email" semantics. If only one sink app 120 is found, the sink app 120 may be launched directly into the "send email" service, with email address extracted from the "business card" source service and/or data structure generated by the camera. If more than one sink app is found, the compatible sink app(s) 120 may be presented for selection by the user.

[0060] In various embodiments, a unified semantics mapping framework is provided across different models and services. The unified semantics mapping may include semantics mapping for both source services and sink services. Source services may refer to those that generate information (data structure or service invocation), and the sink services refer to those that receive information. One application can have both source and sink services at the same time; and one application can have more than one source or sink services. For example, a camera app may have multiple source services and every data type it generates may be a source service, such as "business card," "flower," "people," etc. When the camera app scans an object, it may be configured to generate one or more recognized sources. A virtual assistant app may similarly generate one or more recognized data types when a user says a word or phrase. For example, when a user says "what is the price?" the voice app may generate a source service "query price" and/or "compute cost," etc. In further examples, an email app may have multiple sink services, corresponding to "send email," "search email," etc.

[0061] In some embodiments, all services may be defined in a standard format like XML or JSON, in a way that defines the service attributes, including an indication of whether it is a source or sink service; one or more input parameters (e.g., data structures) (also referred to as attribute / value pairs), and whether an attribute (e.g., input parameter) is optional, etc. As previously described, in some examples, all of the services of an application can be put together in a single file as the service manifest. In some examples, only the sink services may be declared in the service manifest. The service manifest may be registered to the system service framework 110 when the application is installed or loaded at runtime. When an application is running and generates source service(s), the data of every source service may be packaged in a respective data object in memory, e.g., as an Android® Intent. The one or more intents may be transmitted to the system service framework 110 and parsed by the system service framework 110. The system service framework 110 may identify matching sink services from the intents from one or multiple source services (such as voice and camera) according to its defined rules. If there are multiple matched sink services, the user may be prompted to select a sink app 120 to provide the matching sink service. The system service framework 110 may then launch the matched application to execute services to a point specified by the sink service attributes in its manifest. The attribute values from the source services (e.g., service invocations and/or data structures) may be passed to the sink service as input parameters.

[0062] Fig. 2A is a sequence diagram of a system 200A for service actions triggered by multiple applications, in accordance with various embodiments. The system 200 includes one or more source apps 205a-205n, including a first source app 205a and an nth source app 205n. The system 200 further includes system service framework 210 and sink app 220. It should be noted that the various components of the system 200A are schematically illustrated in Fig. 2A, and that modifications to the various components and other arrangements of system 200A may be possible and in accordance with the various embodiments.

[0063] According to various embodiments, the sequence diagram begins with a user action being provided to the source apps 205a-205n. In some examples, a user action may include various user inputs, such as a user command (e.g., a voice command), user query (e.g., a typed or spoken), a user selection, application launch, or any other suitable action by the user. In some embodiments, a user action may trigger a first source app 205a to generate a data object. As previously described, a source app 205a-205n may include an app providing a source service. Source services may generate a data object, which may include information such as a service invocation and/or data structure. In some embodiments, as previously described, the data object may be referred to as an intent. Thus, the data object generated by the first source app 205a may be encapsulated as a system message, known as an intent. In various embodiments, intents provided as an input to the system service framework 210 may be referred to as an input intent. Thus, the first source app 205a may be configured to transmit a first input intent to the system service framework 210. Similarly, a user action may trigger an nth source app to generate an nth input intent to the system service framework 210.

[0064] Alternatively, the system service framework 210 may be configured to collect or otherwise obtain input intents from one or more of the source apps 205. In some examples, the MSP 215 may be configured to collect data from source services (e.g., data structures which the source services may encapsulate into an input intents) responsive to receiving a service invocation. In some examples, the MSP 215 may collect data from source services (e.g., data structures encapsulated into input intents) using a sliding window algorithm, in which grouped messages (e.g., input intents) from various source services within a sliding window of time are collected. For example, a sliding window may collect all data objects input intents from all source services within a sliding window of 1 second. In some examples, the MSP 215 may collect data from all source services (e.g., in the form of input intents) within a time window centered at a time when an app is launched and/or a service invocation is received. For example, using a time window of Is, the MSP 215 may collect all input intents from all source services from 500ms before the app is launched and/or the service invocation is received by the system service framework 210, and 500ms after the app is launched and/or service invocation is received. In other embodiments, the time window may be centered differently, and implemented with different lengths of time that may be shorter or longer than Is (for example, a sliding window in the range of 500ms to 10 seconds). In yet further embodiments, the sliding window and/or time window may be adaptive in length and centering. For example, a different length time window may be used for different respective service IDs (or for different apps). Alternatively, the MSP 215 may utilize a query interface to query a top app and/or additional apps of the one or more source apps 205a-205n for data objects / input intents, as previously described and as will be described in greater detail below with respect to Fig. 2B. [0065] In various embodiments, each of the input intents from the source apps 205a- 205n may be parsed by the MSP 215 to identify the various data objects. Specifically, the MSP 215 may parse the input intents to extract a service invocation (e.g., a service ID), one or more accepted input parameters for the function associated with the service ID, and/or one or more data structures. In various embodiments, based on the data objects parsed from the input intents, the MSP 215 may further be configured to identify data objects from other input intents that may provide additional accepted input parameters for a given service ID. The MSP 215 may thus compose a new system message (e.g. intent) based on the service ID and data structures parsed from the input parameters. The composed message, in this case, the intent, may be referred to as a composite intent. The composite intent may thus comprise not only the service invocation, but also data structures corresponding to the one or more accepted input parameters available to the system framework 210 from the one or more source apps 205a-205n.

[0066] Accordingly, in some embodiments, the composite intent may be provided to the sink app 220, and the sink app 220 configured to execute the appropriate sink service with one or more required and/or accepted input parameters as obtained from the composite intent. In various embodiments, the sink app 220 may be configured to provide one or more sink services, which may further be declared and registered with the system service framework. Thus, if a sink app 220 is found providing a sink service matching the service ID of the service invocation, the system service framework 210 may be configured to invoke the sink service, for example, by launching the sink app. In some embodiments, the system service framework 210 may be configured to identify sink apps offering matching sink services by using, for example, an intent filter to filter for service IDs registered as sink services by various apps, including sink app 220. Accordingly, in some examples, more than one sink app may be identified offering a matching sink service. In some embodiments, if more than one sink app is identified by the system service framework 210, the user may be prompted to select a sink app 220 to provide the sink service. If no sink app is selected by the user, the system 200 may enter a fail state. Alternatively, a sink app 220 may be selected based on a previous selection by the user, or an association created by the user to use the sink app 220 to handle particular sink services. For example, in some embodiments, a user may indicate to the system service framework particular apps for handling particular service IDs. Thus, while apps may overlap in providing the same sink services, a user may have previously indicated a preference for a particular app to handle a particular sink service. In yet further embodiments, the system service framework 210 may be configured to automatically select a sink app 210 to handle the sink service based on one or more selection factors. The one or more selection factors may include, for example, historic usage, usage patterns, user-specific preferences, the user currently logged in to a session (e.g., for devices with multiple users), or other suitable criteria.

[0067] Fig. 2B is a sequence diagram of a system 200B for service actions triggered by multiple applications with a query interface, in accordance with various embodiments. Like Fig. 2A, the system 200B includes one or more source apps 205a-205n, including a first source app 205a and an nth source app 205n. The system 200 further includes system service framework 210 and sink app 220. It should be noted that the various components of the system 200B are schematically illustrated in Fig. 2B, and that modifications to the various components and other arrangements of system 200B may be possible and in accordance with the various embodiments.

[0068] According to various embodiments, the sequence diagram begins with a user action being provided to the first source app 205a. As previously described with respect to Fig. 2A, in one example, the user action may be a voice command. The user action may cause the first source app 205a to generate an input intent that includes a service invocation and data structures as indicated by the user action. The input intent may be transmitted by the first source app 205a to the system service framework 210. embodiments, as previously described, the data object may be referred to as an intent. Thus, the data object generated by the first source app 205a may be encapsulated as a system message, known as an intent.

[0069] In various embodiments, intents provided as an input to the system service framework 210 may be referred to as an input intent. Thus, the first source app 205a may be configured to transmit a first input intent to the system service framework 210. The MSP 215 may parse the first input intent for the service invocation and any data structures. In some embodiments, the MSP 215 may parse a service ID of the service invocation from the first input parameter. [0070] As previously described, sink services may be registered with the system service framework 210 to further specify any required and/or accepted input parameters. Thus, the MSP 215 may further determine any input parameters (e.g., data structures) that may not have been provided with the first input intent. In some embodiments, one or more of the source apps 205a-205n may be queried via a query interface for the one or more accepted input parameters. In some examples, a top app (e.g., an app that is currently used by the user) may be queried for any missing accepted input parameters. In yet further embodiments, as previously described, source services may be registered and/or broadcast to the system service framework 210. Thus, using an intent filter, the system service framework 210 may further identify apps providing corresponding source services capable of providing one or more accepted input parameters.

[0071] In some examples, the identified source apps may be queried, by the MSP 215, via the query interface for one or more accepted input parameters. As depicted in Fig. 2B, source app 205n may be queried, via the query interface, for one or more accepted input parameters. The source app 205n may be configured to generate a data object - in this example a data structure - corresponding to the queried input parameter. The source app 205n may encapsulate the data object as an nth input intent, and further transmit the nth input intent to the system service framework.

[0072] In various embodiments, the MSP 215 may further parse the nth input intent for the data structure(s) corresponding to the one or more accepted input parameters. In some embodiments, the one or more accepted input parameters from the nth source app 205n may be data structures / input parameters missing from the first input intent. Thus, the MSP 215 may compose a new composite intent, including the service invocation and all data structures obtained from the one or more source apps 205a-205n, and corresponding to input parameters of the service ID indicated in the service invocation.

[0073] In various embodiments, a sink service and corresponding sink app providing the sink service may be identified as previously described with respect to Fig. 2A, and the sink service invoked. The composite intent may thus be sent to the appropriate sink app 220 for further handling. [0074] Fig. 3 is a schematic diagram of a system 300 in which service actions are triggered by two applications, in accordance with various embodiments. Fig. 4 is a sequence diagram of the system 400 in which service actions are triggered by two applications, in accordance with various embodiments. With reference to both Figs. 3 & 4, the system 300, 400 includes a virtual assistant app 305a, 405a, camera app 305b, 405b, Android® framework 310, 410, MSP 315, 415, and an email app 320, 420. It should be noted that the various components of the system 300 are schematically illustrated in Figs. 3 & 4, and that modifications to the various components and other arrangements of system 300, 400 may be possible and in accordance with the various embodiments. Specifically, in Figs. 3 & 4, examples of source apps and sink apps are provided for purposes of explanation only. In other embodiments, other apps may comprise the source apps and sink apps, and further other services may comprise the source services and sink services.

[0075] In various embodiments, a user action may be provided to the source apps. For example, a voice input may be provided to a virtual assistant app 305a, 405a while a camera app 305b, 405b is open (e.g., launched by the user and running). Thus, the user actions may include launching of the camera app 305b and voice input provided to a virtual assistant app 305a, 405a. In one example, the voice input may be a wake word configured to launch the virtual assistant app (e.g., "Okay Google," "Hey Siri," "Alexa," etc.) in combination with a voice command to send an email, for example, "[wake word] send an email." The voice assistant app 305a, 405a may be configured to generate a data object based on the voice command. Specifically, the voice assistant app 305a, 405a may generate a service invocation to send an email, with a service ID corresponding to the sink service with a function of sending an email. In some examples, the virtual assistant app 305a, 405a may further be configured to generate additional data objects, and specifically data structures, based on the voice input of the user. For example, in some embodiments, the voice input may specify a name, an attachment (e.g., file, picture, video, etc.), a preferred email app, or other information. According to various embodiments, the data objects generated by the virtual assistant app 305a, 405a may be encapsulated in a system message (e.g., an intent) and transmitted to the Android® framework 310, 410. [0076] In various embodiments, the user may launch a camera app 305b, 405b. In some examples, the voice input to the virtual assistant app 305a, 405a may be provided while the camera app 305b, 405b is open, or concurrently with the launch of the camera app 305b, 405b. Like the voice assistant app 305a, 405a, the camera app 305b, 405b may be configured to generate one or more data objects as source services. Data objects may include, for example, data objects corresponding to objects in a captured picture and/or video, or alternatively, a live view of what the camera captures. In some embodiments, the camera app 305b, 405b may scan an object for text. In some examples, the object may include a business card, signage, documents, or the like. Accordingly, the camera app 305b may be configured to generate data structures corresponding to a business card, signage, or a document.

[0077] In one example, the camera app 305b, 405b may be pointed at a business card by a user. Concurrently, the user may provide a voice command to the virtual assistant app 305a, 405a to "send an email." The camera app 305b may, in some examples, generate a data structure for a "business card," further including data for a name, phone number, email address, and other suitable information. In other examples, the data structure may be configured to specify the type of text as well as the content of the text. Thus, the camera app 305b, 405b may generate data structures specifically for a name, email address, mailing address, phone numbers, uniform resource locator (URL), or other textual information identified in the business card. Data objects extracted from an image and/or live view of the business card may be encapsulated in a system message (in this example, an intent), and further transmitted to the Android® framework 310.

[0078] As previously described, the MSP 315, 415 may be configured to collect or otherwise obtain input intents from the virtual assistant app 305a, 405a and camera app 305b, 405b based on a sliding window algorithm, or alternatively, to obtain input intents over a time window from when a service invocation is received. Alternatively, the MSP 315, 415 may utilize a query interface to query a top app, in this example the camera app 305b, 405b, for data objects in response to receiving the service invocation.

[0079] In various embodiments, the MSP 315, 415 of the Android® framework 310, 410 may be configured to parse the input intents from the voice assistant app 305a, 405a and camera app 305b, 405b. From the voice assistant app 305a, 405a, the MSP 315, 415 may be configured to parse the service invocation for a service ID, and further any other data structures from the input parameter. As previously described, apps may declare what services are supported, whether a supported service is a source or a sink service, what input parameters or data structure services may accept or generate, respectively, and whether an input parameter is required or optional. The MSP 315, 415 may, thus, determine accepted input parameters for the service ID and further determine if any accepted input parameters are available from the parsed input intents. Alternatively, the MSP 315, 415 may utilize a query interface to query a top app, in this example the camera app 305b, 405b, for data objects in response to determining that the input intent(s) do not include required and/or optionally accepted input parameters.

[0080] The MSP 315, 415 may then compose a composite intent that includes the service invocation and data structures corresponding to any accepted input parameters from the parsed input intents. As previously described, the Android® framework 310, 410 may search for sink apps capable of providing sink services that match the service ID of the service invocation based, for example, on an intent filter. The Android® framework 310, 410 may then transmit the composite intent to the matching sink app, in this example, the email app 320, 420 for handling of the composite intent. Thus, the email app 320, 420 may be provided with a composite intent containing the service invocation for a corresponding sink service, and accepted input parameters from multiple source apps (e.g,. the virtual assistant app 305a, 405a and camera app 305b, 405b). In some embodiments, if more than one matching sink app is found, the Android® framework 310, 410 may prompt a user to select the sink app to handle the command / request. Alternatively, if no matching sink apps are found, the Android® framework 310, 410 may enter a fail state, for example, by indicating to the user that the action cannot be completed.

[0081] Fig. 5A is a flow diagram of a method for implementing service actions triggered by multiple applications, in accordance with various embodiments. The method 500A begins, at decision block 505, by determining, with a system service framework, whether there are any messages from any source services. As previously described, in various embodiments, source services may be configured to generate data objects, such as service invocations and/or data structures. The source services may encapsulate the data objects into system messages, called intents. Intents generated by source services are referred to herein as input intents. If it is determined there are no messages, the method 500A may continue, at block 510, by entering the system service framework into an idle state, where the system service framework is in standby until it is determined that a message is received from a source service.

[0082] If it is determined that a message is received form a source service, the method 500A may continue, at block 515 by obtaining source service messages. Specifically, the MSP may obtain data objects, encapsulated as input intents, from one or more relevant source services. As previously described, the system service framework may collect input intents using a sliding window algorithm or a time window based on when a service invocation is received and/or a source app is launched. In further embodiments, an input intent may be parsed for a service invocation, and input parameters accepted for the service ID may be identified, and data structures corresponding to the input parameters may be found. In some examples, the MSP may obtain the data structures from input intents already received by the system service framework.

[0083] Once the input intents (e.g., source service messages) are obtained, as previously described, in various embodiments, the MSP may parse each of the input intents for corresponding service invocations and data structures. The MSP may then compose a new message, a composite intent, based on the input intents. The method 500 may continue, at block 520 by matching a service invocation to a sink service. The system service framework may, in some examples, identify sink services that match the service ID of the service invocation. As previously described, in some embodiments, the system service framework may utilize an intent filter to identify sink services capable of supporting the service ID. In further examples, a sink app providing the sink service may further be identified.

[0084] At decision block 525, it is determined whether the service ID and data structure match. In some examples, the MSP and/or system service framework may determine whether a service ID and data structure of the composite intent match, and further whether the service ID of the service invocation matches the service ID of a sink service. As previously described, services may be declared and/or registered to the system service framework indicating accepted input parameters. Thus, the MSP may determine whether the input parameters of the service ID are matched to the data structures. In some examples, a service ID and data structures may be considered to match if all required input parameters are present as data structures in an intent. Thus, in various embodiments, it may be determined whether the service ID and data structures of the composite intent are matched. If it is determined that the service ID and data structures are matched, the method 500A may continue, at block 530, by invoking the sink service via the sink app. If it is determined that the service ID and data structure do not match, or if the service ID does not match the service ID of the sink service, the method 500A may continue, at decision block 535, by determining whether the data structures match, but the service ID of the service invocation is mismatched from the service ID of any sink services. Specifically, it may be determined whether data structures of the composite intent and match the input parameters of the invoked service ID of the composite intent. If it is determined that the data structures match the input parameters of the invoked service ID, it is determined whether any sink services match the service ID of the service invocation. A mismatch may be considered to be present when no sink services are identified to support the invoked service. Thus, the system service framework may, at block 540, provide a sink service list to the user. Thus, if no sink services are found with matching service IDs, the sink service list may include recommended sink apps for selection by a user, or alternatively no sink apps for selection. Accordingly, the method 500A may continue, at decision block 545, by determining if there is a user selection of a sink app. If no sink app is selected by the user, at block 550, a service fail state may be entered. If a user selection of a sink app is selected, at block 530, the sink service may be invoked.

[0085] If it is determined that a data match and service mismatch does not exist, the method 500A continues, at decision block 555, by determining whether a service match and data mismatch exists. If a service match and data mismatch does not exist, then it is a condition where both service and data are mismatched. Accordingly, at block 550, a service fail state may be entered by the system service framework. In some examples, a service fail state may include, without limitation, an indication to the user that a requested action cannot be completed, and a return to the idle state. If it is determined that a service match and data mismatch exist, the method 500A may continue, at block 560 of Fig 5B, to resolve the data mismatch. [0086] Fig. 5B is a flow diagram of a method 500B for implementing service actions triggered by multiple applications, in accordance with various embodiments. Continuing from block 555 of Fig. 5A, the method 500B begins, at block 560, by invoking, via the MSP and/or system service framework, a query interface. As previously described, a query interface of one or more source apps may be invoked. In some examples, a query interface of a top app may be invoked. In some embodiments, data objects of one or more source apps may be obtained via the query interface. The MSP, in some examples, may query one or more source apps for all source services, and corresponding data objects available via the source services. The data objects may include data structures corresponding to input parameters of a service ID of a service invocation. At decision block 565, the method 500B continues by determining whether a valid result is returned. In this case, a valid result may be a data structure matching a missing input parameter (required or optional) of the service ID. If a valid result is returned, the method 500B may continue, at decision block 570a, by determining whether a data match is present. Specifically, the MSP may be configured to determine whether all required input parameters for the service ID are now present. If it is determined that a data match is now present, the method 500B may continue, at block 530, by invoking the sink service via the sink app. If, at decision block 565, a valid result is not returned from the query interface query of the one or more source apps, in some embodiments, the method 500B may continue, at block 575, by capturing a screenshot of a user's screen and invoking a CV algorithm to identify objects on the user's screen. Similarly, if it is determined, at decision block 570a, that a data match is not present from the query of the top app and/or one or more source apps, the method 500B may continue by invoking, at block 575, a CV algorithm to detect objects on the user's screen.

[0087] In some embodiments, a CV algorithm may be invoked by the system service framework as a source service to identify and generate data objects, such as data structures. For example, a user may have open, on their screen, an image, text, or other information. The CV algorithm may thus be a source service configured to generate a data structure, which may be passed along as an intent. For example, the user may have on their screen an email address. The CV algorithm may identify the email address, and encapsulate the email address as a data structure to the MSP. Accordingly, at decision block 580, if a valid object is recognized on the user's screen (e.g., an object recognized as by / as a corresponding source service and data structure), then the data object may be encapsulated and transmitted as an input intent to the MSP. The method 500B may continue, at decision block 570b, by determining if a data match is now present. As previously described, this may include determining that data structures corresponding to all required input parameters have been obtained by the system service framework. If there is a data match, the method 500B continue, at block 530, by invoking the appropriate sink service matching the service ID of the service invocation.

[0088] If there is no data match, the method 500B may continue, at block 585, by prompting a user for the missing data to have a data match. Similarly, if the CV algorithm does not recognize a valid object, at decision block 580, the method 500B may continue, at block 585, by prompting a user to provide the missing data for a data match. In various embodiments, the user may be prompted for missing data. The missing data may be entered by the user manually or otherwise provided to the system service framework by the user. Thus, in some examples, the prompt to the user may itself be a source service configured to generate a data object corresponding to an input parameter, which may in turn be encapsulated as an intent.

[0089] The method 500B may continue, at decision block 570c, by then determining if a data match exists. If a data match still does not exist, the method 500B may, at block 550, cause the system service framework to enter a service fail state. If a data match exists. If a data exists, the method 500B may, at block 530, invoke the sink service via the sink app.

[0090] Fig. 6 is a flow diagram of a method 600 for resolving multiple sink services for providing a service action triggered by multiple applications, in accordance with various embodiments. As previously described, in some embodiments, the system service framework may utilize an intent filter to identify sink services capable of supporting the service ID. In further examples, a sink app providing the sink service may further be identified. The method 600 begins, at decision block 605, where it is determined whether more than one sink service matches the service ID of the service invocation. If more than one matching sink service is found, a user may be prompted to select a sink service from a list of matching sink services. In some examples, such as the example of Fig. 5A, if at decision block 525 a service ID and data structure match are found, but more than one matching sink service is identified, the system service framework may first present the user with a list of matching sink services and/or sink apps providing the matching sink services. Thus, the method 600 may further include, at block 610, prompting the user to select a sink service from the list of matching sink services and/or list of sink apps providing the matching sink services. Block 615 the method 600 may continue by invoking the sink service and/or the sink app associated with the sink service, as selected by the user. If at decision block 605, more than one matching sink service is not found (e.g., only one matching sink service is found), the sink service may be invoked, at block 615, by the system service framework.

[0091] The techniques and processes described above with respect to various embodiments may be performed by a computer system, such as a mobile device. Fig. 7 is a schematic block diagram of a computer system for providing service actions triggered by multiple applications, in accordance with various embodiments. Fig. 7 provides a schematic illustration of one embodiment of a computer system 700, such as the systems 100, 200A, 200B, 300, 400 or subsystems thereof, which may perform the methods provided by various other embodiments, as described herein. It should be noted that Fig. 7 only provides a generalized illustration of various components, of which one or more of each may be utilized as appropriate. Fig. 7, therefore, broadly illustrates how individual system elements may be implemented in a relatively separated or relatively more integrated manner.

[0092] The computer system 700 includes multiple hardware elements that may be electrically coupled via a bus 705 (or may otherwise be in communication, as appropriate). The hardware elements may include one or more processors 710, including, without limitation, one or more general-purpose processors and/or one or more special-purpose processors (such as microprocessors, digital signal processing chips, graphics acceleration processors, and microcontrollers); one or more input devices 715, which include, without limitation, a mouse, a keyboard, one or more sensors, and/or the like; and one or more output devices 720, which can include, without limitation, a display device, and/or the like.

[0093] The computer system 700 may further include (and/or be in communication with) one or more storage devices 725, which can comprise, without limitation, local and/or network accessible storage, and/or can include, without limitation, a disk drive, a drive array, an optical storage device, solid-state storage device such as a random-access memory ("RAM") and/or a read-only memory ("ROM"), which can be programmable, flash- updateable, and/or the like. Such storage devices may be configured to implement any appropriate data stores, including, without limitation, various file systems, database structures, and/or the like.

[0094] The computer system 700 might also include a communications subsystem 730, which may include, without limitation, a modem, a network card (wireless or wired), an IR communication device, a wireless communication device and/or chipset (such as a Bluetooth™ device, an 802.11 device, a WiFi device, a WiMax device, a WWAN device, a Z- Wave device, a ZigBee device, cellular communication facilities, etc.), and/or a low-power wireless device. The communications subsystem 730 may permit data to be exchanged with a network (such as the network described below, to name one example), with other computer or hardware systems, between data centers or different cloud platforms, and/or with any other devices described herein. In many embodiments, the computer system 700 further comprises a working memory 735, which can include a RAM or ROM device, as described above.

[0095] The computer system 700 also may comprise software elements, shown as being currently located within the working memory 735, including an operating system 740, device drivers, executable libraries, and/or other code, such as one or more application programs 745, which may comprise computer programs provided by various embodiments, and/or may be designed to implement methods, and/or configure systems, provided by other embodiments, as described herein. Merely by way of example, one or more procedures described with respect to the method(s) discussed above might be implemented as code and/or instructions executable by a computer (and/or a processor within a computer); in an aspect, then, such code and/or instructions can be used to configure and/or adapt a general purpose computer (or other device) to perform one or more operations in accordance with the described methods.

[0096] A set of these instructions and/or code might be encoded and/or stored on a non-transitory computer readable storage medium, such as the storage device(s) 725 described above. In some cases, the storage medium might be incorporated within a computer system, such as the system 700. In other embodiments, the storage medium might be separate from a computer system (i.e., a removable medium, such as a compact disc, etc.), and/or provided in an installation package, such that the storage medium can be used to program, configure, and/or adapt a general purpose computer with the instructions/code stored thereon. These instructions might take the form of executable code, which is executable by the computer system 700 and/or might take the form of source and/or installable code, which, upon compilation and/or installation on the computer system 700 (e.g., using any of a variety of generally available compilers, installation programs, compression/decompression utilities, etc.) then takes the form of executable code.

[0097] It will be apparent to those skilled in the art that substantial variations may be made in accordance with specific requirements. For example, customized hardware (such as programmable logic controllers, single board computers, FPGAs, ASICs, and SoCs) might also be used, and/or particular elements might be implemented in hardware, software (including portable software, such as applets, etc.), or both. Further, connection to other computing devices such as network input/output devices may be employed.

[0098] As mentioned above, in one aspect, some embodiments may employ a computer or hardware system (such as the computer system 700) to perform methods in accordance with various embodiments of the invention. According to a set of embodiments, some or all of the procedures of such methods are performed by the computer system 700 in response to processor 710 executing one or more sequences of one or more instructions (which might be incorporated into the operating system 740 and/or other code, such as an application program 745) contained in the working memory 735. Such instructions may be read into the working memory 735 from another computer readable medium, such as one or more of the storage device(s) 725. Merely by way of example, execution of the sequences of instructions contained in the working memory 735 might cause the processor(s) 710 to perform one or more procedures of the methods described herein.

[0099] The terms "machine readable medium" and "computer readable medium," as used herein, refer to any medium that participates in providing data that causes a machine to operate in a specific fashion. In an embodiment implemented using the computer system 700, various computer readable media might be involved in providing instructions/code to processor(s) 710 for execution and/or might be used to store and/or carry such instructions/code (e.g., as signals). In many implementations, a computer readable medium is a non-transitory, physical, and/or tangible storage medium. In some embodiments, a computer readable medium may take many forms, including, but not limited to, non-volatile media, volatile media, or the like. Non-volatile media includes, for example, optical and/or magnetic disks, such as the storage device(s) 725. Volatile media includes, without limitation, dynamic memory, such as the working memory 735. In some alternative embodiments, a computer readable medium may take the form of transmission media, which includes, without limitation, coaxial cables, copper wire and fiber optics, including the wires that comprise the bus 705, as well as the various components of the communication subsystem 730 (and/or the media by which the communications subsystem 730 provides communication with other devices). In an alternative set of embodiments, transmission media can also take the form of waves (including, without limitation, radio, acoustic, and/or light waves, such as those generated during radio-wave and infra-red data communications).

[0100] Common forms of physical and/or tangible computer readable media include, for example, a floppy disk, a flexible disk, a hard disk, magnetic tape, or any other magnetic medium, a CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, any other memory chip or cartridge, a carrier wave as described hereinafter, or any other medium from which a computer can read instructions and/or code.

[0101] Various forms of computer readable media may be involved in carrying one or more sequences of one or more instructions to the processor(s) 710 for execution. Merely by way of example, the instructions may initially be carried on a magnetic disk and/or optical disc of a remote computer. A remote computer might load the instructions into its dynamic memory and send the instructions as signals over a transmission medium to be received and/or executed by the computer system 700. These signals, which might be in the form of electromagnetic signals, acoustic signals, optical signals, and/or the like, are all examples of carrier waves on which instructions can be encoded, in accordance with various embodiments of the invention.

[0102] The communications subsystem 730 (and/or components thereof) generally receives the signals, and the bus 705 then might carry the signals (and/or the data, instructions, etc. carried by the signals) to the working memory 735, from which the processor(s) 710 retrieves and executes the instructions. The instructions received by the working memory 735 may optionally be stored on a storage device 725 either before or after execution by the processor(s) 710.

[0103] While some features and aspects have been described with respect to the embodiments, one skilled in the art will recognize that numerous modifications are possible. For example, the methods and processes described herein may be implemented using hardware components, software components, and/or any combination thereof. Further, while various methods and processes described herein may be described with respect to particular structural and/or functional components for ease of description, methods provided by various embodiments are not limited to any particular structural and/or functional architecture but instead can be implemented on any suitable hardware, firmware and/or software configuration. Similarly, while some functionality is ascribed to one or more system components, unless the context dictates otherwise, this functionality can be distributed among various other system components in accordance with the several embodiments.

[0104] Moreover, while the procedures of the methods and processes described herein are described in a particular order for ease of description, unless the context dictates otherwise, various procedures may be reordered, added, and/or omitted in accordance with various embodiments. Moreover, the procedures described with respect to one method or process may be incorporated within other described methods or processes; likewise, system components described according to a particular structural architecture and/or with respect to one system may be organized in alternative structural architectures and/or incorporated within other described systems. Hence, while various embodiments are described with or without some features for ease of description and to illustrate aspects of those embodiments, the various components and/or features described herein with respect to a particular embodiment can be substituted, added and/or subtracted from among other described embodiments, unless the context dictates otherwise. Consequently, although several embodiments are described above, it will be appreciated that the invention is intended to cover all modifications and equivalents within the scope of the following claims.

Claims

WHAT IS CLAIMED IS:

1. A method comprising: obtaining, via a system service framework, a first system message from a first source service of a first source app, wherein the first system message comprises one or more first data objects generated by the first source service, wherein the one or more first data objects includes a service invocation; obtaining, via the system service framework, a second system message from a second source service of a second source app, wherein the second system message comprises one or more second data objects generated by the second source service; parsing, via a multiple source processor of the system service framework, the first system message and the second system message, wherein parsing the first system message comprises determining a service identification of the service invocation, and wherein parsing the second system message comprises determining a data structure of the one or more second data objects, wherein the data structure is an input parameter accepted by sink services associated with the service identification; composing, via the multiple source processor, a composite system message, wherein the composite system message includes the service invocation from the one or more data objects and the data structure from the one or more second data objects; determining, via the system service framework, among the sink services associated with the service identification, a matching sink service that matches the service identification of the service invocation; and invoking, via the system service framework, the determined matching sink service, wherein invoking the determined matching sink service includes launching a sink app providing the matching sink service and providing the sink app with the composite system message.

2. The method of claim 1, wherein the first system message and second system message are obtained by the system service framework during a sliding window.

39

3. The method of claim 1, wherein the second system message is obtained during a time window that includes at least one of a duration of time before and a duration of time after the service invocation of the first system message is received.

4. The method of claim 1, further comprising: registering, by the sink app, the service identification of the matching sink service and one or more input parameters accepted by the matching sink service, the one or more input parameters including at least one required input parameter; and in response to receiving the service invocation of the first system message, determining whether the one or more first data objects includes the at least one required input parameter; wherein the second system message is obtained in response to determining that the one or more first data objects does not include the at least one required input parameter.

5. The method of claim 1, further comprising: determining, via the system service framework, one or more required input parameters of the sink service associated with the service identification; determining, via the multiple sources processor, whether all of the one or more required input parameters have been received, wherein determining whether all of the one or more input parameters have been received comprises determining whether the one or more first data objects and one or more second data objects includes data structures corresponding to each of the one or more required input parameters; and in response to determining that one or more of the one or more required input parameters are missing, invoking, via the multiple sources processor, a query interface of a top app, wherein the top app is the second source app, wherein invoking the query interface comprises querying the top app for data structures corresponding to one or more missing input parameters of the one or more required input parameters.

40

6. The method of claim 5, further comprising: determining, via the multiple sources processor, whether the one or more missing input parameters are returned from the top app via the query interface; in response to determining that the one or more missing parameters are not returned from the top app, invoking a computer vision (CV) algorithm, wherein invoking the computer vision algorithm further comprises: capturing, via the CV algorithm, a screenshot of a user device currently used by the user; identifying, via the CV algorithm, objects in the screenshot; generating, via the CV algorithm, one or more third data objects corresponding to the objects identified in the screen shot; and obtaining, via the multiple sources processor, the one or more third data objects; determining, via the multiple sources processor, whether the one or more third data objects includes each of the one or more missing parameters.

7. The method of claim 1 further comprising: in response to determining that more than one matching sink service among the sink services associated with the service identification matches the service identification, presenting a user with a list of two or more sink apps, each of the two or more sink apps respectively providing a respective matching sink service of the more than one determined matching sink service; and prompting, via the system service framework, the user to select one of the two or more sink apps to provide the respective matching sink service.

8. An apparatus, comprising: a non-transitory computer readable medium in communication with the processor, the non-transitory computer readable medium having encoded thereon a set of instructions executable by the processor to:

41 obtain, via a system service framework, a first system message from a first source service of a first source app, wherein the first system message comprises one or more first data objects generated by the first source service, wherein the one or more first data objects includes a service invocation; obtain, via the system service framework, a second system message from a second source service of a second source app, wherein the second system message comprises one or more second data objects generated by the second source service; parse, via a multiple source processor of the system service framework, the first system message and the second system message, wherein parsing the first system message comprises determining a service identification of the service invocation, and wherein parsing the second system message comprises determining a data structure of the one or more second data objects, wherein the data structure is an input parameter accepted by sink services associated with the service identification; compose, via the multiple source processor, a composite system message, wherein the composite system message includes the service invocation from the one or more data objects and the data structure from the one or more second data objects; determine, via the system service framework, among the sink services associated with the service identification, a matching sink service that matches the service identification of the service invocation; invoke, via the system service framework, the determined matching sink service, wherein invoking the determined matching sink service includes launching a sink app providing the matching sink service and providing the sink app with the composite system message.

9. The apparatus of claim 8, wherein the first system message and second system message are respectively a first input intent and second input intent.

10. The apparatus of claim 8, wherein the one or more first data objects and the one or more second data objects each comprise at least one of a service invocation or data structure.

11. The apparatus of claim 8, wherein the first system message and second system message are obtained by the system service framework during a sliding window.

12. The apparatus of claim 8, wherein the set of instructions is further executable by the processor to: register, via the sink app, the service identification of the matching sink service and one or more input parameters accepted by the matching sink service with the system service framework, wherein the one or more input parameters includes at least one required input parameter; and in response to receiving the service invocation of the first system message, determine whether the one or more first data objects includes the at least one required input parameter; wherein the second system message is obtained in response to determining that the one or more first data objects does not include the at least one required input parameter.

13. The apparatus of claim 8, wherein the set of instructions is further executable by the processor to: determine, via the system service framework, one or more required input parameters of the sink service associated with the service identification; determine, via the multiple sources processor, whether all of the one or more required input parameters have been received, wherein determining whether all of the one or more input parameters have been received comprises determining whether the one or more first data objects and one or more second data objects includes data structures corresponding to each of the one or more required input parameters; and in response to determining that one or more of the one or more required input parameters are missing, invoke, via the multiple sources processor, a query interface of a top app, wherein the top app is the second source app, wherein invoking the query interface comprises querying the top app for data structures corresponding to one or more missing input parameters of the one or more required input parameters.

14. The apparatus of claim 13, wherein the set of instructions is further executable by the processor to: determine, via the multiple sources processor, whether the one or more missing input parameters are returned from the top app via the query interface; in response to determining that the one or more missing parameters are not returned from the top app, invoke a computer vision (CV) algorithm, wherein invoking the computer vision algorithm further comprises: capturing, via the CV algorithm, a screenshot of a user device currently used by the user; identifying, via the CV algorithm, objects in the screenshot; generating, via the CV algorithm, one or more third data objects corresponding to the objects identified in the screen shot; obtaining, via the multiple sources processor, the one or more third data objects; and determine, via the multiple sources processor, whether the one or more third data objects includes each of the one or more missing parameters.

15. The apparatus of claim 8, wherein the set of instructions is further executable by the processor to: in response to determining that more than one matching sink service among the sink services associated with the service identification matches the service identification, present a user with a list of two or more sink apps, each of the two or more sink apps respectively providing a respective matching sink service of the more than one determined matching sink service; and prompt, via the system service framework, the user to select one of the two or more sink apps to provide the respective matching sink service.

44

16. A system for triggering service actions by multiple applications, the system comprising: one or more user devices, the one or more user devices comprising a first user device; a system service framework operating on one of the one or more user devices, the system service framework comprising a multiple sources processor configured to parse system messages from one or more source services and compose new system messages from data objects parsed from the system messages; one or more source apps operating on one of the one or more user devices; and a sink app operating on one of the one or more user devices; wherein the first user device comprises: a processor; and a non-transitory computer readable medium in communication with the processor, the non-transitory computer readable medium having encoded thereon a set of instructions executable by the processor to: obtain, via the system service framework, a first system message from a first source service of a first source app of the one or more source apps, wherein the first system message comprises one or more first data objects generated by the first source service, wherein the one or more first data objects includes a service invocation; obtain, via the system service framework, a second system message from a second source service of a second source app of the one or more source services, wherein the second system message comprises one or more second data objects generated by the second source service; parse, via the multiple source processor, the first system message and the second system message, wherein parsing the first system message comprises determining a service identification of the service invocation, and wherein parsing the second system message comprises determining a data structure of the one or more second

45 data objects, wherein the data structure is an input parameter accepted by sink services associated with the service identification; compose, via the multiple source processor, a composite system message, wherein the composite system message includes the service invocation from the one or more data objects and the data structure from the one or more second data objects; determine, via the system service framework, among the sink services associated with the service identification, a matching sink service that matches the service identification of the service invocation; invoke, via the system service framework, the determined matching sink service, wherein invoking the determined matching sink service includes launching a sink app providing the matching sink service and providing the sink app with the composite system message.

17. The system of claim 16, wherein the set of instructions is further executable by the processor to: register, via the sink app, the service identification of the matching sink service and one or more input parameters accepted by the matching sink service with the system service framework, wherein the one or more input parameters includes at least one required input parameter; and in response to receiving the service invocation of the first system message, determine whether the one or more first data objects includes the at least one required input parameter; wherein the second system message is obtained in response to determining that the one or more first data objects does not include the at least one required input parameter.

18. The system of claim 16, wherein the set of instructions is further executable by the processor to:

46 determine, via the system service framework, one or more required input parameters of the sink service associated with the service identification; determine, via the multiple sources processor, whether all of the one or more required input parameters have been received, wherein determining whether all of the one or more input parameters have been received comprises determining whether the one or more first data objects and one or more second data objects includes data structures corresponding to each of the one or more required input parameters; and in response to determining that one or more of the one or more required input parameters are missing, invoke, via the multiple sources processor, a query interface of the second source app, wherein invoking the query interface comprises querying the top app for data structures corresponding to one or more missing input parameters of the one or more required input parameters.

19. The system of claim 18, wherein the set of instructions is further executable by the processor to: determine, via the multiple sources processor, whether the one or more missing input parameters are returned from the top app via the query interface; in response to determining that the one or more missing parameters are not returned from the top app, invoke a computer vision (CV) algorithm, wherein invoking the computer vision algorithm further comprises: capturing, via the CV algorithm, a screenshot of a user device currently used by the user; identifying, via the CV algorithm, objects in the screenshot; generating, via the CV algorithm, one or more third data objects corresponding to the objects identified in the screen shot; obtaining, via the multiple sources processor, the one or more third data objects; and determine, via the multiple sources processor, whether the one or more third data objects includes each of the one or more missing parameters.

47

20. The system of claim 16, wherein the first user device is configured to run the first source app and system service framework, and a second user device of the one or more user devices is configured to run the second source app.

48