WO2022145951A1

WO2022145951A1 - Method and apparatus for providing a remote assistance

Info

Publication number: WO2022145951A1
Application number: PCT/KR2021/020021
Authority: WO
Inventors: Vipul Gupta; Abhinav Gaba; Akul TANEJA; Sushant Vobbilisetty; Nitesh Goyal; Sonam CHAWLA
Original assignee: Samsung Electronics Co., Ltd.
Priority date: 2020-12-29
Filing date: 2021-12-28
Publication date: 2022-07-07

Abstract

Provided is a second electronic device for providing remote assistance to a first electronic device, the second electronic device comprising a processor configured to receive a request for an assistance associated with a first functionality from the first electronic device, identify a second functionality corresponding to the first functionality at the second electronic device, perform at least one action associated with the second functionality, convert data related to the at least one action into encrypted data, and transmit the encrypted data to the first electronic device for performing the first functionality; and a memory stores the encrypted data.

Description

METHOD AND APPARATUS FOR PROVIDING A REMOTE ASSISTANCE

The present disclosure relates to a user interface, and more specifically related to a method and an electronic device for remote help to manage complexity of advanced features of the electronic device.

In general, an electronic device can access a large quantity of information and services through Internet and from other sources (e.g. Wireless Fidelity (Wi-Fi), Bluetooth, etc.). Nowadays functionality of the electronic device is rapidly increasing due to various applications (e.g. social media application, business application, etc.) which are available in the electronic device to perform various tasks and provide various types of information to a user of the electronic device. Further, each application has various features which may be difficult to learn or overwhelming for the user, difficult to master all the features. Also, the user may have difficulty even in discovering the features and/or information that is available on the electronic devices' applications. For example, a first user wants to book a cab using a cab application. But the first user may be unable to use the cab application because the first user may not be accustomed to using the cab application by oneself or may be unaware about the features of the cab application (e.g. adding a pickup location, a drop location, multiple payment options, etc.). Therefore, the first user requires help from a second user (remote user). As shown in FIG.1, the second user may need information (e.g. location information) from the first user to book the cab. But the first user does not know how to share the information with the second user. So, in such a situation, the second user requires a mechanism by which the second user can remotely help the first user.

Certain existing mechanisms provide remote support solutions for users of the electronic device but have drawbacks such the existing electronic device requires a stable internet access, same version of application/ software should be enabled at electronic device, etc., and cannot hinder exchange of huge files.

Certain existing mechanisms and electronic device provide a static solution within the electronic device such that a help file can be created and played later on a different or the same device to perform steps recorded in the help files. Where the help file has to be transferred to other user of the electronic device and following provided steps in the help file have to perform/ to complete the task (e.g. booking a cab). Transfer of the help file is data consuming, time-consuming task, and requires a high-speed network to avoid delays. In the static solution, the remote user of the electronic device must manually recognize issues of the other of the electronic device over some medium/application version and must have the same application version on which the other user of the electronic device is working. Thus, it is desired to at least provide a useful alternative for remote help.

The disclosure provides a first electronic device for receiving an assistance from a remote user, the first electronic device including a processor configured to transmit, to a second electronic device, a request for an assistance associated with a first functionality, receive encrypted data related to at least one action associated with a second functionality corresponding to the first functionality, and control to perform the first functionality based on the encrypted data; and a memory stores the encrypted data transmitted from the second electronic device.

The disclosure enables a user unable to perform certain functionality with a device to receive an assistance from a remote user.

This method and system are illustrated in the accompanying drawings, throughout which like reference letters indicate corresponding parts in the various figures. The embodiments herein will be better understood from the following description with reference to the drawings, in which:

FIG. 1 is an example scenario illustrating a user being unable to perform a certain task on his or her electronic device, according to an embodiment of the disclosure;

FIG. 2A illustrates a block diagram of a first electronic device which requests for remote assistance from a second electronic device, according to an embodiment of the disclosure;

FIG. 2B illustrates a block diagram of the second electronic device for providing the remote help to the first electronic device, according to an embodiment of the disclosure;

FIG. 3 is a flow diagram illustrating a method for providing the remote help to a user of the first electronic device by the second electronic device, according to an embodiment of the disclosure;

FIG. 4 is a diagram illustrating performing a companion paring between the first electronic device 100a and the second electronic device 100b, according to an embodiment of the disclosure;

FIG. 5 is a diagram illustrating activating the companion mode between the first electronic device and the second electronic device, according to an embodiment of the disclosure;

FIG. 6 is a diagram illustrating encoding the data related to complete the first functionality associated with the first electronic device, according to an embodiment of the disclosure;

FIGS. 7A, 7B, 7C and 7D are example scenarios illustrating a real-time remote assistance by a remote companion, according to an embodiment of the disclosure;

FIGS. 8A and 8B are flow diagrams illustrating mapping functional information and screen identity of each application in a companion mode, according to an embodiment of the disclosure; and

FIGS. 9A, 9B and 9C illustrate identifying actionable units and inserting them into keymap for extracting functional information, according to an embodiment of the disclosure.

Accordingly, the embodiment discloses a first electronic device for receiving an assistance from a remote user, the first electronic device includes a processor configured to transmit, to a second electronic device, a request for an assistance associated with a first functionality, receive encrypted data related to at least one action associated with a second functionality corresponding to the first functionality, and control to perform the first functionality based on the encrypted data; and a memory stores the encrypted data transmitted from the second electronic device.

Accordingly, the embodiment discloses an embodiment discloses a second electronic device for providing remote assistance to a first electronic device, the second electronic device including: a processor configured to receive a request for an assistance associated with a first functionality from the first electronic device, identify a second functionality corresponding to the first functionality at the second electronic device, perform at least one action associated with the second functionality, convert data related to the at least one action into encrypted data, and transmit the encrypted data to the first electronic device for performing the first functionality; and a memory stores the encrypted data.

Accordingly, embodiments herein disclose a method for providing a remote help to a user of a first electronic device by a remote companion of a second electronic device. The method includes receiving, by the second electronic device, a request from the first electronic device, where the request from the first electronic device indicates that the user of the first electronic device needs a help from the remote companion to complete a first functionality associated with the first electronic device. Further, the method includes identifying, by the second electronic device, a second functionality corresponding to the first functionality in the second electronic device. Further, the method includes performing, by the second electronic device, one action on the second functionality in the second electronic device. Further, the method includes encoding, by the second electronic device, a dataset related to the one action, where the data consists one of one touch point, one functional output, and action completed. Further, the method includes sending, by the second electronic device, the encoded dataset to the first electronic device. Further, the method includes performing, by the second electronic device, the first functionality in the first electronic device.

In an embodiment, the method further includes detecting, by the first electronic device, that one failure action occurs in the first electronic device using a machine learning (ML) model. Further, the method includes detecting, by the first electronic device, a resource availability of the first electronic device. Further, the method includes sending, by the first electronic device, the resource availability of the first electronic device to the second electronic device. Further, the method includes detecting, by the second electronic device, a resource availability of the second electronic device. Further, the method includes receiving, by the first electronic device, a resource availability of the second electronic device. Further, the method includes determining, by the first electronic device, an optimal medium. Further, the method includes activating, by the first electronic device, a companion mode in response to detecting that the one failure action occurs in the first electronic device, the resource availability of the first electronic device, the resource availability of the second electronic device, and the optimal medium. Further, the method includes sending, by the first electronic device, the request to the second electronic device to connect with the first electronic device.

In an embodiment, the optimal medium is determined based on the detected resource availability of the first electronic device and the received resource availability of the second electronic device.

In an embodiment, the method further includes sending, by the first electronic device, a companion request to the second electronic device, where the companion request includes a key containing unique identities of the first electronic device and the second electronic device to bind the first electronic device and the second electronic device in a companionship. Further, the method includes receiving, by the first electronic device, an acknowledgment of the companion request from the second electronic device. Further, the method includes extracting, by the first electronic device, functional information, and a screen identity of each application of the first electronic device. Further, the method includes extracting, by the second electronic device, functional information and a screen identity of each application of the second electronic device. Further, the method includes exchanging, by the first electronic device, the extracted functional information and the extracted screen identity of each application of the first electronic device with the second electronic device. Further, the method includes exchanging, by the second electronic device, the extracted functional information and the extracted screen identity of each application of the second electronic device with the first electronic device. Further, the method includes mapping, by the first electronic device and the second electronic device, the exchanged functional information and the exchanged screen identity and storing the mapped functional information and the mapped screen identity. Further, the method includes performing, by the first electronic device, a companion paring between the first electronic device and the second electronic device based on the mapping.

In an embodiment, the failure action includes a number of attempts to complete the first functionality, a number of failures when performing the first functionality, an entry time associated with the first functionality, and an existing time associated with the first functionality.

In an embodiment, the companion mode is activated through one of a gesture action performed on the first electronic device and by clicking on a quick panel of the first electronic device.

In an embodiment, the optimal medium includes one of a Wireless Fidelity (Wi-Fi), a Bluetooth, a Short Message Service (SMS), an Infrared (IR), and an internet data packet service.

In an embodiment, the method further includes determining, by the second electronic device, whether the extracted functional information and the extracted screen identity of each application of the first electronic device and corresponding functional information and the screen identity of each application of the second electronic device is same. Further, the method includes calculating, by the second electronic device, a delta information in response to determining that the extracted functional information and the extracted screen identity of each application of the first electronic device and corresponding functional information and the screen identity of each application of the second electronic device is not same, wherein the delta information is all mismatched actionable units at the first electronic device and the second electronic device, where the actionable units associates with a quadrangle shape area and type of action on the first electronic device and the second electronic device. Further, the method includes generating, by the second electronic device, a unique key for all the mismatched actionable units, where the unique key includes four corners for the quadrangle shape area and a name of task of the actionable unit. Further, the method includes mapping, by the second electronic device, the unique key to identify the second functionality corresponding to the first functionality in the second electronic device. Further, the method includes sending, by the second electronic device, the mapped unique key to the first electronic device.

In an embodiment, the method further includes initiating, by the first electronic device, chunking of the data set as per the optimal medium. Further, the method includes encrypting, by the first electronic device, the data set. Further, the method includes sending, by the first electronic device, the encrypted data set to the second electronic device using the optimal medium, where the optimal medium is dynamically changes based on the resource availability during the companionship. Further, the method includes receiving, by the second electronic device, the encrypted data set. Further, the method includes performing, by the second electronic device, the at least one action on the second electronic device. Further, the method includes encoding, by the second electronic device, the dataset related to the at least one action.

In an embodiment, the first functionality includes one of sharing an audio file, sharing a video file, sharing a document file, sharing a live location, saving a contact, booking a cab, ordering a food, sharing a blood pressure readings, sharing a diabetes reading, performing an online transaction, sharing a post on a social media, downloading an audio file, downloading a video file, downloading a document file, and online shopping.

In an embodiment, the functional information includes one of a plurality of buttons, a plurality of image views, a plurality of icons associated with each application.

Accordingly, the embodiments herein provide a system for providing the remote help to the user of the first electronic device by the remote companion of the second electronic device. The system includes receiving, by the second electronic device, the request from the first electronic device, where the request from the first electronic device indicates that the user of the first electronic device needs the help from the remote companion to complete the first functionality associated with the first electronic device. Further, the system includes identifying, by the second electronic device, a second functionality corresponding to the first functionality in the second electronic device. Further, the system includes performing, by the second electronic device, one action on the second functionality in the second electronic device. Further, the system includes encoding, by the second electronic device, a dataset related to the one action, where the data consists one of one touch point, one functional output, and action completed. Further, the system includes sending, by the second electronic device, the encoded dataset to the first electronic device. Further, the system includes performing, by the second electronic device, the first functionality in the first electronic device.

These and other aspects of the embodiments herein will be better appreciated and understood when considered in conjunction with the following description and the accompanying drawings. It should be understood, however, that the following descriptions, while indicating preferred embodiments and numerous specific details thereof, are given by way of illustration and not of limitation. Many changes and modifications may be made within the scope of the embodiments herein without departing from the spirit thereof, and the embodiments herein include all such modifications.

The embodiments herein and the various features and advantageous details thereof are explained more fully with reference to the non-limiting embodiments that are illustrated in the accompanying drawings and detailed in the following description. Descriptions of well-known components and processing techniques are omitted so as to not unnecessarily obscure the embodiments herein. Also, the various embodiments described herein are not necessarily mutually exclusive, as some embodiments can be combined with one or more other embodiments to form new embodiments. The term "or" as used herein, refers to a non-exclusive or, unless otherwise indicated. The examples used herein are intended merely to facilitate an understanding of ways in which the embodiments herein can be practiced and to further enable those skilled in the art to practice the embodiments herein. Accordingly, the examples should not be construed as limiting the scope of the embodiments herein.

As is traditional in the field, embodiments may be described and illustrated in terms of blocks which carry out a described function or functions. These blocks, which may be referred to herein as units or modules or the like, are physically implemented by analog or digital circuits such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, hardwired circuits, or the like, and may optionally be driven by firmware. The circuits may, for example, be embodied in one or more semiconductor chips, or on substrate supports such as printed circuit boards and the like. The circuits constituting a block may be implemented by dedicated hardware, or by a processor (e.g., one or more programmed microprocessors and associated circuitry), or by a combination of dedicated hardware to perform some functions of the block and a processor to perform other functions of the block. Each block of the embodiments may be physically separated into two or more interacting and discrete blocks without departing from the scope of the invention. Likewise, the blocks of the embodiments may be physically combined into more complex blocks without departing from the scope of the invention

The accompanying drawings are used to help easily understand various technical features and it should be understood that the embodiments presented herein are not limited by the accompanying drawings. As such, the present disclosure should be construed to extend to any alterations, equivalents and substitutes in addition to those which are particularly set out in the accompanying drawings. Although the terms first, second, etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are generally only used to distinguish one element from another.

In an embodiment, the processor is configured to extract functional information associated with the first functionality and screen identity corresponding to an application associated with the first functionality.

In an embodiment, the processor is configured to transmit the functional information associated with the first functionality and the screen identity corresponding to the application associated with the first functionality to the second electronic device, and wherein the encrypted data relating to the at least one action includes at least one selected from a group of an application ID corresponding to the at least one action, an identity of the second functionality, the screen identity of the second electronic device.

In an embodiment, the application associated with the first functionality is an application performing the first functionality based on the encrypted data.

In an embodiment, the processor is configured to connect with the second electronic device for transmitting the encrypted data in response to the request.

In an embodiment, the first electronic device further includes a display, and the processor is configured to display on the display, based on the encrypted data, a graphical instruction to perform an action of executing an application associated with the first functionality.

In an embodiment, the first electronic device further includes a display, the processor is configured to display on the display, based on the encrypted data, a textual instruction to perform an action of executing an application associated with the first functionality.

Accordingly, an embodiment discloses a second electronic device for providing remote assistance to a first electronic device, the second electronic device including: a processor configured to receive a request for an assistance associated with a first functionality from the first electronic device, identify a second functionality corresponding to the first functionality at the second electronic device, perform at least one action associated with the second functionality, convert data related to the at least one action into encrypted data, and transmit the encrypted data to the first electronic device for performing the first functionality; and a memory stores the encrypted data.

In an embodiment, the processor is configured to receive functional information associated with the first functionality and screen identity of the first electronic device associated with the first functionality, map the functional information and the screen identity of the first electronic device to functional information and screen identity of the second electronic device associated with the second functionality, and generate the data related to the at least one action based on the mapping, wherein the data related to the at least one action includes at least one selected from a group of an application ID corresponding to the at least one action, an identity of the second functionality, the screen identity of the second electronic device.

In an embodiment, the processor is configured to transmit, in response to the request for the assistance, a notification of an acknowledgement to the first electronic device, wherein the receiving of the functional information associated with the first functionality and the screen identity of the first electronic device includes receiving the functional information associated with the first functionality and the screen identity transmitted in response to the notification of the acknowledgement.

In an embodiment, the processor is configured to calculate delta information based on the screen identity of the first electronic device and the screen identity of the second electronic device, wherein the data related to the at least one action further includes the delta information representing coordinate differences between the screen identity of the first electronic device and the screen identity of the second electronic device.

In an embodiment, the screen identity of the first electronic device associated with the first functionality includes coordinate values of an icon of an application performing the first functionality.

In an embodiment, the screen identity of the second electronic device associated with the second functionality includes coordinate values of an icon of an application performing the second functionality.

In an embodiment, the request includes a key to bind the first electronic device and the second electronic device.

In an embodiment, the key includes at least one identity of the first electronic device and at least one identity of the second electronic device.

Accordingly, embodiments herein disclose a method for providing a remote help to a user of a first electronic device by a remote companion of a second electronic device. The method includes receiving, by the second electronic device, a request from the first electronic device, where the request from the first electronic device indicates that the user of the first electronic device needs help from the remote companion to complete a first functionality associated with the first electronic device. Further, the method includes identifying, by the second electronic device, a second functionality corresponding to the first functionality in the second electronic device. Further, the method includes performing, by the second electronic device, one action on the second functionality in the second electronic device. Further, the method includes encoding, by the second electronic device, a dataset related to the one action, where the data consists one of one touch point, one functional output, and action completed. Further, the method includes sending, by the second electronic device, the encoded dataset to the first electronic device. Further, the method includes performing, by the second electronic device, the first functionality in the first electronic device.

Accordingly, the embodiments herein provide a system for providing the remote help to the user of the first electronic device by the remote companion of the second electronic device. The system includes receiving, by the second electronic device, the request from the first electronic device, where the request from the first electronic device indicates that the user of the first electronic device needs the help from the remote companion to complete the first functionality associated with the first electronic device. Further, the system includes identifying, by the second electronic device, a second functionality corresponding to the first functionality in the second electronic device. Further, the system includes performing, by the second electronic device, one action on the second functionality in the second electronic device. Further, the system includes encoding, by the second electronic device, a data related to the one action, where the data consists one of one touch point, one functional output, and action completed. Further, the system includes sending, by the second electronic device, the encoded data to the first electronic device. Further, the system includes performing, by the second electronic device, the first functionality in the first electronic device.

The proposed method and electronic device creates a guidance mechanism which can help a certain group of users who are unable to perform a certain task (e.g. functionality) on his/her electronic device (digitally illiterate), effectively perform the certain task by intelligently selecting the remote companion and appropriately selecting a medium of companionship to complete the certain task in a real-time, where the medium of companionship is selected based on resources available (e.g. Random-access memory (RAM), battery, available network connectivity (Wi-Fi, Bluetooth (BT)), etc.) in both electronic devices. The resources available in both electronic devices provide a solution to an edge over all existing communication technologies (e.g. fourth generation (4G), fifth generation (5G), etc.).

The proposed method and electronic device automatically invokes the remote companion whenever the user is stuck on the certain task to compete in real-time over the medium, there is no such pre-task to understand the problem associated with the certain task. Further, the electronic device provides multiple options for connecting to the most suitable remote companion. Therefore, the certain task is completed more quickly and conveniently in absence of data. The proposed method and electronic device require at least one connection, may be Wi-Fi/BT for close proximity users, SMS for remotely placed users. The proposed solution can work event when there is no data connectivity.

The proposed method and electronic device provide communication between the user and the remote companion using smart chunks of data transfer which can also work on Short Message Service (SMS) and other low bandwidth slow connections without any lag. Where the smart chunks are encrypted by the electronic device.

The proposed method and electronic device dynamically switch the medium even during a current session based on various intelligent factors such as a change in resources of the electronic device, location of the electronic device, etc.

The proposed method and electronic device uses an optimal medium to transfer small chunks b/w electronic devices (e.g. companion device and host device), where the optimal medium is determined based on the detected resource availability at electronic devices and irrespective of hardware compatibility, the companion device can help the host device.

Referring now to the drawings, and more particularly to FIGS. 2A through 9C, there are shown preferred embodiments.

FIG. 2A illustrates a block diagram of a first electronic device which requests for a remote assistance from a second electronic device, according to an embodiment of the disclosure. Examples of the first electronic device 100a include, but not limited to a smart phone, a tablet computer, a Personal Digital Assistance (PDA), a desktop computer, an Internet of Things (IoT) device, a wearable device, etc.

In an embodiment, the first electronic device 100a includes a memory 110a, a processor 120a, a communicator 130a, a display 140a, an application repository 150a and an intelligent task controller 160a.

In an embodiment, the memory 110a stores a list of remote companions, a key containing unique identities of the second electronic device 100b, functional information, and corresponding screen identity of each application of the first electronic device 100a, and mapping exchanged functional information and the exchanged screen identity of the second electronic device 100b. The remote companion may be a user of the second electronic device 100b. The memory 110a stores instructions to be executed by the processor 120a. The memory 110a may include non-volatile storage elements. Examples of such non-volatile storage elements may include magnetic hard discs, optical discs, floppy discs, flash memories, or forms of electrically programmable memories (EPROM) or electrically erasable and programmable (EEPROM) memories. In addition, the memory 110a may, in some examples, be considered a non-transitory storage medium. The term "non-transitory" may indicate that the storage medium is not embodied in a carrier wave or a propagated signal. However, the term "non-transitory" should not be interpreted that the memory 110a is non-movable In certain examples, a non-transitory storage medium may store data that can, over time, change (e.g., in Random Access Memory (RAM) or cache). The memory 110a can be an internal storage unit or it can be an external storage unit of the first electronic device 100a, a cloud storage, or any other type of external storage.

The processor 120a accesses the memory 110a, the communicator 130a, the display 140a, the application repository 150a, and the intelligent task controller 160a. The processor 120a is configured to execute instructions stored in the memory 110a and to perform various processes. The processor 120a may include one or a plurality of processors, maybe a general-purpose processor, such as a central processing unit (CPU), an application processor (AP), or the like, a graphics-only processing unit such as a graphics processing unit (GPU), a visual processing unit (VPU), and/or an Artificial intelligence (AI) dedicated processor such as a neural processing unit (NPU). In an embodiment, the processor 120a may include the intelligent task controller 160a.

The communicator 130a is configured for communicating internally between internal hardware components (e.g. the memory 110a, the processor 120a, the display 140a, the application repository 150a, and intelligent task controller 160a) and with external devices (e.g. second electronic device 100b) via one or more networks (e.g. internet, Wi-Fi, Bluetooth, etc.). The communicator 130a includes an electronic circuit specific to a standard that enables wired or wireless communication.

The application repository 150a is implemented by processing circuitry such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, hardwired circuits, or the like, and may optionally be driven by firmware. The circuits may, for example, be embodied in one or more semiconductor chips, or on substrate supports such as printed circuit boards and the like.

The application repository 150a stores the functional information and corresponding screen identity for each application of the first electronic device 100a. The application repository 150a includes a plurality of applications (Application 150aa to Application 150an). Examples for the application repository 150a are, but not limited to, a media application, a web application, a video player application, a camera application, a game application, a business application, an education application, a lifestyle application, an entertainment application, a utility application, a travel application, etc. The memory 110a may include the application repository 150a as a part of its storage capacity.

The intelligent task controller 160a is implemented by processing circuitry such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, hardwired circuits, or the like, and may optionally be driven by firmware. The circuits may, for example, be embodied in one or more semiconductor chips, or on substrate supports such as printed circuit boards and the like.

In an embodiment, the intelligent task controller 160a is configured to send a companion request (e.g. companion agreement) to the second electronic device 100b. The companion request may include a key (e.g. Mobile Station International Subscriber Directory Number (MSISDN)) containing unique identities of the first electronic device 100a and the second electronic device 100b to bind the first electronic device 100a and the second electronic device 100b in a companionship. Further, the intelligent task controller 160a is configured to extract functional information (e.g. buttons, icons, applications corresponding to the buttons or the icons performing an action, etc.) and screen identity of an application related to the functionality to be performed on the first electronic device 100a by the user thereof and exchange or transmit the extracted functional information and the extracted screen identity of the application related to the functionality with the second electronic device 100b in response to receiving an acknowledgment of the companion request from the second electronic device 100b. Further, the intelligent task controller 160a is configured to receive extracted functional information, and the extracted screen identity of each application of the second electronic device 100b. Further, the intelligent task controller 160a is configured to map and store the exchanged functional information and the exchanged screen identity. In an embodiment, the screen identity of the application may be the screen identity - coordinate values - of the position of an icon corresponding to the application.

Further, the intelligent task controller 160a is configured to detect that one failure action occurs in the first electronic device 100a using a machine learning (ML) model (i.e. ML engine 163a). The failure action may include a number of attempts to complete the first functionality, a number of failures when performing the first functionality, an entry time associated with the first functionality, and an existing time associated with the first functionality. Further, the intelligent task controller 160a is configured to detect a resource availability (e.g. battery, random-access memory (RAM), read-only memory (ROM), etc.) of the first electronic device 100a. Further, the intelligent task controller 160a is configured to send the resource availability of the first electronic device 100a to the second electronic device 100b. Further, the intelligent task controller 160a is configured to receive a resource availability of the second electronic device 100b.

Further, the intelligent task controller 160a is configured to determine an optimal medium (e.g. Wi-Fi, SMS, Bluetooth, etc.). The optimal medium is determined based on the detected resource availability of the first electronic device 100a and the received resource availability of the second electronic device 100b. Further, the intelligent task controller 160a is configured to activate a companion mode in response to detecting that the one failure action occurs in the first electronic device 100a, the resource availability of the first electronic device 100a, the resource availability of the second electronic device 100b, and the optimal medium.

Further, the intelligent task controller 160a is configured to send the request to the second electronic device 100b to connect with the first electronic device 100a, where the request from the first electronic device 100a indicates that the user of the first electronic device 100a needs help from the remote companion to complete a first functionality (e.g. sharing an audio file, performing an online transaction, etc.) associated with the first electronic device 100a. Further, the intelligent task controller 160a is configured to receive an encoded dataset from the second electronic device 100b to complete the first functionality.

In an embodiment, the intelligent task controller 160a includes an authentication engine 161a, a layout inspector 162a, a ML engine 163a, and an encoder 164a.

The authentication engine 161a sends the companion request to the second electronic device 100b and receives the acknowledgment of the companion request from the second electronic device 100b. The authentication engine 161a is implemented by processing circuitry such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, hardwired circuits, or the like, and may optionally be driven by firmware. The circuits may, for example, be embodied in one or more semiconductor chips, or on substrate supports such as printed circuit boards and the like.

The layout inspector 162a extracts functional information and screen identity of each application of the first electronic device 100a and exchange extracted functional information and the extracted screen identity of each application of the first electronic device 100a with the second electronic device 100b in response to receiving an acknowledgment of the companion request from the second electronic device 100b. Further, the layout inspector 162a receives extracted functional information and the extracted screen identity of each application of the second electronic device 100b. Further, the layout inspector 162a maps and stores the exchanged functional information and the exchanged screen identity. The layout inspector 162a is implemented by processing circuitry such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, hardwired circuits, or the like, and may optionally be driven by firmware. The circuits may, for example, be embodied in one or more semiconductor chips, or on substrate supports such as printed circuit boards and the like.

The ML engine 163a detects that the one failure action occurs in the first electronic device 100a. Further, the ML engine 163a detects the resource availability of the first electronic device 100a. Further, the ML engine 163a sends the resource availability of the first electronic device 100a to the second electronic device 100b. Further, the ML engine 163a receives the resource availability of the second electronic device 100b. Further, the ML engine 163a determines the optimal medium. Further, the ML engine 163a activates the companion mode in response to detecting that the failure action occurs in the first electronic device 100a, the resource availability of the first electronic device 100a, the resource availability of the second electronic device 100b, and the optimal medium. The ML engine 163a is implemented by processing circuitry such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, hardwired circuits, or the like, and may optionally be driven by firmware. The circuits may, for example, be embodied in one or more semiconductor chips, or on substrate supports such as printed circuit boards and the like.

The encoder 164a receives the request to the second electronic device 100b to connect with the second electronic device 100b. Further, the encoder 164a receives the encoded data from the second electronic device 100b to complete the first functionality. The encoder 164a is implemented by processing circuitry such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, hardwired circuits, or the like, and may optionally be driven by firmware. The circuits may, for example, be embodied in one or more semiconductor chips, or on substrate supports such as printed circuit boards and the like.

Although FIG. 2A shows various hardware components of the first electronic device 100a but it is to be understood that other embodiments are not limited thereto. In other embodiments, the first electronic device 100a may include less or more number of components. Further, the labels or names of the components are used only for illustrative purpose and does not limit the scope of the invention. One or more components can be combined together to perform same or substantially similar function to receive the remote help from the second electronic device 100b to complete the first functionality.

FIG. 2B illustrates a block diagram of the second electronic device 100b for providing the remote help to the first electronic device 100a, according to an embodiment of the disclosure.

Examples of the first electronic device 100a include, but not limited to a smart phone, a tablet computer, a PDA, a desktop computer, an IoT device, a wearable device, etc.

In an embodiment, the second electronic device 100b includes a memory 110b, a processor 120b, a communicator 130b, a display 140b, an application repository 150b and an intelligent task controller 160b. In an embodiment, the intelligent task controller 160b may be implemented as a part of the processor 120b. In an embodiment, the application repository 150b may be implemented as a part of the memory 110b.

In an embodiment, the memory 110b stores a list of the first electronic device 100a, a key containing unique identities of the first electronic device 100a, functional information, and corresponding screen identity of each application of the second electronic device 100b, mapping exchanged functional information and the exchanged screen identity of the first electronic device 100a, and delta information (difference between the functional information of the first electronic device 100a and the functional information of the second electronic device 100b). The memory 110b stores instructions to be executed by the processor 120b. The memory 110b may include non-volatile storage elements. Examples of such non-volatile storage elements may include magnetic hard discs, optical discs, floppy discs, flash memories, or forms of electrically programmable memories (EPROM) or electrically erasable and programmable (EEPROM) memories. In addition, the memory 110b may, in some examples, be considered a non-transitory storage medium. The term "non-transitory" may indicate that the storage medium is not embodied in a carrier wave or a propagated signal. However, the term "non-transitory" should not be interpreted that the memory 110b is non-movable. In some examples, the memory 110b can be configured to store larger amounts of information than the memory. In certain examples, a non-transitory storage medium may store data that can, over time, change (e.g., in Random Access Memory (RAM) or cache). The memory 110b can be an internal storage unit or it can be an external storage unit of the second electronic device 100b, a cloud storage, or any other type of external storage.

The processor 120b communicates with the memory 110b, the communicator 130b, the display 140b, the application repository 150b, and the intelligent task controller 160b. The processor 120b is configured to execute instructions stored in the memory 110b and to perform various processes. The processor may include one or a plurality of processors, maybe a general-purpose processor, such as a central processing unit (CPU), an application processor (AP), or the like, a graphics-only processing unit such as a graphics processing unit (GPU), a visual processing unit (VPU), and/or an Artificial intelligence (AI) dedicated processor such as a neural processing unit (NPU).

The communicator 130b is configured for communicating internally between internal hardware components (e.g. the memory 110b, the processor 120b, the display 140b, the application repository 150b, and the intelligent task controller 160b) and with external devices (e.g. first electronic device 100a) via one or more networks (e.g. internet, Wi-Fi, Bluetooth, etc.). The communicator 130b includes an electronic circuit specific to a standard that enables wired or wireless communication.

The application repository 150b is implemented by processing circuitry such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, hardwired circuits, or the like, and may optionally be driven by firmware. The circuits may, for example, be embodied in one or more semiconductor chips, or on substrate supports such as printed circuit boards and the like.

The application repository 150b is configured to store the functional information and corresponding screen identity for each application of the second electronic device 100b. The application repository 150b includes a plurality of application 150ba to application 150bn. Examples for the application repository 150b are, but not limited, to a media application, a web application, a video player application, a camera application, a game application, a business application, an education application, a lifestyle application, an entertainment application, a utility application, a travel application, etc.

The intelligent task controller 160b is implemented by processing circuitry such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, hardwired circuits, or the like, and may optionally be driven by firmware. The circuits may, for example, be embodied in one or more semiconductor chips, or on substrate supports such as printed circuit boards and the like.

In an embodiment, the intelligent task controller 160b is configured to receive the companion request from the first electronic device 100a, where the companion request comprises the key containing unique identities of the first electronic device 100a and the second electronic device 100b to bind the first electronic device 100a and the second electronic device 100b in the companionship. Further, the intelligent task controller 160b is configured to extract functional information and screen identity of each application of the second electronic device 100b and exchange extracted functional information and the extracted screen identity of each application of the second electronic device 100b with the first electronic device 100a. Further, the intelligent task controller 160b is configured to receive extracted functional information and the extracted screen identity of each application of the first electronic device 100a. Further, the intelligent task controller 160b is configured to map between the exchanged functional information and the exchanged screen identity. The intelligent task controller 106b is configured to store the exchanged functional information and the exchanged screen identity. Further, the intelligent task controller 160b is configured to calculate the delta information.

Further, the intelligent task controller 160b is configured to detect the resource availability of the second electronic device 100b. Further, the intelligent task controller 160b is configured to send the resource availability of the second electronic device 100b to the first electronic device 100a. Further, the intelligent task controller 160b is configured to receive the resource availability of the first electronic device 100a.

Further, the intelligent task controller 160b is configured to determine the optimal medium. The optimal medium is determined based on the detected resource availability of the first electronic device 100a and the received resource availability of the second electronic device 100b. Further, the intelligent task controller 160b is configured to activate a companion mode in response to receiving the request from the first electronic device 100a, where the request from the first electronic device 100a indicates that the user of the first electronic device 100a needs the help from the remote companion to complete the first functionality associated with the first electronic device 100a. Further, the intelligent task controller 160b is configured to send the encrypted data to the first electronic device 100a to complete the first functionality.

In an embodiment, the intelligent task controller 160b includes an authentication engine 161b, a layout inspector 162b, a ML engine 163b, and an encoder 164b.

The authentication engine 161b receives the companion request from the first electronic device 100a and sends the acknowledgment of the companion request to the first electronic device 100a. The authentication engine 161b is implemented by processing circuitry such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, hardwired circuits, or the like, and may optionally be driven by firmware. The circuits may, for example, be embodied in one or more semiconductor chips, or on substrate supports such as printed circuit boards and the like.

The layout inspector 162b extracts functional information and screen identity of each application of the second electronic device 100b and exchange extracted functional information and the extracted screen identity of each application of the second electronic device 100b with the first electronic device 100a in response to sending the acknowledgment of the companion request to the first electronic device 100a. Further, the layout inspector 162b receives extracted functional information and the extracted screen identity of each application of the first electronic device 100a. Further, the layout inspector 162b maps and stores the functional information and the screen identity of the first electronic device 100a to the functional information and the screen identity of the second electronic device 100b. Further, the layout inspector 162b calculates the delta information. The delta information may be described in detail referring to FIG. 8A. The layout inspector 162b is implemented by processing circuitry such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, hardwired circuits, or the like, and may optionally be driven by firmware. The circuits may, for example, be embodied in one or more semiconductor chips, or on substrate supports such as printed circuit boards and the like.

The ML engine 163b detects the resource availability of the second electronic device 100b. Further, the ML engine 163b sends the resource availability of the second electronic device 100b to the first electronic device 100a. Further, the ML engine 163a receives the resource availability of the first electronic device 100a. Further, the ML engine 163b determines the optimal medium. Further, the ML engine 163b activates the companion mode in response to receiving the request from the first electronic device 100a, the companion mode is activated based on the resource availability of the first electronic device 100a, the resource availability of the second electronic device 100b, and the optimal medium. The ML engine 163b is implemented by processing circuitry such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, hardwired circuits, or the like, and may optionally be driven by firmware. The circuits may, for example, be embodied in one or more semiconductor chips, or on substrate supports such as printed circuit boards and the like.

The encoder 164b receives the request from the first electronic device 100a to connect with the second electronic device 100b. Further, the encoder 164b sends the encrypted data from the second electronic device 100b to the first electronic device 100a to complete the first functionality at the first electronic device 100a. The encoder 164b is implemented by processing circuitry such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, hardwired circuits, or the like, and may optionally be driven by firmware. The circuits may, for example, be embodied in one or more semiconductor chips, or on substrate supports such as printed circuit boards and the like.

Although FIG. 2B shows various hardware components of the second electronic device 100b but it is to be understood that other embodiments are not limited thereto. In other embodiments, the second electronic device 100b may include less or more number of components. Further, the labels or names of the components are used only for illustrative purpose and does not limit the scope of the invention. One or more components can be combined together to perform same or substantially similar function to provide the remote help to the first electronic device 100a to complete the first functionality.

At least one of the plurality of modules/ components of FIG. 2A and FIG. 2B may be implemented through an artificial intelligence (AI) model. A function associated with the AI model may be performed through corresponding memory 110a/110b and the processor 120a/120b.

The one or a plurality of processors controls the processing of the input data in accordance with a predefined operating rule or artificial intelligence (AI) model stored in the non-volatile memory and the volatile memory. The predefined operating rule or artificial intelligence model is provided through training or learning.

Here, being provided through learning means that, by applying a learning process to a plurality of learning data, a predefined operating rule or AI model of a desired characteristic is made. The learning may be performed in a device itself in which AI according to an embodiment is performed, and/o may be implemented through a separate server/system.

The AI model may consist of a plurality of neural network layers. Each layer has a plurality of weight values and performs a layer operation through calculation of a previous layer and an operation of a plurality of weights. Examples of neural networks include, but are not limited to, convolutional neural network (CNN), deep neural network (DNN), recurrent neural network (RNN), restricted Boltzmann Machine (RBM), deep belief network (DBN), bidirectional recurrent deep neural network (BRDNN), generative adversarial networks (GAN), and deep Q-networks.

The learning process is a method for training a predetermined target device (for example, a robot) using a plurality of learning data to cause, allow, or control the target device to make a determination or prediction. Examples of learning processes include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

FIG. 3 is a flow diagram S300 illustrating a method for providing the remote help to the user of the first electronic device 100a by the second electronic device 100b, according to an embodiment of the disclosure.

In operation S302, the method includes receiving a request for assistance from the first electronic device 100a, where the request from the first electronic device 100a indicates that the user of the first electronic device 100a needs the help from a companion using the second electronic device 100b to complete the first functionality associated with the first electronic device 100a. In operation S304, the method includes identifying the second functionality corresponding to the first functionality in the second electronic device 100b. In operation S306, the method includes performing an action associated with the second functionality in the second electronic device 100b. In operation S308, the method includes converting data related to the one action into encrypted data, where the data includes at least one of the touch point on an icon to accomplish the action, the functional output in response to the action, and screen identity corresponding to the icon to accomplish the action completed. In operation S310, the method includes sending the encrypted data to the first electronic device 100a. In operation S312, the method includes performing the first functionality in the first electronic device 100a. In an embodiment, the first functionality in the first electronic device 100a may correspond to the second functionality in the second electronic device 100b such as 'sending location to a third party'. However, the performing the second functionality may be different from the performing the first functionality in that the second functionality enables the first electronic device 100a to send not the location of the second electronic device 100b but the location of the first electronic device 100a to the third party.

The various actions, acts, blocks, steps, or the like in the flow diagram S300 may be performed in the order presented, in a different order or simultaneously. Further, in some embodiments, some of the actions, acts, blocks, steps, or the like may be omitted, added, modified, skipped, or the like without departing from the scope of the invention.

FIG. 4 is a diagram illustrating performing a companion paring between the first electronic device 100a and the second electronic device 100b, according to an embodiment of the disclosure.

At 401, the first electronic device 100a initiates a companion mode and creates the companion agreement to bind the first electronic device 100a and the second electronic device 100b in the companionship. At 402, the authentication engine 161a sends a companion request to the second electronic device 100b, where the companion request comprises the key (e.g. MSISDN). At 403, the second electronic device 100b receives the companion request from the first electronic device 100a and stores the companion agreement in the memory 110b. At 404, the authentication engine 161b sends the acknowledgment of the companion request to the first electronic device 100a.

At 405, the authentication engine 161a receives the acknowledgment of the companion request from the second electronic device 100b and stores the companion agreement in the memory 110a. At 406a, the layout inspector 162a extracts the functional information and the screen identity of each application of the first electronic device 100a. At 406b, the layout inspector 162b extracts the functional information and the screen identity of each application of the second electronic device 100b.

At 407a, the layout inspector 162a exchanges the extracted functional information and the extracted screen identity of each application of the first electronic device 100a with the second electronic device 100b. At 407b, the layout inspector 162b exchanges the extracted functional information and the extracted screen identity of each application of the second electronic device 100b with the first electronic device 100a. The layout inspector 162a interacts with a hierarchy viewer, the hierarchy viewer allows to debug and optimize user interface of the

electronic device

100a and 100b, provides a visual representation of the layout's view hierarchy, and interacts with an application framework.

At 408a, the layout inspector 162a maps the exchanged functional information and the exchanged screen identity of the first electronic device 100a to the exchanged functional information and the exchanged screen identity of the second electronic device 100b. At 409a, the first electronic device 100a completes the setup for the companion mode. At 408b, the layout inspector 162b also maps and stores the exchanged functional information and the exchanged screen identity of the second electronic device 100b to the exchanged functional information and the exchanged screen identity of the first electronic device 100a. The first electronic device 100a and the second electronic device 100b communicates with each other by network medium 400 (e.g. Wi-Fi, data packets, etc.) At 409b, the second electronic device 100b completes the setup for the companion mode.

FIG. 5 is a diagram illustrating activating the companion mode between the first electronic device and the second electronic device, according to an embodiment of the disclosure.

At 501, the ML engine 163a detects a failure action that occurs in the first electronic device 100a. At 502, the ML engine 163a detects the resource availability of the first electronic device 100a. At 503, the ML engine 163a sends the resource availability of the first electronic device 100a to the second electronic device 100b. At 504, the ML engine 163b stores the resource availability of the first electronic device 100a in the memory 110b. At 505, the ML engine 163b detects the resource availability of the second electronic device 100b. At 506, the ML engine 163b sends the resource availability of the second electronic device 100b to the first electronic device 100b. At 507, the ML engine 163a evaluates all the available resources. At 508, the ML engine 163a determines the optimal medium and activates the companion mode in response to detecting that the failure action occurred in the first electronic device 100a, the resource availability of the first electronic device 100a, the resource availability of the second electronic device 100b, and the optimal medium. At 509, the encoder 164a sends the request to the second electronic device 100b to connect with the first electronic device 100a, and the second electronic device 100b controls the first functionality of the first electronic device 100a.

FIG. 6 is a diagram illustrating encoding the data related to complete the first functionality associated with the first electronic device, according to an embodiment of the disclosure.

At 601, the encoder 164a initiates data chunking as per the optimal medium. At 692, the encoder 164a may initiate secure data sharing by encrypting data related to the first functionality. At 603, the encoder 164a exchanges the encrypted data with the second electronic device 100b. At 604, the encoder 164b receives the encrypted data and the remote companion of the second electronic device 100b. The second electronic device 100b performs actions corresponding to the first functionality on the second electronic device 100b. At 606, the encoder 164b fits response into data as per the received encrypted data. At 607, the second electronic device 100b exchanges the response data with the first electronic device 100a.

FIGS. 7A, 7B, 7C and 7D are example scenarios illustrating a real-time remote assistance by a remote companion, according to an embodiment of the disclosure.

Referring to FIG. 7A, at 701, the user of the first electronic device 100a (e.g. Jack) initiates the companion mode and creates the companion agreement to bind the first electronic device 100a and the second electronic device 100b in the companionship. At 702, the user of the first electronic device 100a adds remote companion(s) (e.g. a contact list) and sends the companion request to a second electronic device. In an embodiment, the companion request may include a companion agreement. At 703, the second electronic device 100b receives the companion request from the first electronic device 100a, and stores the companion agreement in the memory 110b, and sends an acknowledgment of the companion request to the first electronic device 100a. Further, the user of the first electronic device 100a receives the acknowledgment of the companion request from the second electronic device 100b and stores the companion agreement in the memory 110a. Further, corresponding functional information and the screen identity of each application of both

devices

100a and 100b will be extracted, exchanged, mapped, and stored. The application related to the screen identity may be associated with a first functionality which the user of the first electronic device would like to perform but fails to do at the first electronic device. In particular, the user of the first electronic device may fail to take a certain action appropriate to accomplish the first functionality.

Referring to FIG. 7B, at 704, the user of the first electronic device 100a chats with another user (e.g. Oliver), the user of the first electronic device 100a wants to share a location (i.e. first functionality/task) with Oliver. However, at 705, the user of the first electronic device 100a is unable to perform the task requested by another user and/or complete the first functionality requested by another user. At 706, the first electronic device 100a may manually or automatically activate the companion mode to receive an assistance with respect to the task or the requested first functionality from a remote user. The activated companion mode may cause to display a list of remote companions on a display of the first electronic device. At 707, the user of the first electronic device 100a selects the suitable remote companion and sends the request for an assistance to the selected companion using the second electronic device 100b. At 708, the first electronic device 100a receives, from the second electronic device 100b, a request of connecting with the first electronic device 100a. The connection between the first electronic device 100a and the second electronic device 100b enables the second electronic device 100b to control the first electronic device 100a for providing the requested assistance or to transmit encrypted data for providing the requested assistance.

Referring to FIG. 7C, at 709, the second electronic device 100b identifies a second functionality corresponding to the first functionality in the second electronic device 100b. At 710, the second electronic device 100b performs an action (e.g. an action such as clicking on the pin 7003) on the second functionality in the second electronic device 100b. Further, the second electronic device 100b converts the action into the encrypted data. In an embodiment, the encrypted data may include an application ID corresponding to the action, an identity of the second functionality, the screen identity of the second electronic device corresponding to a position of an icon of the application on the screen of the second electronic device 100b. The second electronic device 100b transmits the encrypted data to the first electronic device 100a via the established connection. The transmission of the encrypted data may be conducted via an optimal medium which is determined based on the resource availability of the first electronic device 100a and the second electronic device 100b. Further, the first electronic device 100a receives the encrypted data and provides a recommendation to perform an action (e.g. showing 'first, click here' at 709) corresponding to the received encrypted data on the first electronic device 100a. In an embodiment, the recommendation may be implemented as a graphical instruction (e.g. a graphical arrow pointing to the pin 7001 corresponding to the pin 7003 of the second electronic device 100b) or a combination of textual instructions ('first, click here') and the graphical instruction. In an embodiment, the action may execute the application associated with the first functionality at the first electronic device 100a.

Referring to FIG. 7D, at 712, the user of the second electronic device 100b performs another action (e.g. clicking on an icon of location) on the second functionality in the second electronic device 100b. Further, the second electronic device 100b converts the action into the encrypted data and transmits the encrypted data to the first electronic device 100a via the optimal medium. At 711, the first electronic device 100a receives the encrypted data and provides a recommendation of performing a corresponding to an action (e.g. showing 'second, click here') on the first electronic device 100a as per the received encrypted data and completes the first function in real-time. In an embodiment, a graphical indicator 7005 is shown in instruct the user of the first electronic device 100a to push an icon for the first functionality.

FIG. 8A and FIG. 8B are flow diagrams illustrating mapping functional information and screen identity of each application in a companion mode, according to an embodiment of the disclosure.

Referring to FIG. 8A, in operation S802a, the first electronic device 100a initiates the companion mode and creates the companion agreement to bind the first electronic device 100a and the second electronic device 100b in the companionship. In operation 804a, the layout inspector 162a extracts the functional information and the screen identity of each application of the first electronic device 100a, and the layout inspector 162b extracts the functional information and the screen identity of each application of the second electronic device 100b. An example of the functional information is {X1Y1, X2Y2, X3Y3, X4Y4, attach video} where the first four values indicate coordinates values of four corners on a display of the first electronic device 100a and the fifth value of 'attach video' indicates the functionality to be performed. In operation 806a, the layout inspector 162a sends the extracted functional information and the extracted screen identity of each application of the first electronic device 100a to the second electronic device 100b. In operation 808a, the layout inspector 162b determines whether the extracted functional information and the extracted screen identity of each application of the first electronic device 100a and corresponding functional information and the screen identity of each application of the second electronic device 100b is same or not.

In an embodiment, the functional information may include at least one functionality associated with at least one application installed in the first electronic device 100a and the screen identity of the first electronic device 100a may include positional values - coordinate values - associated with an icon of the at least one application on the screen of the first electronic device 100a. In another embodiment, the functional information may include all the functionalities each of which is associated with each of corresponding application installed in the first electronic device 100a and the screen identity of each region of the screen included in the first electronic device 100a. In an embodiment, the functional information may include application IDs corresponding to the functionalities. In an embodiment, the second electronic device 100b may map functionality of the first electronic device 100a to the functionality of the second electronic device 100b based on comparing the applications installed in each of the first electronic device 100a and the second electronic device 100b. In an embodiment, the second electronic device 100b may map the screen identity of the first electronic device 100a to the screen identity of the second electronic device 100b by comparing the screen sizes and coordinate values of each of the first electronic device 100a and the second electronic device 100b, respectively. By mapping the functional information and the screen identity of the first electronic device 100a to the functional information and the screen identity of the second electronic device 100b, a click of an icon corresponding to an application performing the functionality on the screen of the second electronic device 100b enables an action, e.g., a click of an icon corresponding to an application performing the same functionality on the screen of the first electronic device 100a in an embodiment of the disclosure by encrypting the data related to the clicking of the icon corresponding to the application performing the functionality on the screen of the second electronic device 100b. In an embodiment, the data related to the clicking of the icon corresponding to the application performing the functionality on the screen of the second electronic device 100b may include at least one of the application ID, the functionality ID, and the screen identity - coordinate values of four corners of the icon displayed on the screen of the second electronic device 100b.

In operation 810a, the layout inspector 162b calculates delta information in response to determining that the extracted functional information and the extracted screen identity of each application of the first electronic device 100a and corresponding functional information and the screen identity of each application of the second electronic device 100b is not same. For example, if the functional information of the first electronic device 100a is {X1Y1, X2Y2, X3Y3, X4Y4, attach video} and the functional information of the second electronic device 100b is {X10Y10, X20Y20, X30Y30, X40Y40, attach video}, then the delta information is {△X1Y1, △X2Y2, △X3Y3, △X4Y4, attach video} where △X1Y1 represents coordinate value representing the positional difference between X1Y1 and X10Y10, X2Y2 and X20Y20, X3Y3 and X30Y30, and X4Y4 and X40Y40. In operations 812a-814a, the process will repeat for each screen identity (ID) associated with both of the first electronic device 100a and the second electronic device 100b.

Referring to FIG. 8B, in operation S802b, the second electronic device 100b receives the request from the first electronic device 100a to perform the first functionality, where the request includes at least one of the MSISDN, the application ID, and the screen ID associated with the first functionality of the first electronic device 100a. In operation S804b, the layout inspector 162b may perform mapping the functionalities and calculating the delta information, if required, for the received request. In operation S806b, the layout inspector 162b detects an action (e.g. user input) at the second electronic device 100b to perform the first functionality and perform a corresponding action at the first electronic device 100a based on the mapped functionalities, and the calculated delta information. In operation S808b, the second electronic device 100b determines whether the output screen of the second electronic device is different from the output screen of the first electronic device.

Referring to FIG. 9A, an example scenario where the same android application package (APK) version is present at both

electronic devices

100a and 100b. At 901a, the user of the first electronic device 100a wants to enable a call forwarding setting but the user of the first electronic device 100a is unable to complete that task - the first functionality (i.e. call forwarding setting). So, the ML engine 163a activates the companion mode or provides a recommendation to activate the companion mode to complete the first functionality. Alternatively, the user of the first electronic device 100a may activate the companion mode. The encoder 164a sends the request and encrypted data to the second electronic device 100b to connect with the first electronic device 100a and the second electronic device 100b controls the first functionality of the first electronic device 100a.

At 902a, the encoder 164b receives the encrypted data and provides a recommendation to perform actions on the first electronic device 100a and fits response into data as per the received encrypted data and exchanges the response data with the first electronic device 100a. As an example of the response, encrypted touch co-ordinates ([12, 23; 12, 28; 18, 23, 18, 28, Other Call settings]) of the second electronic device 100b is transmitted to the first electronic device 100a via intelligently selected medium and a click is instructed to the user of the first electronic device 100a with the encrypted touch coordinates on the first electronic device 100a, where the second electronic device 100b performs the second function corresponding to the first function in the second electronic device 100b. Further, a resolution of both

electronic devices

100a and 100b is taken into consideration to extract the click point from the encrypted touch coordinates.

Referring to FIG. 9B, an example scenario where different APK versions are present at each of the first electronic device 100a and the second electronic device 100b. At 901b, consider a scenario where the user of the first electronic device 100a wants to make an online payment to another user but the user of the first electronic device 100a is unable to complete the first function (i.e. online payment). So, the first electronic device 100a sends the request to the remote companion of the second electronic device 100b. At 902b, it is detected that the feature/action (i.e. payment feature, audio feature) including an icon corresponding to the first function is not present in the second electronic device 100b. So the layout inspector 162b adds a unique ID, the function (i.e. payment feature, audio feature) and the icon corresponding to the first function based on the companion agreement. A touch on these features of the second electronic device 100b will effectively click on their mapped real features - first function of the payment feature - in the first electronic device 100a.

Referring to FIG. 9C, an example scenario where APK is not present at the second electronic device 100b is illustrated. The layout inspector 162b adds a unique ID and functionality for each functionality and screen ID while the companion agreement is made.

Adding the unique ID and functionality while the companion agreement is made mapping of functionality is performed may include the followings. Initial scanning operation divides each screen region into actionable units that perform a certain task. Each actionable unit associates with a quadrangle shape, circular area, or any bounded area in the screen and a type of action performed on both of the first electronic device 100a and the second electronic device 100b in which the APK is not present or different each other. In an embodiment, when the bounded area is touched, the associated certain task may be performed. Further, the initial scanning operation generates a unique key for all the mismatched actionable units at both of the

electronic devices

100a and 100b. The value of each key may be a list of five items (e.g. {X1Y1, X2Y2, X3Y3, X4Y4, attach video}) when the bounded area has the quadrangle shape. The first four items represent four corners for the quadrangle shape area and the fifth item represents a name of the task of the actionable unit. In an embodiment, when the bounded area has the circular shape, the value of each key may be a list of three items (e.g., { X1Y1, 10, attach video} where X1Y1 indicates the coordinate value of the center point of the circle, 10 indicates the radius and 'attach video' indicates a name of the task of the actionable unit). Once the mapped key is created, then the layout inspector 162b sends the mapped key to the second electronic device 100b via the optimal medium. Further, the first electronic device 100b compares the mapped key.

The embodiments disclosed herein can be implemented using at least one hardware device and performing network management functions to control the elements.

The foregoing description of the specific embodiments will so fully reveal the general nature of the embodiments herein that others can, by applying current knowledge, readily modify and/or adapt for various applications such specific embodiments without departing from the generic concept, and, therefore, such adaptations and modifications should and are intended to be comprehended within the meaning and range of equivalents of the disclosed embodiments. It is to be understood that the phraseology or terminology employed herein is for the purpose of description and not of limitation. Therefore, while the embodiments herein have been described in terms of preferred embodiments, those skilled in the art will recognize that the embodiments herein can be practiced with modification within the spirit and scope of the embodiments as described herein.

Claims

A first electronic device for receiving an assistance from a remote user, the first electronic device comprising:

a processor configured to

transmit, to a second electronic device, a request for an assistance associated with a first functionality,

receive encrypted data related to at least one action associated with a second functionality corresponding to the first functionality, and

control to perform the first functionality based on the encrypted data; and

a memory stores the encrypted data transmitted from the second electronic device.
The first electronic device of claim 1, wherein the processor is configured to extract functional information associated with the first functionality and screen identity corresponding to an application associated with the first functionality.
The first electronic device of claim 2, wherein the processor is configured to transmit the functional information associated with the first functionality and the screen identity corresponding to the application associated with the first functionality to the second electronic device, and wherein the encrypted data relating to the at least one action comprises at least one selected from a group of an application ID corresponding to the at least one action, an identity of the second functionality, the screen identity of the second electronic device.
The first electronic device of claim 2, wherein the application associated with the first functionality is an application performing the first functionality based on the encrypted data.
The first electronic device of claim 1, wherein the processor is configured to connect with the second electronic device for transmitting the encrypted data in response to the request.
The first electronic device of claim 1, further comprising a display,

wherein the processor is configured to display on the display, based on the encrypted data, a graphical instruction to perform an action of executing an application associated with the first functionality.
The first electronic device of claim 1, further comprising a display,

wherein the processor is configured to display on the display, based on the encrypted data, a textual instruction to perform an action of executing an application associated with the first functionality.
A second electronic device for providing remote assistance to a first electronic device, the second electronic device comprising:

a processor configured to:

receive a request for an assistance associated with a first functionality from the first electronic device,

identify a second functionality corresponding to the first functionality at the second electronic device,

perform at least one action associated with the second functionality,

convert data related to the at least one action into encrypted data, and

transmit the encrypted data to the first electronic device for performing the first functionality; and

a memory stores the encrypted data.
The second electronic device of claim 8, wherein the processor is configured to

receive functional information associated with the first functionality and screen identity of the first electronic device associated with the first functionality,

map the functional information and the screen identity of the first electronic device to functional information and screen identity of the second electronic device associated with the second functionality, and

generate the data related to the at least one action based on the mapping,

wherein the data related to the at least one action comprises at least one selected from a group of an application ID corresponding to the at least one action, an identity of the second functionality, the screen identity of the second electronic device.
The second electronic device of claim 9, wherein the processor is configured to

transmit, in response to the request for the assistance, a notification of an acknowledgement to the first electronic device,

wherein the receiving of the functional information associated with the first functionality and the screen identity of the first electronic device comprises receiving the functional information associated with the first functionality and the screen identity transmitted in response to the notification of the acknowledgement.
The second electronic device of claim 9, wherein the processor is configured to calculate delta information based on the screen identity of the first electronic device and the screen identity of the second electronic device,

wherein the data related to the at least one action further comprises the delta information representing coordinate differences between the screen identity of the first electronic device and the screen identity of the second electronic device.
The second electronic device of claim 9, wherein the screen identity of the first electronic device associated with the first functionality comprises coordinate values of an icon of an application performing the first functionality.
The second electronic device of claim 9, wherein the screen identity of the second electronic device associated with the second functionality comprises coordinate values of an icon of an application performing the second functionality.
The second electronic device of claim 8, wherein the request comprises a key to bind the first electronic device and the second electronic device.
The second electronic device of claim 14, wherein the key comprises at least one identity of the first electronic device and at least one identity of the second electronic device.