WO2021075919A1 - Procédé de distribution d'interface utilisateur pour une interaction multi-dispositifs - Google Patents

Procédé de distribution d'interface utilisateur pour une interaction multi-dispositifs Download PDF

Info

Publication number
WO2021075919A1
WO2021075919A1 PCT/KR2020/014187 KR2020014187W WO2021075919A1 WO 2021075919 A1 WO2021075919 A1 WO 2021075919A1 KR 2020014187 W KR2020014187 W KR 2020014187W WO 2021075919 A1 WO2021075919 A1 WO 2021075919A1
Authority
WO
WIPO (PCT)
Prior art keywords
host
guest
user interface
guest device
host device
Prior art date
Application number
PCT/KR2020/014187
Other languages
English (en)
Korean (ko)
Inventor
신인식
오상은
김아현
이선재
고용민
Original Assignee
한국과학기술원
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 한국과학기술원 filed Critical 한국과학기술원
Publication of WO2021075919A1 publication Critical patent/WO2021075919A1/fr

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04M—TELEPHONIC COMMUNICATION
    • H04M1/00—Substation equipment, e.g. for use by subscribers
    • H04M1/72—Mobile telephones; Cordless telephones, i.e. devices for establishing wireless links to base stations without route selection
    • H04M1/724—User interfaces specially adapted for cordless or mobile telephones
    • H04M1/72403—User interfaces specially adapted for cordless or mobile telephones with means for local support of applications that increase the functionality
    • H04M1/72409—User interfaces specially adapted for cordless or mobile telephones with means for local support of applications that increase the functionality by interfacing with external accessories
    • H04M1/72412—User interfaces specially adapted for cordless or mobile telephones with means for local support of applications that increase the functionality by interfacing with external accessories using two-way short-range wireless interfaces
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04M—TELEPHONIC COMMUNICATION
    • H04M1/00—Substation equipment, e.g. for use by subscribers
    • H04M1/72—Mobile telephones; Cordless telephones, i.e. devices for establishing wireless links to base stations without route selection
    • H04M1/724—User interfaces specially adapted for cordless or mobile telephones
    • H04M1/72469—User interfaces specially adapted for cordless or mobile telephones for operating the device by selecting functions from two or more displayed items, e.g. menus or icons
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04M—TELEPHONIC COMMUNICATION
    • H04M2201/00—Electronic components, circuits, software, systems or apparatus used in telephone systems
    • H04M2201/34—Microprocessors

Definitions

  • the present invention relates to a method of distributing a user interface for multi-device interaction, and more particularly, using screens of a plurality of smart devices as a user interface in a flexible and fast manner without modifying the source code of an application used in the device. It relates to a method of distributing a user interface to support this.
  • FIG. 1 is a diagram illustrating a situation in which a user interface (UI) of a chat window for chatting with other viewers and a broadcast video screen overlap while a user is watching a live broadcast through a live streaming app on a mobile device.
  • UI user interface
  • the present invention is invented to solve such a problem, and by separating a subtree including a target UI object of a host device and moving it to an external device, the target UI object is modified without modifying the app. It allows rendering in external devices, and interactions between UI objects of distributed devices and host app logic are performed in the form of local function calls, so local function calls within devices are performed by RPC between devices. Its purpose is to provide a way to maintain interactions between devices, also without modifying the app, by making it transparent to calls.
  • a program that serves as a platform of the host device (hereinafter referred to as'host platform')
  • a computer-implemented method for distributing a user interface (UI) to a second smart device (hereinafter referred to as a'guest device') by the operation of (a) Displaying a list of devices that are located and that can be used as guest devices and a list of applications (hereinafter referred to as "interaction applications") that can interact using multiple UI screens with other devices on the host device screen; (b) transmitting a package file of an application selected according to a user's input to a device selected according to a user's input (hereinafter referred to as a'guest device'); And, (c) during operation of one of the applications in step (b) (hereinafter referred to as'host app'), a UI object used by the
  • step (a0) a static code analysis in which candidates of UI objects used by the application are extracted and stored from the code of the interactive application installed in the host device.
  • the step of performing the static code analysis may be the entire application code, and the output of the static code analysis may be candidates of UI objects and UI resources used by the corresponding application.
  • the step (c) includes: (c11) receiving a user's input for selecting a UI element for sending to a guest device from among UIs used by the host app during operation of the host app; (c12) extracting a UI object actually used to execute the UI element selected in step (c11) from the UI object candidate group extracted in step (a0) in the guest device; (c13) transmitting the UI object extracted in step (c12) to the guest device; And (c14) storing the UI element selected in step (c11) in the platform layout file of the host platform.
  • the step (c) includes: (c21) parsing a layout file of a corresponding app or a platform layout file of a host platform (hereinafter collectively referred to as a “layout file”) during operation of the host app; (c22) extracting a UI object set in the layout file; And (c23) transmitting the UI object extracted in step (c22) to the guest device.
  • step (c) mirroring in which the entire screen including the UI object is sent to the guest device may be performed.
  • the host device and the guest device may use the same platform as a base platform, or, when different base platforms are used, the host app may be an app using a platform-independent code.
  • the information transmitted to the guest device selected in step (c) may further include information on whether to migrate or copy the UI to the guest device according to the user's selection, When it is selected as replication, after the step (c), (d) rendering the selected UI and displaying it on the screen of the host device may be further included.
  • step (c) update of the UI state displayed on the screen of the host device (hereinafter, the update of the UI state is referred to as'UI update')
  • the host app calls a function (local function call) for, (e) the function call is intercepted while the host app does not recognize it, and cross-device function calls (hereinafter referred to as ⁇ remote procedure call (RPC)) )') and transmitting the message to the guest device.
  • RPC remote procedure call
  • the It may further include receiving data input by the user through the UI and displaying it on the screen of the host device.
  • step (e), (e02) the UI of the host device is It may further include the step of receiving data input by the user through.
  • a step of performing UI update in the host device may be further included.
  • (h1) when receiving user input data from the guest device, displaying it through a UI of the host device screen may be further included.
  • the step (e) includes the steps of: (e11) calling a code conversion function set in advance when the function is called; (e12) converting the function into an RPC message by the code conversion function; And (e13) transmitting the RPC message to the guest device.
  • step (e) After the step (e), (f1) storing a return address to be returned after performing the RPC call; (f2) receiving a return value or an error code for the RPC call from the guest device; And (f3) returning to the return address and performing the next step.
  • a data value indicated by the reference type factor may be transmitted to the guest device together with the RPC message.
  • a'host device' when a first smart device (hereinafter referred to as a'host device') executes a specific application, the platform of the second smart device (hereinafter referred to as a'guest device')
  • a'guest platform' A computer-implemented method for performing the UI in the guest device by receiving a user interface (UI) from the host device by the operation of a program that performs a role (hereinafter referred to as a'guest platform') is , (a) receiving, from a host device, a package file of an application selected according to a user's input (hereinafter, referred to as a'host app'); (b) receiving a UI object from a host device; And (c) rendering the UI according to the UI object received in step (b) and displaying it on the screen of the guest device.
  • UI user interface
  • a'host app' a package file of an application selected according to a user's input
  • step (b1) the user's selection of UI elements on the received screen It may further include receiving an input, and the UI object received in step (c) may be a UI object for the UI element selected in step (b1).
  • the information received in step (b) may further include information on whether the UI is being migrated or copied to the guest device.
  • a cross-device function is called for updating the UI state displayed on the screen of the guest device (hereinafter, the update of the UI state is referred to as'UI update').
  • calls) hereinafter referred to as'RPC (remote procedure call)'
  • the step of performing a UI update on the UI of the host device screen may be further included.
  • the computer-implemented method for performing a user interface distributed from the host device includes (f) recognizing a local function call for executing a host app according to the data received in step (e).
  • a step may be further included, and in this case, the called function may be suppressed to a remote procedure call (RPC).
  • RPC remote procedure call
  • step (g) receiving a return value of the function called in step (f), received from the host device, and proceeding to the next step of executing the called function may be further included.
  • a computer-implemented method for performing a user interface distributed from the host device includes: (i) recognizing a local function call for executing a host app according to the data received in step (h).
  • a step may be further included, and in this case, the called function may be suppressed to a remote procedure call (RPC).
  • RPC remote procedure call
  • step (j1) When the UI has been migrated to the guest device, (j1) receiving data input by a user through the UI of the guest device; And, (j2) when a function call for executing a host app is recognized according to the data input in step (j1), it is intercepted and converted into a remote procedure call (RPC) to the host device. It may further include the step of transmitting.
  • RPC remote procedure call
  • the step (j2) includes: (j21) when the function is called, a code conversion function set in advance is called; (j22) converting the function into an RPC message by the code conversion function; And (j23) transmitting the RPC message to the host device.
  • step (f) or after the step (i) by receiving the return value of the function called in the step (f) or in the step (i), received from the host device, and then executing the called function It may further include a step of proceeding to the step.
  • a first smart device that performs a function of distributing a user interface (UI) to a second smart device (hereinafter referred to as a'guest device')
  • the device includes at least one processor; And at least one memory for storing a computer-executable instruction, wherein the computer-executable instruction stored in the at least one memory is (a) located in a short distance to a guest device by the at least one processor.
  • step (b) Displaying a list of devices that can be used and a list of applications that can interact using multiple UI screens with other devices (hereinafter referred to as "interaction applications") on a host device screen; (b) transmitting a package file of an application selected according to a user's input to a device selected according to a user's input (hereinafter referred to as a'guest device'); And, (c) during operation of one of the applications in step (b) (hereinafter referred to as'host app'), a UI object used by the host app is selected in step (b). Allow the step of sending to the device to be executed.
  • a user interface is received from the host device, and the UI is
  • the second smart device to perform includes at least one processor; And at least one memory for storing a computer-executable instruction, wherein the computer-executable instruction stored in the at least one memory is provided by the at least one processor, (a) from a host device, a user input Receiving a package file of the selected application (hereinafter referred to as'host app') according to the method; (b) receiving a UI object from a host device; And (c) rendering the UI according to the UI object received in step (b) and displaying it on the screen of the guest device.
  • a'host device' when a first smart device (hereinafter referred to as a'host device') executes a specific application, the second smart device (hereinafter referred to as a'guest device') is used.
  • a computer program hereinafter referred to as a'host platform'
  • a'host platform' stored in a non-transitory storage medium operating on the host device in order to distribute a user interface (UI) is stored in a non-transitory storage medium, and by a processor,
  • UI user interface
  • Display on the host device screen a list of devices that are located in a short distance and can be used as guest devices, and applications that can interact using multiple UI screens with other devices (hereinafter referred to as'interactive applications').
  • step of doing (b) transmitting a package file of an application selected according to a user's input to a device selected according to a user's input (hereinafter referred to as a'guest device'); And, (c) during operation of one of the applications in step (b) (hereinafter referred to as'host app'), a UI object used by the host app is selected in step (b). It includes an instruction to cause the step of sending to the device to be executed.
  • a second smart device when a first smart device (hereinafter referred to as a'host device') executes a specific application, a second smart device receives a user interface (UI) from the host device.
  • a computer program hereinafter referred to as a'guest platform'
  • a non-transitory storage medium hereinafter referred to as'guest platform'
  • 'guest divece' is a non-transitory storage medium.
  • a host app a package file of an application selected according to a user's input
  • a UI object from a host device
  • the target UI object can be rendered on an external device without modifying the app. Also, since the interaction between the UI objects of the distributed device and the host app logic is in the form of local function calls, it is possible to transparently convert local function calls within devices into RPC calls between devices. , Also has the effect of providing a way to maintain interactions between devices without modifying the app.
  • FIG. 1 is a diagram illustrating a situation in which a user interface (UI) of a chat window for chatting with other viewers and a broadcast video screen overlap while a user is watching a live broadcast through a live streaming app on a mobile device.
  • UI user interface
  • FIG. 2 is a diagram showing a situation in which a chat window UI is distributed to another device (guest device) by using a user interface distribution method according to the present invention.
  • FIG. 3 is a diagram showing a UI structure of Android, which is an embodiment of a basic platform to which the present invention is applied.
  • FIG. 4 is a flow chart showing the overall process of performing the user interface distribution method according to the present invention.
  • FIG. 5 is a schematic diagram for explaining UI distribution in the user interface distribution method according to the present invention.
  • FIG. 6 is a schematic diagram for explaining UI interaction in the user interface distribution method according to the present invention.
  • FIG. 7 is a flow chart showing UI update during a UI migration process in a user interface distribution method according to the present invention.
  • FIG. 8 is a flow chart showing a host app function call through a user's UI input during a UI migration process in a user interface distribution method according to the present invention.
  • FIG. 9 is a schematic diagram of a UI migration process in a user interface distribution method according to the present invention.
  • FIG. 10 is a flow chart showing UI update during a UI replication process in a user interface distribution method according to the present invention.
  • FIG. 11 is a flowchart illustrating a host app function call through a user's UI input during a UI replication process in a user interface distribution method according to the present invention.
  • FIG. 12 is a schematic diagram of a UI copy process in a user interface distribution method according to the present invention.
  • FIG. 13 is a diagram showing an embodiment of a method in which an app developer presets UI objects to be migrated or copied.
  • FIG. 2 is a diagram illustrating a situation in which a chat window UI is distributed to another device (guest device) by using a user interface distribution method according to the present invention.
  • the present invention supports the user to move the chat window UI of FIG. 1 to another device as shown in FIG. 2(b), and allows the user to enjoy streaming in full screen on the original device as shown in FIG. 2(a), It can provide better usability.
  • various useful embodiments can be made by flexibly arranging application UIs on various devices.
  • FIG. 3 is a diagram showing a UI structure of Android, which is an embodiment of a basic platform to which the present invention is applied.
  • UI is composed of a collection of UI elements (hereinafter referred to as'UI elements').
  • the UI element means, for example, a text input window, each button, and the like as shown in FIG. 3. From the user's point of view, the UI element is the smallest unit through which the user can interact with the application.
  • Android which is an embodiment of the basic platform to which the present invention is applied, has two types of UI objects, that is, a layout and a widget as shown in FIG. 3. These layouts and widgets are maintained in a tree structure called'UI tree', and these UI objects include program code and data.
  • the widget is a graphical component mapped one-to-one with each UI element (eg, button) as shown in FIG. 3, and the layout manages how each UI object is positioned on the screen of the device. ) Is the graphic component data.
  • Each UI object manages state data collected into two categories of states: graphical states and non-graphical states.
  • the graphic state is data necessary during rendering, such as color, text, picture, animation, etc.
  • the non-graphical state refers to all other data that is not related to rendering, and is mainly data related to application logic.
  • UI resources' such graphic state data
  • the UI object has a layout and a widget, and includes program code and data.
  • program code and data are collectively referred to as data
  • UI graphic state data
  • UI The resource and non-graphic state data are summed and expressed as a UI object.
  • Each UI element is a term mainly representing each UI element appearing on a screen
  • a UI object is a term mainly viewed from the viewpoint of program code and data, but hereinafter, it will be used without making a big distinction.
  • FIG. 4 is a flowchart showing an overall process of performing a user interface distribution method according to the present invention
  • FIG. 5 is a schematic diagram for explaining UI distribution in the user interface distribution method according to the present invention.
  • the app code is platform-independent. It can be applied in cases where it can be executed smoothly (for example, Web App, etc.).
  • the basic platform refers to a platform (eg, Android) that operates on each of the host device 10 and the guest device 50 before the host platform and the guest platform of the present invention described below are implemented.
  • the platform program (hereinafter referred to as'host platform') 100 for performing the user interface distribution method of the present invention in the host device 10, which is a smart device, includes a specific application program (hereinafter referred to as When'app') is installed, the application code of the app is analyzed, and candidates of UI objects and UI resources used by the application are extracted and stored (S401). This is hereinafter referred to as'static code analysis'.
  • the input of the static code analysis process becomes the entire application code
  • the output of the static code analysis process accordingly becomes the candidates of UI objects and UI resources used by the application.
  • UI objects and candidates of UI resources may include an object having shape information of the corresponding UI, text, and color used for the corresponding UI.
  • UI animation information may be included.
  • the host platform 100 includes a list of devices that can be used as guest devices 50, which are other smart devices located in a short distance, and multiple UIs with other devices among the apps installed in the host device 10 according to a user's request.
  • a list of interactive apps (hereinafter referred to as'interactive apps') using the screen is displayed on the screen of the host device 10 (S402).
  • the user selects a device to interact with the UI (hereinafter, the selected device is referred to as'guest device 50') (S403), and also selects an app to use the UI interaction (S403).
  • the host platform 100 transmits the package file of the selected app (hereinafter referred to as'app package file (APK)') to the selected guest device 50 (S404), and the user interface of the present invention from the guest device 50
  • the platform program (hereinafter referred to as'guest platform') 500 performing the distribution method receives the app package file and installs it on the guest device 50 (S405).
  • the user drives one app (hereinafter referred to as'host app') from the list of interaction apps mentioned in step S402 (S406), and drives one of the UIs used by the host app during the host app operation (S407).
  • a chat UI for chatting with other users may be driven.
  • a user's input for selecting a UI element to be sent to the guest device 50 is received (S408). For example, among UIs that are driven and displayed on the screen of the host device 10, only some UI elements can be selected and sent to the guest device 50, a schematic diagram of which is shown in FIG. 5.
  • each UI element is a term that refers mainly to each UI element that appears on the screen, and UI object is a term mainly viewed from the point of view of program code and data.
  • the host device 10 when the host device 10 is a device in which it is difficult for the user to select a UI element, such as a smart TV, the user of the guest device 50 may directly select a UI element to be used.
  • the host device 10 may perform mirroring in which the entire screen including the UI object is sent to the guest device 50, and the user of the guest device 50 selects a UI element to be used on the received screen. You can choose and use it.
  • the UI object for sending to the guest device 50 is set by the app developer in advance, not by the user's selection (S408) as described above. There is a way. 13 shows an embodiment of a method in which an app developer presets UI objects to be moved or copied as such.
  • the app developer pre-stores the UI object (including UI resources and non-graphical state data) to be transferred to the guest device 50 in a migration or replication method at runtime. It is specified (set) in the app layout file, which is a file.
  • Such an app layout file may be written in an XML format, and is parsed when the app is operated, and UI objects corresponding thereto are configured and drawn on the screen.
  • a UI object to be transferred to the guest device 50 in a migration or replication method at runtime is created by the host platform developer and stored in the host device 10, not by the app layout file. It can also be specified (set) in the platform layout file.
  • the method of setting the UI object in the platform layout file is the same as the method of setting it in the above-described app layout file.
  • the host platform can automatically select which UI objects to move to which external devices by parsing newly added tags and properties without user intervention. That is, the app user of the host device 10 simply selects to deliver the UI to the guest device after running the app, and the host platform transfers the UI object of the corresponding UI set in the app layout file or the platform layout file. It is transmitted to the device 50.
  • the host platform 100 transmits the selected UI element to the platform layout file of the host platform 100. Can be saved. In this way, afterwards, the host platform can transmit the UI object for the UI element stored in the platform layout file to the guest device 50 without the need for selection by the user.
  • the host platform 100 transmits the extracted UI object and UI resource to the guest device 50 (S410), and the guest platform 500 renders the UI according to the UI object for the UI element received in step S410. It is displayed on the screen of the guest device 50 by (rendering) (S411).
  • the information transmitted to the selected guest device 50 may further include information on whether to migrate or copy the UI selected by the user to the guest device 50. .
  • FIG. 6 is a schematic diagram for explaining UI interaction in the user interface distribution method according to the present invention.
  • The'UI thread' shown in FIG. 6 refers to a function of handling the UI tree and updating the state of the UI object of each app. For example, when a user clicks a button, the UI thread executes an event handler related to a button click.
  • the host platform and the guest platform allow the user to interact with the host app as if the UI was on the same device through the host and guest UI.
  • the user may swipe a video progress bar on the screen of the guest device 50 to determine which part of the video clip to display on the host device 10.
  • the host platform and the guest platform enable this interaction, which is accomplished by converting local function calls into cross-device function calls, if necessary.
  • a wrapper app 60 of the guest device 50 when the UI state is updated on the guest device screen by a user's input on the guest device screen, a wrapper app 60 of the guest device 50 according to the user's input.
  • the user input data received by the UI thread of is sent to the host app 20 of the host device 10 by the guest platform 500 (request), and the host app 20 calls a function for updating the UI (local function). call), where the host platform 100 intercepts this and converts it into a cross-device function call and sends it to the guest device 50, and the guest platform 500 calls a function for updating the UI. It converts back to (local function call) and performs UI update.
  • a function call of the host app 20 by the operation of the host platform 100 (local function call)
  • the intercepted device is converted into a cross-device function call and sent to the guest device 50 so that the UI can be used in another device. That is, without modifying the code of the existing host app 20 that has only a function call function for using the UI within the same device, the host app 20 does not recognize it at all.
  • Cross-device function calls are made by the operation of the platform 100.
  • the guest platform 500 intercepts cross-device function calls from the host device 10 thus received, and calls the functions within the guest device 50. By changing to (local function call), UI processing is naturally performed on the guest device as well.
  • a local function call of the guest device 50 is converted into a cross-device function call to the host device, and such cross-device function calls are converted to the host device.
  • a concrete implementation method of cross-device function calls made by the interception operation of the host platform 100 and the guest platform 500 without modifying the existing app that is intended to use the UI in the device As an embodiment of, a code conversion function method may be used, which will be described later.
  • Such UI interactions are classified into UI migration and UI replication, and will be described later with reference to FIGS. 7 to 9 and 10 to 12, respectively.
  • FIG. 7 is a flow chart showing UI update during a UI migration process in the user interface distribution method according to the present invention.
  • FIG. 8 is a user interface distribution method in the UI migration process according to the present invention. Is a flow chart showing a host app function call through UI input, and
  • FIG. 9 is a schematic diagram of a UI migration process in the user interface distribution method according to the present invention.
  • the UI selected by the user is transferred to the guest device 50, or whether the UI selected by the user is to be migrated or copied. Information may be further included, and FIGS. 7 to 9 are processes when migration is selected.
  • FIG. 7 is a flow chart showing a UI update in general among the UI migration process
  • FIG. 9 mainly shows a process of A-->B.
  • the update of the UI state displayed on the screen of the guest device 50 (hereinafter referred to as'UI update') is performed by a local function call of the host app (S701).
  • the UI update is, for example, during broadcast streaming through the host app, the video progress bar continues to progress.
  • This video progress bar is also a kind of UI, and the UI state is constantly changing, and UI update is taking place. .
  • This UI update is necessarily performed by calling the host app's function. That is, in this case, the UI update occurs by calling a function of the host app according to the execution of the host app without a user's input through the UI in the guest device 50. Step S701 of FIG. 7 may be regarded as such a case.
  • the UI may be updated in the guest device by transmitting it to the host device 10 and calling a function by the host app accordingly.
  • the host device 10 For example, when a user inputs a specific button on the guest device, there may be a case where the color of the UI on the screen of the guest device changes. Such a case is shown in FIG. 8, and will be described later in detail with reference to FIG. 8.
  • the host platform 100 converts such a local function call of the host app into a cross-device function call, that is, a remote procedure call (RPC) message (S702), and the guest device 50 ) Is transmitted (S703).
  • RPC remote procedure call
  • the host platform 100 intercepts a local function call called from an app inside the host device 10, and cross-device function calls are not recognized by the app.
  • a wrapper app 60 to the guest platform ( 500) is intercepted, converted into cross-device function calls, and sent to the host device 10 while the app does not recognize them. It is used as a term for cross-device function calls in a case.
  • the argument of the RPC message is a device-dependent factor, that is, a reference type argument
  • the data value indicated by the reference type factor together with the RPC message is sent to the guest device 50 together with the RPC message. Will be transmitted.
  • the guest platform 500 when the guest platform 500 receives an RPC (remote procedure call) message for updating the UI status displayed on the screen of the guest device 50 from the host device 10 (hereinafter referred to as'UI update') ( S703), the guest platform 500 converts the RPC message back to a local function call in the guest device to update the corresponding UI (S704).
  • the host platform 100 After sending the RPC message in this way, the host platform 100 stores the return address to be returned after performing the RPC call, and when it receives the return value or error code for the RPC call from the guest device 50, it returns to the return address. You're going to take the next step.
  • FIG. 8 is a flow chart showing a host app function call through a user's UI input among UI migration processes
  • FIG. 9 mainly shows a process B-->C.
  • the guest platform 500 converts this to a remote procedure call (RPC) (S803) is transmitted to the host device 10 (S804), and the host platform 100 receiving it converts the RPC message back to a local function call in the host device to make a corresponding host app function call.
  • RPC remote procedure call
  • S805 To perform (S805).
  • the user presses a toggle button for play-pause on the guest device there may be a situation in which the streaming app screen changes to stop-play on the host device.
  • step S801 If the UI needs to be updated in the guest device according to the user's input in step S801, the guest device ( In 50), a function call (S806) for UI update occurs. Accordingly, the host platform 100 converts this to a remote procedure call (RPC) (S807) and transmits the RPC message to the guest device 50 (S808), and in the guest device 50, the guest platform 500 UI update is performed (S809).
  • RPC remote procedure call
  • FIG. 10 is a flow chart showing UI update during a UI copy process in the user interface distribution method according to the present invention.
  • FIG. 11 is a UI copy process in the user interface distribution method according to the present invention. It is a flow chart showing the host app function call through UI input.
  • 12 is a schematic diagram of a UI copy process in a method for distributing a user interface according to the present invention.
  • the UI selected by the user is transferred to the guest device 50, or whether the UI selected by the user is to be migrated or copied. Information may be further included, and FIGS. 10 to 12 are processes in the case of selecting replication.
  • FIG. 10 is a flow chart showing UI update among the UI replication process
  • FIG. 12 mainly shows a process of A-->B.
  • the UI update by user input through the UI may occur not only by the guest device 50 but also by a user input from the host device 10, and steps S1002 to 10 of FIG. S1009 represents a UI update generated by a user input through the guest device 50 UI, and steps S1010 to S1017 of FIG. 10 represent UI updates generated by a user input through the host device 10 UI. but.
  • FIG. 12 for convenience, only a UI update process generated by a user input through the host device 10 UI, that is, A-->B (Local Call, RPC) is illustrated.
  • the host platform 100 transmits the same UI as the UI object and UI resource of the UI element selected by the user to the guest device in step S410 of FIG. 4.
  • the UI is rendered and displayed on the screen of the host device (S1001). That is, the same UI as on the screen of the guest device 50 is also displayed on the screen of the host device 10.
  • the guest platform 500 continuously transmits the data input by the user (S1002) through the guest device 50 UI (S1003), and the host device 10 receives the data.
  • the host device 10 is displayed on the screen (S1004). That is, of course, the user's input data is also displayed on the input guest device 50 (S1002) and at the same time also displayed on the screen of the host device 10 (S1004).
  • the host platform 100 is performed by the called function.
  • the UI update is also performed in the host device 10 (S1009), and the host platform 100 converts the same function call into a remote procedure call (RPC) message (S1006) and transmits it to the guest device 50 (S1007). Accordingly, the same UI update is performed in the guest device 50 (S1008).
  • RPC remote procedure call
  • the argument of the RPC message is a reference type argument that is a device-dependent factor
  • the data value indicated by the reference type factor together with the RPC message is sent to the guest device 50 together. Send.
  • step S1010 a case in which the user inputs data through the host device 10 is indicated.
  • the host platform 100 continuously transmits data input by the user (S1010) through the host device 10 UI to the guest device 50 (S1011), and the guest device 50 receives the data to the guest device.
  • the transmission of the inputted data (S1010) to the guest device 50 (S1011) may be performed by an RPC message, and the display of such data on the screen of the guest device 50 (S1012) is a UI update method. Can be done.
  • UI update may occur by calling a function of the host app (S1013) according to the execution of the host app. have. This is similar to step S701 of FIG. 7 of UI migration.
  • FIG. 11 is a flow chart showing a host app function call through a user's UI input among the UI replication process
  • FIG. 12 shows a process of B-->C (local call, suppression). .
  • the host app function call by user input through the UI may occur not only by the guest device 50 but also by the user input from the host device 10, and the step of FIG. 11 S1101 to S1104 represent host app function calls generated by a user input through the guest device 50 UI, and steps S1107, S1108, and S1112 of FIG. 11 are hosts generated by a user input through the host device 10 UI.
  • App function call (B-->C, Local call) is shown.
  • the guest platform 500 continuously transmits data input by the user through the guest device 50 UI (S1101) to the host device 10 (S1102), and the host device 10 Receives it and displays it on the screen of the host device 10 (S1103). That is, of course, the user's input data is also displayed on the input guest device 50 (S1101) and at the same time also displayed on the screen of the host device 10 (S1103). Thereafter, the host device 10 performs a host app function call (local call, see FIG. 12) (S1104, B->C, see FIG. 12).
  • a host app function call local call, see FIG. 12
  • the host platform 100 continuously transmits data input by the user through the host device 50 UI (S1107) to the guest device 10 (S1108), and the guest device 10 receives it Display on the screen of the guest device 10 (S1109). That is, of course, the user's input data is also displayed on the input host device 50 (S1107) and at the same time also displayed on the screen of the guest device 50 (S1109).
  • transmission of the inputted data (S1107) to the guest device 50 (S1108) may be performed by an RPC message, and the display of such data on the screen of the guest device 50 (S1109) is performed in a UI update method. Can be done.
  • the host device 10 performs a host app function call (local call, see FIG. 12) (S1112, B-->C, see FIG. 12).
  • the guest device 50 receives the return value of the function called in steps S1105 and S1110, received from the host device 10, and proceeds to the next step of executing the called function (S1106, S1111).

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Human Computer Interaction (AREA)
  • Signal Processing (AREA)
  • User Interface Of Digital Computer (AREA)
  • Communication Control (AREA)

Abstract

La présente invention concerne un procédé de distribution d'interface utilisateur pour une interaction multi-dispositifs. Plus particulièrement, l'invention porte sur un procédé de distribution d'interface utilisateur destiné à assurer une prise en charge de telle sorte que des écrans d'une pluralité de dispositifs intelligents servent d'interfaces utilisateur de manière souple et rapide sans corriger le code d'une application utilisée dans un dispositif. Selon la présente invention, le procédé comprend : un sous-arbre comprenant un objet d'IU cible d'un dispositif hôte étant séparé de façon à être déplacé vers un dispositif invité, ce qui permet le rendu d'un objet d'IU cible dans le dispositif invité sans nécessité de corriger une application ; et un appel de fonction local dans un dispositif étant converti de manière transparente en un appel RPC inter-dispositifs étant donné qu'une interaction entre des objets d'IU de dispositifs distribués et une logique d'application hôte est réalisée sous la forme d'un appel de fonction local, et ainsi l'interaction entre dispositifs est également maintenue sans nécessité de corriger l'application.
PCT/KR2020/014187 2019-10-16 2020-10-16 Procédé de distribution d'interface utilisateur pour une interaction multi-dispositifs WO2021075919A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
KR20190128772 2019-10-16
KR10-2019-0128772 2019-10-16

Publications (1)

Publication Number Publication Date
WO2021075919A1 true WO2021075919A1 (fr) 2021-04-22

Family

ID=75537980

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/KR2020/014187 WO2021075919A1 (fr) 2019-10-16 2020-10-16 Procédé de distribution d'interface utilisateur pour une interaction multi-dispositifs

Country Status (2)

Country Link
KR (2) KR102426198B1 (fr)
WO (1) WO2021075919A1 (fr)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2023218824A1 (fr) * 2022-05-10 2023-11-16 Sony Group Corporation Appareil de traitement d'informations, procédé de traitement d'informations, et programme

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2023195556A1 (fr) * 2022-04-07 2023-10-12 엘지전자 주식회사 Dispositif d'affichage

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20130081068A (ko) * 2012-01-06 2013-07-16 삼성전자주식회사 복수의 휴대 단말들을 이용하여 멀티 비전 시스템을 구축하는 방법 및 이를 위한 장치
KR20140139550A (ko) * 2012-03-14 2014-12-05 퀄컴 인코포레이티드 2차 디스플레이들을 통한 사용자 정보의 공유
KR20150002883A (ko) * 2010-04-08 2015-01-07 노키아 코포레이션 다수의 디바이스의 디스플레이들을 결합하기 위한 방법, 장치 및 컴퓨터 프로그램 제품
KR101653922B1 (ko) * 2015-04-20 2016-09-05 한국과학기술원 복수의 스크린 간의 상대 위치를 이용하여 양방향 멀티 스크린을 제공하는 방법 및 시스템
KR101923255B1 (ko) * 2011-10-27 2018-11-28 마이크로소프트 테크놀로지 라이센싱, 엘엘씨 모바일 장치로부터의 원격 액세스 기법

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20150002883A (ko) * 2010-04-08 2015-01-07 노키아 코포레이션 다수의 디바이스의 디스플레이들을 결합하기 위한 방법, 장치 및 컴퓨터 프로그램 제품
KR101923255B1 (ko) * 2011-10-27 2018-11-28 마이크로소프트 테크놀로지 라이센싱, 엘엘씨 모바일 장치로부터의 원격 액세스 기법
KR20130081068A (ko) * 2012-01-06 2013-07-16 삼성전자주식회사 복수의 휴대 단말들을 이용하여 멀티 비전 시스템을 구축하는 방법 및 이를 위한 장치
KR20140139550A (ko) * 2012-03-14 2014-12-05 퀄컴 인코포레이티드 2차 디스플레이들을 통한 사용자 정보의 공유
KR101653922B1 (ko) * 2015-04-20 2016-09-05 한국과학기술원 복수의 스크린 간의 상대 위치를 이용하여 양방향 멀티 스크린을 제공하는 방법 및 시스템

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2023218824A1 (fr) * 2022-05-10 2023-11-16 Sony Group Corporation Appareil de traitement d'informations, procédé de traitement d'informations, et programme

Also Published As

Publication number Publication date
KR102426198B1 (ko) 2022-07-28
KR20210045342A (ko) 2021-04-26
KR20220106946A (ko) 2022-08-01
KR102660531B1 (ko) 2024-04-24

Similar Documents

Publication Publication Date Title
WO2021075919A1 (fr) Procédé de distribution d'interface utilisateur pour une interaction multi-dispositifs
US11036345B2 (en) System and method for on-screen graphical user interface encapsulation and reproduction
WO2014142471A1 (fr) Procédé et système de commande multi-entrées, et dispositif électronique les prenant en charge
WO2014116005A1 (fr) Procédé de fourniture d'un environnement de développement d'application et dispositif associé
WO2014069882A1 (fr) Procédé et appareil de traitement de page web dans un dispositif terminal à l'aide d'un serveur infonuagique
WO2011014040A2 (fr) Procédé et dispositif pour la création d'une interface utilisateur intégrée
WO2013042921A1 (fr) Appareil et procédé d'exécution d'une application dans un terminal mobile
WO2010113160A1 (fr) Procédé et système permettant d'émuler des applications de logiciel de bureau dans un réseau de communication mobile
EP3931692A1 (fr) Procédé et système d'apprentissage de tâche intelligent sémantique et d'exécution adaptative
KR101416100B1 (ko) 하이브리드 웹 어플리케이션에서 자바스크립트 연동을 위한 인터페이스
CN104038657B (zh) 信息处理系统、信息处理设备和信息处理方法
WO2018105804A1 (fr) Système diy iot fondé sur bpm et son procédé de mise en œuvre
WO2019168387A1 (fr) Dispositifs, procédés et programme d'ordinateur destinés à afficher des interfaces utilisateur
CN114020194A (zh) 显示数据的修改方法、装置、计算机设备和介质
CN114510170B (zh) 一种组件展示方法及显示设备
WO2018048117A1 (fr) Appareil d'affichage et procédé de commande correspondant
JP2008131388A (ja) 画像形成装置、表示制御方法、及び表示制御プログラム
WO2014171613A1 (fr) Procédé de prestation de service de messagerie, support d'enregistrement enregistré avec un programme afférent et terminal correspondant
WO2015199299A1 (fr) Procédé de réalisation de service en nuage, support d'enregistrement lisible par ordinateur sur lequel est enregistrée une application web prenant celui-ci en charge, et système et dispositif apparentés de service en nuage
KR20010024488A (ko) 드래그 앤드 드롭 인터페이스를 이용한 응용 프로그램제어 시스템
US20230325263A1 (en) User Interface Distribution Method for Multi-Device Interaction
WO2013180505A1 (fr) Terminal de restructuration d'applications et serveur pour personnaliser une application
WO2016108677A1 (fr) Appareil et procédé de sortie de contenu vidéo
WO2013133500A1 (fr) Procédé et appareil de traitement de contenu
WO2015142007A1 (fr) Procédé d'affichage de message pour afficher de façon distinctive un message extrait sur une fenêtre de messagerie instantanée, et terminal mobile et serveur de messagerie instantanée pour mettre en œuvre le procédé

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20877792

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20877792

Country of ref document: EP

Kind code of ref document: A1