WO2023163566A1 - Method of transmitting ui display information to proximal clients by local server - Google Patents

Method of transmitting ui display information to proximal clients by local server Download PDF

Info

Publication number
WO2023163566A1
WO2023163566A1 PCT/KR2023/002793 KR2023002793W WO2023163566A1 WO 2023163566 A1 WO2023163566 A1 WO 2023163566A1 KR 2023002793 W KR2023002793 W KR 2023002793W WO 2023163566 A1 WO2023163566 A1 WO 2023163566A1
Authority
WO
WIPO (PCT)
Prior art keywords
local server
information
client
content
display
Prior art date
Application number
PCT/KR2023/002793
Other languages
English (en)
French (fr)
Inventor
In Sik Shin
Daehee Lee
Sun Jae Lee
Original Assignee
Fluiz Co. Ltd.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Fluiz Co. Ltd. filed Critical Fluiz Co. Ltd.
Publication of WO2023163566A1 publication Critical patent/WO2023163566A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
    • G06F3/1423—Digital output to display device ; Cooperation and interconnection of the display device with other functional units controlling a plurality of local displays, e.g. CRT and flat panel display
    • G06F3/1438—Digital output to display device ; Cooperation and interconnection of the display device with other functional units controlling a plurality of local displays, e.g. CRT and flat panel display using more than one graphics controller
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/451—Execution arrangements for user interfaces
    • G06F9/452—Remote windowing, e.g. X-Window System, desktop virtualisation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
    • G06F3/1423—Digital output to display device ; Cooperation and interconnection of the display device with other functional units controlling a plurality of local displays, e.g. CRT and flat panel display
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
    • G06F3/1454—Digital output to display device ; Cooperation and interconnection of the display device with other functional units involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/451—Execution arrangements for user interfaces
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G2354/00—Aspects of interface with display user
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G2370/00—Aspects of data communication
    • G09G2370/02—Networking aspects
    • G09G2370/022—Centralised management of display operation, e.g. in a server instead of locally

Definitions

  • the present invention relates to a method of transmitting UI display information to proximal clients by a local server, and more particularly, to a method of transmitting UI display information, which transmits UI display information of a content application in a unit of a UI element group, thereby variously combining UI element groups and providing the combined UI element groups to a plurality of devices.
  • the mirroring technology is inefficient because the same display is output in various display devices, and thus the inconvenience that the mobile device transmitting the mirroring cannot be used for other purposes occurs.
  • the screen is divided into a plurality of areas and used, since the display in one device is divided and used, there is a problem in that the UI display information cannot be distributed and used in several display devices.
  • Prior Art 1 Korean Patent Application Laid-Open No. 10-2018-0081133
  • a display UI of a display is divided into at least two display areas in response to the finger joint touch gesture.
  • Prior Art 2 (Korean Patent Application Laid-Open No. 10-2010-0081383) is related to the touch screen dividing method, and is characterized in that a divided area layout of a touch screen is formed by receiving an input of two positions on the touch screen and a separate display is allocated to each of the formed divided areas.
  • Prior Art 2 also discloses the method of dividing the screen in one device, so that there is no motivation to display the divided screen on the plurality of devices in cooperation with the plurality of devices.
  • Prior Art 3 (US Patent Application Publication No. US20190373058 A1) is characterized in that user interaction information is synchronized by transmitting user interaction information between a plurality of devices to a remote server.
  • Prior Art 3 there is no motivation to display a divided screen in a plurality of devices in cooperation with the plurality of devices.
  • the UI display is provided through separate hardware (for example, Chromecast), but a function of reconfiguring a UI display desired by a user is not provided. Further, in the case of different OSs between display devices, it is difficult to share a UI display, so that there is difficulty in sharing UI information between multiple devices.
  • the first problem to be solved by the present invention is to provide a method of transmitting, by a local server capable of variously combining UI element groups and distributing the combined UI element groups to a plurality of devices, UI display information together with proximal clients' devices by performing UI display information transception of a content application in a unit of a dividable UI element group.
  • the second problem to be solved by the present invention is to provide a system for controlling UI display information, which is capable of transmitting UI display information independently to an OS by transmitting a UI element group included in UI distribution information as a video streaming.
  • the present invention provides a computer readable recording medium in which a program for executing the method in a computer is recorded.
  • the present invention provides a method of transmitting UI display information to proximal clients by a local server capable of performing data communication, the method including: dividing a single UI provided by a content application into a plurality of UI element groups after the content application selected by any one client is executed in the local server; generating UI distribution information by matching the proximal clients to the plurality of UI element groups, respectively; and transmitting one or more UI element groups to the matched proximal clients respectively, or transmitting video streams converted from one or more UI element groups to the matched proximal clients respectively by using the UI distribution information.
  • an authority to divide the UI of the content application into UI element groups may be authorized or delegated to any one client among the proximal clients.
  • UI element groups matched to the proximal clients respectively
  • the UI distribution information and the display size and resolution information of the proximal clients are stored in a cloud server.
  • the one or more UI element groups matching to the proximal clients respectively may be transmitted to the proximal clients, and then providing the received UI control information of a user may be received from the proximal clients and provide the content application with the UI control information.
  • the UI arrangement information may be determined according to a physical distance to the proximal client and a direction of the location of the proximal client based on the local server.
  • the UI distribution information may be stored by matching UI element groups of different content applications to the proximal clients.
  • the present invention provides a local server for transmitting UI display information to proximal clients; the local server including: a UI element dividing module divides a single UI, provided by a content application, into a plurality of UI element groups after the content application selected by a user is executed in any one client having an authority to divide a UI of the content application; a UI distribution information generating module which generates UI distribution information by matching the proximal clients to the plurality of UI element groups respectively; and a UI element transmitting module which transmits one or more UI element groups to the matched proximal clients respectively, or transmits a video streams for one or more UI element groups to the matched proximal clients respectively, by using the UI distribution information.
  • a UI element dividing module divides a single UI, provided by a content application, into a plurality of UI element groups after the content application selected by a user is executed in any one client having an authority to divide a UI of the content application
  • the present invention provides a computer-readable recording medium in which a program for executing the method of transmitting the UI display information to the proximal clients by the local server in a computer, a mobile device, a smart home device, and the like is recorded.
  • UI display information independently from an OS by transmitting a video stream converted from UI element group included in UI distribution information.
  • FIG. 1 is a conceptual diagram illustrating transmission of UI display information to proximal clients by a local server in a network consisting of a local server 100, a proximal client 200, a cloud server 300, and a content providing server 400.
  • FIG. 2 illustrates a concept in which the local server 100 divides a UI of an original content into UI element groups by using UI tree information and transmitting the UI element groups to proximal clients 21 and 22 according to an exemplary embodiment of the present invention.
  • FIG. 3 is a configuration diagram illustrating the local server and a client according to the exemplary embodiment of the present invention.
  • FIG. 4 is a flowchart illustrating a method of transmitting, by the local server, UI display information to proximal clients according to the exemplary embodiment of the present invention.
  • FIG. 5 is a flowchart illustrating a method of transmitting, by the local server, UI display information to proximal clients according to another exemplary embodiment of the present invention.
  • FIG. 6 is a flowchart illustrating a method of transmitting, by the local server, UI display information to proximal clients according to still another exemplary embodiment of the present invention.
  • FIG. 7 illustrates a concept of transmitting a UI element group by a relay method when the local server transmits UI display information to proximal clients according to the exemplary embodiment of the present invention.
  • FIG. 8 is a flowchart illustrating a method of dividing, by the local server, UI tree information illustrated in FIG. 7 and transmitting the divided UI tree information to the client according to the exemplary embodiment of the present invention.
  • FIG. 9 illustrates a flow of a method of editing a UI of a content app by a client according to the exemplary embodiment of the present invention.
  • FIG. 10 illustrates a flow of a method of transmitting, by the local server, Internet of Things (IoT) state information to the client according to another exemplary embodiment of the present invention.
  • IoT Internet of Things
  • FIG. 11 illustrates a flow of a method of transmitting, by the local server, IoT state information to the client according to still another exemplary embodiment of the present invention.
  • FIG. 12 illustrates a flow of a method of transmitting, by the local server, IoT state information to an unauthorized client according to still another exemplary embodiment of the present invention.
  • FIG. 13 illustrates a flow of a method of receiving an NFT content and a UI app, by the local server, and transmitting UI display information according to still another exemplary embodiment of the present invention.
  • a method of transmitting UI display information of a user device including: dividing, by a first user device, a single UI of a content application executed in the first user device into a plurality of UI element groups in an OS level; generating, by the first user device or an application of the first user device, UI distribution information by matching the one UI element group to a second user device; transmitting, by the first user device, the matched UI element group to the second user device, or converting the matched UI element group, by using the UI distribution information, into a video stream and transmitting it to the second user.
  • the present invention maximizes service scalability of a content application in consideration of a display size or resolution of a user display device in order to transmit free UI display information between display devices connected through a communication network.
  • the present invention presents an efficient interactive data transmitting method for improving user experience between devices of a user group in which a plurality of display devices registered by a user or display devices registered by multiple users are included.
  • a video, audio, image, touch, keyboard, and voice data may be shared or mirrored and forwarded between display devices connected through a communication network.
  • a content UI of a content application executed in the display device is composed of UI element groups, and provides a function capable of transmitting the UI in the unit of one or more individual UI element groups to a proximal display device and providing a UI element management authority function to enable various UI changes and combinations.
  • a local server having an authority to divide and distribute a content UI may transmit a video stream of the UI element group constituting the UI to the proximal display device.
  • each proximal display device when only UI element group information is transmitted to a proximal display device, each proximal display device generates a video stream by itself, and then each proximal display device transmits interaction information of a user to a local server.
  • the local server may input user interaction information received from each proximal display device to a content application.
  • a user app installed in the proximal display device for the present invention may virtualize a display by using the UI element group information received from the server and transmit UI information to a plurality of user devices.
  • a function supported at an OS level in the local server may be minimized to and configured as a UI dividing function (UI virtualization), UI distribution information storage, and UI element group transmission function, and then the UI transmission function using a virtual display configuration and video streaming required in the present invention may be configured to be performed in a user app installed in a user device.
  • UI virtualization UI dividing function
  • UI distribution information storage UI distribution information storage
  • UI element group transmission function a virtual display configuration and video streaming required in the present invention
  • the local server may update the UI element group information to be transmitted to each proximal display device by reflecting UI distribution information of a content application received from a cloud server.
  • the local server converts the UI element group information into a video stream and transmits the video stream to each proximal client device, there is an advantage in that it is possible to transmit the video stream independent to an OS of the client device in terms of a data communication aspect.
  • the present invention presents a method of easily monitoring or controlling, by a user, a proximal client device and an IoT device.
  • the local server transmits a UI of an IoT application capable of controlling an IoT device to a proximal client by a video streaming method, so that it is possible to monitor state information of the IoT device even in a user's device in which the IoT application is not installed, or control the IoT device by transmitting interaction information of the user from the user device to the local server.
  • the present invention presents a method of managing a content authority which is capable of various user experience environments, and forwarding a content.
  • the present invention may provide a platform environment providing a UI change authority to change a UI of a content, an authority to provide a new UI of an application, and a content access authority assignment function to a user group capable of accessing a digital content of a user.
  • FIG. 1 is a conceptual diagram illustrating transmission of UI display information to proximal clients by a local server in a network consisting of a local server 100, a proximal client 200, a cloud server 300, and a content providing server 400.
  • a content application may request a content from the content providing server 400.
  • the content application installed in the local server 100 may directly access the content providing server 400 and request the corresponding content or request the content via the cloud server 300.
  • the local server 100 may install the content application by requesting the content application from the content providing server 400 and installing the received content application.
  • the proximal client 200 includes various user devices, such as a smart display 21, smart phones 22 and 23, a smart light bulb 25, and a smart camera 26.
  • the smart light bulb 25 and the smart camera 26 are an IoT device group.
  • the proximal client 200 may register its user device information in the local server 100.
  • Any one client in the proximal client 200 may request to register its own device information with the local server 100, and the local server 100 may transmit the registered user device information to the cloud server 300 which stores the user device information.
  • the devices of the proximal client 200 may be registered in the local server 100 as a user device group, and the corresponding user device information may be transmitted to and re-stored in the cloud server 300.
  • the user device group information a Service Set Identifier (SSID), a MAC address, a Universally Unique Identifier (UUID), or the like may be used, the user device group information may be differently configured according to a session from the execution of an app to the end of the app, or a time period from an installation of an app to a deletion of the app.
  • the user device group information may further include display size and resolution information.
  • the proximal client 200 and the local server 100 may be connected by using near-field communication, such as Bluetooth, ultrasound, or Zigbee, and an initial communication connection and authentication may be performed. Thereafter, UI display information between the authorized devices can be transmitted by using a home network connected by Wi-Fi.
  • near-field communication such as Bluetooth, ultrasound, or Zigbee
  • a new user device may be added to a Wi-Fi communication network by transmitting, by an initial smart phone or set top box connected with the Wi-Fi router, Wi-Fi connection information to another smart device or IoT device.
  • the Wi-Fi router serves as the local server 100
  • the set top box serves as the proximal client 200.
  • the local server 100 transmits UI display information to the proximal clients 200
  • the divided UIs of the content application may be shared and viewed by multiple clients.
  • the local server 100 receives a content of the content providing server 400 and transmits the received content to the proximal clients 200.
  • the UI display information of the content may be divided, matched, and transmitted for each client by a UI distribution control module 105 (see FIG. 3) implemented in the local server 100.
  • the UI distribution control module 105 installed in the local server 100 does not divide UI display information of the content by modifying a source code of an application of the content, but has a function of dividing UI tree information, constituting UI display information received at an Operating System (OS) level, into UI element groups; and rendering module is implemented, thereby dividing the UI display information and transmitting the divided UI display information to the plurality of user devices.
  • the content application installed in the local server 100 is received from the content providing server 400, and the local server is implemented in the form of a microkernel providing a minimum function required at the OS level to divide single UI information of the content to a plurality of UI element groups.
  • the local server or an application which is installed in the local server and includes a partial function of a UI editing control module, may match the plurality of UI element groups to proximal client devices and transmits the plurality of UI element groups by a video streaming method.
  • the proximal clients 200 may receive a UI video stream without installing the content application installed in the local server.
  • the local server or the application installed in the local server may transmit the matched UI element groups to the proximal client 200 devices without the conversion to the video stream.
  • the user application installed in the client for the present invention may convert the UI element group into a video stream and provide the UI display information to the client device.
  • the local server in order for the local server to easily distribute and transmit the UI to the proximal clients 200, an authority or a function of dividing and virtualizing UI tree information in the content at the OS level is required.
  • the local server 100 or the separate application installed in the local server may generate UI distribution information that matches the proximal clients 200 to the UI element group constituting the content UI display information by using the virtualized UI tree information.
  • an image of a movie content may be matched and transmitted to the smart display 21, a chatting window may be matched and transmitted to the first smart phone 22, and the subtitles of the movie may be matched and transmitted to the second smart phone 23.
  • the function of the present invention divides the single UI display information of the content application into each separable UI element group in the OS of the user device corresponding to the local server. Thereafter, the local server or the application installed in the local server may generate UI distribution information of the user device group by matching other user devices corresponding to the client devices and the UI element groups.
  • the UI distribution information may be backed up to the cloud server 300, and the backed-up UI distribution information may be stored as related information of a device group for each user.
  • a UI element to be matched to the added client may be updated in the UI distribution information.
  • the client of another user may also be stored to the UI distribution information and updated.
  • the UI distribution information stored in the cloud server 300 may be shared or copied to the client of the invited user, and the UI distribution information may be updated by reflecting only the client that needs to be added or changed to the UI distribution information or be stored as separate UI distribution information selectable by the user.
  • a user device group information and the UI distribution information stored in the cloud server 300 may be used when the local server 100 has a failure or a physical change of the local server is required. Further, the UI distribution information of the local server 100 and the proximal clients may be easily modified by copying or modifying existing UI distribution information of the user stored in the cloud server 300. In this case, the clients have be successfully authenticated transmit the Wi-Fi network information to the proximal client and the local server to simplify local network setting.
  • the user generally monitors the state of the smart light 25 or the smart camera 26 registered in the device group for each user or removes the smart light 25 or the smart camera 26 through the content application.
  • UI element group when the content image is dark, a control signal for brightening the smart light may be included in the UI distribution information, and in the movie mode, a control signal for dimming the smart light may be included in the UI distribution information.
  • the smart light 25 or the smart camera 26 among the proximal clients 200 does not have a separate display, so that it is necessary to download and install a necessary application (app) to the smart phones 22 and 23 or the smart display 21 of the user and sets the smart phones 22 and 23 or the smart display 21, or it is necessary to set the smart light 25 or the smart camera 26 by accessing a homepage of a manufacturing company.
  • another user may receive required content information of the IoT device (lighting state, camera information, and IoT device information), from the local server as.
  • a new user is capable of receiving the required UI display as a video stream only by installing a UI application of the present invention without installing the applications of various IoT devices, so that the new user is capable of receiving and controlling the video stream regardless of the OS (Android, IOS, Window, Linux, and the like) of the user client device.
  • the IoT devices 25 and 26 are capable of transmitting the state change information to the local server 100 connected through the home network or transmitting the cloud server 300 connected through the Internet network.
  • the cloud server 300 may provide the user who is allowed to access IoT state information with the state information of the IoT device through the Internet network corresponding to the present invention.
  • the two IoT apps are automatically synchronized.
  • the IoT app required for monitoring and controlling the IOT device may also be installed in the local server 100.
  • the local server 100 may allocate the UI element group for each proximal client 200 by using the UI tree information included in the content and match the UI element group to the proximal client 200.
  • the local server 100 configures required UI display information corresponding to the UI distribution information and display sizes and resolution of the proximal clients 200, and converts the UI element groups, matched to the proximal clients 200, into video streams or images, and transmits the video streams or images to the user device.
  • the UI distribution information required for controlling the IoT device may be shared with other user devices invited to the device group for each user.
  • the invited user device may receive a video stream or image information for monitoring and UI control information of the IoT device that is the same as that included in the user device group.
  • the user devices receive the UI control information as the video stream or the image information, so that there is an advantage in that it is possible to transmit the data independent to the OS systems.
  • FIG. 2 illustrates a concept in which the local server 100 divides a UI of an original content into UI element groups by using UI tree information and transmitting a UI element group to the proximal client 21 and 22 by a video streaming according to an exemplary embodiment of the present invention.
  • the original content is transmitted to the local server 100 from the content providing server 400 and is transmitted to the proximal clients 21 and 22.
  • the local server 100 may modify a composition and a layout of the UI element group by using the UI tree information included in the original content.
  • the local server 100 may divide the original content into a first UI element group and a second UI element group, match the first UI element group and the second UI element group to the smart display 21 and the smart phone 22 respectively, and then transmit the matched first UI element group and second UI element group by pixel streaming.
  • the data transmitted to the smart display is a video included in the original content, and is decoded in the local server and is transmitted to the smart display as a video stream, and when the second UI element is transmitted to the smart phone by encoded pixel streaming, and the smart phone decodes the received pixel stream.
  • the reproduction and the stop of the video stream reproduced in the smart display are determined according to an interaction of the user in the smart phone.
  • the decoding time of the video stream of the smart display needs to be synchronized based on the time in which the video stream is decoded in the smart phone.
  • the local server 100 includes the UI distribution control module 105
  • the smart display 21 includes a UI element display module 320 to a UI editing module 350
  • the smart phone 22 includes a UI element display module 321 or a UI editing module 351
  • FIG. 3 is a configuration diagram illustrating the local server and a client according to the exemplary embodiment of the present invention.
  • the local server 100 includes a receiving module 110, the UI distribution control module 105, and a transmitting module 190.
  • the UI distribution control module 105 includes a content app executing module 120, a UI element dividing module 130, a UI distribution information generating module 140, a display requesting module 150, a UI element transmitting module 160, a virtual display generating module 170, and a UI control information converting module 180.
  • the UI application installed in the local server 100 includes the entirety or a part of the configurations of the UI distribution control module 105.
  • the local server 100 and clients A and B 21 and 22 may be expressed as user devices.
  • clients A and B 21 and 22 includes app selecting modules 370 and 301, transmitting modules 310 and 311, UI element display modules 320 and 321, virtual display generating modules 330 and 331, UI control information input modules 340 and 341, UI editing modules 350 and 351, and receiving modules 360 and 361.
  • the local server 100 transmits UI display information to clients A and B 21 and 22 by video streaming or transmits the UI element groups to clients A and B 21 and 22, and receives UI control information by a user interaction from the client and generates a control signal of the content app.
  • Client B 22 selects a content app for which UI distribution information is to be changed and transmits the selected content app to the local server 100, and displays UI display information according to the changed UI distribution information.
  • Client A 21 is the client added for changing the UI distribution information of client B 22, and receives the UI element group of the content app from the local server 100 and displays the received UI element group.
  • the receiving module 110 receives whether to select the content app for which the UI distribution information is to be changed from client B 22.
  • the number of content apps selectable by client B 22 may be two or more.
  • the UI distribution control module 105 divides the UI display information of the content app into UI element groups, matches the divided UI element groups to each client, and generates UI distribution information.
  • the content app executing module 120 executes the content app selected by client B 22.
  • the content app is not required in clients A 21 and B 22 when the UI information is transmitted from the local server 100 by pixel streaming.
  • the content app executing module 120 may be replaced with an IoT app executing module and a UI app executing module combined with a Non-Fungible Token (NFT) content.
  • NFT Non-Fungible Token
  • the UI element dividing module 130 divides the UI, provided in the content app executed in the content app executing module 120, into a plurality of UI element groups.
  • the UI distribution information generating module 140 matches each proximal client to the each UI element group respectively, and generates UI distribution information.
  • the UI distribution information generating module 140 may determine the UI distribution information by a physical distance to the proximal client based on the local server 100 or a direction of the location of the proximal client. For example, the UI distribution information generating module 140 may determine to distribute the UI element group having a large display size in the client relatively far from the local server 100 and to arrange the UI element group having a small display size, such as a character input display, to the client relatively close to the local server 100.
  • the display requesting module 150 requests a UI element display to the client A 21 or the client B 22.
  • the local server 100 may request the UI element display module 320 to execute the UI element display through a push message.
  • the display requesting module 150 may make the client ready for the display of the received UI element group.
  • the UI element transmitting module 160 by using the UI distribution information, transmits one or more UI element groups which match to the proximal each client respectively, or transmits video stream, converted from the one or more UI element groups, matched to the proximal each client respectively.
  • the virtual display generating module 170 generates virtual display information by using the UI distribution information generated by the UI distribution information generating module 140.
  • the UI control information converting module 180 When the UI control information converting module 180 receives the UI control information from clients A 21 and B 22, the UI control information converting module 180 converts the received UI control information to a control signal of the content app.
  • the transmitting module 190 performs the display request of the display requesting module 150, and the transmission of the UI element group of the UI element transmitting module 160, and the like.
  • client A 21 and client B 22 are the same, so that client A 21 will be mainly described.
  • the app selecting module 370 selects a content app of which UI display information needs to be changed. In this case, it is not necessary to execute the content app as a result of the selection.
  • the transmitting unit 310 transmits user interaction information about which content app the app selecting unit 300 has selected to the local server 100, and the UI control information input unit 341 transmits the received UI control information to the local server 100.
  • the UI element display module 320 is executed in client A 21 by the request of the display requesting module 150 of the local server 100. It is preferable to receive the UI element group or the video streaming for the UI element group in the state where the UI element display module 320 is executed in client A 21.
  • the virtual display generating module 330 renders the UI element group received from the local server 100 and stores the UI element group in a graphic buffer.
  • the UI control information input module 340 receives the input of the UI control information through a touch screen, a keyboard, a voice, a picture, and the like.
  • original data may be transmitted to a separate storage server, and information about the storage space may be changed to a web URL and transmitted.
  • the UI editing modules 350 and 351 may select at least one UI element group of the content app, and select a client to display the selected UI element group and change UI distribution information.
  • the receiving module 360 directly receives at least one UI element groups from the local server 100 or receives the video stream of at least one of UI element groups.
  • FIG. 4 is a flowchart illustrating a method of transmitting, by the local server, UI display information to proximal clients according to the exemplary embodiment of the present invention.
  • the method of transmitting UI display information according to the exemplary embodiment includes operations time-serially processed in the local server and the client illustrated in FIG. 3. Accordingly, even if omitted below, the contents described above with respect to the local server and the client illustrated in FIG. 3 are also applied to the method of transmitting the UI display information according to the exemplary embodiment.
  • client A 21, client B 22, and the local server 100 are connected through the home network, and may be mutually called by using the HTTP-based web API (REST service or Remote Procedure Calls (RPC) mechanism), and the TCP/IP protocol.
  • REST API may be applied for synchronization of the content apps
  • RPC may be applied for video streaming between devices
  • TCP/IP protocol may be applied to the transmit of the control information according to user interaction between devices.
  • the local server 100 receives UI control information according to a selection of a content app of a user from client B 22.
  • the local server transmits the UI control information to a content application.
  • the present invention is described based on one content application, but the app selecting module may select a plurality of content applications.
  • the UI distribution control module 105 of the local server 100 may divide the plurality of content application UIs by matching the clients to each UI element group of the plurality of content application UIs.
  • the content application is an application used for displaying the original content, and may be a video application, such as YouTube or Twitch.
  • client B 22 may receive the UI element group from the local server 100 by video streaming, or may receive UI element group information and then the client may render and display the UI element group.
  • the local server 100 executes the content app selected in operation 400.
  • the local server 100 divides the UI of the executed content app into a plurality of UI element groups.
  • the UI provided from the content application executed in the local server 100 may be divided into a plurality of UI elements by using the UI distribution control module 105 installed in the local server 100.
  • an authority for the functions of dividing the single UI of the content app into the plurality of UI element groups and generating and correcting the UI distribution information is granted only to a specific device of a user possessing the corresponding right, so that a security function of the system may be strengthened.
  • the local server 100 matches the proximal clients (21 and 22) to the plurality of UI element groups respectively, and generates UI distribution information.
  • client B 22 selects the content app in operation 400, it is preferable to select which client among the proximal clients and which UI element group will be matched for displaying the UI element group.
  • the local server 100 requests client A (21) to display the UI element group.
  • the request to display in operation 440 may be made before operation 440.
  • the local server 100 may make the UI element display module 320 to execute the UI element display by transmitting a push message to the user's app.
  • the user's app of client A (21) receiving a request to display the UI element group executes the UI element group display module 320.
  • client A may display the UI element group by receiving the video streaming received from the local server 100 in operation 472.
  • the local server 100 In operation 460, the local server 100 generates virtual display information by using the UI distribution information generated in operation 430.
  • the virtual display information means information stored to render UI element group information in a graphic buffer. Therefore, the graphic buffer may reflect the UI distribution information by using the virtual display information and transmit the video stream to each device.
  • the local server 100 transmits one or more UI element groups corresponding to client A (21) and client B (22) respectively, as a video stream by using the UI distribution information. Since the local server 100 transmits the UI element group as a video stream, it is not necessary to install and execute the content application in client A (21) and client B (22).
  • the local server 100 receives UI control information from client A (21) and client B (22).
  • the local server 100 converts the received UI control information to a control signal of the content app.
  • the UI control information may be transmitted to the local server through the TCP/IP protocol or the web API.
  • the local server 100 may convert the UI control information received from client A (21) and client B (22) into the control signal of the content app in consideration of display size and resolution information of each client possessed by the local server 100. Further, the user control information may be transmitted to the local server by providing a separate UI together with the video stream received from the user's app installed in the client device. In this case, the client device needs to convert the user control signal and transmit the converted user control signal to the local server.
  • the local server 100 when the local server 100 receives a coordinate value input by the user on the display of client A (21), it is necessary to determine which element of the content app UI corresponds to the received coordinate value in consideration of the display size and resolution information. The determination may also be performed in client A (21) and may also be performed in the local server 100. Even though the local server 100 receives the coordinate value input by the user on the display of client A (21), the local server 100 has the display size and resolution information of client A (21), so that the local server 100 may determine which part of the content app may corresponds to the received coordinate value.
  • the UI control information may include content control information and UI distribution editing information, and the content control and the UI distribution editing are performed by the following method.
  • a context state value (or a parameter setting value of the content app) of the result of the UI distribution and its editing of selected UI by the clients, 21 and 22 may be transmitted to the local server 100.
  • the local server 100 performs with the authority to call the functions of the client 21 and 22 necessary for collecting the context state value.
  • the local server 100 it is not necessary for the local server 100 to transmit display data for the UI distribution editing to the client 21 and 22, and when the display information is generated and the UI editing is completed in the client 21 and 22, it is preferable that the client 21 and 22 change the context state value and provide the changed context state value to the content app of the local server 100.
  • the local server 100 provides the information necessary for the information of UI distribution and its editing to the clients, 21 and 22 as a video stream.
  • the client 21 and 22 receives the UI distribution display transmitted by the local server 100 and transmits touch screen response results of the users in the client 21 and 22 to the local server 100
  • the local server 100 transmits the touch screen response as user interaction information to the content app.
  • the local server applies the updated UI information of the content app to the UI distribution information and transmits the UI distribution information as a video stream.
  • the user is capable of performing to edit a UI distribution.
  • the transmission of the video stream is possible independent to the OSs of the client 21 and 22, the client 21 and 22 may analyze the user's control intention based on the response information of the touch screen or the touch pad and transmit a corresponding result to the local server 100.
  • FIG. 5 is a flowchart illustrating a method of transmitting, by the local server, UI display information to proximal clients according to another exemplary embodiment of the present invention.
  • the method of transmitting UI display information according to the exemplary embodiment includes operations time-serially processed in the local server and the client illustrated in FIG. 3. Accordingly, even if omitted below, the contents described above with respect to the local server and the client illustrated in FIG. 3 are also applied to the method of transmitting the UI display information according to the exemplary embodiment.
  • client A (21) receives the UI element group from the local server 100.
  • Client A (21) and client B (22) are connected with the local server 100 through the home network, and are capable of mutually calling through the HTTP-based web API (REST service or RPC mechanism).
  • HTTP-based web API REST service or RPC mechanism
  • Operation 400 to operation 460 are as illustrated in FIG. 4.
  • the local server 100 transmits a video stream for at least one UI element group to the matched client B (22) by using the UI distribution information.
  • the local server 100 transmits at least one UI element group corresponding to client A 21 by using the UI distribution information.
  • the virtual display generating module 330 since the local server 100 transmits the UI element group, the virtual display generating module 330 generates virtual display information by using the UI element group.
  • client A (21) since client A (21) generates the virtual display information by using the received UI element group, client A (21) may have a function of a sub local server performing pixel streaming on another client.
  • client A (21) may have a function of a sub local server performing pixel streaming on another client.
  • An example of the case where the client has the function of the sub local server will be described in detail with reference to FIG. 7.
  • the local server 100 receives UI control information from client A (21) and client B (22).
  • the local server 100 converts the received UI control information to a control signal of the content app.
  • the local server 100 analyzes user interaction of the received UI control information, converts the user interaction into a control signal, and inputs the control signal to the content app that is being executed in the local server 100.
  • FIG. 6 is a flowchart illustrating a method of transmitting, by the local server, UI display information to proximal clients according to still another exemplary embodiment of the present invention.
  • the difference between the method of transmitting UI display information to the proximal clients by the local server 100 illustrated in FIG. 6 and the method illustrated in FIG. 5 is that both client A (21) and client B (22) receive the UI element groups from the local server 100.
  • clients 21 and 22 may have a function of a sub local server performing pixel streaming on another client.
  • Operation 400 to operation 440 are as illustrated in FIG. 4.
  • the local server 100 requests client B 22 to display the UI element group similar to operation 440.
  • client A (21) and client B (22) receiving the request to display the UI element group execute the UI element display modules 320 and 321, respectively.
  • the local server 100 In operation 460, the local server 100 generates virtual display information by using the UI distribution information generated in operation 430.
  • the client 21 and 22 may receive the UI element groups from the local server 100 in operations 473 and 474.
  • client A (21) and client B (22) Similar to operation 460 in which the local server 100 generates the virtual display information, in operations 475 and 476, client A (21) and client B (22) generate virtual display information by using the UI element groups received from the local server 100. In this case, client A/B may transmit a first UI element to itself as a video stream, and may transmit a second UI element to another user device as a video stream by using the generated virtual display information.
  • the local server 100 receives UI control information from client A 21 and client B 22.
  • the local server 100 converts the received UI control information to a control signal of the content app.
  • client A (21) or client B (22) may be defined as a local server candidate having the same authority as that of the local server 100.
  • the UI distribution information generating module 140 of the local server is synchronized to the client defined as the local server candidate by the web RPC or REST service method, the content app installed in the local server needs to be installed even in the client, and the local server 100 and the content app of the client need to be synchronized by the web API method.
  • the local server candidate client having the authority of the local server may serve as the local server by using the information of the synchronized elements.
  • the matched UI element group information may be transmitted to the user's app for the present invention of the client device by using a transport layer for each OS, and then, the client device may generate a virtual display and perform video streaming by using the UI element group.
  • FIG. 7 illustrates a concept of transmitting a UI element group by a relay method when the local server transmits UI display information to proximal clients according to the exemplary embodiment of the present invention.
  • the local server extracts UI tree information included in the original content, virtualizes the single UI, and then divides the UI element groups constituting the UI tree information into group 1 and group 2.
  • the UI elements group divided into group 1 are stored as pixel data of a first graphic buffer, and the UI element groups divided into group 2 are stored without being converted to the pixel data.
  • the client device A forwards the received UI element group 2 to devices of clients B and C.
  • Clients B and C may forward and provide the UI display information between the devices by rendering the UI element group 2 by themselves and providing a UI display.
  • clients B and C of group 2 may generate UI control information and transmit the generated UI control information to client A, and client A may transmit its UI control information and the control information received from clients B and C to the local server.
  • client A may transmit its UI control information and the control information received from clients B and C to the local server.
  • the user's app of the present invention may be configured so that the UI control information of clients B and C is directly transmitted to the local server without passing through client A.
  • FIG. 8 is a flowchart illustrating a method of dividing, by the local server, UI tree information illustrated in FIG. 7 and transmitting the divided UI tree information to the client according to the exemplary embodiment of the present invention.
  • the method of dividing UI tree information and transmitting the divided UI tree information to the client according to the exemplary embodiment includes operations time-serially processed in the local server and the client illustrated in FIG. 3. Accordingly, even if omitted below, the contents described above with respect to the local server and the client illustrated in FIG. 3 are also applied to the method of transmitting the UI display information according to the exemplary embodiment.
  • the local server 100 divides the UI tree information included in the content in consideration of the number of clients.
  • the local server 100 virtualizes the single UI tree.
  • the virtualization means managing the UI display information of the user's device as a single UI tree in the content application, and managing the single UI tree as a UI group divided into UI element groups at the OS level. Accordingly, the main function of the virtualization is to manage the divided UI display information to be transmitted to the plurality of user devices as the information of UI element groups, and it is possible to prevent a malfunction according to the UI tree virtualization by transmitting the content application in which user interaction information for the UI display information, divided into the plurality of UI groups and transmitted to the plurality of devices, is collected and managed as the single UI tree.
  • the update of the user interaction information provides the user interaction information to the content application when a variable value of the main function of the UI element group is changed within a predetermined update cycle.
  • the local server 100 receives display resolution from client A (21).
  • the local server 100 generates a graphic buffer (for example, the virtual display) having the same resolution as the received display resolution.
  • a graphic buffer for example, the virtual display
  • the local server 100 renders the UI tree information divided in operation 800 to the graphic buffer corresponding to the client.
  • the local server 100 encodes pixel data stored in the graphic buffer and transmits the encoded pixel data to the corresponding client.
  • client A 21 decodes the received pixel data to a video stream and renders the video stream.
  • FIG. 9 illustrates a flow of a method of editing a UI of a content app by a client according to the exemplary embodiment of the present invention.
  • the method of editing a UI of a content app by the client includes operations time-serially processed in the local server and the client illustrated in FIG. 3. Accordingly, even if omitted below, the contents described above with respect to the local server and the client illustrated in FIG. 3 are also applied to the method of transmitting the UI display information according to the exemplary embodiment.
  • client A (21) transmits selected content app to the local server 100.
  • the local server 100 executes the selected content app.
  • the local server 100 may download the content app through the cloud server 300 and execute the downloaded content app.
  • client A (21) receives the UI of the content app from the local server 100 by video streaming.
  • client A (21) displays the received UI of the content app.
  • client A (21) executes a UI setting mode of the content app.
  • client A (21) selects at least one UI element group of the content app, and selects a client which is to display the selected UI element group.
  • client A (21) transmits UI distribution information matching client information to the selected UI element group for display to the local server 100.
  • the local server 100 transmits a UI element group to the matched client that displays the UI element group with reference to the UI distribution information.
  • FIG. 10 illustrates a flow of a method of transmitting, by the local server, IoT state information to the client device in an OS level according to another exemplary embodiment of the present invention.
  • the method of transmitting IoT state information to the client according to the exemplary embodiment includes operations time-serially processed in the local server and the client illustrated in FIG. 3. Accordingly, even if omitted below, the contents described above with respect to the local server and the client illustrated in FIG. 3 are also applied to the method of transmitting the IoT state information according to the exemplary embodiment.
  • the local server 100 receives information about the IoT application selected for receiving IoT state information by the user of client A (21).
  • the selectable IoT applications are listed in client A (21), and when the user selects any one IoT application, the local server 100 receives which IoT application is selected as user UI control information.
  • the local server 100 executes the selected IoT application.
  • the local server 100 may download the IoT application through the cloud server 300 of the IoT device.
  • the local server 100 divides a UI of the executed IoT application into a plurality of UI element groups.
  • the local server 100 generates UI distribution information by using the plurality of divided UI elements groups.
  • the local server 100 In operation 1040, the local server 100 generates virtual display information by using the generated UI distribution information.
  • the local server 100 may transmit the virtual display information to client A (21) by video streaming or transmit the matched UI element group to the client device.
  • the local server 100 receives UI control information from the user's app of client A (21).
  • the local server 100 converts the received UI control information to a control signal of the IoT application.
  • the local server 100 transmits the IoT control signal to an IoT device 25 through the cloud or the home network.
  • the IoT device 25 may be controlled through the home network using ZigBee/Wi-Fi.
  • the IoT cloud server transmits the IoT control signal to the IoT device through the IoT hub by the TCP/IP communication by referring to a IP address information uniquely matched to the IoT device 25.
  • the IoT app of the local server 100 receives IoT state information through the IoT device 25 or the IoT cloud server.
  • the local server 100 transmits the IoT state information to the user's app of client A (21) as the video stream or the UI element group information.
  • FIG. 11 illustrates a flow of a method of transmitting, by the cloud server, IoT state information to the client device according to another exemplary embodiment of the present invention.
  • the method of transmitting IoT state information to the client according to the exemplary embodiment includes operations time-serially processed in the local server and the user of the client illustrated in FIG. 3. Accordingly, even if omitted below, the contents described above with respect to the local server and the user app of the client illustrated in FIG. 3 are also applied to the method of transmitting the IoT state information according to the exemplary embodiment.
  • FIG. 11 is different from FIG. 10 in that client A (21) located outside the home network receives the IoT state information through the cloud server 300.
  • the UI virtualization information of the cloud server 300 and the local server 100, the UI distribution information, the user device information matched to the UI distribution information, and the like are synchronized through the web API.
  • the cloud server 300 receives user control information for selecting an IoT application by the user from client A (21).
  • the cloud server 300 determines whether the device of client A (21) is a registered user device in the user device group information.
  • the local server 100 receives information about the IoT application selected by the user of client A (21) from the cloud server 300.
  • the local server 100 executes the selected IoT application.
  • the local server 100 divides a UI of the executed IoT application into a plurality of UI element groups at an OS level.
  • the local server 100 In operation 1150, the local server 100 generates UI distribution information by using the plurality of divided UI elements groups. Further, the local server 100transmits the updated UI distribution information of the local server to the cloud server to synchronizes the UI distribution information between two devices.
  • the local server 100 In operation 1160, the local server 100 generates the UI of the IoT app of the local server, synchronized with the IoT app of the cloud server, as virtual display information by using the generated UI distribution information.
  • the local server 100 transmits a UI display of the matched IoT app to the device of client A (21) by video streaming by using the virtual display information, or transmit the matched UI element group information.
  • client A (21) transmits UI control information to the cloud server 300, and when client A (21) transmits the UI control information to the IoT app installed in the cloud server, context state information of the IoT app of the local server is updated by the cloud server of the IoT device.
  • the IoT app of the local server 100 transmits the received UI control information to the IoT device.
  • the IoT app of the local server 100 transmits IoT control signal to the IoT cloud server through the Internet web, or when the local server serves as the IoT hub, the local server directly transmits the IoT control signal to the IoT device through the home network.
  • the IoT app of the local server 100 receives the state information of the IoT device via the cloud of the IoT app and the IoT hub, or directly receives the state information from the IoT device through the home network when the local server serves as the IoT hub.
  • the IoT app when the context state information of the IoT app of the local server 100 is changed, the IoT app provides the cloud of the IoT device with the changed state information, or transmits the changed state information to the IoT app of the cloud server 300 when the local server serves as the IoT hub.
  • the hub for controlling the IoT device controls the IoT device on the home network by using ZigBee, Bluetooth, or Wi-Fi network.
  • FIG. 12 illustrates a flow of a method of transmitting, by the local server, IoT state information to an unauthorized client according to another exemplary embodiment of the present invention.
  • the method of transmitting, by the local server, the IoT state information to a non-authorized client is different from the methods illustrated in FIGS. 10 and 11 in that a non-authorized client 30 requests the local server 100 to transmit the IoT state information.
  • the local server 100 sets IoT state information providing range according to a distance to the client device.
  • the local server 100 may differentiate and provide the IoT state information according to the location of the non-authorized client 30. There is a case where the non-authorized client 30 is physically close to the local server 100 and there is a case where the non-authorized client 30 requests the IoT state information through the home network or the Internet at a remote distance, and it is possible to differentiate and provide the IoT state information according to the physical distance between the local server 100 and the non-authorized client 30.
  • IoT state information when the location of the non-authorized client 30 is physically present within a predetermined distance from the local 100, more IoT state information may be set to be provided.
  • the local server 100 may differentiate and provide the state information of the IoT device 25.
  • the local server 100 receives a transmission request of an IoT state information from the user's app of the non-authorized client 30.
  • the local server 100 calculates a distance to the non-authorized client 30 based on the transmission request of IoT state information received in operation 1210.
  • the paired antennas may be implemented to receive horizontal polarization and some other antennas may be implemented to receive vertical polarization.
  • the local server 100 may receive a wireless communication signal from the non-authorized client 30 through the paired vertical polarization/horizontal polarization multi-antennas, and accurately calculate a location of the non-authorized client 30 by a arrival time difference according to a difference in the incident angles of the received signals between the paired multi-antennas.
  • the local server 100 may determine the IoT state information to be provided to the user's app of the non-authorized client 30 by calculating the distance between the local server 100 and the non-authorized client 30.
  • the local server 100 transmits the transmission request of the IoT state information to the IoT device 25.
  • the local server 100 receives the IoT state information from the IoT device 25.
  • the local server 100 determines the IoT state information and the UI distribution information to be transmitted to the non-authorized client 30 according to the distance calculated in operation 1220.
  • the local server 100 In operation 1260, the local server 100 generates virtual display information by using the UI distribution information.
  • the local server 100 transmits the IoT state information as a video streaming to the user's app of a non-authorized client device.
  • the IoT state information is transmitted from the local server 100 to the non-authorized client 30 as a video stream and is displayed on a screen. Accordingly, the installation of the IoT application, required for receiving the IoT state information of plurality of IoT devices, is not required in the non-authorized client 30.
  • FIG. 13 illustrates a flow of a method of receiving an NFT content and a UI app, by the local server, and transmitting UI display information according to another exemplary embodiment of the present invention.
  • the method of transmitting UI display information according to the exemplary embodiment includes operations time-serially processed in the local server and the client illustrated in FIG. 3. Accordingly, even if omitted below, the contents described above with respect to the local server and the client illustrated in FIG. 3 are also applied to the method of transmitting the UI display information according to the exemplary embodiment.
  • FIG. 13 illustrates a method in which a local server 20 or client A (21) uses a UI app including NFT content available for a predetermined period by transmitting the NFT content of an NFT content sever 500 and the UI app of a UI app server 600 to the local server 20.
  • the user mutually concludes a smart contract with an NFT content provider 500 and a UI app provider 600.
  • Digital assets are managed by blockchain-based smart contracts, and digital asset tokens are largely divided into Fungible Tokens (FT) and Non-Fungible Tokens (NFT).
  • FT Fungible Tokens
  • NFT Non-Fungible Tokens
  • ERC-20 is defined as a fungible token
  • ERC-721 token is defined as a non-fungible token
  • ERC-1155 is a token that can manage ERC-20 and ERC-721 tokens through a single smart contract.
  • the NFT content provider/server 500, the UI app provider/server 600, and the user conclude a smart contract related to ownership/use of the NFT content and the UI app based on ERC-1155.
  • the UI app provider/server 600, the NFT content provider/server 500, and the user/local server 20 conclude the smart contract effective for a specific period of time
  • the NFT content and the UI app may be transmitted to the local server 20.
  • the local server 20 is granted to access the NFT contents for the predetermined period of time and is allowed to access as a sole user.
  • the smart contract may include a condition of limiting display transmission of the NFT content transmitted to the plurality of user devices , which is a present invention limiting the number of authorized user devices included in the user device group.
  • client A (21) selects the NFT content and the UI app to be used and transmits the selected NFT content and UI app to the local server 100.
  • the local server 20 may generate UI distribution information by dividing a single UI of the UI app into the units of UI element groups at the OS level and matching the divided UI element groups to the client devices of the user, and transmit one UI element group to the matched client A (21).
  • Client A (21) may select one of the plurality of selectable UI apps stored in the UI app server 600, combine the selected UI app with the NFT content, and display the combined NFT content.
  • the local server 100 checks whether the smart contract is concluded between the UI app server 600 and the NFT content server 500, and then receives the UI app selected by client A (21) from the UI app server 600.
  • the local server 100 checks whether the smart contract is concluded between the UI app server 600 and the NFT content server 500 and then receives the NFT content from the NFT content server 500.
  • the local server 100 executes the selected UI app.
  • the local server 100 divides a UI of the executed UI app into a plurality of UI element groups at an OS level.
  • the local server 100 matches the divided each UI element group to the each client device of the user respectively, and generates UI distribution information.
  • the UI distribution control module 105 transmits the generated UI distribution information to the cloud server 300.
  • the local server 100 In operation 1360, the local server 100 generates virtual display information by using the generated UI distribution information.
  • the local server 100 transmits the UI element group to client A (21). Instead of transmitting the UI element group to client A (21), the local server 100 may transmit a video stream for the at least one UI element group matching to client A (21).
  • client A (21) generates virtual display information by using the UI element group received from the local server 100.
  • the local server 100 receives UI control information from client A (21).
  • the local server 100 converts the UI control information to a control signal of the UI app.
  • the local server 100 may apply the control signal of the UI app and re-transmit the UI element group or the video stream.
  • the local server 100 may delegate the authority to use the UI app and the NFT content to client A (21).
  • the smart contract between the UI app server 600 and the NFT content server 500 is updated including client A (21), instead of the local server 100, and the UI distribution control module 105 is executable in client A (21).
  • the cloud server 300 may manage information, such as the user's authority to access to the NFT content, the authority to provide the UI by using the UI app, the authority to correct the UI distribution information, and the delegation of the authority to use.
  • the NFT content user may be delegated all of the above authorities, use the NFT content for a predetermined period of time, and delegate the authority to another user to distribute the NFT content.
  • the NFT content user may stream and use the NFT content in his/her display device by changing the UI with a style that he/she wants for a predetermined period of time. Then, when the predetermined period of time elapses based on the contents of the smart contract, the authorities to use of all of the users, except for the content user may be reset.
  • the exemplary embodiments of the present invention may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer readable medium.
  • the computer readable medium may include a program command, a data file, a data structure, and the like alone or in combination.
  • the program command recorded in the medium may be specially designed and configured for the present invention, or may also be known and usable to those skilled in computer software.
  • Examples of the computer readable recording medium include a magnetic medium, such as a hard disk, a floppy disk, or a magnetic tape, an optical recording medium, such as a CD-ROM or a DVD, a magneto-optical medium, such as a floptical disk, and a hardware device which is specifically configured to store and execute the program command such as a ROM, a RAM, and a flash memory.
  • An example of the program command includes a high-level language code executable by a computer by using an interpreter, and the like, as well as a machine language code created by a compiler.
  • the hardware device may be configured to be operated with one or more software modules in order to perform the operation of the present invention, and an opposite situation thereof is available.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Software Systems (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Graphics (AREA)
  • Information Transfer Between Computers (AREA)
PCT/KR2023/002793 2022-02-28 2023-02-28 Method of transmitting ui display information to proximal clients by local server WO2023163566A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
KR1020220026426A KR20230128913A (ko) 2022-02-28 2022-02-28 로컬 서버가 주변 클라이언트들로 ui 화면 정보를 전송하는 방법
KR10-2022-0026426 2022-02-28

Publications (1)

Publication Number Publication Date
WO2023163566A1 true WO2023163566A1 (en) 2023-08-31

Family

ID=87766506

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/KR2023/002793 WO2023163566A1 (en) 2022-02-28 2023-02-28 Method of transmitting ui display information to proximal clients by local server

Country Status (2)

Country Link
KR (1) KR20230128913A (ko)
WO (1) WO2023163566A1 (ko)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110154287A1 (en) * 2009-12-18 2011-06-23 Sybase, Inc. Visual Generation of Mobile Applications Based on Data Models
US20170161031A1 (en) * 2014-07-07 2017-06-08 Samsung Electronics Co., Ltd. Method for sharing application program and electronic device therefor
US20190373058A1 (en) * 2018-06-03 2019-12-05 Apple Inc. Data synchronization across multiple devices
US20200028753A1 (en) * 2018-07-20 2020-01-23 Paypal, Inc. System architecture for dynamically rendering a customized user interface on a mobile device

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110154287A1 (en) * 2009-12-18 2011-06-23 Sybase, Inc. Visual Generation of Mobile Applications Based on Data Models
US20170161031A1 (en) * 2014-07-07 2017-06-08 Samsung Electronics Co., Ltd. Method for sharing application program and electronic device therefor
US20190373058A1 (en) * 2018-06-03 2019-12-05 Apple Inc. Data synchronization across multiple devices
US20200028753A1 (en) * 2018-07-20 2020-01-23 Paypal, Inc. System architecture for dynamically rendering a customized user interface on a mobile device

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
SANGEUN OH , AHYEON KIM , SUNJAE LEE , KILHO LEE , SAE R JEONG , STEVEN Y KO , INSIK SHIN: "FLUID Flexible User Interface Distribution for Ubiquitous Multi-device Interaction", MOBILE COMPUTING AND NETWORKING, ACM, 2 PENN PLAZA, SUITE 701NEW YORKNY10121-0701USA, 11 October 2019 (2019-10-11) - 25 October 2019 (2019-10-25), 2 Penn Plaza, Suite 701New YorkNY10121-0701USA , pages 1 - 16, XP058475514, ISBN: 978-1-4503-6169-9, DOI: 10.1145/3300061.3345443 *

Also Published As

Publication number Publication date
KR20230128913A (ko) 2023-09-05

Similar Documents

Publication Publication Date Title
WO2020190016A1 (en) Method and device for providing authentication in network-based media processing (nbmp) system
WO2015041438A1 (en) Method for screen mirroring and source device thereof
WO2014014302A1 (en) Method and device for sharing content
WO2016068622A1 (en) Terminal device and method of controlling same
WO2022050652A1 (en) Method, apparatus, and computer readable storage medium for controlling account
WO2013151374A1 (en) Method and system for transfering data between plurality of devices
WO2016027933A1 (ko) 디지털 디바이스 및 그 제어 방법
WO2012112011A2 (ko) 심리스 콘텐트 재생 방법 및 장치
WO2015016627A1 (ko) 동일한 네트워크 상의 복수개의 ap 장치 중 하나의 ap 장치와 단말기를 연결하는 방법 및 장치
WO2009126009A2 (ko) 유비쿼터스 환경의 양방향 컨텐츠 서비스를 제공하기 위한 방법, 시스템 및 컴퓨터 판독 가능한 기록 매체
WO2015137604A1 (ko) 클라우드 스트리밍 서버 테스트 방법, 이를 위한 장치 및 시스템
WO2021133042A1 (en) Electronic device and method of operating the same
WO2021150060A1 (en) Method and apparatus for edge computing service
WO2020032465A1 (en) Method for contents playback with continuity and electronic device therefor
WO2022060175A1 (ko) 엣지 컴퓨팅 서비스를 이용한 영상 컨텐츠 전송 방법 및 장치
WO2019013415A1 (ko) Vr 콘텐츠에 대한 라이브 스트리밍 방법 및 시스템
WO2017034298A1 (ko) 디지털 디바이스 및 상기 디지털 디바이스에서 데이터 처리 방법
WO2022075737A1 (ko) Mec 서비스를 제공하기 위한 방법 및 장치
WO2022211465A1 (en) Apparatus and method for providing media streaming
WO2023163566A1 (en) Method of transmitting ui display information to proximal clients by local server
WO2023163569A1 (en) Method of transceiving ui display between user devices
WO2023163567A1 (en) Method of transmitting state information of iot device by local server
WO2023163568A1 (en) Method of transmitting ui display information based on smart contract
WO2020149722A1 (en) Working environment provisioning method and apparatus for execution of application program between electronic device and external server
WO2014092289A1 (ko) 클라우드 서비스 제공 방법, 이를 위한 시스템 및 장치

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23760440

Country of ref document: EP

Kind code of ref document: A1