WO2023163569A1

WO2023163569A1 - Method of transceiving ui display between user devices

Info

Publication number: WO2023163569A1
Application number: PCT/KR2023/002798
Authority: WO
Inventors: In Sik Shin; Daehee Lee; Sun Jae Lee
Original assignee: Fluiz Co. Ltd.
Priority date: 2022-02-28
Filing date: 2023-02-28
Publication date: 2023-08-31
Also published as: KR20230128916A

Abstract

The present invention relates to a method of controlling, by a local server, UI display information together with proximal clients, the method including: after a content application selected by any one client is executed in the local server, dividing a single UI, the UI provided by the content application, into a plurality of UI element groups, generating UI distribution information by matching each proximal client to one or more of the plurality of UI element groups respectively, requesting, by the local server, the proximal clients to receive the matched UI element group or a video stream, receiving, by the proximal clients, the matched UI element group or the video stream, transmitting one or more of the UI element groups to the matched proximal clients respectively or transmitting video streams for the one or more UI element groups to the matched proximal clients respectively by using the UI distribution information, displaying, by the proximal clients, the one or more received UI element groups or the video streams and receiving UI control information and transmitting the received UI control information to the local server, and receiving, by the local server, the UI control information from the proximal client and transmitting the received UI control information to the content application, in which the UI display information of the content application is transceived in the unit of the UI element group, thereby variously combining and providing the UI element groups to a plurality of devices.

Description

METHOD OF TRANSCEIVING UI DISPLAY BETWEEN USER DEVICES

The present invention relates to a method of controlling UI display information together with proximal clients by a local server, and more particularly, to a method of controlling UI display information, which transmits UI display information of a content application in a unit of a UI element group to variously combine UI element groups and provide the combined UI element groups to a plurality of devices.

In the prior art, in a technology for sharing User Interface (UI) display information between two or more display devices, it is common to mirror one display to another display or divide the display into a plurality of areas to display different contents in each divided area.

In particular, the mirroring technology is inefficient because the same display is output on multiple display devices and there are devices that output the display unnecessarily. Further, when the screen is divided into a plurality of areas and used, since the display in one device is divided and used, there is a problem in that the UI display information cannot be distributed and used in several display devices.

Further, in order to control an IoT device in a smart device, or check state information of the IoT device, a separate application capable of accessing the IoT device needs to be inconveniently installed and there is a problem in that as the number of IoT devices increases, the number of installed applications continues to increase.

Prior Art 1 (Korean Patent Application Laid-Open No. 10-2018-0081133) is the prior art related to a rapid screen dividing method, and is characterized in that when a finger joint touch gesture acting on a touch-sensitive surface is detected, a display UI of a display is divided into at least two display areas in response to the finger joint touch gesture.

Referring to Prior Art 1, only the method of easily dividing the screen is disclosed, and the configuration of whether the divided screen can be efficiently displayed in cooperation with proximal devices is not disclosed.

Prior Art 2 (Korean Patent Application Laid-Open No. 10-2010-0081383) is related to the touch screen dividing method, and is characterized in that a divided area layout of a touch screen is formed by receiving an input of two positions on the touch screen and a separate display is allocated to each of the formed divided areas. Prior Art 2 also discloses the method of dividing the screen in one device, so that there is no motivation to display the divided screen on the plurality of devices in cooperation with the plurality of devices.

Prior Art 3 (US Patent Application Publication No. US20190373058 A1) is characterized in that user interaction information is synchronized by transmitting user interaction information between a plurality of devices to a remote server. In Prior Art 3, there is no motivation to display a divided screen in a plurality of devices in cooperation with the plurality of devices.

Further, in the related art, when various display devices share a UI display, there is a problem in that an application for displaying the corresponding UI display must be installed in all of the display devices for the execution. Further, in the case of different OSs between display devices, it is difficult to share a UI display, so that there is difficulty in sharing UI information between multiple devices.

Therefore, the first problem to be solved by the present invention is to provide a method of controlling, by a local server capable of variously combining UI element groups and distributing the combined UI element groups to a plurality of devices, UI display information together with proximal clients’ devices by performing UI display information transception of a content application in a unit of a dividable UI element group.

The second problem to be solved by the present invention is to provide a system for controlling UI display information, which is capable of transmitting UI display information independently from an OS by transmitting a UI element group included in UI distribution information as a video streaming.

Further, the present invention provides a computer readable recording medium in which a program for executing the method in a computer is recorded.

In order to achieve the first problem, the present invention provides a method of controlling, by a local server, UI display information together with proximal clients’ devices, the method including: after a content application selected in any one client is executed in the local server, dividing a single UI provided by the content application into a plurality of UI element groups, generating UI distribution information by matching the proximal clients to the plurality of UI element groups, respectively, requesting, by the local server, the proximal clients to receive the UI element group or a video stream, receiving, by the proximal clients, the UI element group or the video stream, transmitting one or more UI element groups to the matched proximal clients respectively to the proximal clients or transmitting video streams converted from one or more UI element groups matched to the proximal clients respectively by using the UI distribution information, displaying, by the proximal clients, the one or more received UI element groups or the video streams and receiving UI control information and transmitting the received UI control information to the local server, and transmitting, by the local server, the received UI control information from the proximal clients to the content application.

According to an exemplary embodiment of the present invention, the method may further include authorizing or delegating an authority to divide the UI of the content application into UI element groups to any one client among the proximal clients.

Further, when one or more UI element groups, matched to the proximal clients respectively, are transmitted as video streams, it is preferable to render and transmit the UI element group as the video streams in association with a display size and resolution of each of the proximal clients.

According to another exemplary embodiment of the present invention, the method may further include storing the UI distribution information and the display size and resolution information of the proximal clients in a cloud server.

Further, the method may further include transmitting the one or more UI element groups matched to the proximal clients respectively, receiving UI control information from the proximal clients, and then providing the received UI control information to the content application.

In order to resolve the second problem, the present invention provides a system for controlling UI display information, the system including: a local server including: a UI element dividing module which, after a content application selected in any one client is executed in the local server, divides a single UI provided by the content application into a plurality of UI element groups; a UI distribution information generation module which generates UI distribution information by matching the proximal clients to the plurality of UI element groups respectively; a display requesting module which requests, by the local server, the proximal clients to receive the UI element group or a video stream; a UI element transmitting module which transmits the one or more UI element groups to the matched plurality of proximal clients respectively, or transmitting video streams converted from the one or more UI element groups to the matched plurality of proximal clients respectively by using the UI distribution information; and a UI control information converting module which receives UI control information from the proximal clients and transmits the received UI control information to the content application; and one or more clients including: a UI element display module which receives the UI element group or receives the video stream; a UI control information input module which receives an input of the UI control information when the one or more received UI element groups or the video stream is displayed; and a transmitting module which transmits the input UI control information to the local server.

In order to solve another technical problem, the present invention provides a computer-readable recording medium in which a program for executing a method of controlling, by the local server, the UI display information together with the proximal clients in a computer is recorded.

According to the present invention, it is possible to provide various combinations of UI displays to a plurality of devices by transmitting UI display information of a content application in the unit of a UI element group.

Further, according to the present invention, it is possible to transmit UI display information independently from an OS by transmitting a video stream converted from UI element group included in UI distribution information.

Further, according to the present invention, it is possible to minimize unnecessary app installation, and monitoring and controlling a server-client by installing a content app in one local server and transmitting UI distribution information to each client device as UI display information without the need to install the same content app in all of the multiple mobile devices participating as clients.

Further, it is possible to provide content services to which various UXs are employed by granting an authority to forward an authority to access all contents to a third user, and to provide a new content-application to a digital content user with authorities to provide a new UI app, to divide a UI of the UI app, and to forward a authority to access a content of the UI app.

FIG. 1 is a conceptual diagram illustrating transmission of UI display information to proximal clients by a local server in a network consisting of a local server 100, a proximal client 200, a cloud server 300, and a content providing server 400.

FIG. 2 illustrates a concept in which the local server 100 divides a UI of an original content into UI element groups by using UI tree information and transmitting the UI element groups to

proximal clients

21 and 22 according to an exemplary embodiment of the present invention.

FIG. 3 is a configuration diagram illustrating the local server and a client according to the exemplary embodiment of the present invention.

FIG. 4 is a flowchart illustrating a method of transmitting, by the local server, UI display information to proximal clients according to the exemplary embodiment of the present invention.

FIG. 5 is a flowchart illustrating a method of transmitting, by the local server, UI display information to proximal clients according to another exemplary embodiment of the present invention.

FIG. 6 is a flowchart illustrating a method of transmitting, by the local server, UI display information to proximal clients according to still another exemplary embodiment of the present invention.

FIG. 7 illustrates a concept of transmitting a UI element group by a relay method when the local server transmits UI display information to proximal clients according to the exemplary embodiment of the present invention.

FIG. 8 is a flowchart illustrating a method of dividing, by the local server, UI tree information illustrated in FIG. 7 and transmitting the divided UI tree information to the client according to the exemplary embodiment of the present invention.

FIG. 9 illustrates a flow of a method of editing a UI of a content app by a client according to the exemplary embodiment of the present invention.

FIG. 10 illustrates a flow of a method of transmitting, by the local server, Internet of Things (IoT) state information to the client according to another exemplary embodiment of the present invention.

FIG. 11 illustrates a flow of a method of transmitting, by the local server, IoT state information to the client according to still another exemplary embodiment of the present invention.

FIG. 12 illustrates a flow of a method of transmitting, by the local server, IoT state information to an unauthorized client according to still another exemplary embodiment of the present invention.

FIG. 13 illustrates a flow of a method of receiving an NFT content and a UI app, by the local server, and transmitting UI display information according to still another exemplary embodiment of the present invention.

According to the present invention, there is a method of distributing a UI of a user device and transmitting a UI display, the method including: dividing, by a first user device, a single UI of a UI app into a plurality of UI element groups at an OS level; generating, by the first user device, UI distribution information by matching the one UI element group to a second user device; transmitting, by the first user device, the mapped UI element group or converting the matched UI element group, by using the UI distribution information, into a video stream and transmitting the video stream to the second user device; transmitting, by the first user device, user device group information including the first user device and the second user device or the UI distribution information to a cloud server; and when the user device group information or the UI distribution information is changed, receiving, by the first user device, the changed information from the cloud server and updating its UI distribution information.

Prior to the description of the specific contents of the present invention, for the convenience of understanding, the outline of the solution to the problem to be solved by the present invention or the core of the technical spirit is first presented.

The present invention maximizes service scalability of a content application in consideration of a display size or resolution of a user display device in order to transmit free UI display information between display devices connected through a communication network.

The present invention presents an efficient interactive data transmitting method for improving user experience between devices of a user group in which a plurality of display devices registered by a user or display devices registered by multiple users are included.

A video, audio, image, touch, keyboard, and voice data may be shared or mirrored and forwarded between display devices connected through a communication network. In this case, a content UI of a content application executed in the display device is composed of UI element groups, and provides a function capable of transmitting the UI in the unit of one or more individual UI element groups to a proximal display device and providing a UI element management authority function to enable various UI changes and combinations.

A local server having an authority to divide and distribute a content UI may transmit a video stream of the UI element group constituting the UI to the proximal display device. As another method, when only UI element group information is transmitted to a proximal display device, each proximal display device generates a video stream by itself, and then each proximal display device transmits interaction information of a user to a local server. The local server may input user interaction information received from each proximal display device to a content application. In this case, a user app installed in the proximal display device for the present invention may virtualize a display by using the UI element group information received from the server and transmit UI information to a plurality of user devices. Accordingly, a function supported at an OS level in the local server may be minimized to and configured as a UI dividing function (UI virtualization), UI distribution information storage, and UI element group transmission function, and then the UI transmission function using a virtual display configuration and video streaming required in the present invention may be configured to be performed in a user app installed in a user device.

Further, the local server may update the UI element group information to be transmitted to each proximal display device by reflecting UI distribution information of a content application received from a cloud server. In the case where the local server converts the UI element group information into a video stream and transmits the video stream to each proximal client device, there is an advantage in that it is possible to transmit the video stream independent to an OS of the client device in terms of a data communication aspect.

Further, the present invention presents a method of easily monitoring or controlling, by a user, a proximal client device and an IoT device.

In particular, the local server transmits a UI of an IoT application capable of controlling an IoT device to a proximal client by a video streaming method, so that it is possible to monitor state information of the IoT device even in a user’s device in which the IoT application is not installed, or control the IoT device by transmitting interaction information of the user from the user device to the local server.

Further, the present invention presents a method of managing a content authority which is capable of various user experience environments, and forwarding a content. The present invention may provide a platform environment providing a UI change authority to change a UI of a content, an authority to provide a new UI of an application, and a content access authority assignment function to a user group capable of accessing a digital content of a user.

Hereinafter, an exemplary embodiment in which the present invention can be easily carried out by those skilled in the art will be described in detail with reference to the accompanying drawings. However, the exemplary embodiments are for explaining the present invention in more detail, and it will be apparent to those skilled in the art that the scope of the present invention is not limited by the exemplary embodiments.

The configuration of the invention for clarifying the solution of the problem to be solved by the present invention will be described in detail with reference to the accompanying drawings based on the exemplary embodiment of the present invention, in assigning reference numbers to the components of the drawings, the same reference numbers are given to the same components even if they are in different drawings, and it should be noted in advance that components of other drawings can be cited when necessary in the description of the drawings. In addition, when it is determined that detailed descriptions of well-known functions or configurations related to the present invention and other matters may unnecessarily obscure the gist of the present invention in explaining the principle of operation of the exemplary embodiment of the present invention in detail, the detailed description is omitted.

In order for the local server 100 to provide display information of a content application (CA) selected by a user to a proximal client 200 device, a content application may request a content from the content providing server 400. In this case, the content application installed in the local server 100 may directly access the content providing server 400 and request the corresponding content or request the content via the cloud server 300. In this case, when the content application required for receiving the content of the content providing server 400 is not installed in the local server 100, the local server 100 may install the content application by requesting the content application from the content providing server 400 and installing the received content application.

Referring to FIG. 1, the proximal client 200 includes various user devices, such as a smart display 21,

smart phones

22 and 23, a smart light bulb 25, and a smart camera 26. The smart light bulb 25 and the smart camera 26 are an IoT device group.

The proximal client 200 may register its user device information in the local server 100.

Any one client in the proximal client 200 may request to register its own device information with the local server 100, and the local server 100 may transmit the registered user device information to the cloud server 300 which stores the user device information.

That is, the devices of the proximal client 200, such as a set top box or a hub, may be registered in the local server 100 as a user device group, and the corresponding user device information may be transmitted to and re-stored in the cloud server 300. As the user device group information, a Service Set Identifier (SSID), a MAC address, a Universally Unique Identifier (UUID), or the like may be used, the user device group information may be differently configured according to a session from the execution of an app to the end of the app, or a time period from an installation of an app to a deletion of the app. The user device group information may further include display size and resolution information.

The proximal client 200 and the local server 100 may be connected by using near-field communication, such as Bluetooth, ultrasound, or Zigbee, and an initial communication connection and authentication may be performed. Thereafter, UI display information between the authorized devices can be transmitted by using a home network connected by Wi-Fi.

Further, in a connection of a Wi-Fi router between the proximal client 200 and the local server 100 for data communication, a new user device may be added to a Wi-Fi communication network by transmitting, by an initial smart phone or set top box connected with the Wi-Fi router, Wi-Fi connection information to another smart device or IoT device. When the Wi-Fi router serves as the local server 100, the set top box serves as the proximal client 200.

When the local server 100 transmits UI display information to the proximal clients 200, the divided UIs of the content application may be shared and viewed by multiple clients.

The local server 100 receives a content of the content providing server 400 and transmits the received content to the proximal clients 200. In this case, the UI display information of the content may be divided, matched, and transmitted for each client by a UI distribution control module 105 (see FIG. 3) implemented in the local server 100.

In particular, according to the exemplary embodiment of the present invention, the UI distribution control module 105 installed in the local server 100 does not divide UI display information of the content by modifying a source code of an application of the content, but has a function of dividing UI tree information constituting UI display information, at an Operating System (OS) level, into UI element groups; and rendering module is implemented, thereby dividing the UI display information and transmitting the divided UI display information to the plurality of user devices. The content application installed in the local server 100 is received from the content providing server 400, and the local server is implemented in the form of a microkernel providing a minimum function required at the OS level to divide single UI information of the content to a plurality of UI element groups. Further, the local server or an application, which is installed in the local server and includes a partial function of a UI editing control module, may match the plurality of UI element groups to proximal client devices and transmits the plurality of UI element groups by a video streaming method. Accordingly, the proximal clients 200 may receive a UI video stream without installing the content application installed in the local server. Further, the local server or the application installed in the local server may transmit the matched UI element groups to the proximal client 200 devices without the conversion to the video stream. In this case, the user application installed in the client for the present invention may convert the UI element group into a video stream and provide the UI display information to the client device.

In the present invention, in order for the local server to easily distribute and transmit the UI to the proximal clients 200, an authority or a function of dividing and virtualizing UI tree information in the content at the OS level is required. The local server 100 or the separate application installed in the local server may generate UI distribution information that matches the proximal clients 200 to the UI element group constituting the content UI display information by using the virtualized UI tree information.

For example, an image of a movie content may be matched and transmitted to the smart display 21, a chatting window may be matched and transmitted to the first smart phone 22, and the subtitles of the movie may be matched and transmitted to the second smart phone 23.

Therefore, the function of the present invention divides the single UI display information of the content application into each separable UI element group in the OS of the user device corresponding to the local server. Thereafter, the local server or the application installed in the local server may generate UI distribution information of the user device group by matching other user devices, corresponding to the client devices, to the UI element groups. The UI distribution information may be backed up to the cloud server 300, and the backed-up UI distribution information may be stored as related information of a device group for each user. When another client of the user is added to the user device group, a UI element to be matched to the added client may be updated in the UI distribution information.

When a client of another user is invited and included in a device group for each user, the client of another user may also be stored to the UI distribution information and updated. The UI distribution information stored in the cloud server 300 may be shared or copied to the client of the invited user, and the UI distribution information may be updated by applying only the client that needs to be added or changed to the UI distribution information or be stored as separate UI distribution information selectable by a user.

A user device group information and the UI distribution information stored in the cloud server 300 may be used when the local server 100 has a failure or a physical change of the local server is required. Further, the UI distribution information of the local server 100 and the proximal clients may be easily modified by copying or modifying existing UI distribution information of the user stored in the cloud server 300. In this case, the clients may be successfully authenticated to transmit the Wi-Fi network information to the proximal client and the local server to simplify local network setting.

The user generally monitors the state of the smart light 25 or the smart camera 26 registered in the device group for each user or removes the smart light 25 or the smart camera 26 through the content application. As an example of UI element group, when the content image is dark, a control signal for brightening the smart light may be included in the UI distribution information, and in the movie mode, a control signal for dimming the smart light may be included in the UI distribution information.

In general, the smart light 25 or the smart camera 26 among the proximal clients 200 does not have a separate display, so that it is necessary to download and install a necessary application (app) to the

smart phones

22 and 23 or the smart display 21 of the user and sets the

smart phones

22 and 23 or the smart display 21, or it is necessary to set the smart light 25 or the smart camera 26 by accessing a homepage of a manufacturing company.

In this case, it is necessary to install an app for each manufacturing company required for the smart light 25 or the smart camera 26, and the corresponding app may be installed in a smart device linked to a user account. Further, in order to control the smart light 25 or the smart camera 26, the inconvenience of having to perform the same operation by each user.

In the present invention, in the case where a user configuring the initial UI distribution information invites another user to a device group after a content app required is installed only in the local server, another user may receive UI information for IoT monitoring and control, required content information of the IoT device (lighting state, camera information, and IoT device information), from the local server. In this case, a new user is capable of receiving the required UI display as a video stream only by installing a UI application of the present invention without installing the applications of various IoT devices, so that the new user is capable of receiving and controlling the video stream independent to the OS (Android, IOS, Window, Linux, and the like) of the user client device.

The

IoT devices

25 and 26 are capable of transmitting the state change information to the local server 100 connected through the home network or transmitting the cloud server 300 connected through the Internet network. When the IoT app is installed in the cloud server 300, the cloud server 300 may provide the user who is allowed to access IoT state information with the state information of the IoT device through the Internet network corresponding to the present invention. In this case, when a context state of the IoT app installed in the local server or the cloud is updated, the two IoT apps are automatically synchronized.

As another exemplary embodiment, the IoT app required for monitoring and controlling the IOT device, such as the smart light 25 or the smart camera 26, may also be installed in the local server 100.

The local server 100 may allocate the UI element group for each proximal client 200 by using the UI tree information included in the content and match the UI element group to the proximal client 200.

The local server 100 configures required UI display information corresponding to the UI distribution information and display sizes and resolution of the proximal clients 200, and converts the UI element groups, matched to the proximal clients 200, into video streams or images, and transmits the video streams or images to the user device.

The UI distribution information required for controlling the IoT device may be shared with other user devices invited to the device group for each user.

The invited user device may receive a video stream or image information for monitoring and UI control information of the IoT device that is the same as that included in the user device group. In this case, even when a OS systems used in the user devices are different, the user devices receive the UI control information as the video stream or the image information, so that there is an advantage in that it is possible to transmit the data independent to the OS systems.

FIG. 2 illustrates a concept in which the local server 100 divides a UI of an original content into UI element groups by using UI tree information and transmitting a UI element group to the

proximal clients

21 and 22 by a video streaming according to an exemplary embodiment of the present invention.

Referring to FIG. 2, the original content is transmitted to the local server 100 from the content providing server 400 and is transmitted to the

proximal clients

21 and 22. The local server 100 may modify a composition and a layout of the UI element group by using the UI tree information included in the original content.

The local server 100 may divide the original content into a first UI element group and a second UI element group, match the first UI element group and the second UI element group to the smart display 21 and the smart phone 22 respectively, and then transmit the matched first UI element group and second UI element group by pixel streaming. In this case, the data transmitted to the smart display is a video included in the original content, and is decoded in the local server and is transmitted to the smart display as a video stream, and when the second UI element is transmitted to the smart phone by encoded pixel streaming, and the smart phone decodes the received pixel stream. In this case, the reproduction and the stop of the video stream reproduced in the smart display are determined according to an interaction of the user in the smart phone. However, when it is necessary to synchronize the reproduction time of the video streams of two devices, the decoding time of the video stream of the smart display needs to be synchronized based on the time in which the video stream is decoded in the smart phone.

As illustrated in FIG. 3, the local server 100 includes the UI distribution control module 105, the smart display 21 includes a UI element display module 320 to a UI editing module 350, the smart phone 22 includes a UI element display module 321 or a UI editing module 351, and it is possible to provide an expanded UI function through a element group consisting of the UI displays of various content applications by generating and storing the UI distribution information which is using the UI element group information provided from the two or more content applications.

Referring to FIG. 3, the local server 100 according to the present exemplary embodiment includes a receiving module 110, the UI distribution control module 105, and a transmitting module 190. The UI distribution control module 105 includes a content app executing module 120, a UI element dividing module 130, a UI distribution information generating module 140, a display requesting module 150, a UI element transmitting module 160, a virtual display generating module 170, and a UI control information converting module 180. The UI application installed in the local server 100 includes the entirety or a part of the configurations of the UI distribution control module 105. The local server 100 and clients A and

B

21 and 22 may be expressed as user devices.

In the meantime, clients A and

B

21 and 22 includes

app selecting modules

370 and 301, transmitting

modules

310 and 311, UI

element display modules

320 and 321, virtual

display generating modules

330 and 331, UI control

information input modules

340 and 341,

UI editing modules

350 and 351, and receiving

modules

360 and 361.

The local server 100 transmits UI display information to clients A and

B

21 and 22 by video streaming or transmits the UI element groups to clients A and

B

21 and 22, and receives UI control information by a user interaction from the client and generates a control signal of the content app.

Client B 22 selects a content app for which UI distribution information is to be changed and transmits the selected content app to the local server 100, and displays UI display information according to the changed UI distribution information.

Client A 21 is the client added for changing the UI distribution information of client B 22, and receives the UI element group of the content app from the local server 100 and displays the received UI element group.

Hereinafter, the configuration of the local server 100 will be described in detail.

The receiving module 110 receives whether to select the content app for which the UI distribution information is to be changed from client B 22. In this case, the number of content apps selectable by client B 22 may be two or more.

The UI distribution control module 105 divides the UI display information of the content app into UI element groups, matches the divided UI element groups to each client, and generates UI distribution information.

The content app executing module 120 executes the content app selected by client B 22. The content app is not required in clients A 21 and B 22 when the UI information is transmitted from the local server 100 by pixel streaming.

The content app executing module 120 may be replaced with an IoT app executing module and a UI app executing module combined with a Non-Fungible Token (NFT) content.

The UI element dividing module 130 divides the UI provided in the content app executed in the content app executing module 120 into a plurality of UI element groups.

The UI distribution information generating module 140 matches each of proximal clients to each of the UI element groups respectively, and generates UI distribution information.

The UI distribution information generating module 140 may determine the UI distribution information by a physical distance to the proximal client based on the local server 100 or a direction of the location of the proximal client. For example, the UI distribution information generating module 140 may determine to distribute the UI element group having a large display size in the client relatively far from the local server 100 and to distribute the UI element group having a small display size, such as a character input display, to the client relatively close to the local server 100.

The display requesting module 150 requests a UI element display to the client A 21 or the client B 22. When the local server 100 requests client A 21 to display the UI element, the local server 100 may request the UI element display module 320 to execute the UI element display through a push message. When the client receives the UI element group, the display requesting module 150 may make the client ready for the display of the received UI element group.

The UI element transmitting module 160, by using the UI distribution information, transmits one or more UI element groups, which corresponds to the proximal each client respectively, or transmits video stream, converted from one or more UI element groups, which corresponds to the each proximal client respectively.

When the local server 100 transmits the UI element group by the video stream, it is not necessary to install and execute the content application in client A 21 and client B 22.

The virtual display generating module 170 generates virtual display information by using the UI distribution information generated by the UI distribution information generating module 140.

When the UI control information converting module 180 receives the UI control information from clients A 21 and B 22, the UI control information converting module 180 converts the received UI control information to a control signal of the content app.

The transmitting module 190 performs the display request of the display requesting module 150, and the transmission of the UI element group of the UI element transmitting module 160, and the like.

Hereinafter, the configurations of clients A 21 and B 22 will be described in detail.

The configurations of client A 21 and client B 22 are the same, so that client A 21 will be mainly described.

The app selecting module 370 selects a content app of which UI display information needs to be changed. In this case, it is not necessary to execute the content app as a result of the selection.

The transmitting module 310 transmits user interaction information about which content app has selected to the local server 100, and the UI control information input module 340 transmits the received UI control information to the local server 100.

The UI element display module 320 is executed in client A 21 by the request of the display requesting module 150 of the local server 100. It is preferable to receive the UI element group or the UI element video streaming in the state where the UI element display module 320 is executed in client A 21.

The virtual display generating module 330 renders the UI element group received from the local server 100 and stores the UI element group in a graphic buffer.

The UI control information input module 340 receives the input of the UI control information through a touch screen, a keyboard, a voice, a picture, and the like. When a size of the data, such as pictures and voices, is relatively large, original data may be transmitted to a separate storage server, and information about the storage space may be changed to a web URL and transmitted.

The

UI editing modules

350 and 351 may select at least one UI element group of the content app, and select a client to display the selected UI element group and change UI distribution information.

The receiving module 360 directly receives at least one UI element groups from the local server 100 or receives video stream of at least one UI element groups.

Referring to FIG. 4, the method of transmitting UI display information according to the exemplary embodiment includes operations time-serially processed in the local server and the client illustrated in FIG. 3. Accordingly, even if omitted below, the contents described above with respect to the local server and the client illustrated in FIG. 3 are also applied to the method of transmitting the UI display information according to the exemplary embodiment.

Referring to FIG. 4, client A 21, client B 22, and the local server 100 are connected through the home network, and may be mutually called by using the HTTP-based web API (REST service or Remote Procedure Calls (RPC) mechanism), and the TCP/IP protocol. As a specific example, the REST API may be applied for synchronization of the content apps, the RPC may be applied for video streaming between devices, and the TCP/IP protocol may be applied to the transmit of the control information according to user interaction between devices.

In operation 400, the local server 100 receives UI control information according to a selection of a content app of a user from client B 22.

Once client B 22 transmits keyboard input and touch screen input information that the user reacts on the received video stream to the local server through the app selecting module 301 as UI control information of the user, the local server transmits the UI control information to a content application. The present invention is described based on one content application, but the app selecting module may select a plurality of content applications. In this case, the UI distribution control module 105 of the local server 100 may divide the plurality of content application UIs by matching the clients to each UI element group of the plurality of content application UIs. The content application is an application used for displaying the original content, and may be a video application, such as YouTube or Twitch.

In the meantime, even in the case where the content application is not installed in client B 22, client B 22 may receive the UI element group from the local server 100 by video streaming, or may receive UI element group information and then the client may render and display the UI element group.

In operation 410, the local server 100 executes the content app selected in operation 400.

In operation 420, the local server 100 divides the UI of the executed content app into a plurality of UI element groups. The UI provided from the content application executed in the local server 100 may be divided into a plurality of UI elements by using the UI distribution control module 105 installed in the local server 100. Herein, an authority for the functions of dividing the single UI of the content app into the plurality of UI element groups and generating and correcting the UI distribution information is granted only to a specific device of a user possessing the corresponding right, so that a security function of the system may be strengthened.

In operation 430, the local server 100 matches the proximal clients (21 and 22) to the plurality of UI element groups respectively, and generates UI distribution information.

When client B 22 selects the content app in operation 400, it is preferable to select which client among the proximal clients and which UI element group will be matched for displaying the UI element group.

In operation 440, the local server 100 requests client A (21) to display the UI element group.

The request to display in operation 440 may be made before operation 440. When the local server 100 requests client A (21) to display the UI element group, the local server 100 may make the UI element display module 320 to execute the UI element display by transmitting a push message to the user’s app.

In operation 450, the user’s app of client A (21) receiving a request to display the UI element group executes the UI element group display module 320.

When the UI element group display module 320 is executed, client A may display the UI element group by receiving the video streaming received from the local server 100 in operation 472.

In operation 460, the local server 100 generates virtual display information by using the UI distribution information generated in operation 430. The virtual display information means information stored to render UI element group information in a graphic buffer. Therefore, the graphic buffer may reflect the UI distribution information by using the virtual display information and transmit the video stream to each device.

In operation 471 and 472, the local server 100 transmits one or more UI element groups corresponding to client A (21) and client B (22), respectively, as a video stream by using the UI distribution information. Since the local server 100 transmits the UI element group as a video stream, it is not necessary to install and execute the content application in client A (21) and client B (22).

In operation S481 and 482, the local server 100 receives UI control information from client A (21) and client B (22).

In operation 490, the local server 100 converts the received UI control information to a control signal of the content app.

When the user inputs the UI control information through a touch screen, a keyboard, a voice, and the like of client A (21) and client B (22), the UI control information may be transmitted to the local server through the TCP/IP protocol or the web API.

The local server 100 may convert the UI control information received from client A (21) and client B (22) into the control signal of the content app in consideration of display size and resolution information of each client possessed by the local server 100. Further, the user control information may be transmitted to the local server by providing a separate UI together with the video stream received from the user’s app installed in the client device. In this case, the client device needs to convert the user control signal and transmit the converted user control signal to the local server.

For example, when the local server 100 receives a coordinate value input by the user on the display of client A (21), it is necessary to determine which element of the content app UI corresponds to the received coordinate value in consideration of the display size and resolution information. The determination may also be performed in client A (21) and may also be performed in the local server 100. Even though the local server 100 receives the coordinate value input by the user on the display of client A (21), the local server 100 has the display size and resolution information of client A (21), so that the local server 100 may determine which part of the content app may corresponds to the received coordinate value.

Further, the UI control information may include content control information and UI distribution editing information, and the content control and the UI distribution editing are performed by the following method.

As an example, when the same content app is installed in the local server 100 and the

clients

21 and 22, a context state value (or a parameter setting value of the content app) of the result of the distribution and its editing of selected UI by the

clients

21 and 22 may be transmitted to the local server 100.

The local server 100 performs with the authority to call the functions of the clients, 21 and 22, necessary for collecting the context state value.

In this case, it is not necessary for the local server 100 to transmit display data for the UI distribution editing to the clients, 21 and 22, and when the display information is generated and the UI editing is completed in the clients, 21 and 22, it is preferable that the clients, 21 and 22, change the context state value and provide the changed context state value to the content app of the local server 100.

As another example, when the content app is installed only in the local server 100, it is preferable that the local server 100 provides the information necessary for the information of UI distribution and its editing to the clients, 21 and 22, as a video stream. When the clients, 21 and 22, receives the UI distribution display transmitted by the local server 100 and transmits touch screen response results of the users in the clients, 21 and 22, to the local server 100, the local server 100 transmits the touch screen response as user interaction information to the content app. Thereafter, when the information of the single UI of the content app is updated, the local server applies the updated UI information of the content app to the UI distribution information and transmits the UI distribution information as a video stream. By repeating this process, the user is capable of performing UI distribution editing.

Further, the transmission of the video stream is possible independent to the OSs of the clients, 21 and 22, the clients, 21 and 22, may analyze the user’s control intention based on the response information of the touch screen or the touch pad and transmit a corresponding result to the local server 100.

Referring to FIG. 5, the method of transmitting UI display information according to the exemplary embodiment includes operations time-serially processed in the local server and the client illustrated in FIG. 3. Accordingly, even if omitted below, the contents described above with respect to the local server and the client illustrated in FIG. 3 are also applied to the method of transmitting the UI display information according to the exemplary embodiment.

The difference between the method of transmitting UI display information to the

proximal clients

21 and 22 by the local server 100 illustrated in FIG. 5 and the method illustrated in FIG. 4 is that client A (21) receives the UI element group from the local server 100.

Client A (21) and client B (22) are connected with the local server 100 through the home network, and are capable of mutually calling through the HTTP-based web API (REST service or RPC mechanism).

Operation 400 to operation 460 are as illustrated in FIG. 4.

In operation 471, the local server 100 transmits a video stream for at least one UI element group matched to client B (22) by using the UI distribution information.

In the meantime, in operation 473, the local server 100 transmits at least one UI element group corresponding to client A 21 by using the UI distribution information.

In operation 475, since the local server 100 transmits the UI element group, the virtual display generating module 330 generates virtual display information by using the UI element group.

In the meantime, since client A (21) generates the virtual display information by using the received UI element group, client A (21) may have a function of a sub local server performing pixel streaming on another client. An example of the case where the client has the function of the sub local server will be described in detail with reference to FIG. 7.

In operation 490, the local server 100 converts the received UI control information to a control signal of the content app. The local server 100 analyzes user interaction of the received UI control information, converts the user interaction into a control signal, and inputs the control signal to the content app that is being executed in the local server 100.

The difference between the method of transmitting UI display information to the proximal clients by the local server 100 illustrated in FIG. 6 and the method illustrated in FIG. 5 is that both client A (21) and client B (22) receive the UI element groups from the local server 100.

Since both client A (21) and client B (22) receive the UI element groups and generate the virtual display information,

clients

21 and 22 may have a function of a sub local server performing pixel streaming on another client.

Operation 400 to operation 440 are as illustrated in FIG. 4.

In operation 441 illustrated in FIG. 6, the local server 100 requests client B 22 to display the UI element group similar to operation 440.

In operations 450 and 451, client A (21) and client B (22) receiving the request to display the UI element group execute the UI

element display modules

320 and 321, respectively.

In operation 460, the local server 100 generates virtual display information by using the UI distribution information generated in operation 430.

In the meantime, when the clients, 21 and 22, execute the UI

element display modules

320 and 321 respectively, the clients, 21 and 22, may receive the UI element groups from the local server 100 in

operations

473 and 474.

Similar to operation 460 in which the local server 100 generates the virtual display information, in operations 475 and 476, client A (21) and client B (22) generate virtual display information by using the UI element groups received from the local server 100. In this case, client A/B may transmit a first UI element to itself as a video stream, and may transmit a second UI element to another user device as a video stream by using the generated virtual display information.

In operation S481 and 482, the local server 100 receives UI control information from client A 21 and client B 22.

In the meantime, when an error is generated in the operation of the local server 100, in order to build a resilient system, client A (21) or client B (22) may be defined as a local server candidate having the same authority as that of the local server 100.

In the situation where the local server 100 is operated, the UI distribution information generating module 140 of the local server is synchronized to the client defined as the local server candidate by the web RPC or REST service method, the content app installed in the local server needs to be installed even in the client, and the local server 100 and the content app of the client need to be synchronized by the web API method.

When an error is generated in the operation of the local server 100, the local server candidate client having the authority of the local server, the local server candidate client having the authority of the local server may serve as the local server by using the information of the synchronized elements.

In the meantime, in the case where the OS system of the local server 100 is different from the OS of the client, the matched UI element group information may be transmitted to the user’s app for the present invention of the client device by using a transport layer for each OS, and then, the client device may generate a virtual display and perform video streaming by using the UI element group.

Referring to FIG. 7, the local server extracts UI tree information included in the original content, virtualizes the single UI, and then divides the UI element groups constituting the UI tree information into group 1 and group 2.

The UI elements group divided into group 1 are stored as pixel data of a first graphic buffer, and the UI element groups divided into group 2 are stored without being converted to the pixel data. In this case, when the local server renders and transmits for the UI element group 1, among the UI element groups divided into group 1 and group 2, and transmits for the UI element group 2 without rending to the device of client A, the device of client A forwards the received UI element group 2 to devices of clients B and C. Clients B and C may forward and provide the UI display information between the devices by rendering the UI element group 2 by themselves and providing a UI display.

Further, clients B and C of group 2 may generate UI control information and transmit the generated UI control information to client A, and client A may transmit its UI control information and the control information received from clients B and C to the local server. Herein, when information discriminating the local server and the forwarding client is included in the UI element information, the user’s app of the present invention may be configured so that the UI control information of clients B and C is directly transmitted to the local server without passing through client A.

Referring to FIG. 8, the method of dividing UI tree information and transmitting the divided UI tree information to the client according to the exemplary embodiment includes operations time-serially processed in the local server and the client illustrated in FIG. 3. Accordingly, even if omitted below, the contents described above with respect to the local server and the client illustrated in FIG. 3 are also applied to the method of transmitting the UI display information according to the exemplary embodiment.

In operation 800, the local server 100 divides the UI tree information included in the content in consideration of the number of clients.

In operation 810, the local server 100 virtualizes the single UI tree. The virtualization means managing the UI display information of the user’s device as a single UI tree in the content application, and managing the single UI tree as a UI group divided into UI element groups at the OS level. Accordingly, the main function of the virtualization is to manage the divided UI display information to be transmitted to the plurality of user devices as the information of UI element groups, and it is possible to prevent a malfunction according to the UI tree virtualization by transmitting the content application in which user interaction information for the UI display information, being divided into the plurality of UI groups and being transmitted to the plurality of devices, is collected and managed as the single UI tree.

Therefore, in order to prevent the malfunction, it is necessary to synchronize an update cycle of a main variable value of the divided UI element groups, the update of the user interaction information provides the user interaction information to the content application when a variable value of the main function of the UI element group is changed within a predetermined update cycle.

In operation 820, the local server 100 receives display resolution from client A (21).

In operation 830, the local server 100 generates a graphic buffer (for example, the virtual display) having the same resolution as the received display resolution.

In operation 840, the local server 100 renders the UI tree information divided in operation 800 to the graphic buffer corresponding to the client.

In operation 850, the local server 100 encodes pixel data stored in the graphic buffer and transmits the encoded pixel data to the corresponding client.

In operation 860, client A 21 decodes the received pixel data to a video stream and renders the video stream.

Referring to FIG. 9, the method of editing a UI of a content app by the client according to the exemplary embodiment includes operations time-serially processed in the local server and the client illustrated in FIG. 3. Accordingly, even if omitted below, the contents described above with respect to the local server and the client illustrated in FIG. 3 are also applied to the method of transmitting the UI display information according to the exemplary embodiment.

In operation 900, client A (21) transmits selected content app to the local server 100.

In operation 910, the local server 100 executes the selected content app. When there is no selected content app, the local server 100 may download the content app through the cloud server 300 and execute the downloaded content app.

In operation 920, client A (21) receives the UI of the content app from the local server 100 by video streaming.

In operation 930, client A (21) displays the received UI of the content app.

In operation 940, client A (21) executes a UI setting mode of the content app.

In operation 950, client A (21) selects at least one UI element group of the content app, and selects a client which is to display the selected UI element group.

In operation 960, client A (21) transmits UI distribution information matching a client information to the selected UI element group for display to the local server 100.

In operation 970, the local server 100 transmits a UI element group to the matched client that displays the UI element group with reference to the UI distribution information.

FIG. 10 illustrates a flow of a method of transmitting, by the local server, IoT state information to the client device in an OS level according to another exemplary embodiment of the present invention.

Referring to FIG. 10, the method of transmitting IoT state information to the client according to the exemplary embodiment includes operations time-serially processed in the local server and the client illustrated in FIG. 3. Accordingly, even if omitted below, the contents described above with respect to the local server and the client illustrated in FIG. 3 are also applied to the method of transmitting the IoT state information according to the exemplary embodiment.

In operation 1000, the local server 100 receives information about the IoT application selected for receiving IoT state information by a user from the client A (21). The selectable IoT applications are listed in client A (21), and when the user selects any one IoT application, the local server 100 receives which IoT application is selected as user UI control information.

In operation 1010, the local server 100 executes the selected IoT application. In the case where the IoT application selected in operation 1000 is not installed, the local server 100 may download the IoT application through the cloud server 300 of the IoT device.

In operation 1020, the local server 100 divides a UI of the executed IoT application into a plurality of UI element groups.

In operation 1030, the local server 100 generates UI distribution information by using the plurality of divided UI elements groups.

In operation 1040, the local server 100 generates virtual display information by using the generated UI distribution information.

In operation 1050, the local server 100 may transmit the virtual display information to client A (21) by video streaming or transmit the matched UI element group to the client device.

In operation 1060, the local server 100 receives UI control information from the user’s app of client A (21).

In operation 1070, the local server 100 converts the received UI control information to a control signal of the IoT application.

In operation 1071, the local server 100 transmits the IoT control signal to an IoT device 25 through the cloud or the home network. When the user device serves as a hub of the IoT device, the IoT device 25 may be controlled through the home network using ZigBee/Wi-Fi.

In the meantime, when the IoT app installed in the local server 100 transmits the IoT control signal to an IoT cloud server, the IoT cloud server transmits the IoT control signal to the IoT device through the IoT hub by the TCP/IP communication by referring to a IP address information uniquely matched to the IoT device 25.

In operation 1072, the IoT app of the local server 100 receives IoT state information through the IoT device 25 or the IoT cloud server.

In operation 1073, the local server 100 transmits the IoT state information to the user’s app of client A (21) as the video stream or the UI element group information.

FIG. 11 illustrates a flow of a method of transmitting, by the cloud server, IoT state information to the client device according to another exemplary embodiment of the present invention.

Referring to FIG. 11, the method of transmitting IoT state information to the client according to the exemplary embodiment includes operations time-serially processed in the local server and the user of the client illustrated in FIG. 3. Accordingly, even if omitted below, the contents described above with respect to the local server and the user app of the client illustrated in FIG. 3 are also applied to the method of transmitting the IoT state information according to the exemplary embodiment.

FIG. 11 is different from FIG. 10 in that client A (21) located outside the home network receives the IoT state information through the cloud server 300.

In FIG. 11, for the remote control of the IoT device, it is preferable that the UI virtualization information of the cloud server 300 and the local server 100, the UI distribution information, the user device information matched to the UI distribution information, and the like are synchronized through the web API.

In operation 1100, the cloud server 300 receives user control information for selecting an IoT application by the user from client A (21).

In operation 1110, the cloud server 300 determines whether the device of client A (21) is a registered user device in the user device group information.

In operation 1120, when client A (21) is the registered user device as a result of the determination of the cloud server 300, the local server 100 receives information about the IoT application selected by the user of client A (21) from the cloud server 300.

In operation 1130, the local server 100 executes the selected IoT application.

In operation 1140, the local server 100 divides a UI of the executed IoT application into a plurality of UI element groups at an OS level.

In operation 1150, the local server 100 generates UI distribution information by using the plurality of divided UI elements groups. Further, the local server transmits an updated UI distribution information of the local server to the cloud server to synchronizes the UI distribution information between two devices.

In operation 1160, the local server 100 generates the UI of the IoT app of the local server synchronized with the IoT app of the cloud server as virtual display information by using the generated UI distribution information.

In operation 1170, the local server 100 transmits a UI display of the matched IoT app to the device of client A (21) by video streaming by using the virtual display information, or transmit the matched UI element group information.

In operation 1180, client A (21) transmits UI control information to the cloud server 300, and when client A (21) transmits the UI control information to the IoT app installed in the cloud server, context state information of the IoT app of the local server is updated by the cloud server of the IoT device.

Therefore, the IoT app of the local server 100 transmits the received UI control information to the IoT device.

In operation 1191, the IoT app of the local server 100 transmits IoT control signal to the IoT cloud server through the Internet web, or when the local server serves as the IoT hub, the local server directly transmits the IoT control signal to the IoT device through the home network.

In operation 1192, the IoT app of the local server 100 receives the state information of the IoT device via the cloud of the IoT app and the IoT hub, or directly receives the state information from the IoT device through the home network when the local server serves as the IoT hub.

In operation 1193, when the context state information of the IoT app of the local server 100 is changed, the IoT app provides the cloud of the IoT device with the changed state information, or transmits the changed state information to the IoT app of the cloud server 300 when the local server serves as the IoT hub.

The hub for controlling the IoT device controls the IoT device on the home network by using ZigBee, Bluetooth, or Wi-Fi network.

FIG. 12 illustrates a flow of a method of transmitting, by the local server, IoT state information to an unauthorized client according to another exemplary embodiment of the present invention.

The method of transmitting, by the local server, the IoT state information to a non-authorized client is different from the methods illustrated in FIGS. 10 and 11 in that a non-authorized client 30 requests the local server 100 to transmit the IoT state information.

In operation 1200, the local server 100 sets IoT state information providing range according to a distance to the client device.

The local server 100 may differentiate and provide the IoT state information according to the location of the non-authorized client 30. There is a case where the non-authorized client 30 is physically close to the local server 100 and there is a case where the non-authorized client 30 requests the IoT state information through the home network or the Internet at a remote distance, and it is possible to differentiate and provide the IoT state information according to the physical distance between the local server 100 and the non-authorized client 30.

In this case, when the location of the non-authorized client 30 is physically present within a predetermined distance from the local 100, more IoT state information may be set to be provided.

Further, depending on whether the IoT state information is requested from inside or outside the place in which the local server 100 is installed, the local server 100 may differentiate and provide the state information of the IoT device 25.

In operation 1210, the local server 100 receives a transmission request of an IoT state information from the user’s app of the non-authorized client 30.

In operation 1220, the local server 100 calculates a distance to the non-authorized client 30 based on the transmission request of IoT state information received in operation 1210.

In order to accurately calculate the distance between the local server 100 and the non-authorized client 30, when an antenna of a wireless communication module (Bluetooth, Wi-Fi, and ultra-wide band) of the local server 100 is implemented with multi-antennas having a predetermined reference distance, some of the paired antennas may be implemented to receive horizontal polarization and some other antennas may be implemented to receive vertical polarization.

In this case, the local server 100 may receive a wireless communication signal from the non-authorized client 30 through the paired vertical polarization/horizontal polarization multi-antennas, and accurately calculate a location of the non-authorized client 30 by a arrival time difference according to a difference in the incident angles of the received signals between the paired multi-antennas.

Accordingly, the local server 100 may determine the IoT state information to be provided to the user’s app of the non-authorized client 30 by calculating the distance between the local server 100 and the non-authorized client 30.

In operation 1230, the local server 100 transmits the transmission request of the IoT state information to the IoT device 25.

In operation 1240, the local server 100 receives the IoT state information from the IoT device 25.

In operation 1250, the local server 100 determines the IoT state information and the UI distribution information to be transmitted to the non-authorized client 30 according to the distance calculated in operation 1220.

In operation 1260, the local server 100 generates virtual display information by using the UI distribution information.

In operation 1270, the local server 100 transmits the IoT state information as a video streaming to the user’s app of the non-authorized client device. In this case, the IoT state information is transmitted from the local server 100 to the non-authorized client 30 as a video stream and is displayed on a screen. Accordingly, the installation of the IoT application, required for receiving the IoT state information of plurality of IoT devices, is not required in the non-authorized client 30.

FIG. 13 illustrates a flow of a method of receiving an NFT content and a UI app, by the local server, and transmitting UI display information according to another exemplary embodiment of the present invention.

Referring to FIG. 13, the method of transmitting UI display information according to the exemplary embodiment includes operations time-serially processed in the local server and the client illustrated in FIG. 3. Accordingly, even if omitted below, the contents described above with respect to the local server and the client illustrated in FIG. 3 are also applied to the method of transmitting the UI display information according to the exemplary embodiment.

FIG. 13 illustrates a method in which a local server 20 or client A (21) uses a UI app including NFT content for a predetermined period by transmitting the NFT content of an NFT content sever 500 and the UI app of a UI app server 600 to the local server 20.

In operation 1300, the user mutually concludes a smart contract with an NFT content provider 500 and a UI app provider 600.

Digital assets are managed by blockchain-based smart contracts, and digital asset tokens are largely divided into Fungible Tokens (FT) and Non-Fungible Tokens (NFT). In the Ethereum token, ERC-20 is defined as a fungible token and ERC-721 token is defined as a non-fungible token. In addition, ERC-1155 is a token that can manage ERC-20 and ERC-721 tokens through a single smart contract.

Accordingly, in the present invention, it is preferable that the NFT content provider/server 500, the UI app provider/server 600, and the user conclude a smart contract related to ownership/use of the NFT content and the UI app based on ERC-1155.

When the UI app provider/server 600, the NFT content provider/server 500, and the user/local server 20 conclude the smart contract effective for a specific period of time, the NFT content and the UI app may be transmitted to the local server 20. Accordingly, the local server 20 is granted to access the NFT contents for the predetermined period of time and is allowed to access as a sole user. In this case, the smart contract may include a condition of limiting display transmission of the NFT content transmitted to the plurality of user devices, which is a present invention limiting the number of authorized user devices included in the user device group.

In operation 1310, client A (21) selects the NFT content and the UI app to be used and transmits the selected NFT content and UI app to the local server 100.

In this case, it is possible to match the plurality of selectable UI apps stored in the UI app server 600 to a fungible token and provide the matched token to the local server 20 based on the contents of the smart contract, and the local server 20 may generate UI distribution information by dividing a single UI of the UI app into the units of UI element groups at the OS level and matching the divided UI element groups to the client devices of the user, and transmit one UI element group to the matched client A (21).

Client A (21) may select one of the plurality of selectable UI apps stored in the UI app server 600, combine the selected UI app with the NFT content, and display the combined NFT content.

In operation 1320, the local server 100 checks whether the smart contract is concluded between the UI app server 600 and the NFT content server 500, and then receives the UI app selected by client A (21) from the UI app server 600.

In operation 1321, the local server 100 checks whether the smart contract is concluded between the UI app server 600 and the NFT content server 500 and then receives the NFT content from the NFT content server 500.

In operation 1330, the local server 100 executes the selected UI app.

In operation 1340, the local server 100 divides a UI of the executed UI app into a plurality of UI element groups at an OS level.

In operation 1350, the local server 100 matches the divided each UI element group and each of the client user device respectively, and generates UI distribution information.

When the local server 100 is connected with the proximal clients through the home network, it is preferable that the UI distribution control module 105 transmits the generated UI distribution information to the cloud server 300.

In operation 1360, the local server 100 generates virtual display information by using the generated UI distribution information.

In operation 1370, the local server 100 transmits the UI element group to client A (21). Instead of transmitting the UI element group to client A (21), the local server 100 may transmit a video stream for the at least one UI element group corresponding to client A (21) in operation 1370.

In operation 1371, client A (21) generates virtual display information by using the UI element group received from the local server 100.

In operation 1380, the local server 100 receives UI control information from client A (21).

In operation 1381, the local server 100 converts the UI control information to a control signal of the UI app. The local server 100 may apply the control signal of the UI app and re-transmit the UI element group or the video stream.

In operation 1390, the local server 100 may delegate the authority to use the UI app and the NFT content to client A (21).

During the period of time in which the smart contract, between the NFT content server 500 and the UI app server 600 concluded, in operation 1300 is effective, a situation may arise in which the NFT content is transmitted to another client. In this case, when the authority to use is delegated to another client, the smart contract is updated, and each operation of FIG. 13 is executed so that the UI application is executable in a new client.

When the authority to use is delegated to client A (21), the smart contract between the UI app server 600 and the NFT content server 500 is updated including client A (21), instead of the local server 100, and the UI distribution control module 105 is executable in client A (21).

The cloud server 300 may manage information, such as the user’s authority to access to the NFT content, the authority to provide the UI by using the UI app, the authority to correct the UI distribution information, and the delegation of the authority to use. The NFT content user may be delegated all of the above authorities, use the NFT content for a predetermined period of time, and delegate the authority to another user to distribute the NFT content.

Further, the NFT content user may stream and use the NFT content in his/her display device by changing the UI with a style that he/she wants for a predetermined period of time. Then, when the predetermined period of time elapses based on the contents of the smart contract, the authorities to use of all of the users, except for the content user may be reset.

The exemplary embodiments of the present invention may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer readable medium. The computer readable medium may include a program command, a data file, a data structure, and the like alone or in combination. The program command recorded in the medium may be specially designed and configured for the present invention, or may also be known and usable to those skilled in computer software. Examples of the computer readable recording medium include a magnetic medium, such as a hard disk, a floppy disk, or a magnetic tape, an optical recording medium, such as a CD-ROM or a DVD, a magneto-optical medium, such as a floptical disk, and a hardware device which is specifically configured to store and execute the program command such as a ROM, a RAM, and a flash memory. An example of the program command includes a high-level language code executable by a computer by using an interpreter, and the like, as well as a machine language code created by a compiler. The hardware device may be configured to be operated with one or more software modules in order to perform the operation of the present invention, and an opposite situation thereof is available.

As described above, the present invention has been described with reference to the specific matters, such as a specific component, limited embodiments, and drawings, but these are provided only for helping general understanding of the present invention, and the present invention is not limited to the aforementioned exemplary embodiments, and those skilled in the art will appreciate that various changes and modifications are possible from the description.

Accordingly, the spirit of the present invention shall not be determined while being limited to the foregoing exemplary embodiments, and it will be considered that all matters having equivalent or equivalent modifications to the claims are within the scope of the present invention.

Claims

A method of distributing and controlling a UI of user device comprising:

dividing, by a first user device, UIs of a content application into a plurality of UI element groups at an OS level;

generating, by the first user device, a UI distribution information by matching a UI element group, among the divided plurality of UI element groups, to a second user device;

transmitting, by the first user device, the matched UI element group or a video stream for the matched UI element group to the second device based on the UI distribution information;

transmitting, by the first user device, user device group information including the first user device and the second user device or the UI distribution information to a cloud server; and

receiving, by the first user device, updated user device group information or updated UI distribution information from the cloud server.
The method of claim 1, further comprising:

authorizing or delegating, by the first user device included in the user group information, another device not included in the user device group to divide the matched UI element groups into a plurality of UI element groups.
The method of claim 1, further comprising:

receiving, by the first user device, a user control information from a user device included in the user device group information, and apply a user control information, in association with the received the user control information, to the content application.
The method of claim 1, wherein the video streaming, by the first user device, in associated with a display size and resolution of the second user device.
The method of claim 1, further comprising: the matched UI element group to second user device is transmitted, by the second user device, to a third user device included in the user device group information.
The method of claim 1, wherein the UIs of the content application, the content application comprised of a plurality of content applications, is generated from a UI distribution information comprised of UI element groups of plurality of UI element groups, the plurality of UI element groups divided from UIs of each content application of the plurality of content applications.
The method of claim 1, wherein the matched UI element group is matched only to a one user device included in the user device group information.
A method of distributing and controlling a UI of a UI application of a user device, the method comprising:

generating, by a UI application of a first user device, UI distribution information by matching a UI element group of plurality of UI element groups to a second user device, the plurality of the UI element groups divided from UIs of content application at an OS level;

transmitting, by the UI application of the first user device, UI display information of the matched UI element group or a video stream in association with the matched UI element group to the second user device;

transmitting, by the UI application of the first user device, user device group information including the first user device and the second user device or the UI distribution information to a cloud server; and

receiving, by the UI application of the first user device, updated user device group information or updated UI distribution from the cloud server.
The method of claim 8, further comprising:

authorizing or delegating, by the UI application of the first user device included in the user device group information, a third user device not included in the user device group information to divide the matched UI element group into a plurality of UI element groups.
The method of claim 8, further comprising:

receiving, by the UI application of the first user device, a user control information from a user device included in the user device group information, and applying user control information, by the application of the first user device, to the content application with association with the received user control information.
The method of claim 8, wherein the video streaming is associated with a display size and resolution of the second user device.
The method of claim 8, further comprising:

transmitting, by a UI application of the second user device, the received UI element group to the third user device included in the user device group information.
The method of claim 8, wherein UIs of the content application, comprised of a plurality of content applications, is generated from UI distribution information comprised of UI element groups of plurality of UI element groups, the plurality of UI element groups divided from UIs of each content application of the plurality of content applications.
The method of claim 8, wherein the matched UI element group is distributed to only the second user device included in the user device group information.
The method of claim 8, wherein the UI distribution information further comprising:

the UI distribution information is determined by a physical distance or an angle from the first user device to the second user device.