WO2023163568A1 - Method of transmitting ui display information based on smart contract - Google Patents

Method of transmitting ui display information based on smart contract Download PDF

Info

Publication number
WO2023163568A1
WO2023163568A1 PCT/KR2023/002796 KR2023002796W WO2023163568A1 WO 2023163568 A1 WO2023163568 A1 WO 2023163568A1 KR 2023002796 W KR2023002796 W KR 2023002796W WO 2023163568 A1 WO2023163568 A1 WO 2023163568A1
Authority
WO
WIPO (PCT)
Prior art keywords
user
local server
information
content
client
Prior art date
Application number
PCT/KR2023/002796
Other languages
French (fr)
Inventor
In Sik Shin
Daehee Lee
Sun Jae Lee
Sangwook Lee
Original Assignee
Fluiz Co. Ltd.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Fluiz Co. Ltd. filed Critical Fluiz Co. Ltd.
Publication of WO2023163568A1 publication Critical patent/WO2023163568A1/en

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/4302—Content synchronisation processes, e.g. decoder synchronisation
    • H04N21/4307—Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen
    • H04N21/43079—Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen of additional data with content streams on multiple devices
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/25—Management operations performed by the server for facilitating the content distribution or administrating data related to end-users or client devices, e.g. end-user or client device authentication, learning user preferences for recommending movies
    • H04N21/258—Client or end-user data management, e.g. managing client capabilities, user preferences or demographics, processing of multiple end-users preferences to derive collaborative data
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
    • G06F3/1423—Digital output to display device ; Cooperation and interconnection of the display device with other functional units controlling a plurality of local displays, e.g. CRT and flat panel display
    • G06F3/1438—Digital output to display device ; Cooperation and interconnection of the display device with other functional units controlling a plurality of local displays, e.g. CRT and flat panel display using more than one graphics controller
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/10—Office automation; Time management
    • G06Q10/101—Collaborative creation, e.g. joint development of products or services
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
    • G06Q50/01—Social networking
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/21—Server components or server architectures
    • H04N21/222—Secondary servers, e.g. proxy server, cable television Head-end
    • H04N21/2223—Secondary servers, e.g. proxy server, cable television Head-end being a public access point, e.g. for downloading to or uploading from clients
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/25—Management operations performed by the server for facilitating the content distribution or administrating data related to end-users or client devices, e.g. end-user or client device authentication, learning user preferences for recommending movies
    • H04N21/258—Client or end-user data management, e.g. managing client capabilities, user preferences or demographics, processing of multiple end-users preferences to derive collaborative data
    • H04N21/25866—Management of end-user data
    • H04N21/25875—Management of end-user data involving end-user authentication
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/436—Interfacing a local distribution network, e.g. communicating with another STB or one or more peripheral devices inside the home
    • H04N21/43615—Interfacing a Home Network, e.g. for connecting the client to a plurality of peripherals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/436—Interfacing a local distribution network, e.g. communicating with another STB or one or more peripheral devices inside the home
    • H04N21/4363—Adapting the video stream to a specific local network, e.g. a Bluetooth® network
    • H04N21/43637—Adapting the video stream to a specific local network, e.g. a Bluetooth® network involving a wireless protocol, e.g. Bluetooth, RF or wireless LAN [IEEE 802.11]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/47—End-user applications
    • H04N21/485—End-user interface for client configuration
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/81—Monomedia components thereof
    • H04N21/8166—Monomedia components thereof involving executable data, e.g. software
    • H04N21/8173—End-user applications, e.g. Web browser, game
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q2220/00—Business processing using cryptography

Definitions

  • the present invention relates to a method of transmitting UI display information based on a smart contract, and more particularly, to a method of transmitting UI display information, which grants an authority to provide a new UI for an NFT content user by providing an authority to forward an NFT content and an authority to change an UI to the NFT content user and forwarding and assigning an authority to access the corresponding NFT content to a third user, thereby providing content services to which various UXs are reflected.
  • the mirroring technology is inefficient because the same display is output on multiple display devices and there are devices that output the display unnecessarily. Further, when the screen is divided into a plurality of areas and used, since the display in one device is divided and used, there is a problem in that the UI display information cannot be distributed and used in several display devices.
  • Prior Art 1 Korean Patent Application Laid-Open No. 10-2018-0081133
  • a display UI of a display is divided into at least two display areas in response to the finger joint touch gesture.
  • Prior Art 2 (Korean Patent Application Laid-Open No. 10-2010-0081383) is related to the touch screen dividing method, and is characterized in that a divided area layout of a touch screen is formed by receiving an input of two positions on the touch screen and a separate display is allocated to each of the formed divided areas.
  • Prior Art 2 also discloses the method of dividing the screen in one device, so that there is no motivation to display the divided screen on the plurality of devices in cooperation with the plurality of devices.
  • Prior Art 3 (US Patent Application Publication No. US20190373058 A1) is characterized in that user interaction information is synchronized by transmitting user interaction information between a plurality of devices to a remote server.
  • Prior Art 3 there is no motivation to display a divided screen in a plurality of devices in cooperation with the plurality of devices.
  • the first problem to be solved by the present invention is to provide a method of transmitting UI display information based on a smart contract, which grants an authority to provide a new UI for an Non-Fungible Token (NFT) content user by providing an authority to forward an NFT content and an authority to change an UI to the NFT content user and forwarding an authority to access the corresponding NFT content to a third user, thereby providing content services to which various UXs are reflected.
  • NFT Non-Fungible Token
  • the second problem to be solved by the present invention is to provide a local server transmitting UI display information based on a smart contract, which transmits UI display information independently from an OS by transmitting a UI element group included in UI distribution information as a video streaming.
  • the present invention provides a computer readable recording medium in which a program for executing the method in a computer is recorded.
  • the present invention provides a method of transmitting UI display information based on a smart contract, the method including: when an UI app provider, an NFT content provider, and a user conclude a smart contract effective for a specific period of time, dividing an UI provided by an UI app executed in a local server of the user into a plurality of UI element groups in an OS level; generating UI distribution information by matching the proximal clients to the plurality of UI element groups respectively; and transmitting one or more UI element groups to the matched proximal clients respectively, by using the UI distribution information or transmitting one or more UI element groups to the matched proximal clients as a video stream.
  • the method may further include authorizing or delegating an authority to divide the UI of the UI app into UI element groups to any one client among the proximal clients.
  • the UI arrangement information may be stored by matching UI element groups of different applications to the proximal clients.
  • the UI arrangement information may be transmitted to a cloud server.
  • the present invention provides local server transmitting UI display information based on a smart contract
  • the local server including: an UI element dividing module, which divides a UI provided by an UI app executed in a local server into a plurality of UI element groups in an OS level when an UI app server, an NFT content server, and the local server conclude a smart contract effective for a specific period of time; a UI distribution information generation module which generates UI distribution information by matching the proximal clients to the plurality of UI element groups respectively; and an UI element transmitting module which transmits one or more UI element groups to the matched proximal clients respectively by using the UI arrangement information or transmits video stream, for one or more UI element groups, to the proximal clients respectively.
  • the present invention provides a computer-readable recording medium in which a program for executing a method of transmitting the UI display information based on the smart contract by the local server of the user in a computer is recorded.
  • the present invention it is possible to provide content services to which various UXs are employed by granting an authority to forward an authority to access all NFT contents to a third user, and to provide a new NFT content-application to a digital content user with authorities to provide a new UI app, to divide a UI of the UI app, and to forward a authority to access a NFT content of the UI app.
  • UI display information independently from an OS by transmitting a video stream converted from UI element group included in UI distribution information.
  • the local server transmits an UI of an application capable of controlling an IoT device to a proximal client by a video streaming method, so that even a user device in which an IoT application is not installed is capable of monitoring and controlling the device.
  • FIG. 1 is a conceptual diagram illustrating transmission of UI display information to proximal clients by a local server in a network consisting of a local server 100, a proximal client 200, a cloud server 300, and a content providing server 400.
  • FIG. 2 illustrates a concept in which the local server 100 divides a UI of an original content into UI element groups by using UI tree information and transmitting the UI element groups to proximal clients 21 and 22 according to an exemplary embodiment of the present invention.
  • FIG. 3 is a configuration diagram illustrating the local server and a client according to the exemplary embodiment of the present invention.
  • FIG. 4 is a flowchart illustrating a method of transmitting, by the local server, UI display information to proximal clients according to the exemplary embodiment of the present invention.
  • FIG. 5 is a flowchart illustrating a method of transmitting, by the local server, UI display information to proximal clients according to another exemplary embodiment of the present invention.
  • FIG. 6 is a flowchart illustrating a method of transmitting, by the local server, UI display information to proximal clients according to still another exemplary embodiment of the present invention.
  • FIG. 7 illustrates a concept of transmitting a UI element group by a relay method when the local server transmits UI display information to proximal clients according to the exemplary embodiment of the present invention.
  • FIG. 8 is a flowchart illustrating a method of dividing, by the local server, UI tree information illustrated in FIG. 7 and transmitting the divided UI tree information to the client according to the exemplary embodiment of the present invention.
  • FIG. 9 illustrates a flow of a method of editing a UI of a content app by a client according to the exemplary embodiment of the present invention.
  • FIG. 10 illustrates a flow of a method of transmitting, by the local server, Internet of Things (IoT) state information to the client according to another exemplary embodiment of the present invention.
  • IoT Internet of Things
  • FIG. 11 illustrates a flow of a method of transmitting, by the local server, IoT state information to the client according to still another exemplary embodiment of the present invention.
  • FIG. 12 illustrates a flow of a method of transmitting, by the local server, IoT state information to an unauthorized client according to still another exemplary embodiment of the present invention.
  • FIG. 13 illustrates a flow of a method of receiving an NFT content and a UI app, by the local server, and transmitting UI display information according to still another exemplary embodiment of the present invention.
  • a method of transmitting UI information to a user device by using smart contract information including: when a smart contract is concluded between an NFT content provider, an UI app provider, and a user, dividing, by an OS of a first device of the user, a single UI of an UI app of the UI app provider into a plurality of UI element groups; generating, by the first user device or an application executed in the first user device, UI distribution information by matching the one UI element group to a second user device; transmitting, by the first user device or the application, the matched UI element group to the second user device by using the UI distribution information or converting the one matched UI element group, by using the UI distribution information, into a video stream and transmitting UI display information.
  • the present invention maximizes service scalability of a content application in consideration of a display size or resolution of a user display device in order to transmit free UI display information between display devices connected through a communication network.
  • the present invention presents an efficient interactive data transmitting method for improving user experience between devices of a user group in which a plurality of display devices registered by a user or display devices registered by multiple users are included.
  • a video, audio, image, touch, keyboard, and voice data may be shared or mirrored and forwarded between display devices connected through a communication network.
  • a content UI of a content application executed in the display device is composed of UI element groups, and provides a function capable of transmitting the UI in the unit of one or more individual UI element groups to a proximal display device and providing a UI element management authority function to enable various UI changes and combinations.
  • a local server having an authority to divide and distribute a content UI may transmit a video stream of the UI element group constituting the UI to the proximal display device.
  • each proximal display device when only UI element group information is transmitted to a proximal display device, each proximal display device generates a video stream by itself, and then each proximal display device transmits interaction information of a user to a local server.
  • the local server may input user interaction information received from each proximal display device to a content application.
  • a user app installed in the proximal display device for the present invention may virtualize a display by using the UI element group information received from the server and transmit UI information to a plurality of user devices.
  • a function supported at an OS level in the local server may be minimized to and configured as a UI dividing function (UI virtualization), UI distribution information storage, and UI element group transmission function, and then the UI transmission function using a virtual display configuration and video streaming required in the present invention may be configured to be performed in a user app installed in a user device.
  • UI virtualization UI dividing function
  • UI distribution information storage UI distribution information storage
  • UI element group transmission function a virtual display configuration and video streaming required in the present invention
  • the local server may update the UI element group information to be transmitted to each proximal display device by reflecting UI distribution information of a content application received from a cloud server.
  • the local server converts the UI element group information into a video stream and transmits the video stream to each proximal client device, there is an advantage in that it is possible to transmit the video stream independent to an OS of the client device in terms of a data communication aspect.
  • the present invention presents a method of easily monitoring or controlling, by a user, a proximal client device and an IoT device.
  • the local server transmits a UI of an IoT application capable of controlling an IoT device to a proximal client by a video streaming method, so that it is possible to monitor state information of the IoT device even in a user’s device in which the IoT application is not installed, or control the IoT device by transmitting interaction information of the user from the user device to the local server.
  • the present invention presents a method of managing a content authority which is capable of various user experience environments, and forwarding a content.
  • the present invention may provide a platform environment providing a UI change authority to change a UI of a content, an authority to provide a new UI of an application, and a content access authority assignment function to a user group capable of accessing a digital content of a user.
  • FIG. 1 is a conceptual diagram illustrating transmission of UI display information to proximal clients by a local server in a network consisting of a local server 100, a proximal client 200, a cloud server 300, and a content providing server 400.
  • a content application may request a content from the content providing server 400.
  • the content application installed in the local server 100 may directly access the content providing server 400 and request the corresponding content or request the content via the cloud server 300.
  • the local server 100 may install the content application by requesting the content application from the content providing server 400 and installing the received content application.
  • the proximal client 200 includes various user devices, such as a smart display 21, smart phones 22 and 23, a smart light bulb 25, and a smart camera 26.
  • the smart light bulb 25 and the smart camera 26 are an IoT device group.
  • the proximal client 200 may register its user device information in the local server 100.
  • Any one client in the proximal client 200 may request to register its own device information with the local server 100, and the local server 100 may transmit the registered user device information to the cloud server 300 which stores the user device information.
  • the devices of the proximal client 200 may be registered in the local server 100 as a user device group, and the corresponding user device information may be transmitted to and re-stored in the cloud server 300.
  • the user device group information a Service Set Identifier (SSID), a MAC address, a Universally Unique Identifier (UUID), or the like may be used, the user device group information may be differently configured according to a session from the execution of an app to the end of the app, or a effective time period from an installation of an app to a deletion of the app.
  • the user device group information may further include display size and resolution information.
  • the proximal client 200 and the local server 100 may be connected by using near-field communication, such as Bluetooth, ultrasound, or Zigbee, and an initial communication connection and authentication may be performed. Thereafter, UI display information between the authorized devices can be transmitted by using a home network connected by Wi-Fi.
  • near-field communication such as Bluetooth, ultrasound, or Zigbee
  • a new user device may be added to a Wi-Fi communication network by transmitting Wi-Fi connection information to another smart device or IoT device by an initial smart phone or set top box connected with the Wi-Fi router.
  • the Wi-Fi router serves as the local server 100
  • the set top box serves as the proximal client 200.
  • the local server 100 transmits UI display information to the proximal clients 200
  • the divided UIs of the content application may be shared and viewed by multiple clients.
  • the local server 100 receives a content of the content providing server 400 and transmits the received content to the proximal clients 200.
  • the UI display information of the content may be divided, matched, and transmitted for each client by a UI distribution control module 105 (see FIG. 3) implemented in the local server 100.
  • the UI distribution control module 105 installed in the local server 100 does not divide UI display information of the content by modifying a source code of an application of the content, but has a function of dividing UI tree information constituting UI display information at an Operating System (OS) level, into UI element groups; and rendering module is implemented, thereby dividing the UI display information and transmitting the divided UI display information to the plurality of user devices.
  • the content application installed in the local server 100 is received from the content providing server 400, and the local server is implemented in the form of a microkernel providing a minimum function required at the OS level to divide single UI information of the content to a plurality of UI element groups.
  • the local server or an application which is installed in the local server and includes a partial function of a UI editing control module, may match the plurality of UI element groups to proximal client devices and transmits the plurality of UI element groups by a video streaming method.
  • the proximal clients 200 may receive a UI video stream without installing the content application installed in the local server.
  • the local server or the application installed in the local server may transmit the matched UI element groups to the proximal client 200 devices without the conversion to the video stream.
  • the user application installed in the client for the present invention may convert the UI element group into a video stream and provide the UI display information to the client device.
  • the local server in order for the local server to easily distribute and transmit the UI to the proximal clients 200, an authority or a function of dividing and virtualizing UI tree information in the content at the OS level is required.
  • the local server 100 or the separate application installed in the local server may generate UI distribution information that matches the proximal clients 200 to the UI element group constituting the content UI display information by using the virtualized UI tree information.
  • an image of a movie content may be matched and transmitted to the smart display 21, a chatting window may be matched and transmitted to the first smart phone 22, and the subtitles of the movie may be matched and transmitted to the second smart phone 23.
  • the function of the present invention divides the single UI display information of the content application into each separable UI element group in the OS of the user device corresponding to the local server. Thereafter, the local server or the application installed in the local server may generate UI distribution information of the user device group by matching other user devices corresponding to the client devices to the UI element groups.
  • the UI distribution information may be stored up to the cloud server 300, and the backed-up UI distribution information may be stored as related information of a device group for each user.
  • a UI element to be matched to the added client may be updated in the UI distribution information.
  • the client of another user may also be added to the UI distribution information and updated.
  • the UI distribution information stored in the cloud server 300 may be shared or copied to the client of the invited user, and the UI distribution information may be updated by reflecting only the client that needs to be added or changed to the UI distribution information or be stored as separate UI distribution information selectable by a user.
  • a user device group information and the UI distribution information stored in the cloud server 300 may be used when the local server 100 has a failure or a physical change of the local server is required. Further, the UI distribution information of the local server 100 and the proximal clients may be easily modified by copying or modifying existing UI distribution information of the user stored in the cloud server 300. In this case, the clients have be successfully authenticated transmit the Wi-Fi network information to the proximal client and the local server to simplify local network setting.
  • the user generally monitors the state of the smart light 25 or the smart camera 26 registered in the device group for each user or removes the smart light 25 or the smart camera 26 through the content application.
  • UI element group when the content image is dark, a control signal for brightening the smart light may be included in the UI distribution information, and in the movie mode, a control signal for dimming the smart light may be included in the UI distribution information.
  • the smart light 25 or the smart camera 26 among the proximal clients 200 does not have a separate display, so that it is necessary to download and install a necessary application (app) to the smart phones 22 and 23 or the smart display 21 of the user and sets the smart phones 22 and 23 or the smart display 21, or it is necessary to set the smart light 25 or the smart camera 26 by accessing a homepage of a manufacturing company.
  • another user may receive UI information for IoT monitoring and control, required content information of the IoT device (lighting state, camera information, and IoT device information), from the local server.
  • the new user is capable of receiving the required UI display as a video stream only by installing the UI application of the present invention without installing the applications of various IoT devices, so that the new user is capable of receiving and controlling the video stream independent to the OS (Android, IOS, Window, Linux, and the like) of the user client device.
  • the IoT devices 25 and 26 are capable of transmitting the state change information to the local server 100 connected through the home network or transmitting the cloud server 300 connected through the Internet network.
  • the cloud server 300 may provide the user who is allowed to access IoT state information with the state information of the IoT device to be provided to the user through the Internet network corresponding to the present invention.
  • the two IoT apps are automatically synchronized.
  • the IoT app required for monitoring and controlling the IOT device may also be installed in the local server 100.
  • the local server 100 may allocate the UI element group for each proximal client 200 by using the UI tree information included in the content and match the UI element group to the proximal client 200.
  • the local server 100 configures required UI display information corresponding to the UI distribution information and display sizes and resolution of the proximal clients 200, and converts the UI element groups, matched to the proximal clients 200, into video streams or images, and transmits the video streams or images to the user device.
  • the UI distribution information required for controlling the IoT device may be shared with other user devices invited to the device group for each user.
  • the invited user device may receive a video stream or image information for monitoring and UI control information of the IoT device that is the same as that included in the user device group.
  • the user devices receive the UI control information as the video stream or the image information, so that there is an advantage in that it is possible to transmit the data independent to the OS systems.
  • FIG. 2 illustrates a concept in which the local server 100 divides a UI of an original content into UI element groups by using UI tree information and transmitting a UI element group to the proximal clients 21 and 22 by a video streaming according to an exemplary embodiment of the present invention.
  • the original content is transmitted to the local server 100 from the content providing server 400 and is transmitted to the proximal clients 21 and 22.
  • the local server 100 may modify a composition and a layout of the UI element groups by using the UI tree information included in the original content.
  • the local server 100 may divide the original content into a first UI element group and a second UI element group, match the first UI element group and the second UI element group to the smart display 21 and the smart phone 22 respectively, and then transmit the matched first UI element group and second UI element group by pixel streaming.
  • the data transmitted to the smart display is a video included in the original content, and is decoded in the local server and is transmitted to the smart display as a video stream, and when the second UI element is transmitted to the smart phone by encoded pixel streaming, and the smart phone decodes the received pixel stream.
  • the reproduction and the stop of the video stream reproduced in the smart display are determined according to an interaction of the user in the smart phone.
  • the decoding time of the video stream of the smart display needs to be synchronized based on the time in which the video stream is decoded in the smart phone.
  • the local server 100 includes the UI distribution control module 105
  • the smart display 21 includes a UI element display module 320 to a UI editing module 350
  • the smart phone 22 includes a UI element display module 321 or a UI editing module 351
  • FIG. 3 is a configuration diagram illustrating the local server and a client according to the exemplary embodiment of the present invention.
  • the local server 100 includes a receiving module 110, the UI distribution control module 105, and a transmitting module 190.
  • the UI distribution control module 105 includes a content app executing module 120, a UI element dividing module 130, a UI distribution information generating module 140, a display requesting module 150, a UI element transmitting module 160, a virtual display generating module 170, and a UI control information converting module 180.
  • the UI application installed in the local server 100 includes the entirety or a part of the configurations of the UI distribution control module 105.
  • the local server 100 and clients A and B 21 and 22 may be expressed as user devices.
  • clients A and B 21 and 22 includes app selecting modules 370 and 301, transmitting modules 310 and 311, UI element display modules 320 and 321, virtual display generating modules 330 and 331, UI control information input modules 340 and 341, UI editing modules 350 and 351, and receiving modules 360 and 361.
  • the local server 100 transmits UI display information to clients A and B 21 and 22 by video streaming or transmits the UI element groups to clients A and B 21 and 22, and receives UI control information by a user interaction from the client and generates a control signal of the content app.
  • Client B 22 selects a content app for which UI distribution information is to be changed and transmits the selected content app to the local server 100, and displays UI display information according to the changed UI distribution information.
  • Client A 21 is the client added for changing the UI distribution information of client B 22, and receives the UI element group of the content app from the local server 100 and displays the received UI element group.
  • the receiving module 110 receives whether to select the content app for which the UI distribution information is to be changed from client B 22.
  • the number of content apps selectable by client B 22 may be two or more.
  • the UI distribution control module 105 divides the UI display information of the content app into UI element groups, matches the divided UI element groups to each client, and generates UI distribution information.
  • the content app executing module 120 executes the content app selected by client B 22.
  • the content app is not required in clients A 21 and B 22 when the UI information is transmitted from the local server 100 by pixel streaming.
  • the content app executing module 120 may be replaced with an IoT app executing module and a UI app executing module combined with a Non-Fungible Token (NFT) content.
  • NFT Non-Fungible Token
  • the UI element dividing module 130 divides the UI provided in the content app executed in the content app executing module 120 into a plurality of UI element groups.
  • the UI distribution information generating module 140 matches the proximal clients and the plurality of UI element groups respectively, and generates UI distribution information.
  • the UI distribution information generating module 140 may determine the UI distribution information by a physical distance to the proximal client based on the local server 100 or a direction of the location of the proximal client. For example, the UI distribution information generating module 140 may determine to arrange the UI element group having a large display size in the client relatively far from the local server 100 and to arrange the UI element group having a small display size, such as a character input display, to the client relatively close to the local server 100.
  • the display requesting module 150 requests a UI element display to the client A 21 or the client B 22.
  • the local server 100 may request the UI element display module 320 to execute the UI element display through a push message.
  • the display requesting module 150 may make the client ready for the display of the received UI element group.
  • the UI element transmitting module 160 by using the UI distribution information, transmits one or more UI element groups, which is matched to the each proximal client respectively, or transmits video stream, converted from one or more UI element groups, which is matched to the each proximal client respectively.
  • the virtual display generating module 170 generates virtual display information by using the UI distribution information generated by the UI distribution information generating module 140.
  • the UI control information converting module 180 When the UI control information converting module 180 receives the UI control information from clients A 21 and B 22, the UI control information converting module 180 converts the received UI control information to a control signal of the content app.
  • the transmitting module 190 performs the display request of the display requesting module 150, and the transmission of the UI element group of the UI element transmitting module 160, and the like.
  • client A 21 and client B 22 are the same, so that client A 21 will be mainly described.
  • the app selecting module 370 selects a content app of which UI display information needs to be changed. In this case, it is not necessary to execute the content app as a result of the selection.
  • the transmitting module 310 transmits user interaction information about which the app selecting module 370 of content app has selected to the local server 100, and the UI control information input module 340 transmits the received UI control information to the local server 100.
  • the UI element display module 320 is executed in client A 21 by the request of the display requesting module 150 of the local server 100. It is preferable to receive the UI element group or the UI element video streaming in the state where the UI element display module 320 is executed in client A 21.
  • the virtual display generating module 330 renders the UI element group received from the local server 100 and stores the UI element group in a graphic buffer.
  • the UI control information input module 340 receives the input of the UI control information through a touch screen, a keyboard, a voice, a picture, and the like.
  • original data may be transmitted to a separate storage server, and information about the storage space may be changed to a web URL and transmitted.
  • the UI editing modules 350 and 351 may select at least one UI element group of the content app, and select a client to display the selected UI element group and change UI distribution information.
  • the receiving module 360 directly receives at least one UI element groups from the local server 100 or receives video stream for at least one UI element groups.
  • FIG. 4 is a flowchart illustrating a method of transmitting, by the local server, UI display information to proximal clients according to the exemplary embodiment of the present invention.
  • the method of transmitting UI display information according to the exemplary embodiment includes operations time-serially processed in the local server and the client illustrated in FIG. 3. Accordingly, even if omitted below, the contents described above with respect to the local server and the client illustrated in FIG. 3 are also applied to the method of transmitting the UI display information according to the exemplary embodiment.
  • client A 21, client B 22, and the local server 100 are connected through the home network, and may be mutually called by using the HTTP-based web API (REST service or Remote Procedure Calls (RPC) mechanism), and the TCP/IP protocol.
  • REST API may be applied for synchronization of the content apps
  • RPC may be applied for video streaming between devices
  • TCP/IP protocol may be applied to the transmit of the control information according to user interaction between devices.
  • the local server 100 receives UI control information according to a selection of a content app of a user from client B 22.
  • the local server transmits the UI control information to a content application.
  • the present invention is described based on one content application, but the app selecting module may select a plurality of content applications.
  • the UI distribution control module 105 of the local server 100 may divide the plurality of content application UIs into UI elements groups, and match the each UI element group to clients.
  • the content application is used for displaying the original content, and may be a video application, such as YouTube or Twitch.
  • client B 22 may receive the UI element group from the local server 100 by video streaming, or may receive UI element group information and then the client may render and display the UI element group.
  • the local server 100 executes the content app selected in operation 400.
  • the local server 100 divides the UI of the executed content app into a plurality of UI element groups.
  • the UI provided from the content application executed in the local server 100 may be divided into a plurality of UI elements by using the UI distribution control module 105 installed in the local server 100.
  • an authority for the functions of dividing the single UI of the content app into the plurality of UI element groups and generating and modifying the UI distribution information is granted only to a specific device of a user possessing the corresponding right, so that a security function of the system may be strengthened.
  • the local server 100 matches the proximal clients (21 and 22) and the plurality of UI element groups respectively, and generates UI distribution information.
  • client B 22 selects the content app in operation 400, it is preferable to select which client among the proximal clients and which UI element group will be matched to displaying the UI element group.
  • the local server 100 requests client A (21) to display the UI element group.
  • the request to display in operation 440 may be made before operation 440.
  • the local server 100 may make the UI element display module 320 to execute the UI element display by transmitting a push message to the user’s app.
  • the user’s app of client A (21) receiving a request to display the UI element group executes the UI element group display module 320.
  • client A may display the UI element group by receiving the video streaming received from the local server 100 in operation 472.
  • the local server 100 In operation 460, the local server 100 generates virtual display information by using the UI distribution information generated in operation 430.
  • the virtual display information means information stored to render UI element group information in a graphic buffer. Therefore, the graphic buffer may reflect the UI distribution information by using the virtual display information and transmit the video stream to each device.
  • the local server 100 transmits one or more UI element groups corresponding to client A (21) and client B (22), respectively, as a video stream by using the UI distribution information. Since the local server 100 transmits the UI element group as a video stream, it is not necessary to install and execute the content application in client A (21) and client B (22).
  • the local server 100 receives UI control information from client A (21) and client B (22).
  • the local server 100 converts the received UI control information to a control signal of the content app.
  • the UI control information may be transmitted to the local server through the TCP/IP protocol or the web API.
  • the local server 100 may convert the UI control information received from client A (21) and client B (22) into the control signal of the content app in consideration of display size and resolution information of each client possessed by the local server 100. Further, the user control information may be transmitted to the local server by providing a separate UI together with the video stream received from the user’s app installed in the client device. In this case, the client device needs to convert the user control signal and transmit the converted user control signal to the local server.
  • the local server 100 when the local server 100 receives a coordinate value input by the user on the display of client A (21), it is necessary to determine which element of the content app UI corresponds to the received coordinate value in consideration of the display size and resolution information. The determination may also be performed in client A (21) and may also be performed in the local server 100. Even though the local server 100 receives the coordinate value input by the user on the display of client A (21), the local server 100 has the display size and resolution information of client A (21), so that the local server 100 may determine which part of the content app may corresponds to the received coordinate value.
  • the UI control information may include content control information and UI distribution editing information, and the content control and the UI distribution editing are performed by the following method.
  • a context state value (or a parameter setting value of the content app) of the result of the distribution and its editing of selected UI by the client 21 and 22 may be transmitted to the local server 100.
  • the local server 100 performs with the authority to call the functions of the clients 21 and 22 necessary for collecting the context state value.
  • the local server 100 it is not necessary for the local server 100 to transmit display data for the UI distribution editing to the client 21 and 22, and when the display information is generated and the UI editing is completed in the client 21 and 22, it is preferable that the client 21 and 22 change the context state value and provide the changed context state value to the content app of the local server 100.
  • the local server 100 provides the information necessary for the information of UI distribution and its editing to the client 21 and 22 as a video stream.
  • the client 21 and 22 receives the UI distribution display transmitted by the local server 100 and transmits touch screen response results of the users in the client 21 and 22 to the local server 100
  • the local server 100 transmits the touch screen response as user interaction information to the content app.
  • the local server applies the updated UI information of the content app to the UI distribution information and transmits the UI distribution information as a video stream.
  • the user is capable of performing UI distribution editing.
  • the transmission of the video stream is possible independent to the OSs of the client 21 and 22, the client 21 and 22 may analyze the user’s control intention based on the response information of the touch screen or the touch pad and transmit a corresponding result to the local server 100.
  • FIG. 5 is a flowchart illustrating a method of transmitting, by the local server, UI display information to proximal clients according to another exemplary embodiment of the present invention.
  • the method of transmitting UI display information according to the exemplary embodiment includes operations time-serially processed in the local server and the client illustrated in FIG. 3. Accordingly, even if omitted below, the contents described above with respect to the local server and the client illustrated in FIG. 3 are also applied to the method of transmitting the UI display information according to the exemplary embodiment.
  • client A (21) receives the UI element group from the local server 100.
  • Client A (21) and client B (22) are connected with the local server 100 through the home network, and are capable of mutually calling through the HTTP-based web API (REST service or RPC mechanism).
  • HTTP-based web API REST service or RPC mechanism
  • Operation 400 to operation 460 are as illustrated in FIG. 4.
  • the local server 100 transmits a video stream for at least one UI element group corresponding to client B (22) by using the UI distribution information.
  • the local server 100 transmits at least one UI element group corresponding to client A 21 by using the UI distribution information.
  • the virtual display generating module 330 since the local server 100 transmits the UI element group, the virtual display generating module 330 generates virtual display information by using the UI element group.
  • client A (21) since client A (21) generates the virtual display information by using the received UI element group, client A (21) may have a function of a sub local server performing pixel streaming on another client.
  • client A (21) may have a function of a sub local server performing pixel streaming on another client.
  • An example of the case where the client has the function of the sub local server will be described in detail with reference to FIG. 7.
  • the local server 100 receives UI control information from client A (21) and client B (22).
  • the local server 100 converts the received UI control information to a control signal of the content app.
  • the local server 100 analyzes user interaction of the received UI control information, converts the user interaction into a control signal, and inputs the control signal to the content app that is being executed in the local server 100.
  • FIG. 6 is a flowchart illustrating a method of transmitting, by the local server, UI display information to proximal clients according to still another exemplary embodiment of the present invention.
  • the difference between the method of transmitting UI display information to the proximal clients by the local server 100 illustrated in FIG. 6 and the method illustrated in FIG. 5 is that both client A (21) and client B (22) receive the UI element groups from the local server 100.
  • clients 21 and 22 may have a function of a sub local server performing pixel streaming on another client.
  • Operation 400 to operation 440 are as illustrated in FIG. 4.
  • the local server 100 requests client B 22 to display the UI element group similar to operation 440.
  • client A (21) and client B (22) receiving the request to display the UI element group execute the UI element display modules 320 and 321, respectively.
  • the local server 100 In operation 460, the local server 100 generates virtual display information by using the UI distribution information generated in operation 430.
  • the client 21 and 22 may receive the UI element groups from the local server 100 in operations 473 and 474.
  • client A (21) and client B (22) Similar to operation 460 in which the local server 100 generates the virtual display information, in operations 475 and 476, client A (21) and client B (22) generate virtual display information by using the UI element groups received from the local server 100. In this case, client A/B may transmit a first UI element to itself as a video stream, and may transmit a second UI element to another user device as a video stream by using the generated virtual display information.
  • the local server 100 receives UI control information from client A 21 and client B 22.
  • the local server 100 converts the received UI control information to a control signal of the content app.
  • client A (21) or client B (22) may be defined as a local server candidate having the same authority as that of the local server 100.
  • the UI distribution information generating module 140 of the local server is synchronized to the client defined as the local server candidate by the web RPC or REST service method, the content app installed in the local server needs to be installed even in the client, and the local server 100 and the content app of the client need to be synchronized by the web API method.
  • the local server candidate client having the authority of the local server may serve as the local server by using the information of the synchronized elements.
  • the matched UI element group information may be transmitted to the user’s app for the present invention of the client device by using a transport layer for each OS, and then, the client device may generate a virtual display and perform video streaming by using the UI element group.
  • FIG. 7 illustrates a concept of transmitting a UI element group by a relay method when the local server transmits UI display information to proximal clients according to the exemplary embodiment of the present invention.
  • the local server extracts UI tree information included in the original content, virtualizes the single UI, and then divides the UI element groups constituting the UI tree information into group 1 and group 2.
  • the UI elements group divided into group 1 are stored as pixel data of a first graphic buffer, and the UI element groups divided into group 2 are stored without being converted to the pixel data.
  • the local server when the local server renders and transmits the UI element group 1, among the UI element groups divided into group 1 and group 2, and transmits. without rending, the UI element group 2 to the client device A, the device of client A forwards the received UI element group 2 to devices of clients B and C.
  • Clients B and C may forward and provide the UI display information between the devices by rendering the UI element group 2 by themselves and providing a UI display.
  • clients B and C of group 2 may generate UI control information and transmit the generated UI control information to client A, and client A may transmit its UI control information and the control information received from clients B and C to the local server.
  • client A may transmit its UI control information and the control information received from clients B and C to the local server.
  • the user’s app of the present invention may be configured so that the UI control information of clients B and C is directly transmitted to the local server without passing through client A.
  • FIG. 8 is a flowchart illustrating a method of dividing, by the local server, UI tree information illustrated in FIG. 7 and transmitting the divided UI tree information to the client according to the exemplary embodiment of the present invention.
  • the method of dividing UI tree information and transmitting the divided UI tree information to the client according to the exemplary embodiment includes operations time-serially processed in the local server and the client illustrated in FIG. 3. Accordingly, even if omitted below, the contents described above with respect to the local server and the client illustrated in FIG. 3 are also applied to the method of transmitting the UI display information according to the exemplary embodiment.
  • the local server 100 divides the UI tree information included in the content in consideration of the number of clients.
  • the local server 100 virtualizes the single UI tree.
  • the virtualization means managing the UI display information of the user’s device as a single UI tree in the content application, and managing the single UI tree as a UI group divided into UI element groups at the OS level. Accordingly, the main function of the virtualization is to manage the divided UI display information to be transmitted to the plurality of user devices as the information of UI element groups, and it is possible to prevent a malfunction according to the UI tree virtualization by transmitting the content application in which user interaction information for the UI display information, being divided into the plurality of UI groups and being transmitted to the plurality of devices, is collected and managed as the single UI tree.
  • the update of the user interaction information provides the user interaction information to the content application when a variable value of the main function of the UI element group is changed within a predetermined update cycle.
  • the local server 100 receives display resolution from client A (21).
  • the local server 100 generates a graphic buffer (for example, the virtual display) having the same resolution as the received display resolution.
  • a graphic buffer for example, the virtual display
  • the local server 100 renders the UI tree information divided in operation 800 to the graphic buffer corresponding to the client.
  • the local server 100 encodes pixel data stored in the graphic buffer and transmits the encoded pixel data to the corresponding client.
  • client A 21 decodes the received pixel data to a video stream and renders the video stream.
  • FIG. 9 illustrates a flow of a method of editing a UI of a content app by a client according to the exemplary embodiment of the present invention.
  • the method of editing a UI of a content app by the client includes operations time-serially processed in the local server and the client illustrated in FIG. 3. Accordingly, even if omitted below, the contents described above with respect to the local server and the client illustrated in FIG. 3 are also applied to the method of transmitting the UI display information according to the exemplary embodiment.
  • client A (21) transmits selected content app to the local server 100.
  • the local server 100 executes the selected content app.
  • the local server 100 may download the content app through the cloud server 300 and execute the downloaded content app.
  • client A (21) receives the UI of the content app from the local server 100 by video streaming.
  • client A (21) displays the received UI of the content app.
  • client A (21) executes a UI setting mode of the content app.
  • client A (21) selects at least one UI element group of the content app, and selects a client which is to display the selected UI element group.
  • client A (21) transmits UI distribution information matching a client information to the selected UI element group for display to the local server 100.
  • the local server 100 transmits a UI element group to the matched client that displays the UI element group with reference to the UI distribution information.
  • FIG. 10 illustrates a flow of a method of transmitting, by the local server, IoT state information to the client device in an OS level according to another exemplary embodiment of the present invention.
  • the method of transmitting IoT state information to the client according to the exemplary embodiment includes operations time-serially processed in the local server and the client illustrated in FIG. 3. Accordingly, even if omitted below, the contents described above with respect to the local server and the client illustrated in FIG. 3 are also applied to the method of transmitting the IoT state information according to the exemplary embodiment.
  • the local server 100 receives information about the IoT application selected for receiving IoT state information by a user from the client A (21).
  • the selectable IoT applications are listed in client A (21), and when the user selects any one IoT application, the local server 100 receives which IoT application is selected as user UI control information.
  • the local server 100 executes the selected IoT application.
  • the local server 100 may download the IoT application through the cloud server 300 of the IoT device.
  • the local server 100 divides a UI of the executed IoT application into a plurality of UI element groups.
  • the local server 100 generates UI distribution information by using the plurality of divided UI elements groups.
  • the local server 100 In operation 1040, the local server 100 generates virtual display information by using the generated UI distribution information.
  • the local server 100 may transmit the virtual display information to client A (21) by video streaming or transmit the matched UI element group to the client device.
  • the local server 100 receives UI control information from the user’s app of client A (21).
  • the local server 100 converts the received UI control information to a control signal of the IoT application.
  • the local server 100 transmits the IoT control signal to an IoT device 25 through the cloud or the home network.
  • the IoT device 25 may be controlled through the home network using ZigBee/Wi-Fi.
  • the IoT cloud server transmits the IoT control signal to the IoT device through the IoT hub by the TCP/IP communication by referring to an IP address information uniquely matched to the IoT device 25.
  • the IoT app of the local server 100 receives IoT state information through the IoT device 25 or the IoT cloud server.
  • the local server 100 transmits the IoT state information to the user’s app of client A (21) as the video stream or the UI element group information.
  • FIG. 11 illustrates a flow of a method of transmitting, by the cloud server, IoT state information to the client device according to another exemplary embodiment of the present invention.
  • the method of transmitting IoT state information to the client according to the exemplary embodiment includes operations time-serially processed in the local server and the user of the client illustrated in FIG. 3. Accordingly, even if omitted below, the contents described above with respect to the local server and the user app of the client illustrated in FIG. 3 are also applied to the method of transmitting the IoT state information according to the exemplary embodiment.
  • FIG. 11 is different from FIG. 10 in that client A (21) located outside the home network receives the IoT state information through the cloud server 300.
  • the UI virtualization information of the cloud server 300 and the local server 100, the UI distribution information, the user device information matched to the UI distribution information, and the like are synchronized through the web API.
  • the cloud server 300 receives user control information for selecting an IoT application by the user from client A (21).
  • the cloud server 300 determines whether the device of client A (21) is a registered user device in the user device group information.
  • the local server 100 receives information about the IoT application selected by the user of client A (21) from the cloud server 300.
  • the local server 100 executes the selected IoT application.
  • the local server 100 divides a UI of the executed IoT application into a plurality of UI element groups at an OS level.
  • the local server 100 In operation 1150, the local server 100 generates UI distribution information by using the plurality of divided UI elements groups. Further, the local server transmits a updated UI distribution information of the local server to the cloud server to synchronizes the UI distribution information between two devices.
  • the local server 100 In operation 1160, the local server 100 generates the UI of the IoT app of the local server synchronized with the IoT app of the cloud server as virtual display information by using the generated UI distribution information.
  • the local server 100 transmits a UI display of the matched IoT app to the device of client A (21) by video streaming by using the virtual display information, or transmit the matched UI element group information.
  • client A (21) transmits UI control information to the cloud server 300, and when client A (21) transmits the UI control information to the IoT app installed in the cloud server, context state information of the IoT app of the local server is updated by the cloud server of the IoT device.
  • the IoT app of the local server 100 transmits the received UI control information to the IoT device.
  • the IoT app of the local server 100 transmits IoT control signal to the IoT cloud server through the Internet web, or when the local server serves as the IoT hub, the local server directly transmits the IoT control signal to the IoT device through the home network.
  • the IoT app of the local server 100 receives the state information of the IoT device via the cloud of the IoT app and the IoT hub, or directly receives the state information from the IoT device through the home network when the local server serves as the IoT hub.
  • the IoT app when the context state information of the IoT app of the local server 100 is changed, the IoT app provides the cloud of the IoT device with the changed state information, or transmits the changed state information to the IoT app of the cloud server 300 when the local server serves as the IoT hub.
  • the hub for controlling the IoT device controls the IoT device on the home network by using ZigBee, Bluetooth, or Wi-Fi network.
  • FIG. 12 illustrates a flow of a method of transmitting, by the local server, IoT state information to an unauthorized client according to another exemplary embodiment of the present invention.
  • the method of transmitting, by the local server, the IoT state information to a non-authorized client is different from the methods illustrated in FIGS. 10 and 11 in that a non-authorized client 30 requests the local server 100 to transmit the IoT state information.
  • the local server 100 sets IoT state information providing range according to a distance to the client device.
  • the local server 100 may differentiate and provide the IoT state information according to the location of the non-authorized client 30. There is a case where the non-authorized client 30 is physically close to the local server 100 and there is a case where the non-authorized client 30 requests the IoT state information through the home network or the Internet at a remote distance, and it is possible to differentiate and provide the IoT state information according to the physical distance between the local server 100 and the non-authorized client 30.
  • IoT state information when the location of the non-authorized client 30 is physically present within a predetermined distance from the local 100, more IoT state information may be set to be provided.
  • the local server 100 may differentiate and provide the state information of the IoT device 25.
  • the local server 100 receives a transmission request of an IoT state information from the user’s app of the non-authorized client 30.
  • the local server 100 calculates a distance to the non-authorized client 30 based on the transmission request of IoT state information received in operation 1210.
  • the paired antennas may be implemented to receive horizontal polarization and some other antennas may be implemented to receive vertical polarization.
  • the local server 100 may receive a wireless communication signal from the non-authorized client 30 through the paired vertical polarization/horizontal polarization multi-antennas, and accurately calculate a location of the non-authorized client 30 by a arrival time difference according to a difference in the incident angles of the received signals between the paired multi-antennas.
  • the local server 100 may determine the IoT state information to be provided to the user’s app of the non-authorized client 30 by calculating the distance between the local server 100 and the non-authorized client 30.
  • the local server 100 transmits the transmission request of the IoT state information to the IoT device 25.
  • the local server 100 receives the IoT state information from the IoT device 25.
  • the local server 100 determines the IoT state information and the UI distribution information to be transmitted to the non-authorized client 30 according to the distance calculated in operation 1220.
  • the local server 100 In operation 1260, the local server 100 generates virtual display information by using the UI distribution information.
  • the local server 100 transmits the IoT state information as a video streaming to the user’s app of the non-authorized client device.
  • the IoT state information is transmitted from the local server 100 to the non-authorized client 30 as a video stream and is displayed on a screen. Accordingly, the installation of the IoT application, required for receiving the IoT state information of plurality of IoT devices, is not required in the non-authorized client 30.
  • FIG. 13 illustrates a flow of a method of receiving an NFT content and a UI app, by the local server, and transmitting UI display information according to another exemplary embodiment of the present invention.
  • the method of transmitting UI display information according to the exemplary embodiment includes operations time-serially processed in the local server and the client illustrated in FIG. 3. Accordingly, even if omitted below, the contents described above with respect to the local server and the client illustrated in FIG. 3 are also applied to the method of transmitting the UI display information according to the exemplary embodiment.
  • FIG. 13 illustrates a method in which a local server 20 or client A (21) uses a UI app including NFT content for a predetermined period by transmitting the NFT content of an NFT content sever 500 and the UI app of a UI app server 600 to the local server 20.
  • the user mutually concludes a smart contract with an NFT content provider 500 and a UI app provider 600.
  • Digital assets are managed by blockchain-based smart contracts, and digital asset tokens are largely divided into Fungible Tokens (FT) and Non-Fungible Tokens (NFT).
  • FT Fungible Tokens
  • NFT Non-Fungible Tokens
  • ERC-20 is defined as a fungible token
  • ERC-721 token is defined as a non-fungible token
  • ERC-1155 is a token that can manage ERC-20 and ERC-721 tokens through a single smart contract.
  • the NFT content provider/server 500, the UI app provider/server 600, and the user conclude a smart contract related to ownership/use of the NFT content and the UI app based on ERC-1155.
  • the UI app provider/server 600, the NFT content provider/server 500, and the user/local server 20 conclude the smart contract effective for a specific period of time
  • the NFT content and the UI app may be transmitted to the local server 20.
  • the local server 20 is granted to access the NFT contents for the predetermined period of time and is allowed to access as a sole user.
  • the smart contract may include a condition of limiting display transmission of the NFT content transmitted to the plurality of user devices , which is a present invention limiting the number of authorized user devices included in the user device group.
  • client A (21) selects the NFT content and the UI app to be used and transmits the selected NFT content and UI app to the local server 100.
  • the local server 20 may generate UI distribution information by dividing a single UI of the UI app into the units of UI element groups at the OS level and matching the divided UI element groups to the client devices of the user, and transmit one UI element group to the matched client A (21).
  • Client A (21) may select one of the plurality of selectable UI apps stored in the UI app server 600, combine the selected UI app with the NFT content, and display the combined NFT content.
  • the local server 100 checks whether the smart contract is concluded between the UI app server 600 and the NFT content server 500, and then receives the UI app selected by client A (21) from the UI app server 600.
  • the local server 100 checks whether the smart contract is concluded between the UI app server 600 and the NFT content server 500 and then receives the NFT content from the NFT content server 500.
  • the local server 100 executes the selected UI app.
  • the local server 100 divides a UI of the executed UI app into a plurality of UI element groups at an OS level.
  • the local server 100 matches the divided UI element groups to each of the client devices of the user respectively, and generates UI distribution information.
  • the UI distribution control module 105 transmits the generated UI distribution information to the cloud server 300.
  • the local server 100 In operation 1360, the local server 100 generates virtual display information by using the generated UI distribution information.
  • the local server 100 transmits the UI element group to client A (21). Instead of transmitting the UI element group to client A (21), the local server 100 may transmit a video stream for the UI element group to the matched client A (21) in operation 1370.
  • client A (21) generates virtual display information by using the UI element group received from the local server 100.
  • the local server 100 receives UI control information from client A (21).
  • the local server 100 converts the UI control information to a control signal of the UI app.
  • the local server 100 may apply the control signal to the UI app and re-transmit the UI element group or the video stream.
  • the local server 100 may delegate the authority to use the UI app and the NFT content to client A (21).
  • the smart contract between the UI app server 600 and the NFT content server 500 is updated including client A (21), instead of the local server 100, and the UI distribution control module 105 is executable in client A (21).
  • the cloud server 300 may manage information, such as the user’s authority to access to the NFT content, the authority to provide the UI by using the UI app, the authority to correct the UI distribution information, and the delegation of the authority to use.
  • the NFT content user may be delegated all of the above authorities, use the NFT content for a predetermined period of time, and delegate the authority to another user to distribute the NFT content.
  • the NFT content user may stream and use the NFT content in his/her display device by changing the UI with a style that he/she wants for a predetermined period of time. Then, when the predetermined period of time elapses based on the contents of the smart contract, the authorities to use of all of the users, except for the content user may be reset.
  • the exemplary embodiments of the present invention may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer readable medium.
  • the computer readable medium may include a program command, a data file, a data structure, and the like alone or in combination.
  • the program command recorded in the medium may be specially designed and configured for the present invention, or may also be known and usable to those skilled in computer software.
  • Examples of the computer readable recording medium include a magnetic medium, such as a hard disk, a floppy disk, or a magnetic tape, an optical recording medium, such as a CD-ROM or a DVD, a magneto-optical medium, such as a floptical disk, and a hardware device which is specifically configured to store and execute the program command such as a ROM, a RAM, and a flash memory.
  • An example of the program command includes a high-level language code executable by a computer by using an interpreter, and the like, as well as a machine language code created by a compiler.
  • the hardware device may be configured to be operated with one or more software modules in order to perform the operation of the present invention, and an opposite situation thereof is available.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Business, Economics & Management (AREA)
  • Databases & Information Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Graphics (AREA)
  • Human Computer Interaction (AREA)
  • Human Resources & Organizations (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Strategic Management (AREA)
  • Economics (AREA)
  • Tourism & Hospitality (AREA)
  • General Business, Economics & Management (AREA)
  • Marketing (AREA)
  • Software Systems (AREA)
  • Computer Security & Cryptography (AREA)
  • Health & Medical Sciences (AREA)
  • Primary Health Care (AREA)
  • General Health & Medical Sciences (AREA)
  • Computing Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Operations Research (AREA)
  • Quality & Reliability (AREA)
  • Information Transfer Between Computers (AREA)

Abstract

The present invention relates to a method of transmitting UI display information based on a smart contract, the method including: when a UI app server, an NFT content server, and a local server conclude a smart contract effective for a specific period of time, dividing an UI, the UI provided by an UI app executed in the local server, into a plurality of UI elements by using a UI distribution control module installed in the local server; generating, by the UI distribution control module, UI distribution information by matching the proximal clients to the plurality of UI element groups respectively; and transmitting, by the UI distribution control module, one or more UI element groups to the matched proximal clients respectively by using the UI distribution information or transmitting a video stream of one or more UI element groups to the matched proximal clients respectively; authorizing a NFT content user to provide a new UI by providing an authorities to forward an NFT content and to change an UI to the NFT content user; and forwarding an authority to access the corresponding NFT content to a third user, thereby providing content services which various UXs are reflected.

Description

METHOD OF TRANSMITTING UI DISPLAY INFORMATION BASED ON SMART CONTRACT
The present invention relates to a method of transmitting UI display information based on a smart contract, and more particularly, to a method of transmitting UI display information, which grants an authority to provide a new UI for an NFT content user by providing an authority to forward an NFT content and an authority to change an UI to the NFT content user and forwarding and assigning an authority to access the corresponding NFT content to a third user, thereby providing content services to which various UXs are reflected.
In the prior art, in a technology for sharing User Interface (UI) display information between two or more display devices, it is common to mirror one display to another display or divide the display into a plurality of areas to display different contents in each divided area.
In particular, the mirroring technology is inefficient because the same display is output on multiple display devices and there are devices that output the display unnecessarily. Further, when the screen is divided into a plurality of areas and used, since the display in one device is divided and used, there is a problem in that the UI display information cannot be distributed and used in several display devices.
Further, in order to control an IoT device in a smart device, or check state information of the IoT device, a separate application capable of accessing the IoT device needs to be inconveniently installed and there is a problem in that as the number of IoT devices increases, the number of installed applications continues to increase.
Prior Art 1 (Korean Patent Application Laid-Open No. 10-2018-0081133) is the prior art related to a rapid screen dividing method, and is characterized in that when a finger joint touch gesture acting on a touch-sensitive surface is detected, a display UI of a display is divided into at least two display areas in response to the finger joint touch gesture.
Referring to Prior Art 1, only the method of easily dividing the screen is disclosed, and the configuration of whether the divided screen can be efficiently displayed in cooperation with proximal devices is not disclosed.
Prior Art 2 (Korean Patent Application Laid-Open No. 10-2010-0081383) is related to the touch screen dividing method, and is characterized in that a divided area layout of a touch screen is formed by receiving an input of two positions on the touch screen and a separate display is allocated to each of the formed divided areas. Prior Art 2 also discloses the method of dividing the screen in one device, so that there is no motivation to display the divided screen on the plurality of devices in cooperation with the plurality of devices.
Prior Art 3 (US Patent Application Publication No. US20190373058 A1) is characterized in that user interaction information is synchronized by transmitting user interaction information between a plurality of devices to a remote server. In Prior Art 3, there is no motivation to display a divided screen in a plurality of devices in cooperation with the plurality of devices.
Further, in the related art, when various display devices share a UI display, there is a problem in that an application for displaying the corresponding UI display must be installed in all of the display devices for the execution. Further, in the case of different OSs between display devices, it is difficult to share a UI display, so that there is difficulty in sharing UI information between multiple devices.
Therefore, the first problem to be solved by the present invention is to provide a method of transmitting UI display information based on a smart contract, which grants an authority to provide a new UI for an Non-Fungible Token (NFT) content user by providing an authority to forward an NFT content and an authority to change an UI to the NFT content user and forwarding an authority to access the corresponding NFT content to a third user, thereby providing content services to which various UXs are reflected.
The second problem to be solved by the present invention is to provide a local server transmitting UI display information based on a smart contract, which transmits UI display information independently from an OS by transmitting a UI element group included in UI distribution information as a video streaming.
Further, the present invention provides a computer readable recording medium in which a program for executing the method in a computer is recorded.
In order to achieve the first problem, the present invention provides a method of transmitting UI display information based on a smart contract, the method including: when an UI app provider, an NFT content provider, and a user conclude a smart contract effective for a specific period of time, dividing an UI provided by an UI app executed in a local server of the user into a plurality of UI element groups in an OS level; generating UI distribution information by matching the proximal clients to the plurality of UI element groups respectively; and transmitting one or more UI element groups to the matched proximal clients respectively, by using the UI distribution information or transmitting one or more UI element groups to the matched proximal clients as a video stream.
According to an exemplary embodiment of the present invention, the method may further include authorizing or delegating an authority to divide the UI of the UI app into UI element groups to any one client among the proximal clients.
Further, when two or more UI apps are selected and installed, the UI arrangement information may be stored by matching UI element groups of different applications to the proximal clients.
According to another exemplary embodiment of the present invention, when the local server and the proximal clients are connected through a home network, the UI arrangement information may be transmitted to a cloud server.
In order to resolve the second problem, the present invention provides local server transmitting UI display information based on a smart contract, the local server including: an UI element dividing module, which divides a UI provided by an UI app executed in a local server into a plurality of UI element groups in an OS level when an UI app server, an NFT content server, and the local server conclude a smart contract effective for a specific period of time; a UI distribution information generation module which generates UI distribution information by matching the proximal clients to the plurality of UI element groups respectively; and an UI element transmitting module which transmits one or more UI element groups to the matched proximal clients respectively by using the UI arrangement information or transmits video stream, for one or more UI element groups, to the proximal clients respectively.
In order to solve another technical problem, the present invention provides a computer-readable recording medium in which a program for executing a method of transmitting the UI display information based on the smart contract by the local server of the user in a computer is recorded.
According to the present invention, it is possible to provide content services to which various UXs are employed by granting an authority to forward an authority to access all NFT contents to a third user, and to provide a new NFT content-application to a digital content user with authorities to provide a new UI app, to divide a UI of the UI app, and to forward a authority to access a NFT content of the UI app.
Further, according to the present invention, it is possible to transmit UI display information independently from an OS by transmitting a video stream converted from UI element group included in UI distribution information.
Further, according to the present invention, it is possible to variously combine UI element groups and provide a plurality of devices with the combined UI element groups by transceiving UI display information of a content application in the unit of an UI element group.
According to the present invention, the local server transmits an UI of an application capable of controlling an IoT device to a proximal client by a video streaming method, so that even a user device in which an IoT application is not installed is capable of monitoring and controlling the device.
FIG. 1 is a conceptual diagram illustrating transmission of UI display information to proximal clients by a local server in a network consisting of a local server 100, a proximal client 200, a cloud server 300, and a content providing server 400.
FIG. 2 illustrates a concept in which the local server 100 divides a UI of an original content into UI element groups by using UI tree information and transmitting the UI element groups to proximal clients 21 and 22 according to an exemplary embodiment of the present invention.
FIG. 3 is a configuration diagram illustrating the local server and a client according to the exemplary embodiment of the present invention.
FIG. 4 is a flowchart illustrating a method of transmitting, by the local server, UI display information to proximal clients according to the exemplary embodiment of the present invention.
FIG. 5 is a flowchart illustrating a method of transmitting, by the local server, UI display information to proximal clients according to another exemplary embodiment of the present invention.
FIG. 6 is a flowchart illustrating a method of transmitting, by the local server, UI display information to proximal clients according to still another exemplary embodiment of the present invention.
FIG. 7 illustrates a concept of transmitting a UI element group by a relay method when the local server transmits UI display information to proximal clients according to the exemplary embodiment of the present invention.
FIG. 8 is a flowchart illustrating a method of dividing, by the local server, UI tree information illustrated in FIG. 7 and transmitting the divided UI tree information to the client according to the exemplary embodiment of the present invention.
FIG. 9 illustrates a flow of a method of editing a UI of a content app by a client according to the exemplary embodiment of the present invention.
FIG. 10 illustrates a flow of a method of transmitting, by the local server, Internet of Things (IoT) state information to the client according to another exemplary embodiment of the present invention.
FIG. 11 illustrates a flow of a method of transmitting, by the local server, IoT state information to the client according to still another exemplary embodiment of the present invention.
FIG. 12 illustrates a flow of a method of transmitting, by the local server, IoT state information to an unauthorized client according to still another exemplary embodiment of the present invention.
FIG. 13 illustrates a flow of a method of receiving an NFT content and a UI app, by the local server, and transmitting UI display information according to still another exemplary embodiment of the present invention.
According to the present invention, there is provide a method of transmitting UI information to a user device by using smart contract information, the method including: when a smart contract is concluded between an NFT content provider, an UI app provider, and a user, dividing, by an OS of a first device of the user, a single UI of an UI app of the UI app provider into a plurality of UI element groups; generating, by the first user device or an application executed in the first user device, UI distribution information by matching the one UI element group to a second user device; transmitting, by the first user device or the application, the matched UI element group to the second user device by using the UI distribution information or converting the one matched UI element group, by using the UI distribution information, into a video stream and transmitting UI display information.
Prior to the description of the specific contents of the present invention, for the convenience of understanding, the outline of the solution to the problem to be solved by the present invention or the core of the technical spirit is first presented.
The present invention maximizes service scalability of a content application in consideration of a display size or resolution of a user display device in order to transmit free UI display information between display devices connected through a communication network.
The present invention presents an efficient interactive data transmitting method for improving user experience between devices of a user group in which a plurality of display devices registered by a user or display devices registered by multiple users are included.
A video, audio, image, touch, keyboard, and voice data may be shared or mirrored and forwarded between display devices connected through a communication network. In this case, a content UI of a content application executed in the display device is composed of UI element groups, and provides a function capable of transmitting the UI in the unit of one or more individual UI element groups to a proximal display device and providing a UI element management authority function to enable various UI changes and combinations.
A local server having an authority to divide and distribute a content UI may transmit a video stream of the UI element group constituting the UI to the proximal display device. As another method, when only UI element group information is transmitted to a proximal display device, each proximal display device generates a video stream by itself, and then each proximal display device transmits interaction information of a user to a local server. The local server may input user interaction information received from each proximal display device to a content application. In this case, a user app installed in the proximal display device for the present invention may virtualize a display by using the UI element group information received from the server and transmit UI information to a plurality of user devices. Accordingly, a function supported at an OS level in the local server may be minimized to and configured as a UI dividing function (UI virtualization), UI distribution information storage, and UI element group transmission function, and then the UI transmission function using a virtual display configuration and video streaming required in the present invention may be configured to be performed in a user app installed in a user device.
Further, the local server may update the UI element group information to be transmitted to each proximal display device by reflecting UI distribution information of a content application received from a cloud server. In the case where the local server converts the UI element group information into a video stream and transmits the video stream to each proximal client device, there is an advantage in that it is possible to transmit the video stream independent to an OS of the client device in terms of a data communication aspect.
Further, the present invention presents a method of easily monitoring or controlling, by a user, a proximal client device and an IoT device.
In particular, the local server transmits a UI of an IoT application capable of controlling an IoT device to a proximal client by a video streaming method, so that it is possible to monitor state information of the IoT device even in a user’s device in which the IoT application is not installed, or control the IoT device by transmitting interaction information of the user from the user device to the local server.
Further, the present invention presents a method of managing a content authority which is capable of various user experience environments, and forwarding a content. The present invention may provide a platform environment providing a UI change authority to change a UI of a content, an authority to provide a new UI of an application, and a content access authority assignment function to a user group capable of accessing a digital content of a user.
Hereinafter, an exemplary embodiment in which the present invention can be easily carried out by those skilled in the art will be described in detail with reference to the accompanying drawings. However, the exemplary embodiments are for explaining the present invention in more detail, and it will be apparent to those skilled in the art that the scope of the present invention is not limited by the exemplary embodiments.
The configuration of the invention for clarifying the solution of the problem to be solved by the present invention will be described in detail with reference to the accompanying drawings based on the exemplary embodiment of the present invention, in assigning reference numbers to the components of the drawings, the same reference numbers are given to the same components even if they are in different drawings, and it should be noted in advance that components of other drawings can be cited when necessary in the description of the drawings. In addition, when it is determined that detailed descriptions of well-known functions or configurations related to the present invention and other matters may unnecessarily obscure the gist of the present invention in explaining the principle of operation of the exemplary embodiment of the present invention in detail, the detailed description is omitted.
FIG. 1 is a conceptual diagram illustrating transmission of UI display information to proximal clients by a local server in a network consisting of a local server 100, a proximal client 200, a cloud server 300, and a content providing server 400.
In order for the local server 100 to provide display information of a content application (CA) selected by a user to a proximal client 200 device, a content application may request a content from the content providing server 400. In this case, the content application installed in the local server 100 may directly access the content providing server 400 and request the corresponding content or request the content via the cloud server 300. In this case, when the content application required for receiving the content of the content providing server 400 is not installed in the local server 100, the local server 100 may install the content application by requesting the content application from the content providing server 400 and installing the received content application.
Referring to FIG. 1, the proximal client 200 includes various user devices, such as a smart display 21, smart phones 22 and 23, a smart light bulb 25, and a smart camera 26. The smart light bulb 25 and the smart camera 26 are an IoT device group.
The proximal client 200 may register its user device information in the local server 100.
Any one client in the proximal client 200 may request to register its own device information with the local server 100, and the local server 100 may transmit the registered user device information to the cloud server 300 which stores the user device information.
That is, the devices of the proximal client 200, such as a set top box or a hub, may be registered in the local server 100 as a user device group, and the corresponding user device information may be transmitted to and re-stored in the cloud server 300. As the user device group information, a Service Set Identifier (SSID), a MAC address, a Universally Unique Identifier (UUID), or the like may be used, the user device group information may be differently configured according to a session from the execution of an app to the end of the app, or a effective time period from an installation of an app to a deletion of the app. The user device group information may further include display size and resolution information.
The proximal client 200 and the local server 100 may be connected by using near-field communication, such as Bluetooth, ultrasound, or Zigbee, and an initial communication connection and authentication may be performed. Thereafter, UI display information between the authorized devices can be transmitted by using a home network connected by Wi-Fi.
Further, in a connection of a Wi-Fi router between the proximal client 200 and the local server 100 for data communication, a new user device may be added to a Wi-Fi communication network by transmitting Wi-Fi connection information to another smart device or IoT device by an initial smart phone or set top box connected with the Wi-Fi router. When the Wi-Fi router serves as the local server 100, the set top box serves as the proximal client 200.
When the local server 100 transmits UI display information to the proximal clients 200, the divided UIs of the content application may be shared and viewed by multiple clients.
The local server 100 receives a content of the content providing server 400 and transmits the received content to the proximal clients 200. In this case, the UI display information of the content may be divided, matched, and transmitted for each client by a UI distribution control module 105 (see FIG. 3) implemented in the local server 100.
In particular, according to the exemplary embodiment of the present invention, the UI distribution control module 105 installed in the local server 100 does not divide UI display information of the content by modifying a source code of an application of the content, but has a function of dividing UI tree information constituting UI display information at an Operating System (OS) level, into UI element groups; and rendering module is implemented, thereby dividing the UI display information and transmitting the divided UI display information to the plurality of user devices. The content application installed in the local server 100 is received from the content providing server 400, and the local server is implemented in the form of a microkernel providing a minimum function required at the OS level to divide single UI information of the content to a plurality of UI element groups. Further, the local server or an application, which is installed in the local server and includes a partial function of a UI editing control module, may match the plurality of UI element groups to proximal client devices and transmits the plurality of UI element groups by a video streaming method. Accordingly, the proximal clients 200 may receive a UI video stream without installing the content application installed in the local server. Further, the local server or the application installed in the local server may transmit the matched UI element groups to the proximal client 200 devices without the conversion to the video stream. In this case, the user application installed in the client for the present invention may convert the UI element group into a video stream and provide the UI display information to the client device.
In the present invention, in order for the local server to easily distribute and transmit the UI to the proximal clients 200, an authority or a function of dividing and virtualizing UI tree information in the content at the OS level is required. The local server 100 or the separate application installed in the local server may generate UI distribution information that matches the proximal clients 200 to the UI element group constituting the content UI display information by using the virtualized UI tree information.
For example, an image of a movie content may be matched and transmitted to the smart display 21, a chatting window may be matched and transmitted to the first smart phone 22, and the subtitles of the movie may be matched and transmitted to the second smart phone 23.
Therefore, the function of the present invention divides the single UI display information of the content application into each separable UI element group in the OS of the user device corresponding to the local server. Thereafter, the local server or the application installed in the local server may generate UI distribution information of the user device group by matching other user devices corresponding to the client devices to the UI element groups. The UI distribution information may be stored up to the cloud server 300, and the backed-up UI distribution information may be stored as related information of a device group for each user. When another client of the user is added to the user device group, a UI element to be matched to the added client may be updated in the UI distribution information.
When a client of another user is invited and included in a device group for each user, the client of another user may also be added to the UI distribution information and updated. The UI distribution information stored in the cloud server 300 may be shared or copied to the client of the invited user, and the UI distribution information may be updated by reflecting only the client that needs to be added or changed to the UI distribution information or be stored as separate UI distribution information selectable by a user.
A user device group information and the UI distribution information stored in the cloud server 300 may be used when the local server 100 has a failure or a physical change of the local server is required. Further, the UI distribution information of the local server 100 and the proximal clients may be easily modified by copying or modifying existing UI distribution information of the user stored in the cloud server 300. In this case, the clients have be successfully authenticated transmit the Wi-Fi network information to the proximal client and the local server to simplify local network setting.
The user generally monitors the state of the smart light 25 or the smart camera 26 registered in the device group for each user or removes the smart light 25 or the smart camera 26 through the content application. As an example of UI element group, when the content image is dark, a control signal for brightening the smart light may be included in the UI distribution information, and in the movie mode, a control signal for dimming the smart light may be included in the UI distribution information.
In general, the smart light 25 or the smart camera 26 among the proximal clients 200 does not have a separate display, so that it is necessary to download and install a necessary application (app) to the smart phones 22 and 23 or the smart display 21 of the user and sets the smart phones 22 and 23 or the smart display 21, or it is necessary to set the smart light 25 or the smart camera 26 by accessing a homepage of a manufacturing company.
In this case, it is necessary to install an app for each manufacturing company required for the smart light 25 or the smart camera 26, and the corresponding app may be installed in a smart device linked to a user account. Further, in order to control the smart light 25 or the smart camera 26, the inconvenience of having to perform the same operation by each user.
In the present invention, in the case where a user configuring the initial UI distribution information invites another user to a device group after a content app required is installed only in the local server, another user may receive UI information for IoT monitoring and control, required content information of the IoT device (lighting state, camera information, and IoT device information), from the local server. In this case, the new user is capable of receiving the required UI display as a video stream only by installing the UI application of the present invention without installing the applications of various IoT devices, so that the new user is capable of receiving and controlling the video stream independent to the OS (Android, IOS, Window, Linux, and the like) of the user client device.
The IoT devices 25 and 26 are capable of transmitting the state change information to the local server 100 connected through the home network or transmitting the cloud server 300 connected through the Internet network. When the IoT app is installed in the cloud server 300, the cloud server 300 may provide the user who is allowed to access IoT state information with the state information of the IoT device to be provided to the user through the Internet network corresponding to the present invention. In this case, when a context state of the IoT app installed in the local server or the cloud is updated, the two IoT apps are automatically synchronized.
As another exemplary embodiment, the IoT app required for monitoring and controlling the IOT device, such as the smart light 25 or the smart camera 26, may also be installed in the local server 100.
The local server 100 may allocate the UI element group for each proximal client 200 by using the UI tree information included in the content and match the UI element group to the proximal client 200.
The local server 100 configures required UI display information corresponding to the UI distribution information and display sizes and resolution of the proximal clients 200, and converts the UI element groups, matched to the proximal clients 200, into video streams or images, and transmits the video streams or images to the user device.
The UI distribution information required for controlling the IoT device may be shared with other user devices invited to the device group for each user.
The invited user device may receive a video stream or image information for monitoring and UI control information of the IoT device that is the same as that included in the user device group. In this case, even when an OS systems used in the user devices are different, the user devices receive the UI control information as the video stream or the image information, so that there is an advantage in that it is possible to transmit the data independent to the OS systems.
FIG. 2 illustrates a concept in which the local server 100 divides a UI of an original content into UI element groups by using UI tree information and transmitting a UI element group to the proximal clients 21 and 22 by a video streaming according to an exemplary embodiment of the present invention.
Referring to FIG. 2, the original content is transmitted to the local server 100 from the content providing server 400 and is transmitted to the proximal clients 21 and 22. The local server 100 may modify a composition and a layout of the UI element groups by using the UI tree information included in the original content.
The local server 100 may divide the original content into a first UI element group and a second UI element group, match the first UI element group and the second UI element group to the smart display 21 and the smart phone 22 respectively, and then transmit the matched first UI element group and second UI element group by pixel streaming. In this case, the data transmitted to the smart display is a video included in the original content, and is decoded in the local server and is transmitted to the smart display as a video stream, and when the second UI element is transmitted to the smart phone by encoded pixel streaming, and the smart phone decodes the received pixel stream. In this case, the reproduction and the stop of the video stream reproduced in the smart display are determined according to an interaction of the user in the smart phone. However, when it is necessary to synchronize the reproduction time of the video streams of two devices, the decoding time of the video stream of the smart display needs to be synchronized based on the time in which the video stream is decoded in the smart phone.
As illustrated in FIG. 3, the local server 100 includes the UI distribution control module 105, the smart display 21 includes a UI element display module 320 to a UI editing module 350, the smart phone 22 includes a UI element display module 321 or a UI editing module 351, and it is possible to provide an expanded UI function through an element group consisting of the UI displays of various content applications by generating and storing the UI distribution information which are using the UI element group information provided from the two or more content applications.
FIG. 3 is a configuration diagram illustrating the local server and a client according to the exemplary embodiment of the present invention.
Referring to FIG. 3, the local server 100 according to the present exemplary embodiment includes a receiving module 110, the UI distribution control module 105, and a transmitting module 190. The UI distribution control module 105 includes a content app executing module 120, a UI element dividing module 130, a UI distribution information generating module 140, a display requesting module 150, a UI element transmitting module 160, a virtual display generating module 170, and a UI control information converting module 180. The UI application installed in the local server 100 includes the entirety or a part of the configurations of the UI distribution control module 105. The local server 100 and clients A and B 21 and 22 may be expressed as user devices.
In the meantime, clients A and B 21 and 22 includes app selecting modules 370 and 301, transmitting modules 310 and 311, UI element display modules 320 and 321, virtual display generating modules 330 and 331, UI control information input modules 340 and 341, UI editing modules 350 and 351, and receiving modules 360 and 361.
The local server 100 transmits UI display information to clients A and B 21 and 22 by video streaming or transmits the UI element groups to clients A and B 21 and 22, and receives UI control information by a user interaction from the client and generates a control signal of the content app.
Client B 22 selects a content app for which UI distribution information is to be changed and transmits the selected content app to the local server 100, and displays UI display information according to the changed UI distribution information.
Client A 21 is the client added for changing the UI distribution information of client B 22, and receives the UI element group of the content app from the local server 100 and displays the received UI element group.
Hereinafter, the configuration of the local server 100 will be described in detail.
The receiving module 110 receives whether to select the content app for which the UI distribution information is to be changed from client B 22. In this case, the number of content apps selectable by client B 22 may be two or more.
The UI distribution control module 105 divides the UI display information of the content app into UI element groups, matches the divided UI element groups to each client, and generates UI distribution information.
The content app executing module 120 executes the content app selected by client B 22. The content app is not required in clients A 21 and B 22 when the UI information is transmitted from the local server 100 by pixel streaming.
The content app executing module 120 may be replaced with an IoT app executing module and a UI app executing module combined with a Non-Fungible Token (NFT) content.
The UI element dividing module 130 divides the UI provided in the content app executed in the content app executing module 120 into a plurality of UI element groups.
The UI distribution information generating module 140 matches the proximal clients and the plurality of UI element groups respectively, and generates UI distribution information.
The UI distribution information generating module 140 may determine the UI distribution information by a physical distance to the proximal client based on the local server 100 or a direction of the location of the proximal client. For example, the UI distribution information generating module 140 may determine to arrange the UI element group having a large display size in the client relatively far from the local server 100 and to arrange the UI element group having a small display size, such as a character input display, to the client relatively close to the local server 100.
The display requesting module 150 requests a UI element display to the client A 21 or the client B 22. When the local server 100 requests client A 21 to display the UI element, the local server 100 may request the UI element display module 320 to execute the UI element display through a push message. When the client receives the UI element group, the display requesting module 150 may make the client ready for the display of the received UI element group.
The UI element transmitting module 160, by using the UI distribution information, transmits one or more UI element groups, which is matched to the each proximal client respectively, or transmits video stream, converted from one or more UI element groups, which is matched to the each proximal client respectively.
When the local server 100 transmits the UI element group by the video stream, it is not necessary to install and execute the content application in client A 21 and client B 22.
The virtual display generating module 170 generates virtual display information by using the UI distribution information generated by the UI distribution information generating module 140.
When the UI control information converting module 180 receives the UI control information from clients A 21 and B 22, the UI control information converting module 180 converts the received UI control information to a control signal of the content app.
The transmitting module 190 performs the display request of the display requesting module 150, and the transmission of the UI element group of the UI element transmitting module 160, and the like.
Hereinafter, the configurations of client A 21 and B 22 will be described in detail.
The configurations of client A 21 and client B 22 are the same, so that client A 21 will be mainly described.
The app selecting module 370 selects a content app of which UI display information needs to be changed. In this case, it is not necessary to execute the content app as a result of the selection.
The transmitting module 310 transmits user interaction information about which the app selecting module 370 of content app has selected to the local server 100, and the UI control information input module 340 transmits the received UI control information to the local server 100.
The UI element display module 320 is executed in client A 21 by the request of the display requesting module 150 of the local server 100. It is preferable to receive the UI element group or the UI element video streaming in the state where the UI element display module 320 is executed in client A 21.
The virtual display generating module 330 renders the UI element group received from the local server 100 and stores the UI element group in a graphic buffer.
The UI control information input module 340 receives the input of the UI control information through a touch screen, a keyboard, a voice, a picture, and the like. When a size of the data, such as pictures and voices, is relatively large, original data may be transmitted to a separate storage server, and information about the storage space may be changed to a web URL and transmitted.
The UI editing modules 350 and 351 may select at least one UI element group of the content app, and select a client to display the selected UI element group and change UI distribution information.
The receiving module 360 directly receives at least one UI element groups from the local server 100 or receives video stream for at least one UI element groups.
FIG. 4 is a flowchart illustrating a method of transmitting, by the local server, UI display information to proximal clients according to the exemplary embodiment of the present invention.
Referring to FIG. 4, the method of transmitting UI display information according to the exemplary embodiment includes operations time-serially processed in the local server and the client illustrated in FIG. 3. Accordingly, even if omitted below, the contents described above with respect to the local server and the client illustrated in FIG. 3 are also applied to the method of transmitting the UI display information according to the exemplary embodiment.
Referring to FIG. 4, client A 21, client B 22, and the local server 100 are connected through the home network, and may be mutually called by using the HTTP-based web API (REST service or Remote Procedure Calls (RPC) mechanism), and the TCP/IP protocol. As a specific example, the REST API may be applied for synchronization of the content apps, the RPC may be applied for video streaming between devices, and the TCP/IP protocol may be applied to the transmit of the control information according to user interaction between devices.
In operation 400, the local server 100 receives UI control information according to a selection of a content app of a user from client B 22.
Once client B 22 transmits keyboard input and touch screen input information that the user reacts on the received video stream to the local server through the app selecting module 301 as UI control information of the user, the local server transmits the UI control information to a content application. The present invention is described based on one content application, but the app selecting module may select a plurality of content applications. In this case, the UI distribution control module 105 of the local server 100 may divide the plurality of content application UIs into UI elements groups, and match the each UI element group to clients. The content application is used for displaying the original content, and may be a video application, such as YouTube or Twitch.
In the meantime, even in the case where the content application is not installed in client B 22, client B 22 may receive the UI element group from the local server 100 by video streaming, or may receive UI element group information and then the client may render and display the UI element group.
In operation 410, the local server 100 executes the content app selected in operation 400.
In operation 420, the local server 100 divides the UI of the executed content app into a plurality of UI element groups. The UI provided from the content application executed in the local server 100 may be divided into a plurality of UI elements by using the UI distribution control module 105 installed in the local server 100. Herein, an authority for the functions of dividing the single UI of the content app into the plurality of UI element groups and generating and modifying the UI distribution information is granted only to a specific device of a user possessing the corresponding right, so that a security function of the system may be strengthened.
In operation 430, the local server 100 matches the proximal clients (21 and 22) and the plurality of UI element groups respectively, and generates UI distribution information.
When client B 22 selects the content app in operation 400, it is preferable to select which client among the proximal clients and which UI element group will be matched to displaying the UI element group.
In operation 440, the local server 100 requests client A (21) to display the UI element group.
The request to display in operation 440 may be made before operation 440. When the local server 100 requests client A (21) to display the UI element group, the local server 100 may make the UI element display module 320 to execute the UI element display by transmitting a push message to the user’s app.
In operation 450, the user’s app of client A (21) receiving a request to display the UI element group executes the UI element group display module 320.
When the UI element group display module 320 is executed, client A may display the UI element group by receiving the video streaming received from the local server 100 in operation 472.
In operation 460, the local server 100 generates virtual display information by using the UI distribution information generated in operation 430. The virtual display information means information stored to render UI element group information in a graphic buffer. Therefore, the graphic buffer may reflect the UI distribution information by using the virtual display information and transmit the video stream to each device.
In operation 471 and 472, the local server 100 transmits one or more UI element groups corresponding to client A (21) and client B (22), respectively, as a video stream by using the UI distribution information. Since the local server 100 transmits the UI element group as a video stream, it is not necessary to install and execute the content application in client A (21) and client B (22).
In operation S481 and 482, the local server 100 receives UI control information from client A (21) and client B (22).
In operation 490, the local server 100 converts the received UI control information to a control signal of the content app.
When the user inputs the UI control information through a touch screen, a keyboard, a voice, and the like of client A (21) and client B (22), the UI control information may be transmitted to the local server through the TCP/IP protocol or the web API.
The local server 100 may convert the UI control information received from client A (21) and client B (22) into the control signal of the content app in consideration of display size and resolution information of each client possessed by the local server 100. Further, the user control information may be transmitted to the local server by providing a separate UI together with the video stream received from the user’s app installed in the client device. In this case, the client device needs to convert the user control signal and transmit the converted user control signal to the local server.
For example, when the local server 100 receives a coordinate value input by the user on the display of client A (21), it is necessary to determine which element of the content app UI corresponds to the received coordinate value in consideration of the display size and resolution information. The determination may also be performed in client A (21) and may also be performed in the local server 100. Even though the local server 100 receives the coordinate value input by the user on the display of client A (21), the local server 100 has the display size and resolution information of client A (21), so that the local server 100 may determine which part of the content app may corresponds to the received coordinate value.
Further, the UI control information may include content control information and UI distribution editing information, and the content control and the UI distribution editing are performed by the following method.
As an example, when the same content app is installed in the local server 100 and the clients 21 and 22, a context state value (or a parameter setting value of the content app) of the result of the distribution and its editing of selected UI by the client 21 and 22 may be transmitted to the local server 100.
The local server 100 performs with the authority to call the functions of the clients 21 and 22 necessary for collecting the context state value.
In this case, it is not necessary for the local server 100 to transmit display data for the UI distribution editing to the client 21 and 22, and when the display information is generated and the UI editing is completed in the client 21 and 22, it is preferable that the client 21 and 22 change the context state value and provide the changed context state value to the content app of the local server 100.
As another example, when the content app is installed only in the local server 100, it is preferable that the local server 100 provides the information necessary for the information of UI distribution and its editing to the client 21 and 22 as a video stream. When the client 21 and 22 receives the UI distribution display transmitted by the local server 100 and transmits touch screen response results of the users in the client 21 and 22 to the local server 100, the local server 100 transmits the touch screen response as user interaction information to the content app. Thereafter, when the information of the single UI of the content app is updated, the local server applies the updated UI information of the content app to the UI distribution information and transmits the UI distribution information as a video stream. By repeating this process, the user is capable of performing UI distribution editing.
Further, the transmission of the video stream is possible independent to the OSs of the client 21 and 22, the client 21 and 22 may analyze the user’s control intention based on the response information of the touch screen or the touch pad and transmit a corresponding result to the local server 100.
FIG. 5 is a flowchart illustrating a method of transmitting, by the local server, UI display information to proximal clients according to another exemplary embodiment of the present invention.
Referring to FIG. 5, the method of transmitting UI display information according to the exemplary embodiment includes operations time-serially processed in the local server and the client illustrated in FIG. 3. Accordingly, even if omitted below, the contents described above with respect to the local server and the client illustrated in FIG. 3 are also applied to the method of transmitting the UI display information according to the exemplary embodiment.
The difference between the method of transmitting UI display information to the proximal clients 21 and 22 by the local server 100 illustrated in FIG. 5 and the method illustrated in FIG. 4 is that client A (21) receives the UI element group from the local server 100.
Client A (21) and client B (22) are connected with the local server 100 through the home network, and are capable of mutually calling through the HTTP-based web API (REST service or RPC mechanism).
Operation 400 to operation 460 are as illustrated in FIG. 4.
In operation 471, the local server 100 transmits a video stream for at least one UI element group corresponding to client B (22) by using the UI distribution information.
In the meantime, in operation 473, the local server 100 transmits at least one UI element group corresponding to client A 21 by using the UI distribution information.
In operation 475, since the local server 100 transmits the UI element group, the virtual display generating module 330 generates virtual display information by using the UI element group.
In the meantime, since client A (21) generates the virtual display information by using the received UI element group, client A (21) may have a function of a sub local server performing pixel streaming on another client. An example of the case where the client has the function of the sub local server will be described in detail with reference to FIG. 7.
In operation S481 and 482, the local server 100 receives UI control information from client A (21) and client B (22).
In operation 490, the local server 100 converts the received UI control information to a control signal of the content app. The local server 100 analyzes user interaction of the received UI control information, converts the user interaction into a control signal, and inputs the control signal to the content app that is being executed in the local server 100.
FIG. 6 is a flowchart illustrating a method of transmitting, by the local server, UI display information to proximal clients according to still another exemplary embodiment of the present invention.
The difference between the method of transmitting UI display information to the proximal clients by the local server 100 illustrated in FIG. 6 and the method illustrated in FIG. 5 is that both client A (21) and client B (22) receive the UI element groups from the local server 100.
Since both client A (21) and client B (22) receive the UI element groups and generate the virtual display information, clients 21 and 22 may have a function of a sub local server performing pixel streaming on another client.
Operation 400 to operation 440 are as illustrated in FIG. 4.
In operation 441 illustrated in FIG. 6, the local server 100 requests client B 22 to display the UI element group similar to operation 440.
In operations 450 and 451, client A (21) and client B (22) receiving the request to display the UI element group execute the UI element display modules 320 and 321, respectively.
In operation 460, the local server 100 generates virtual display information by using the UI distribution information generated in operation 430.
In the meantime, when the client 21 and 22 execute the UI element display modules 320 and 321 respectively, the client 21 and 22 may receive the UI element groups from the local server 100 in operations 473 and 474.
Similar to operation 460 in which the local server 100 generates the virtual display information, in operations 475 and 476, client A (21) and client B (22) generate virtual display information by using the UI element groups received from the local server 100. In this case, client A/B may transmit a first UI element to itself as a video stream, and may transmit a second UI element to another user device as a video stream by using the generated virtual display information.
In operation S481 and 482, the local server 100 receives UI control information from client A 21 and client B 22.
In operation 490, the local server 100 converts the received UI control information to a control signal of the content app.
In the meantime, when an error is generated in the operation of the local server 100, in order to build a resilient system, client A (21) or client B (22) may be defined as a local server candidate having the same authority as that of the local server 100.
In the situation where the local server 100 is operated, the UI distribution information generating module 140 of the local server is synchronized to the client defined as the local server candidate by the web RPC or REST service method, the content app installed in the local server needs to be installed even in the client, and the local server 100 and the content app of the client need to be synchronized by the web API method.
When an error is generated in the operation of the local server 100, the local server candidate client having the authority of the local server, the local server candidate client having the authority of the local server may serve as the local server by using the information of the synchronized elements.
In the meantime, in the case where the OS system of the local server 100 is different from the OS of the client, the matched UI element group information may be transmitted to the user’s app for the present invention of the client device by using a transport layer for each OS, and then, the client device may generate a virtual display and perform video streaming by using the UI element group.
FIG. 7 illustrates a concept of transmitting a UI element group by a relay method when the local server transmits UI display information to proximal clients according to the exemplary embodiment of the present invention.
Referring to FIG. 7, the local server extracts UI tree information included in the original content, virtualizes the single UI, and then divides the UI element groups constituting the UI tree information into group 1 and group 2.
The UI elements group divided into group 1 are stored as pixel data of a first graphic buffer, and the UI element groups divided into group 2 are stored without being converted to the pixel data. In this case, when the local server renders and transmits the UI element group 1, among the UI element groups divided into group 1 and group 2, and transmits. without rending, the UI element group 2 to the client device A, the device of client A forwards the received UI element group 2 to devices of clients B and C. Clients B and C may forward and provide the UI display information between the devices by rendering the UI element group 2 by themselves and providing a UI display.
Further, clients B and C of group 2 may generate UI control information and transmit the generated UI control information to client A, and client A may transmit its UI control information and the control information received from clients B and C to the local server. Herein, when information discriminating the local server and the forwarding client is included in the UI element information, the user’s app of the present invention may be configured so that the UI control information of clients B and C is directly transmitted to the local server without passing through client A.
FIG. 8 is a flowchart illustrating a method of dividing, by the local server, UI tree information illustrated in FIG. 7 and transmitting the divided UI tree information to the client according to the exemplary embodiment of the present invention.
Referring to FIG. 8, the method of dividing UI tree information and transmitting the divided UI tree information to the client according to the exemplary embodiment includes operations time-serially processed in the local server and the client illustrated in FIG. 3. Accordingly, even if omitted below, the contents described above with respect to the local server and the client illustrated in FIG. 3 are also applied to the method of transmitting the UI display information according to the exemplary embodiment.
In operation 800, the local server 100 divides the UI tree information included in the content in consideration of the number of clients.
In operation 810, the local server 100 virtualizes the single UI tree. The virtualization means managing the UI display information of the user’s device as a single UI tree in the content application, and managing the single UI tree as a UI group divided into UI element groups at the OS level. Accordingly, the main function of the virtualization is to manage the divided UI display information to be transmitted to the plurality of user devices as the information of UI element groups, and it is possible to prevent a malfunction according to the UI tree virtualization by transmitting the content application in which user interaction information for the UI display information, being divided into the plurality of UI groups and being transmitted to the plurality of devices, is collected and managed as the single UI tree.
Therefore, in order to prevent the malfunction, it is necessary to synchronize an update cycle of a main variable value of the divided UI element groups, the update of the user interaction information provides the user interaction information to the content application when a variable value of the main function of the UI element group is changed within a predetermined update cycle.
In operation 820, the local server 100 receives display resolution from client A (21).
In operation 830, the local server 100 generates a graphic buffer (for example, the virtual display) having the same resolution as the received display resolution.
In operation 840, the local server 100 renders the UI tree information divided in operation 800 to the graphic buffer corresponding to the client.
In operation 850, the local server 100 encodes pixel data stored in the graphic buffer and transmits the encoded pixel data to the corresponding client.
In operation 860, client A 21 decodes the received pixel data to a video stream and renders the video stream.
FIG. 9 illustrates a flow of a method of editing a UI of a content app by a client according to the exemplary embodiment of the present invention.
Referring to FIG. 9, the method of editing a UI of a content app by the client according to the exemplary embodiment includes operations time-serially processed in the local server and the client illustrated in FIG. 3. Accordingly, even if omitted below, the contents described above with respect to the local server and the client illustrated in FIG. 3 are also applied to the method of transmitting the UI display information according to the exemplary embodiment.
In operation 900, client A (21) transmits selected content app to the local server 100.
In operation 910, the local server 100 executes the selected content app. When there is no selected content app, the local server 100 may download the content app through the cloud server 300 and execute the downloaded content app.
In operation 920, client A (21) receives the UI of the content app from the local server 100 by video streaming.
In operation 930, client A (21) displays the received UI of the content app.
In operation 940, client A (21) executes a UI setting mode of the content app.
In operation 950, client A (21) selects at least one UI element group of the content app, and selects a client which is to display the selected UI element group.
In operation 960, client A (21) transmits UI distribution information matching a client information to the selected UI element group for display to the local server 100.
In operation 970, the local server 100 transmits a UI element group to the matched client that displays the UI element group with reference to the UI distribution information.
FIG. 10 illustrates a flow of a method of transmitting, by the local server, IoT state information to the client device in an OS level according to another exemplary embodiment of the present invention.
Referring to FIG. 10, the method of transmitting IoT state information to the client according to the exemplary embodiment includes operations time-serially processed in the local server and the client illustrated in FIG. 3. Accordingly, even if omitted below, the contents described above with respect to the local server and the client illustrated in FIG. 3 are also applied to the method of transmitting the IoT state information according to the exemplary embodiment.
In operation 1000, the local server 100 receives information about the IoT application selected for receiving IoT state information by a user from the client A (21). The selectable IoT applications are listed in client A (21), and when the user selects any one IoT application, the local server 100 receives which IoT application is selected as user UI control information.
In operation 1010, the local server 100 executes the selected IoT application. In the case where the IoT application selected in operation 1000 is not installed, the local server 100 may download the IoT application through the cloud server 300 of the IoT device.
In operation 1020, the local server 100 divides a UI of the executed IoT application into a plurality of UI element groups.
In operation 1030, the local server 100 generates UI distribution information by using the plurality of divided UI elements groups.
In operation 1040, the local server 100 generates virtual display information by using the generated UI distribution information.
In operation 1050, the local server 100 may transmit the virtual display information to client A (21) by video streaming or transmit the matched UI element group to the client device.
In operation 1060, the local server 100 receives UI control information from the user’s app of client A (21).
In operation 1070, the local server 100 converts the received UI control information to a control signal of the IoT application.
In operation 1071, the local server 100 transmits the IoT control signal to an IoT device 25 through the cloud or the home network. When the user device serves as a hub of the IoT device, the IoT device 25 may be controlled through the home network using ZigBee/Wi-Fi.
In the meantime, when the IoT app installed in the local server 100 transmits the IoT control signal to an IoT cloud server, the IoT cloud server transmits the IoT control signal to the IoT device through the IoT hub by the TCP/IP communication by referring to an IP address information uniquely matched to the IoT device 25.
In operation 1072, the IoT app of the local server 100 receives IoT state information through the IoT device 25 or the IoT cloud server.
In operation 1073, the local server 100 transmits the IoT state information to the user’s app of client A (21) as the video stream or the UI element group information.
FIG. 11 illustrates a flow of a method of transmitting, by the cloud server, IoT state information to the client device according to another exemplary embodiment of the present invention.
Referring to FIG. 11, the method of transmitting IoT state information to the client according to the exemplary embodiment includes operations time-serially processed in the local server and the user of the client illustrated in FIG. 3. Accordingly, even if omitted below, the contents described above with respect to the local server and the user app of the client illustrated in FIG. 3 are also applied to the method of transmitting the IoT state information according to the exemplary embodiment.
FIG. 11 is different from FIG. 10 in that client A (21) located outside the home network receives the IoT state information through the cloud server 300.
In FIG. 11, for the remote control of the IoT device, it is preferable that the UI virtualization information of the cloud server 300 and the local server 100, the UI distribution information, the user device information matched to the UI distribution information, and the like are synchronized through the web API.
In operation 1100, the cloud server 300 receives user control information for selecting an IoT application by the user from client A (21).
In operation 1110, the cloud server 300 determines whether the device of client A (21) is a registered user device in the user device group information.
In operation 1120, when client A (21) is the registered user device as a result of the determination of the cloud server 300, the local server 100 receives information about the IoT application selected by the user of client A (21) from the cloud server 300.
In operation 1130, the local server 100 executes the selected IoT application.
In operation 1140, the local server 100 divides a UI of the executed IoT application into a plurality of UI element groups at an OS level.
In operation 1150, the local server 100 generates UI distribution information by using the plurality of divided UI elements groups. Further, the local server transmits a updated UI distribution information of the local server to the cloud server to synchronizes the UI distribution information between two devices.
In operation 1160, the local server 100 generates the UI of the IoT app of the local server synchronized with the IoT app of the cloud server as virtual display information by using the generated UI distribution information.
In operation 1170, the local server 100 transmits a UI display of the matched IoT app to the device of client A (21) by video streaming by using the virtual display information, or transmit the matched UI element group information.
In operation 1180, client A (21) transmits UI control information to the cloud server 300, and when client A (21) transmits the UI control information to the IoT app installed in the cloud server, context state information of the IoT app of the local server is updated by the cloud server of the IoT device.
Therefore, the IoT app of the local server 100 transmits the received UI control information to the IoT device.
In operation 1191, the IoT app of the local server 100 transmits IoT control signal to the IoT cloud server through the Internet web, or when the local server serves as the IoT hub, the local server directly transmits the IoT control signal to the IoT device through the home network.
In operation 1192, the IoT app of the local server 100 receives the state information of the IoT device via the cloud of the IoT app and the IoT hub, or directly receives the state information from the IoT device through the home network when the local server serves as the IoT hub.
In operation 1193, when the context state information of the IoT app of the local server 100 is changed, the IoT app provides the cloud of the IoT device with the changed state information, or transmits the changed state information to the IoT app of the cloud server 300 when the local server serves as the IoT hub.
The hub for controlling the IoT device controls the IoT device on the home network by using ZigBee, Bluetooth, or Wi-Fi network.
FIG. 12 illustrates a flow of a method of transmitting, by the local server, IoT state information to an unauthorized client according to another exemplary embodiment of the present invention.
The method of transmitting, by the local server, the IoT state information to a non-authorized client is different from the methods illustrated in FIGS. 10 and 11 in that a non-authorized client 30 requests the local server 100 to transmit the IoT state information.
In operation 1200, the local server 100 sets IoT state information providing range according to a distance to the client device.
The local server 100 may differentiate and provide the IoT state information according to the location of the non-authorized client 30. There is a case where the non-authorized client 30 is physically close to the local server 100 and there is a case where the non-authorized client 30 requests the IoT state information through the home network or the Internet at a remote distance, and it is possible to differentiate and provide the IoT state information according to the physical distance between the local server 100 and the non-authorized client 30.
In this case, when the location of the non-authorized client 30 is physically present within a predetermined distance from the local 100, more IoT state information may be set to be provided.
Further, depending on whether the IoT state information is requested from inside or outside the place in which the local server 100 is installed, the local server 100 may differentiate and provide the state information of the IoT device 25.
In operation 1210, the local server 100 receives a transmission request of an IoT state information from the user’s app of the non-authorized client 30.
In operation 1220, the local server 100 calculates a distance to the non-authorized client 30 based on the transmission request of IoT state information received in operation 1210.
In order to accurately calculate the distance between the local server 100 and the non-authorized client 30, when an antenna of a wireless communication module (Bluetooth, Wi-Fi, and ultra-wide band) of the local server 100 is implemented with multi-antennas having a predetermined reference distance, some of the paired antennas may be implemented to receive horizontal polarization and some other antennas may be implemented to receive vertical polarization.
In this case, the local server 100 may receive a wireless communication signal from the non-authorized client 30 through the paired vertical polarization/horizontal polarization multi-antennas, and accurately calculate a location of the non-authorized client 30 by a arrival time difference according to a difference in the incident angles of the received signals between the paired multi-antennas.
Accordingly, the local server 100 may determine the IoT state information to be provided to the user’s app of the non-authorized client 30 by calculating the distance between the local server 100 and the non-authorized client 30.
In operation 1230, the local server 100 transmits the transmission request of the IoT state information to the IoT device 25.
In operation 1240, the local server 100 receives the IoT state information from the IoT device 25.
In operation 1250, the local server 100 determines the IoT state information and the UI distribution information to be transmitted to the non-authorized client 30 according to the distance calculated in operation 1220.
In operation 1260, the local server 100 generates virtual display information by using the UI distribution information.
In operation 1270, the local server 100 transmits the IoT state information as a video streaming to the user’s app of the non-authorized client device. In this case, the IoT state information is transmitted from the local server 100 to the non-authorized client 30 as a video stream and is displayed on a screen. Accordingly, the installation of the IoT application, required for receiving the IoT state information of plurality of IoT devices, is not required in the non-authorized client 30.
FIG. 13 illustrates a flow of a method of receiving an NFT content and a UI app, by the local server, and transmitting UI display information according to another exemplary embodiment of the present invention.
Referring to FIG. 13, the method of transmitting UI display information according to the exemplary embodiment includes operations time-serially processed in the local server and the client illustrated in FIG. 3. Accordingly, even if omitted below, the contents described above with respect to the local server and the client illustrated in FIG. 3 are also applied to the method of transmitting the UI display information according to the exemplary embodiment.
FIG. 13 illustrates a method in which a local server 20 or client A (21) uses a UI app including NFT content for a predetermined period by transmitting the NFT content of an NFT content sever 500 and the UI app of a UI app server 600 to the local server 20.
In operation 1300, the user mutually concludes a smart contract with an NFT content provider 500 and a UI app provider 600.
Digital assets are managed by blockchain-based smart contracts, and digital asset tokens are largely divided into Fungible Tokens (FT) and Non-Fungible Tokens (NFT). In the Ethereum token, ERC-20 is defined as a fungible token and ERC-721 token is defined as a non-fungible token. In addition, ERC-1155 is a token that can manage ERC-20 and ERC-721 tokens through a single smart contract.
Accordingly, in the present invention, it is preferable that the NFT content provider/server 500, the UI app provider/server 600, and the user conclude a smart contract related to ownership/use of the NFT content and the UI app based on ERC-1155.
When the UI app provider/server 600, the NFT content provider/server 500, and the user/local server 20 conclude the smart contract effective for a specific period of time, the NFT content and the UI app may be transmitted to the local server 20. Accordingly, the local server 20 is granted to access the NFT contents for the predetermined period of time and is allowed to access as a sole user. In this case, the smart contract may include a condition of limiting display transmission of the NFT content transmitted to the plurality of user devices , which is a present invention limiting the number of authorized user devices included in the user device group.
In operation 1310, client A (21) selects the NFT content and the UI app to be used and transmits the selected NFT content and UI app to the local server 100.
In this case, it is possible to match the plurality of selectable UI apps stored in the UI app server 600 to a fungible token and provide the matched token to the local server 20 based on the contents of the smart contract, and the local server 20 may generate UI distribution information by dividing a single UI of the UI app into the units of UI element groups at the OS level and matching the divided UI element groups to the client devices of the user, and transmit one UI element group to the matched client A (21).
Client A (21) may select one of the plurality of selectable UI apps stored in the UI app server 600, combine the selected UI app with the NFT content, and display the combined NFT content.
In operation 1320, the local server 100 checks whether the smart contract is concluded between the UI app server 600 and the NFT content server 500, and then receives the UI app selected by client A (21) from the UI app server 600.
In operation 1321, the local server 100 checks whether the smart contract is concluded between the UI app server 600 and the NFT content server 500 and then receives the NFT content from the NFT content server 500.
In operation 1330, the local server 100 executes the selected UI app.
In operation 1340, the local server 100 divides a UI of the executed UI app into a plurality of UI element groups at an OS level.
In operation 1350, the local server 100 matches the divided UI element groups to each of the client devices of the user respectively, and generates UI distribution information.
When the local server 100 is connected with the proximal clients through the home network, it is preferable that the UI distribution control module 105 transmits the generated UI distribution information to the cloud server 300.
In operation 1360, the local server 100 generates virtual display information by using the generated UI distribution information.
In operation 1370, the local server 100 transmits the UI element group to client A (21). Instead of transmitting the UI element group to client A (21), the local server 100 may transmit a video stream for the UI element group to the matched client A (21) in operation 1370.
In operation 1371, client A (21) generates virtual display information by using the UI element group received from the local server 100.
In operation 1380, the local server 100 receives UI control information from client A (21).
In operation 1381, the local server 100 converts the UI control information to a control signal of the UI app. The local server 100 may apply the control signal to the UI app and re-transmit the UI element group or the video stream.
In operation 1390, the local server 100 may delegate the authority to use the UI app and the NFT content to client A (21).
During the period of time in which the smart contract, between the NFT content server 500 and the UI app server 600 concluded, in operation 1300 is effective, a situation may arise in which the NFT content is transmitted to another client. In this case, when the authority to use is delegated to another client, the smart contract is updated, and each operation of FIG. 13 is executed so that the UI application is executable in a new client.
When the authority to use is delegated to client A (21), the smart contract between the UI app server 600 and the NFT content server 500 is updated including client A (21), instead of the local server 100, and the UI distribution control module 105 is executable in client A (21).
The cloud server 300 may manage information, such as the user’s authority to access to the NFT content, the authority to provide the UI by using the UI app, the authority to correct the UI distribution information, and the delegation of the authority to use. The NFT content user may be delegated all of the above authorities, use the NFT content for a predetermined period of time, and delegate the authority to another user to distribute the NFT content.
Further, the NFT content user may stream and use the NFT content in his/her display device by changing the UI with a style that he/she wants for a predetermined period of time. Then, when the predetermined period of time elapses based on the contents of the smart contract, the authorities to use of all of the users, except for the content user may be reset.
The exemplary embodiments of the present invention may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer readable medium. The computer readable medium may include a program command, a data file, a data structure, and the like alone or in combination. The program command recorded in the medium may be specially designed and configured for the present invention, or may also be known and usable to those skilled in computer software. Examples of the computer readable recording medium include a magnetic medium, such as a hard disk, a floppy disk, or a magnetic tape, an optical recording medium, such as a CD-ROM or a DVD, a magneto-optical medium, such as a floptical disk, and a hardware device which is specifically configured to store and execute the program command such as a ROM, a RAM, and a flash memory. An example of the program command includes a high-level language code executable by a computer by using an interpreter, and the like, as well as a machine language code created by a compiler. The hardware device may be configured to be operated with one or more software modules in order to perform the operation of the present invention, and an opposite situation thereof is available.
As described above, the present invention has been described with reference to the specific matters, such as a specific component, limited embodiments, and drawings, but these are provided only for helping general understanding of the present invention, and the present invention is not limited to the aforementioned exemplary embodiments, and those skilled in the art will appreciate that various changes and modifications are possible from the description.
Accordingly, the spirit of the present invention shall not be determined while being limited to the foregoing exemplary embodiments, and it will be considered that all matters having equivalent or equivalent modifications to the claims are within the scope of the present invention.

Claims (13)

  1. A method of transmitting UI information to a user device by using smart contract information, the method comprising:
    dividing, by a first device of a user, UIs of a content application of a content application provider into a plurality of UI element groups at OS level, when a smart contract is effective between a NFT content provider, the content application provider, or the user;
    generating, by the first device of the user, UI distribution information by matching a UI element group of the plurality of UI element groups to a second device of the user; and
    transmitting, by the first device of the user, the matched UI element group or a video stream in association with the matched UI element group to the second device of the user based on the UI distribution information.
  2. The method of claim 1, further comprising:
    authorizing or delegating, by the first device of the user, another user or another user device to divide the UIs of the content application into a plurality of UI element groups.
  3. The method of claim 2, further comprising:
    when the first device of the user authorizes or delegates another user or another user device to divide the UIs of the content application into the plurality of UI element groups, the first device of the user transmits the change of the authority of another user or another device of the user in a cloud server.
  4. The method of claim 1, wherein the content application consists of a plurality of content applications, the UI element group is selected among a plurality of UI element groups, the plurality of UI element groups divided from a UIs of each UI application of the plurality of content applications.
  5. The method of claim 1, wherein in the transmission of the UI information, the UI information is transmitted within an effective term of the smart contract.
  6. The method of claim 1, further comprising:
    transmitting, by the first device of the user, a NFT of the NFT content provider to only one device of the user in association with the UI distribution information.
  7. The method of claim 1, wherein the step of transmission, the matched UI element group is transmitted to the second device of the user, when the UI element group is matched to a third device of the user corresponding to the UI distribution information.
  8. A method of transmitting UI information by a UI application of a user device in association with smart contract information, the method comprising:
    generating, by a UI application of a first device of a user, UI distribution information by matching a UI element group of a plurality of UI element groups to a second device of the user, the plurality of UI element groups divided from UIs of a content application of a content application provider, when a smart contract is effective between a NFT content provider, the content application provider, or the user; and
    transmitting, by the UI application of the first device of the user, the matched UI element group or a video stream in association with the mapped UI element group to the second device of the user based on the UI distribution information.
  9. The method of claim 8, further comprising:
    authorizing or delegating, by the UI application, another user or another device of a user to divide the UIs of the content application into a plurality of UI element groups.
  10. The method of claim 9, further comprising:
    when the authorization delegation performed, the UI application of the first device of the user transmits the change of the authority of another user or another device of the user in a cloud server.
  11. The method of claim 8, wherein the content application consists of a plurality of content applications, the UI element group is selected among a plurality of UI element groups, the plurality of UI element groups divided from a UIs of each content application of the plurality of content applications.
  12. The method of claim 8, further comprising:
    transmitting, by the UI application of the first device of the user, a NFT of the NFT content provider to only one device of the user in association with the UI distribution information.
  13. The method of claim 8, wherein the step of transmission, the matched UI element group is transmitted to the second device of the user, when the UI element group is matched to a third device of the user corresponding to the UI distribution information.
PCT/KR2023/002796 2022-02-28 2023-02-28 Method of transmitting ui display information based on smart contract WO2023163568A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
KR1020220026429A KR20230128915A (en) 2022-02-28 2022-02-28 Method of transmitting UI display information based on smart contract
KR10-2022-0026429 2022-02-28

Publications (1)

Publication Number Publication Date
WO2023163568A1 true WO2023163568A1 (en) 2023-08-31

Family

ID=87766492

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/KR2023/002796 WO2023163568A1 (en) 2022-02-28 2023-02-28 Method of transmitting ui display information based on smart contract

Country Status (2)

Country Link
KR (1) KR20230128915A (en)
WO (1) WO2023163568A1 (en)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20160091622A (en) * 2015-01-26 2016-08-03 엔트릭스 주식회사 System for cloud streaming service, method of image cloud streaming service using division of the change area and apparatus for the same
KR102026586B1 (en) * 2019-04-30 2019-11-26 펑타이그레이터차이나 유한회사 Segmented video wireless display system and method thereof
KR20200144788A (en) * 2019-06-19 2020-12-30 주식회사 비엘디 Vertically Arranged Folder-type Dual Monitor
KR20210058791A (en) * 2014-11-19 2021-05-24 주식회사 알티캐스트 System and method for providing cloud based user interfaces
US20210227275A1 (en) * 2020-01-21 2021-07-22 Dish Network L.L.C. Systems and methods for adapting content items to secured endpoint media device data

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20210058791A (en) * 2014-11-19 2021-05-24 주식회사 알티캐스트 System and method for providing cloud based user interfaces
KR20160091622A (en) * 2015-01-26 2016-08-03 엔트릭스 주식회사 System for cloud streaming service, method of image cloud streaming service using division of the change area and apparatus for the same
KR102026586B1 (en) * 2019-04-30 2019-11-26 펑타이그레이터차이나 유한회사 Segmented video wireless display system and method thereof
KR20200144788A (en) * 2019-06-19 2020-12-30 주식회사 비엘디 Vertically Arranged Folder-type Dual Monitor
US20210227275A1 (en) * 2020-01-21 2021-07-22 Dish Network L.L.C. Systems and methods for adapting content items to secured endpoint media device data

Also Published As

Publication number Publication date
KR20230128915A (en) 2023-09-05

Similar Documents

Publication Publication Date Title
WO2020190016A1 (en) Method and device for providing authentication in network-based media processing (nbmp) system
WO2016068622A1 (en) Terminal device and method of controlling same
WO2022050652A1 (en) Method, apparatus, and computer readable storage medium for controlling account
WO2014014302A1 (en) Method and device for sharing content
WO2016027933A1 (en) Digital device and method for controlling same
WO2013151374A1 (en) Method and system for transfering data between plurality of devices
WO2016085094A1 (en) Multimedia device and control method therefor
EP3047383A1 (en) Method for screen mirroring and source device thereof
WO2015016627A1 (en) Method and device for connecting single ap device among multiple ap devices on same network to terminal
WO2015137604A1 (en) Method for testing cloud streaming server, and apparatus and system therefor
WO2021133042A1 (en) Electronic device and method of operating the same
WO2012099378A2 (en) Method and apparatus for controlling the transceiving of content
WO2009126009A2 (en) Method and system for providing interactive content service of ubiquitous environment and computer-readable recording medium
WO2021101275A1 (en) Method, system and device for sharing contents
WO2018034491A1 (en) A primary device, an accessory device, and methods for processing operations on the primary device and the accessory device
WO2022075737A1 (en) Method and device for providing mec service
WO2018110970A1 (en) Terminal device, electronic device and control method for determining the position of searching object device
WO2022211465A1 (en) Apparatus and method for providing media streaming
WO2017034298A1 (en) Digital device and method of processing data in said digital device
WO2020149722A1 (en) Working environment provisioning method and apparatus for execution of application program between electronic device and external server
WO2014142410A1 (en) Computing system with device interaction mechanism and method of operation thereof
WO2023163568A1 (en) Method of transmitting ui display information based on smart contract
WO2023163567A1 (en) Method of transmitting state information of iot device by local server
WO2023163569A1 (en) Method of transceiving ui display between user devices
WO2024025199A1 (en) Computing device and operation method therefor

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23760442

Country of ref document: EP

Kind code of ref document: A1