WO2023163567A1 - Method of transmitting state information of iot device by local server - Google Patents

Method of transmitting state information of iot device by local server Download PDF

Info

Publication number
WO2023163567A1
WO2023163567A1 PCT/KR2023/002794 KR2023002794W WO2023163567A1 WO 2023163567 A1 WO2023163567 A1 WO 2023163567A1 KR 2023002794 W KR2023002794 W KR 2023002794W WO 2023163567 A1 WO2023163567 A1 WO 2023163567A1
Authority
WO
WIPO (PCT)
Prior art keywords
information
client
local server
iot
user device
Prior art date
Application number
PCT/KR2023/002794
Other languages
French (fr)
Inventor
In Sik Shin
Dae Hee Lee
Sangwook Lee
Original Assignee
Fluiz Co. Ltd.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Fluiz Co. Ltd. filed Critical Fluiz Co. Ltd.
Publication of WO2023163567A1 publication Critical patent/WO2023163567A1/en

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/50—Network services
    • H04L67/75—Indicating network or usage conditions on the user display
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16Y—INFORMATION AND COMMUNICATION TECHNOLOGY SPECIALLY ADAPTED FOR THE INTERNET OF THINGS [IoT]
    • G16Y40/00—IoT characterised by the purpose of the information processing
    • G16Y40/10—Detection; Monitoring
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16Y—INFORMATION AND COMMUNICATION TECHNOLOGY SPECIALLY ADAPTED FOR THE INTERNET OF THINGS [IoT]
    • G16Y40/00—IoT characterised by the purpose of the information processing
    • G16Y40/30—Control
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/12—Protocols specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
    • H04L67/125—Protocols specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks involving control of end-device applications over a network
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/50—Network services
    • H04L67/60—Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources
    • H04L67/63—Routing a service request depending on the request content or context
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/41—Structure of client; Structure of client peripherals
    • H04N21/4104—Peripherals receiving signals from specially adapted client devices
    • H04N21/4131—Peripherals receiving signals from specially adapted client devices home appliance, e.g. lighting, air conditioning system, metering devices
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/81—Monomedia components thereof
    • H04N21/8166—Monomedia components thereof involving executable data, e.g. software
    • H04N21/8173—End-user applications, e.g. Web browser, game
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/41—Structure of client; Structure of client peripherals
    • H04N21/4104—Peripherals receiving signals from specially adapted client devices
    • H04N21/4126—The peripheral being portable, e.g. PDAs or mobile phones
    • H04N21/41265—The peripheral being portable, e.g. PDAs or mobile phones having a remote control device for bidirectional communication between the remote control device and client device
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/436—Interfacing a local distribution network, e.g. communicating with another STB or one or more peripheral devices inside the home
    • H04N21/43615—Interfacing a Home Network, e.g. for connecting the client to a plurality of peripherals

Definitions

  • the present invention relates to a method of transmitting IoT state information to a client device by an IoT application installed in a local server, and more particularly, transmitting IoT state information, in which a local server transmits a UI of an Internet of Things (IoT) application capable of controlling an IoT device to a proximal client device by a video streaming, so that even a user device in which the IoT application is not installed is capable of monitoring or controlling the device.
  • IoT Internet of Things
  • the mirroring technology is inefficient because the same display is output on multiple display devices and there are devices that output the display unnecessarily. Further, when the screen is divided into a plurality of areas and used, since the display in one device is divided and used, there is a problem in that the UI display information cannot be distributed and used in several display devices.
  • Prior Art 1 Korean Patent Application Laid-Open No. 10-2018-0081133
  • a display UI of a display is divided into at least two display areas in response to the finger joint touch gesture.
  • Prior Art 2 (Korean Patent Application Laid-Open No. 10-2010-0081383) is related to the touch screen dividing method, and is characterized in that a divided area layout of a touch screen is formed by receiving an input of two positions on the touch screen and a separate display is allocated to each of the formed divided areas.
  • Prior Art 2 also discloses the method of dividing the screen in one device, so that there is no motivation to display the divided screen on the plurality of devices in cooperation with the plurality of devices.
  • Prior Art 3 (US Patent Application Publication No. US20190373058 A1) is characterized in that user interaction information is synchronized by transmitting user interaction information between a plurality of devices to a remote server.
  • Prior Art 3 there is no motivation to display a divided screen in a plurality of devices in cooperation with the plurality of devices.
  • the first problem to be solved by the present invention is to provide a method of transmitting Internet of Things (IoT) state information to a client by a local server, in which a local server transmits a UI of an IoT application that is installed in the local server and is capable of controlling an IoT device to a proximal client device in an OS level by a video streaming method, so that even a user device in which the IoT application is not installed is capable of monitoring or controlling the device.
  • IoT Internet of Things
  • the second problem to be solved by the present invention is to provide a service for transmitting IoT state information, in which a local server transceives UI display information in a unit of a UI element group in an OS level to combine various UI displays and provide a plurality of user devices with the combined UI display.
  • the present invention provides a computer readable recording medium in which a program for executing the method in a computer is recorded.
  • the present invention provides a method of transmitting Internet of Things (IoT) state information to a client device by an IoT application installed in a local server capable of performing data communication, the method including: dividing a UI, the UI provided by an IoT application executed in the local server, into a plurality of UI element groups by providing a function of a UI distribution control module that virtualizes a single UI of the IoT application in an OS level of the local server and controlling UI distribution; generating, by the UI distribution control module, UI distribution information by matching the client to a UI element group selected among the plurality of UI element groups; generating, by the UI distribution control module, virtual display information by using the UI distribution information; transmitting the selected UI element to the client as a video stream; receiving UI control information according to a user interaction from the client; converting the received UI control information into a control signal of an IoT app and transmitting the control signal to an IoT device; and transmitting
  • the method may further include authorizing or delegating, by the UI distribution control module, an authority to divide the UI of the IoT application into the UI element groups to any one client among proximal clients.
  • the UI distribution control module transmits the selected UI element group to the client as the video stream, it is preferable to render and transmit the UI element group as the video stream to which a display size and resolution of the client are reflected.
  • the method may further include storing the UI distribution information and the display size and resolution information of the proximal clients in a cloud server.
  • the method may include receiving, by the UI distribution control module, the UI control information from the client and providing the UI control information to the IoT application after transmitting the selected UI element group to the client as the video stream.
  • the present invention provides a function of a local server for transmitting IoT state information to a client, the function being supported in an OS level, the local server including: a UI element dividing module which divides a UI, the UI provided by an IoT application, into a plurality of UI element groups after the selected IoT application is executed in a client; a UI distribution information generating and storing module which generates UI distribution information by matching the client to a selected UI element among a plurality of UI elements; a virtual display generating module which generates virtual display information by using the UI distribution information; a UI element transmitting module which transmits the selected UI element group to the client as a video stream; a UI control information converting module which converts received UI control information to a control signal of an IoT app when the UI control information is received from the client; and a transmitting module which transmits the IoT state information to the client when the IoT state information is received from an I
  • the local server may authorize or delegate an authority to divide the UI of the IoT application into the plurality of UI element groups to any one client among proximal clients as the function supported in the OS level.
  • the UI element transmitting module transmits the selected UI element group to the client as the video stream, it is preferable to render and transmit the UI element group as the video stream to which a display size and resolution of the client are reflected.
  • the present invention provides a computer-readable recording medium in which a program for executing the method of transmitting the IoT state information to the client by the local server in a computer is recorded.
  • the local server transmits a UI of an Internet of Things (IoT) application capable of controlling an IoT device to a proximal client by a video streaming, so that even a user device in which an IoT application is not installed is capable of monitoring and controlling the device.
  • IoT Internet of Things
  • UI display information of a content application is transceived in the unit of a UI element, so that it is possible to variously combining the UI elements and providing a plurality of devices with the combined UI elements.
  • UI display information independently from an OS by transmitting a video stream converted from UI element group included in UI distribution information.
  • FIG. 1 is a conceptual diagram illustrating transmission of UI display information to proximal clients by a local server in a network consisting of a local server 100, a proximal client 200, a cloud server 300, and a content providing server 400.
  • FIG. 2 illustrates a concept in which the local server 100 divides a UI of an original content into UI element groups by using UI tree information and transmitting the UI element groups to proximal clients 21 and 22 according to an exemplary embodiment of the present invention.
  • FIG. 3 is a configuration diagram illustrating the local server and a client according to the exemplary embodiment of the present invention.
  • FIG. 4 is a flowchart illustrating a method of transmitting, by the local server, UI display information to proximal clients according to the exemplary embodiment of the present invention.
  • FIG. 5 is a flowchart illustrating a method of transmitting, by the local server, UI display information to proximal clients according to another exemplary embodiment of the present invention.
  • FIG. 6 is a flowchart illustrating a method of transmitting, by the local server, UI display information to proximal clients according to still another exemplary embodiment of the present invention.
  • FIG. 7 illustrates a concept of transmitting a UI element group by a relay method when the local server transmits UI display information to proximal clients according to the exemplary embodiment of the present invention.
  • FIG. 8 is a flowchart illustrating a method of dividing, by the local server, UI tree information illustrated in FIG. 7 and transmitting the divided UI tree information to the client according to the exemplary embodiment of the present invention.
  • FIG. 9 illustrates a flow of a method of editing a UI of a content app by a client according to the exemplary embodiment of the present invention.
  • FIG. 10 illustrates a flow of a method of transmitting, by the local server, Internet of Things (IoT) state information to the client according to another exemplary embodiment of the present invention.
  • IoT Internet of Things
  • FIG. 11 illustrates a flow of a method of transmitting, by the local server, IoT state information to the client according to still another exemplary embodiment of the present invention.
  • FIG. 12 illustrates a flow of a method of transmitting, by the local server, IoT state information to an unauthorized client according to still another exemplary embodiment of the present invention.
  • FIG. 13 illustrates a flow of a method of receiving an NFT content and a UI app, by the local server, and transmitting UI display information according to still another exemplary embodiment of the present invention.
  • a method of receiving and controlling state information of an Internet of Things (IoT) device in a user app of a user device including: receiving UI element group information by a user app executed in a first user device or receiving the UI element group information as a video stream; and transmitting control information of a user in the user app, in which the UI element group information is state information of the IoT device and one UI element group information transmitted after dividing a single UI, the UI of an IoT app installed in a second user device, into a plurality of UI element groups for control, and the one UI element group information is determined by a distance between the second user device and the first user device or direction angle information of the first user device based on the second user device.
  • IoT Internet of Things
  • the present invention maximizes service scalability of a content application in consideration of a display size or resolution of a user display device in order to transmit free UI display information between display devices connected through a communication network.
  • the present invention presents an efficient interactive data transmitting method for improving user experience between devices of a user group in which a plurality of display devices registered by a user or display devices registered by multiple users are included.
  • a video, audio, image, touch, keyboard, and voice data may be shared or mirrored and forwarded between display devices connected through a communication network.
  • a content UI of a content application executed in the display device is composed of UI element groups, and provides a function capable of transmitting the UI in the unit of one or more individual UI element groups to a proximal display device and providing a UI element management authority function to enable various UI changes and combinations.
  • a local server having an authority to divide and distribute a content UI may transmit a video stream of the UI element group constituting the UI to the proximal display device.
  • each proximal display device when only UI element group information is transmitted to a proximal display device, each proximal display device generates a video stream by itself, and then each proximal display device transmits interaction information of a user to a local server.
  • the local server may input user interaction information received from each proximal display device to a content application.
  • a user app installed in the proximal display device for the present invention may virtualize a display by using the UI element group information received from the server and transmit UI information to a plurality of user devices.
  • a function supported at an OS level in the local server may be minimized to and configured as a UI dividing function (UI virtualization), UI distribution information storage, and UI element group transmission function, and then the UI transmission function using a virtual display configuration and video streaming required in the present invention may be configured to be performed in a user app Installed in a user device.
  • UI virtualization UI dividing function
  • UI distribution information storage UI distribution information storage
  • UI element group transmission function a virtual display configuration and video streaming required in the present invention
  • the local server may update the UI element group information to be transmitted to each proximal display device by reflecting UI distribution information of a content application received from a cloud server.
  • the local server converts the UI element group information into a video stream and transmits the video stream to each proximal client device, there is an advantage in that it is possible to transmit the video stream independent to an OS of the client device in terms of a data communication aspect.
  • the present invention presents a method of easily monitoring or controlling, by a user, a proximal client device and an IoT device.
  • the local server transmits a UI of an IoT application capable of controlling an IoT device to a proximal client by a video streaming method, so that it is possible to monitor state information of the IoT device even in a user’s device in which the IoT application is not installed, or control the IoT device by transmitting interaction information of the user from the user device to the local server.
  • the present invention presents a method of managing a content authority which is capable of various user experience environments, and forwarding a content.
  • the present invention may provide a platform environment providing a UI change authority to change a UI of a content, an authority to provide a new UI of an application, and a content access authority for a distribution function to a user group capable of accessing a digital content of a user.
  • FIG. 1 is a conceptual diagram illustrating transmission of UI display information to proximal clients by a local server in a network consisting of a local server 100, a proximal client 200, a cloud server 300, and a content providing server 400.
  • a content application may request a content from the content providing server 400.
  • the content application installed in the local server 100 may directly access the content providing server 400 and request the corresponding content or request the content via the cloud server 300.
  • the local server 100 may install the content application by requesting the content application from the content providing server 400 and installing the received content application.
  • the proximal client 200 includes various user devices, such as a smart display 21, smart phones 22 and 23, a smart light bulb 25, and a smart camera 26.
  • the smart light bulb 25 and the smart camera 26 are an IoT device group.
  • the proximal client 200 may register its user device information in the local server 100.
  • Any one client in the proximal client 200 may request to register its device information with the local server 100, and the local server 100 may transmit the registered user device information to the cloud server 300 which stores the user device information.
  • the devices of the proximal client 200 may be registered in the local server 100 as a user device group, and the corresponding user device information may be transmitted to and re-stored in the cloud server 300.
  • the user device group information a Service Set Identifier (SSID), a MAC address, a Universally Unique Identifier (UUID), or the like may be used, the user device group information may be differently configured according to a session from the execution of an app to the end of the app, or a time period from an installation of an app to a deletion of the app.
  • the user device group information may further include display size and resolution information.
  • the proximal client 200 and the local server 100 may be connected by using near-field communication, such as Bluetooth, ultrasound, or Zigbee, and an initial communication connection and authentication may be performed. Thereafter, UI display information between the authorized devices can be transmitted by using a home network connected by Wi-Fi.
  • near-field communication such as Bluetooth, ultrasound, or Zigbee
  • a new user device may be added to a Wi-Fi communication network by transmitting, by an initial smart phone or set top box connected with the Wi-Fi router, Wi-Fi connection information to another smart device or IoT device.
  • the Wi-Fi router serves as the local server 100
  • the set top box serves as the proximal client 200.
  • the local server 100 transmits UI display information to the proximal clients 200
  • the divided UIs of the content application may be shared and viewed by multiple clients.
  • the local server 100 receives a content of the content providing server 400 and transmits the received content to the proximal clients 200.
  • the UI display information of the content may be divided, matched, and transmitted for each client by a UI distribution control module 105 (see FIG. 3) implemented in the local server 100.
  • the UI distribution control module 105 installed in the local server 100 does not divide UI display information of the content by modifying a source code of an application of the content, but has a function of dividing UI tree information, constituting UI display information at an Operating System (OS) level, into UI element groups; and rendering module is implemented, thereby dividing the UI display information and transmitting the divided UI display information to the plurality of user devices.
  • the content application installed in the local server 100 is received from the content providing server 400, and the local server is implemented in the form of a microkernel providing a minimum function required at the OS level to divide single UI information of the content to a plurality of UI element groups.
  • the local server or an application which is installed in the local server and includes a partial function of a UI editing control module, may match the plurality of UI element groups to proximal client devices and transmits the plurality of UI element groups by a video streaming method.
  • the proximal clients 200 may receive a UI video stream without installing the content application installed in the local server.
  • the local server or the application installed in the local server may transmit the matched UI element groups to the proximal client 200 devices without the conversion to the video stream.
  • the user application installed in the client for the present invention may convert the UI element group into a video stream and provide the UI display information to the client device.
  • the local server in order for the local server to easily distribute and transmit the UI to the proximal clients 200, an authority or a function of dividing and virtualizing UI tree information in the content at the OS level is required.
  • the local server 100 or the separate application installed in the local server may generate UI distribution information that matches the proximal clients 200 to the UI element group constituting the content UI display information by using the virtualized UI tree information.
  • an image of a movie content may be matched and transmitted to the smart display 21, a chatting window may be matched and transmitted to the first smart phone 22, and the subtitles of the movie may be matched and transmitted to the second smart phone 23.
  • the function of the present invention divides the single UI display information of the content application into each separable UI element group in the OS of the user device corresponding to the local server. Thereafter, the local server or the application installed in the local server may generate UI distribution information of the user device group by matching other user devices to the client devices and the UI element groups.
  • the UI distribution information may be backed up to the cloud server 300, and the backed-up UI distribution information may be stored as related information of a device group for each user.
  • a UI element to be matched to the added client may be updated in the UI distribution information.
  • the client of another user may also be stored to the UI distribution information and updated.
  • the UI distribution information stored in the cloud server 300 may be shared or copied to the client of the invited user, and the UI distribution information may be updated by reflecting only the client that needs to be added or changed to the UI distribution information or be stored as separate UI distribution information selectable by the user.
  • the user device group information and the UI distribution information stored in the cloud server 300 may be used when the local server 100 has a failure or a physical change of the local server is required. Further, the UI distribution information of the local server 100 and the proximal clients may be easily modified by copying or modifying existing UI distribution information of the user stored in the cloud server 300. In this case, the clients may be successfully authenticated to transmit the Wi-Fi network information to the proximal client and the local server to simplify local network setting.
  • the user generally monitors the state of the smart light 25 or the smart camera 26 registered in the device group for each user or removes the smart light 25 or the smart camera 26 through the content application.
  • UI element group when the content image is dark, a control signal for brightening the smart light may be included in the UI distribution information, and in the movie mode, a control signal for dimming the smart light may be included in the UI distribution information.
  • the smart light 25 or the smart camera 26 among the proximal clients 200 does not have a separate display, so that it is necessary to download and install a necessary application (app) to the smart phones 22 and 23 or the smart display 21 of the user and sets the smart phones 22 and 23 or the smart display 21, or it is necessary to set the smart light 25 or the smart camera 26 by accessing a homepage of a manufacturing company.
  • another user device may receive UI information for IoT monitoring and control, required content information of the IoT device (lighting state, camera information, and IoT device information), from the local server.
  • the new user is capable of receiving the required UI display as a video stream only by installing the UI application of the present invention without installing the applications of various IoT devices, so that the new user is capable of receiving and controlling the video stream independent to the OS (Android, IOS, Window, Linux, and the like) of the user client device.
  • the IoT devices 25 and 26 are capable of transmitting the state change information to the local server 100 connected through the home network or transmitting the cloud server 300 connected through the Internet network.
  • the cloud server 300 may provide the user who is allowed to access IoT state information with the state information of the IoT device through the Internet network corresponding to the present invention.
  • the two IoT apps are automatically synchronized.
  • the IoT app required for monitoring and controlling the IOT device may also be installed in the local server 100.
  • the local server 100 may allocate the UI element group for each proximal client 200 by using the UI tree information included in the content and matches the UI element group to the proximal client 200.
  • the local server 100 configures required UI display information corresponding to the UI distribution information and display sizes and resolution of the proximal clients 200, and converts the UI element groups, matched to the proximal clients 200, into video streams or images, and transmits the video streams or images to the user device.
  • the UI distribution information required for controlling the IoT device may be shared with other user devices invited to the device group for each user.
  • the invited user device may receive a video stream or image information for monitoring and UI control information of the IoT device that is the same as that included in the user device group.
  • the user devices receive the UI control information as the video stream or the image information, so that there is an advantage in that it is possible to transmit the data independent to the OS systems.
  • FIG. 2 illustrates a concept in which the local server 100 divides a UI of an original content into UI element groups by using UI tree information and transmitting a UI element group to the proximal clients 21 and 22 by a video streaming according to an exemplary embodiment of the present invention.
  • the original content is transmitted to the local server 100 from the content providing server 400 and is transmitted to the proximal clients 21 and 22.
  • the local server 100 may modify a composition and a layout of the UI element group by using the UI tree information included in the original content.
  • the local server 100 may divide the original content into a first UI element group and a second UI element group, match the first UI element group and the second UI element group to the smart display 21 and the smart phone 22, respectively, and then transmit the matched first UI element group and second UI element group by pixel streaming.
  • the data transmitted to the smart display is a video included in the original content, and is decoded in the local server and is transmitted to the smart display as a video stream, and when the second UI element is transmitted to the smart phone by encoded pixel streaming, and the smart phone decodes the received pixel stream.
  • the reproduction and the stop of the video stream reproduced in the smart display are determined according to an interaction of the user in the smart phone.
  • the decoding time of the video stream of the smart display needs to be synchronized based on the time in which the video stream is decoded in the smart phone.
  • the local server 100 includes the UI distribution control module 105
  • the smart display 21 includes a UI element display module 320 to a UI editing module 350
  • the smart phone 22 includes a UI element display module 321 or a UI editing module 351
  • FIG. 3 is a configuration diagram illustrating the local server and a client according to the exemplary embodiment of the present invention.
  • the local server 100 includes a receiving module 110, the UI distribution control module 105, and a transmitting module 190.
  • the UI distribution control module 105 includes a content app executing module 120, a UI element dividing module 130, a UI distribution information generating module 140, a display requesting module 150, a UI element transmitting module 160, a virtual display generating module 170, and a UI control information converting module 180.
  • the UI application installed in the local server 100 includes the entirety or a part of the configurations of the UI distribution control module 105.
  • the local server 100 and clients A and B 21 and 22 may be expressed as user devices.
  • clients A and B 21 and 22 includes app selecting modules 370 and 301, transmitting modules 310 and 311, UI element display modules 320 and 321, virtual display generating modules 330 and 331, UI control information input modules 340 and 341, UI editing modules 350 and 351, and receiving modules 360 and 361.
  • the local server 100 transmits UI display information to clients A and B 21 and 22 by video streaming or transmits the UI element groups to clients A and B 21 and 22, and receives UI control information by a user interaction from the client and generates a control signal of the content app.
  • Client B 22 selects a content app for which UI distribution information is to be changed and transmits the selected content app to the local server 100, and displays UI display information according to the changed UI distribution information.
  • Client A 21 is the client added for changing the UI distribution information of client B 22, and receives the UI element group of the content app from the local server 100 and displays the received UI element group.
  • the receiving module 110 receives whether to select the content app for which the UI distribution information is to be changed from client B 22.
  • the number of content apps selectable by client B 22 may be two or more.
  • the UI distribution control module 105 divides the UI display information of the content app into UI element groups, matches the divided UI element groups to each client, and generates UI distribution information.
  • the content app executing module 120 executes the content app selected by client B 22.
  • the content app is not required in clients A 21 and B 22 when the UI information is transmitted from the local server 100 by pixel streaming.
  • the content app executing module 120 may be replaced with an IoT app executing module and a UI app executing module combined with a Non-Fungible Token (NFT) content.
  • NFT Non-Fungible Token
  • the UI element dividing module 130 divides the UI provided in the content app executed in the content app executing module 120 into a plurality of UI element groups.
  • the UI distribution information generating module 140 matches the each proximal client to the each UI element group respectively, and generates UI distribution information.
  • the UI distribution information generating module 140 may determine the UI distribution information by a physical distance to the proximal client based on the local server 100 or a direction of the location of the proximal client. For example, the UI distribution information generating module 140 may determine to arrange the UI element group having a large display size in the client relatively far from the local server 100 and to arrange the UI element group having a small display size, such as a character input display, to the client relatively close to the local server 100.
  • the display requesting module 150 requests a UI element display to the client A 21 or the client B 22.
  • the local server 100 may request the UI element display module 320 to execute the UI element display through a push message.
  • the display requesting module 150 may make the client ready for the display of the received UI element group.
  • the UI element transmitting module 160 by using the UI distribution information, transmits one or more UI element groups, which is matched to the proximal each client respectively, or transmits video stream, converted from one or more UI element groups, to the matched each proximal client respectively.
  • the virtual display generating module 170 generates virtual display information by using the UI distribution information generated by the UI distribution information generating module 140.
  • the UI control information converting module 180 When the UI control information converting module 180 receives the UI control information from clients A 21 and B 22, the UI control information converting module 180 converts the received UI control information to a control signal of the content app.
  • the transmitting module 190 performs the display request of the display requesting module 150, and the transmission of the UI element group of the UI element transmitting module 160, and the like.
  • client A 21 and client B 22 are the same, so that client A 21 will be mainly described.
  • the app selecting module 370 selects a content app of which UI display information needs to be changed. In this case, it is not necessary to execute the content app as a result of the selection.
  • the transmitting module 310 transmits user interaction information about which the app selecting module 370 of content app has selected to the local server 100, and the UI control information input module 340 transmits the received UI control information to the local server 100.
  • the UI element display module 320 is executed in client A 21 by the request of the display requesting module 150 of the local server 100. It is preferable to receive the UI element group or the UI element video streaming in the state where the UI element display module 320 is executed in client A 21.
  • the virtual display generating module 330 renders the UI element group received from the local server 100 and stores the UI element group in a graphic buffer.
  • the UI control information input module 340 receives the input of the UI control information through a touch screen, a keyboard, a voice, a picture, and the like.
  • original data may be transmitted to a separate storage server, and information about the storage space may be changed to a web URL and transmitted.
  • the UI editing modules 350 and 351 may select at least one UI element group of the content app, and select a client to display the selected UI element group and change UI distribution information.
  • the receiving module 360 directly receives at least one UI element groups from the local server 100 or receives video stream of at least one of UI element groups.
  • FIG. 4 is a flowchart illustrating a method of transmitting, by the local server, UI display information to proximal clients according to the exemplary embodiment of the present invention.
  • the method of transmitting UI display information according to the exemplary embodiment includes operations time-serially processed in the local server and the client illustrated in FIG. 3. Accordingly, even if omitted below, the contents described above with respect to the local server and the client illustrated in FIG. 3 are also applied to the method of transmitting the UI display information according to the exemplary embodiment.
  • client A 21, client B 22, and the local server 100 are connected through the home network, and may be mutually called by using the HTTP-based web API (REST service or Remote Procedure Calls (RPC) mechanism), and the TCP/IP protocol.
  • REST API may be applied for synchronization of the content apps
  • RPC may be applied for video streaming between devices
  • TCP/IP protocol may be applied to the transmit of the control information according to user interaction between devices.
  • the local server 100 receives UI control information according to a selection of a content app of a user from client B 22.
  • the local server transmits the UI control information to a content application.
  • the present invention is described based on one content application, but the app selecting module may select a plurality of content applications.
  • the UI distribution control module 105 of the local server 100 may divide a plurality of content application UIs by matching the clients to each UI element group of the plurality of content application UIs.
  • the content application is used for displaying the original content, and may be a video application, such as YouTube or Twitch.
  • client B 22 may receive the UI element group from the local server 100 by video streaming, or may receive UI element group information and then the client may render and display the UI element group.
  • the local server 100 executes the content app selected in operation 400.
  • the local server 100 divides the UI of the executed content app into a plurality of UI element groups.
  • the UI provided from the content application executed in the local server 100 may be divided into a plurality of UI elements by using the UI distribution control module 105 installed in the local server 100.
  • an authority for the functions of dividing the single UI of the content app into the plurality of UI element groups and generating and modifying a UI distribution information is granted only to the specific device of a user possessing the corresponding right, so that a security function of the system may be strengthened.
  • the local server 100 matches the proximal clients (21 and 22) to the plurality of UI element groups respectively, and generates UI distribution information.
  • client B 22 selects the content app in operation 400, it is preferable to select which client among the proximal clients and which UI element group will be matched for displaying the UI element group.
  • the local server 100 requests client A (21) to display the UI element group.
  • the request to display in operation 440 may be made before operation 440.
  • the local server 100 may make the UI element display module 320 to execute the UI element display by transmitting a push message to the user’s app.
  • the user’s app of client A (21) receiving a request to display the UI element group executes the UI element group display module 320.
  • client A may display the UI element group by receiving the video streaming received from the local server 100 in operation 472.
  • the local server 100 In operation 460, the local server 100 generates virtual display information by using the UI distribution information generated in operation 430.
  • the virtual display information means information stored to render UI element group information in a graphic buffer. Therefore, the graphic buffer may reflect the UI distribution information by using the virtual display information and transmit the video stream to each device.
  • the local server 100 transmits one or more UI element groups corresponding to client A (21) and client B (22), respectively, as a video stream by using the UI distribution information. Since the local server 100 transmits the UI element group as a video stream, it is not necessary to install and execute the content application in client A (21) and client B (22).
  • the local server 100 receives UI control information from client A (21) and client B (22).
  • the local server 100 converts the received UI control information to a control signal of the content app.
  • the UI control information may be transmitted to the local server through the TCP/IP protocol or the web API.
  • the local server 100 may convert the UI control information received from client A (21) and client B (22) into the control signal of the content app in consideration of display size and resolution information of each client possessed by the local server 100. Further, the user control information may be transmitted to the local server by providing a separate UI together with the video stream received from the user’s app installed in the client device. In this case, the client device needs to convert the user control signal and transmit the converted user control signal to the local server.
  • the local server 100 when the local server 100 receives a coordinate value input by the user on the display of client A (21), it is necessary to determine which element of the content app UI corresponds to the received coordinate value in consideration of the display size and resolution information. The determination may also be performed in client A (21) and may also be performed in the local server 100. Even though the local server 100 receives the coordinate value input by the user on the display of client A (21), the local server 100 has the display size and resolution information of client A (21), so that the local server 100 may determine which part of the content app may corresponds to the received coordinate value.
  • the UI control information may include content control information and UI distribution editing information, and the content control and the UI distribution editing are performed by the following method.
  • a context state value (or a parameter setting value of the content app) of the result of the UI distribution and its editing of selected UI by the client 21 and 22 may be transmitted to the local server 100.
  • the local server 100 performs with the authority to call the functions of the client 21 and 22 necessary for collecting the context state value.
  • the local server 100 it is not necessary for the local server 100 to transmit display data for the UI distribution editing to the client 21 and 22, and when the display information is generated and the UI editing is completed in the client 21 and 22, it is preferable that the client 21 and 22 change the context state value and provide the changed context state value to the content app of the local server 100.
  • the local server 100 provides the information necessary for the information of UI distribution and its editing to the client 21 and 22 as a video stream.
  • the client 21 and 22 receives the UI distribution display transmitted by the local server 100 and transmits touch screen response results of the users in the client 21 and 22 to the local server 100
  • the local server 100 transmits the touch screen response as user interaction information to the content app.
  • the local server applies the updated UI information of the content app to the UI distribution information and transmits the UI distribution information as a video stream.
  • the user is capable of performing UI distribution editing.
  • the transmission of the video stream is possible independent to the OSs of the client 21 and 22, the client 21 and 22 may analyze the user’s control intention based on the response information of the touch screen or the touch pad and transmit a corresponding result to the local server 100.
  • FIG. 5 is a flowchart illustrating a method of transmitting, by the local server, UI display information to proximal clients according to another exemplary embodiment of the present invention.
  • the method of transmitting UI display information according to the exemplary embodiment includes operations time-serially processed in the local server and the client illustrated in FIG. 3. Accordingly, even if omitted below, the contents described above with respect to the local server and the client illustrated in FIG. 3 are also applied to the method of transmitting the UI display information according to the exemplary embodiment.
  • client A (21) receives the UI element group from the local server 100.
  • Client A (21) and client B (22) are connected with the local server 100 through the home network, and are capable of mutually calling through the HTTP-based web API (REST service or RPC mechanism).
  • HTTP-based web API REST service or RPC mechanism
  • Operation 400 to operation 460 are as illustrated in FIG. 4.
  • the local server 100 transmits a video stream for at least one UI element group corresponding to client B (22) by using the UI distribution information.
  • the local server 100 transmits at least one UI element group corresponding to client A 21 by using the UI distribution information.
  • the virtual display generating module 330 since the local server 100 transmits the UI element group, the virtual display generating module 330 generates virtual display information by using the UI element group.
  • client A (21) since client A (21) generates the virtual display information by using the received UI element group, client A (21) may have a function of a sub local server performing pixel streaming on another client.
  • client A (21) may have a function of a sub local server performing pixel streaming on another client.
  • An example of the case where the client has the function of the sub local server will be described in detail with reference to FIG. 7.
  • the local server 100 receives UI control information from client A (21) and client B (22).
  • the local server 100 converts the received UI control information to a control signal of the content app.
  • the local server 100 analyzes user interaction of the received UI control information, converts the user interaction into a control signal, and inputs the control signal to the content app that is being executed in the local server 100.
  • FIG. 6 is a flowchart illustrating a method of transmitting, by the local server, UI display information to proximal clients according to still another exemplary embodiment of the present invention.
  • the difference between the method of transmitting UI display information to the proximal clients by the local server 100 illustrated in FIG. 6 and the method illustrated in FIG. 5 is that both client A (21) and client B (22) receive the UI element groups from the local server 100.
  • clients 21 and 22 may have a function of a sub local server performing pixel streaming on another client.
  • Operation 400 to operation 440 are as illustrated in FIG. 4.
  • the local server 100 requests client B 22 to display the UI element group similar to operation 440.
  • client A (21) and client B (22) receiving the request to display the UI element group execute the UI element display module 320 and 321 respectively.
  • the local server 100 In operation 460, the local server 100 generates virtual display information by using the UI distribution information generated in operation 430.
  • the client 21 and 22 may receive the UI element groups from the local server 100 in operations 473 and 474.
  • client A (21) and client B (22) Similar to operation 460 in which the local server 100 generates the virtual display information, in operations 475 and 476, client A (21) and client B (22) generate virtual display information by using the UI element groups received from the local server 100. In this case, client A/B may transmit a first UI element to itself as a video stream, and may transmit a second UI element to another user device as a video stream by using the generated virtual display information.
  • the local server 100 receives UI control information from client A 21 and client B 22.
  • the local server 100 converts the received UI control information to a control signal of the content app.
  • client A (21) or client B (22) may be defined as a local server candidate having the same authority as that of the local server 100.
  • the UI distribution information generating module 140 of the local server is synchronized to the client defined as the local server candidate by the web RPC or REST service method, the content app installed in the local server needs to be installed even in the client, and the local server 100 and the content app of the client need to be synchronized by the web API method.
  • the local server candidate client having the authority of the local server may serve as the local server by using the information of the synchronized elements.
  • the matched UI element group information may be transmitted to the user’s app for the present invention of the client device by using a transport layer for each OS, and then, the client device may generate a virtual display and perform video streaming by using the UI element group.
  • FIG. 7 illustrates a concept of transmitting a UI element group by a relay method when the local server transmits UI display information to proximal clients according to the exemplary embodiment of the present invention.
  • the local server extracts UI tree information included in the original content, virtualizes the single UI, and then divides the UI element groups constituting the UI tree information into group 1 and group 2.
  • the UI elements group divided into group 1 are stored as pixel data of a first graphic buffer, and the UI element groups divided into group 2 are stored without being converted to the pixel data.
  • the local server when the local server renders and transmits for the UI element group 1, among the UI element groups divided into group 1 and group 2, and transmits for the UI element group 2 without rending to the device of client A, the device of client A forwards the received UI element group 2 to devices of clients B and C.
  • Clients B and C may forward and provide the UI display information between the devices by rendering the UI element group 2 by themselves and providing a UI display.
  • clients B and C of group 2 may generate UI control information and transmit the generated UI control information to client A, and client A may transmit its UI control information and the control information received from clients B and C to the local server.
  • client A may transmit its UI control information and the control information received from clients B and C to the local server.
  • the user’s app of the present invention may be configured so that the UI control information of clients B and C is directly transmitted to the local server without passing through client A.
  • FIG. 8 is a flowchart illustrating a method of dividing, by the local server, UI tree information illustrated in FIG. 7 and transmitting the divided UI tree information to the client according to the exemplary embodiment of the present invention.
  • the method of dividing UI tree information and transmitting the divided UI tree information to the client according to the exemplary embodiment includes operations time-serially processed in the local server and the client illustrated in FIG. 3. Accordingly, even if omitted below, the contents described above with respect to the local server and the client illustrated in FIG. 3 are also applied to the method of transmitting the UI display information according to the exemplary embodiment.
  • the local server 100 divides the UI tree information included in the content in consideration of the number of clients.
  • the local server 100 virtualizes the single UI tree.
  • the virtualization means managing the UI display information of the user’s device as a single UI tree in the content application, and managing the single UI tree as a UI group divided into UI element groups at the OS level. Accordingly, the main function of the virtualization is to manage the divided UI display information to be transmitted to the plurality of user devices as UI group information, and it is possible to prevent a malfunction according to the UI tree virtualization by transmitting, to the content application for a single UI tree, collected user interaction information for the UI display information in which is divided into the plurality of UI groups and is transmitted to the plurality of devices.
  • the update of the user interaction information provides the user interaction information to the content application when a variable value of the main function of the UI element group is changed within a predetermined update cycle.
  • the local server 100 receives display resolution from client A (21).
  • the local server 100 generates a graphic buffer (for example, the virtual display) having the same resolution as the received display resolution.
  • a graphic buffer for example, the virtual display
  • the local server 100 renders the UI tree information divided in operation 800 to the graphic buffer corresponding to the client.
  • the local server 100 encodes pixel data stored in the graphic buffer and transmits the encoded pixel data to the corresponding client.
  • client A 21 decodes the received pixel data to a video stream and renders the video stream.
  • FIG. 9 illustrates a flow of a method of editing a UI of a content app by a client according to the exemplary embodiment of the present invention.
  • the method of editing a UI of a content app by the client includes operations time-serially processed in the local server and the client illustrated in FIG. 3. Accordingly, even if omitted below, the contents described above with respect to the local server and the client illustrated in FIG. 3 are also applied to the method of transmitting the UI display information according to the exemplary embodiment.
  • client A (21) transmits selected content app to the local server 100.
  • the local server 100 executes the selected content app.
  • the local server 100 may download the content app through the cloud server 300 and execute the downloaded content app.
  • client A (21) receives the UI of the content app from the local server 100 by video streaming.
  • client A (21) displays the received UI of the content app.
  • client A (21) executes a UI setting mode of the content app.
  • client A (21) selects at least one UI element group of the content app, and selects a client which is to display the selected UI element group.
  • client A (21) transmits UI distribution information matching a client information to the selected UI element group for display to the local server 100.
  • the local server 100 transmits the UI element group to the matched client that displays the UI element group with reference to the UI distribution information.
  • FIG. 10 illustrates a flow of a method of transmitting, by the local server, IoT state information to the client device in an OS level according to another exemplary embodiment of the present invention.
  • the method of transmitting IoT state information to the client according to the exemplary embodiment includes operations time-serially processed in the local server and the client illustrated in FIG. 3. Accordingly, even if omitted below, the contents described above with respect to the local server and the client illustrated in FIG. 3 are also applied to the method of transmitting the IoT state information according to the exemplary embodiment.
  • the local server 100 receives information about the IoT application selected for receiving IoT state information by a user from the client A (21).
  • the selectable IoT applications are listed in client A (21), and when the user selects any one IoT application, the local server 100 receives which IoT application is selected as user UI control information.
  • the local server 100 executes the selected IoT application.
  • the local server 100 may download the IoT application through the cloud server 300 of the IoT device.
  • the local server 100 divides a UI of the executed IoT application into a plurality of UI element groups.
  • the local server 100 generates UI distribution information by using the plurality of divided UI elements groups.
  • the local server 100 In operation 1040, the local server 100 generates virtual display information by using the generated UI distribution information.
  • the local server 100 may transmit the virtual display information to client A (21) by video streaming or transmit the matched UI element group to the client device.
  • the local server 100 receives UI control information from the user’s app of client A (21).
  • the local server 100 converts the received UI control information to a control signal of the IoT application.
  • the local server 100 transmits the IoT control signal to an IoT device 25 through the cloud or the home network.
  • the IoT device 25 may be controlled through the home network using ZigBee/Wi-Fi.
  • the IoT cloud server transmits the IoT control signal to the IoT device through the IoT hub by the TCP/IP communication by referring to a IP address information uniquely mapped to the IoT device 25.
  • the IoT app of the local server 100 receives IoT state information through the IoT device 25 or the IoT cloud server.
  • the local server 100 transmits the IoT state information to the user’s app of client A (21) as the video stream or the UI element group information.
  • FIG. 11 illustrates a flow of a method of transmitting, by the cloud server, IoT state information to the client device according to another exemplary embodiment of the present invention.
  • the method of transmitting IoT state information to the client according to the exemplary embodiment includes operations time-serially processed in the local server and the user of the client illustrated in FIG. 3. Accordingly, even if omitted below, the contents described above with respect to the local server and the user app of the client illustrated in FIG. 3 are also applied to the method of transmitting the IoT state information according to the exemplary embodiment.
  • FIG. 11 is different from FIG. 10 in that client A (21) located outside the home network receives the IoT state information through the cloud server 300.
  • the UI virtualization information of the cloud server 300 and the local server 100, the UI distribution information, the user device information matched to the UI distribution information, and the like are synchronized through the web API.
  • the cloud server 300 receives user control information for selecting an IoT application by the user from client A (21).
  • the cloud server 300 determines whether the device of client A (21) is a registered user device in the user device group information.
  • the local server 100 receives information about the IoT application selected by the user of client A (21) from the cloud server 300.
  • the local server 100 executes the selected IoT application.
  • the local server 100 divides a UI of the executed IoT application into a plurality of UI element groups at an OS level.
  • the local server 100 In operation 1150, the local server 100 generates UI distribution information by using the plurality of divided UI elements groups. Further, the local server transmits an updated UI distribution information of the local server to the cloud server to synchronizes the UI distribution information between two devices.
  • the local server 100 In operation 1160, the local server 100 generates the UI of the IoT app of the local server synchronized with the IoT app of the cloud server as virtual display information by using the generated UI distribution information.
  • the local server 100 transmits a UI display of the matched IoT app to the device of client A (21) by video streaming by using the virtual display information, or transmit the matched UI element group information.
  • client A (21) transmits UI control information to the cloud server 300, and when client A (21) transmits the UI control information to the IoT app installed in the cloud server, context state information of the IoT app of the local server is updated by the cloud server of the IoT device (1181).
  • the IoT app of the local server 100 transmits the received UI control information to the IoT device.
  • the IoT app of the local server 100 transmits IoT control signal to the IoT cloud server through the Internet web, or when the local server serves as the IoT hub, the local server directly transmits the IoT control signal to the IoT device through the home network.
  • the IoT app of the local server 100 receives the state information of the IoT device via the cloud of the IoT app and the IoT hub, or directly receives the state information from the IoT device through the home network when the local server serves as the IoT hub.
  • the IoT app when the context state information of the IoT app of the local server 100 is changed, the IoT app provides the cloud of the IoT device with the changed state information, or transmits the changed state information to the IoT app of the cloud server 300 when the local server serves as the IoT hub.
  • the hub for controlling the IoT device controls the IoT device on the home network by using ZigBee, Bluetooth, or Wi-Fi network.
  • FIG. 12 illustrates a flow of a method of transmitting, by the local server, IoT state information to an unauthorized client according to another exemplary embodiment of the present invention.
  • the method of transmitting, by the local server, the IoT state information to a non-authorized client is different from the methods illustrated in FIGS. 10 and 11 in that a non-authorized client 30 requests the local server 100 to transmit the IoT state information.
  • the local server 100 sets IoT state information providing range according to a distance to the client device.
  • the local server 100 may differentiate and provide the IoT state information according to the location of the non-authorized client 30. There is a case where the non-authorized client 30 is physically close to the local server 100 and there is a case where the non-authorized client 30 requests the IoT state information through the home network or the Internet at a remote distance, and it is possible to differentiate and provide the IoT state information according to the physical distance between the local server 100 and the non-authorized client 30.
  • IoT state information when the location of the non-authorized client 30 is physically present within a predetermined distance from the local 100, more IoT state information may be set to be provided.
  • the local server 100 may differentiate and provide the state information of the IoT device 25.
  • the local server 100 receives a transmission request of an IoT state information from the user’s app of the non-authorized client 30.
  • the local server 100 calculates a distance to the non-authorized client 30 based on the transmission request of IoT state information received in operation 1210.
  • the paired antennas may be implemented to receive horizontal polarization and some other antennas may be implemented to receive vertical polarization.
  • the local server 100 may receive a wireless communication signal from the non-authorized client 30 through the paired vertical polarization/horizontal polarization multi-antennas, and accurately calculate a location of the non-authorized client 30 by a arrival time difference according to a difference in the incident angles of the received signals between the paired multi-antennas.
  • the local server 100 may determine the IoT state information to be provided to the user’s app of the non-authorized client 30 by calculating the distance between the local server 100 and the non-authorized client 30.
  • the local server 100 transmits the transmission request of the IoT state information to the IoT device 25.
  • the local server 100 receives the IoT state information from the IoT device 25.
  • the local server 100 determines the IoT state information and the UI distribution information to be transmitted to the non-authorized client 30 according to the distance calculated in operation 1220.
  • the local server 100 In operation 1260, the local server 100 generates virtual display information by using the UI distribution information.
  • the local server 100 transmits the IoT state information as a video streaming to the user’s app of the non-authorized client device.
  • the IoT state information is transmitted from the local server 100 to the non-authorized client 30 as a video stream and is displayed on a screen. Accordingly, the installation of the IoT application, required for receiving the IoT state information of plurality of IoT devices, is not required in the non-authorized client 30.
  • FIG. 13 illustrates a flow of a method of receiving an NFT content and a UI app, by the local server, and transmitting UI display information according to another exemplary embodiment of the present invention.
  • the method of transmitting UI display information according to the exemplary embodiment includes operations time-serially processed in the local server and the client illustrated in FIG. 3. Accordingly, even if omitted below, the contents described above with respect to the local server and the client illustrated in FIG. 3 are also applied to the method of transmitting the UI display information according to the exemplary embodiment.
  • FIG. 13 illustrates a method in which a local server 20 or client A (21) uses a UI app including NFT content available for a predetermined period by transmitting the NFT content of an NFT content sever 500 and the UI app of a UI app server 600 to the local server 20.
  • the user mutually concludes a smart contract with an NFT content provider 500 and a UI app provider 600.
  • Digital assets are managed by blockchain-based smart contracts, and digital asset tokens are largely divided into Fungible Tokens (FT) and Non-Fungible Tokens (NFT).
  • FT Fungible Tokens
  • NFT Non-Fungible Tokens
  • ERC-20 is defined as a fungible token
  • ERC-721 token is defined as a non-fungible token
  • ERC-1155 is a token that can manage ERC-20 and ERC-721 tokens through a single smart contract.
  • the NFT content provider/server 500, the UI app provider/server 600, and the user conclude a smart contract related to ownership/use of the NFT content and the UI app based on ERC-1155.
  • the UI app provider/server 600, the NFT content provider/server 500, and the user/local server 20 conclude the smart contract effective for a specific period of time
  • the NFT content and the UI app may be transmitted to the local server 20.
  • the local server 20 is granted to access the NFT contents for the predetermined period of time and is allowed to access as a sole user.
  • the smart contract may include a condition of limiting display transmission of the NFT content transmitted to the plurality of user devices , which is a present invention limiting the number of authorized user devices included in the user device group.
  • client A (21) selects the NFT content and the UI app to be used and transmits the selected NFT content and UI app to the local server 100.
  • the local server 20 may generate UI distribution information by dividing a single UI of the UI app into the units of UI element groups at the OS level and matching the divided UI element groups to the client devices of the user, and transmit one UI element group to the matched client A (21).
  • Client A (21) may select one of the plurality of selectable UI apps stored in the UI app server 600, combine the selected UI app with the NFT content, and display the combined NFT content.
  • the local server 100 checks whether the smart contract is concluded between the UI app server 600 and the NFT content server 500, and then receives the UI app selected by client A (21) from the UI app server 600.
  • the local server 100 checks whether the smart contract is concluded between the UI app server 600 and the NFT content server 500 and then receives the NFT content from the NFT content server 500.
  • the local server 100 executes the selected UI app.
  • the local server 100 divides a UI of the executed UI app into a plurality of UI element groups at an OS level.
  • the local server 100 matches the divided each UI element group to each client device of the user respectively, and generates UI distribution information.
  • the UI distribution control module 105 transmits the generated UI distribution information to the cloud server 300.
  • the local server 100 In operation 1360, the local server 100 generates virtual display information by using the generated UI distribution information.
  • the local server 100 transmits the UI element group to client A (21). Instead of transmitting the UI element group to client A (21), the local server 100 may transmit a video stream for the at least one UI element group corresponding to client A (21) in operation 1370.
  • client A (21) generates virtual display information by using the UI element group received from the local server 100.
  • the local server 100 receives UI control information from client A (21).
  • the local server 100 converts the UI control information to a control signal of the UI app.
  • the local server 100 may apply the control signal of the UI app and re-transmit the UI element group or the video stream.
  • the local server 100 may delegate the authority to use the UI app and the NFT content to client A (21).
  • the smart contract between the UI app server 600 and the NFT content server 500 is updated including client A (21), instead of the local server 100, and the UI distribution control module 105 is executable in client A (21).
  • the cloud server 300 may manage information, such as the user’s authority to access to the NFT content, the authority to provide the UI by using the UI app, the authority to correct the UI distribution information, and the delegation of the authority to use.
  • the NFT content user may be delegated all of the above authorities, use the NFT content for a predetermined period of time, and delegate the authority to another user to distribute the NFT content.
  • the NFT content user may stream and use the NFT content in his/her display device by modifying the UI with a style that he/she wants for a predetermined period of time. Then, when the predetermined period of time elapses based on the contents of the smart contract, the authorities to use of all of the users, except for the content user may be reset.
  • the exemplary embodiments of the present invention may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer readable medium.
  • the computer readable medium may include a program command, a data file, a data structure, and the like alone or in combination.
  • the program command recorded in the medium may be specially designed and configured for the present invention, or may also be known and usable to those skilled in computer software.
  • Examples of the computer readable recording medium include a magnetic medium, such as a hard disk, a floppy disk, or a magnetic tape, an optical recording medium, such as a CD-ROM or a DVD, a magneto-optical medium, such as a floptical disk, and a hardware device which is specifically configured to store and execute the program command such as a ROM, a RAM, and a flash memory.
  • An example of the program command includes a high-level language code executable by a computer by using an interpreter, and the like, as well as a machine language code created by a compiler.
  • the hardware device may be configured to be operated with one or more software modules in order to perform the operation of the present invention, and an opposite situation thereof is available.

Landscapes

  • Engineering & Computer Science (AREA)
  • Signal Processing (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Computing Systems (AREA)
  • Multimedia (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • General Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Automation & Control Theory (AREA)
  • Information Transfer Between Computers (AREA)

Abstract

The present invention relates to a method of transmitting IoT state information to a client by a local server, the method including: dividing, by the local server, a UI, provided by an IoT application executed in the local server, into a plurality of UI elements; generating, by the UI distribution control module, UI distribution information by matching a client to a selected UI element from the plurality of UI elements; generating, by the UI distribution control module, virtual display information by using the UI distribution information; transmitting the selected UI element to the client as a video stream and receiving UI control information from the client; converting the received UI control information to a control signal of an IoT application, and transmitting the control signal to an IoT device; and transmitting the IoT state information to the client when the IoT state information is received from the IoT device, in which the local server transmits a video stream of the UI of the application capable of controlling the IoT device to a matched proximal client, so that the user client device of which the IoT application is not installed is capable of monitoring or controlling the device.

Description

METHOD OF TRANSMITTING STATE INFORMATION OF IOT DEVICE BY LOCAL SERVER
The present invention relates to a method of transmitting IoT state information to a client device by an IoT application installed in a local server, and more particularly, transmitting IoT state information, in which a local server transmits a UI of an Internet of Things (IoT) application capable of controlling an IoT device to a proximal client device by a video streaming, so that even a user device in which the IoT application is not installed is capable of monitoring or controlling the device.
In the prior art, in a technology for sharing User Interface (UI) display information between two or more display devices, it is common to mirror one display to another display or divide the display into a plurality of areas to display different contents in each divided area.
In particular, the mirroring technology is inefficient because the same display is output on multiple display devices and there are devices that output the display unnecessarily. Further, when the screen is divided into a plurality of areas and used, since the display in one device is divided and used, there is a problem in that the UI display information cannot be distributed and used in several display devices.
Further, in order to control an IoT device in a smart device, or check state information of the IoT device, a separate application capable of accessing the IoT device needs to be inconveniently installed and there is a problem in that as the number of IoT devices increases, the number of installed applications continues to increase.
Prior Art 1 (Korean Patent Application Laid-Open No. 10-2018-0081133) is the prior art related to a rapid screen dividing method, and is characterized in that when a finger joint touch gesture acting on a touch-sensitive surface is detected, a display UI of a display is divided into at least two display areas in response to the finger joint touch gesture.
Referring to Prior Art 1, only the method of easily dividing the screen is disclosed, and the configuration of whether the divided screen can be efficiently displayed in cooperation with proximal devices is not disclosed.
Prior Art 2 (Korean Patent Application Laid-Open No. 10-2010-0081383) is related to the touch screen dividing method, and is characterized in that a divided area layout of a touch screen is formed by receiving an input of two positions on the touch screen and a separate display is allocated to each of the formed divided areas. Prior Art 2 also discloses the method of dividing the screen in one device, so that there is no motivation to display the divided screen on the plurality of devices in cooperation with the plurality of devices.
Prior Art 3 (US Patent Application Publication No. US20190373058 A1) is characterized in that user interaction information is synchronized by transmitting user interaction information between a plurality of devices to a remote server. In Prior Art 3, there is no motivation to display a divided screen in a plurality of devices in cooperation with the plurality of devices.
Further, in the related art, when various display devices share a UI display, there is a problem in that an application for displaying the corresponding UI display must be installed in all of the display devices for the execution. Further, in the case of different OSs between display devices, it is difficult to share a UI display, so that there is difficulty in sharing UI information between multiple devices.
Accordingly, the first problem to be solved by the present invention is to provide a method of transmitting Internet of Things (IoT) state information to a client by a local server, in which a local server transmits a UI of an IoT application that is installed in the local server and is capable of controlling an IoT device to a proximal client device in an OS level by a video streaming method, so that even a user device in which the IoT application is not installed is capable of monitoring or controlling the device.
The second problem to be solved by the present invention is to provide a service for transmitting IoT state information, in which a local server transceives UI display information in a unit of a UI element group in an OS level to combine various UI displays and provide a plurality of user devices with the combined UI display.
Further, the present invention provides a computer readable recording medium in which a program for executing the method in a computer is recorded.
In order to achieve the first problem, the present invention provides a method of transmitting Internet of Things (IoT) state information to a client device by an IoT application installed in a local server capable of performing data communication, the method including: dividing a UI, the UI provided by an IoT application executed in the local server, into a plurality of UI element groups by providing a function of a UI distribution control module that virtualizes a single UI of the IoT application in an OS level of the local server and controlling UI distribution; generating, by the UI distribution control module, UI distribution information by matching the client to a UI element group selected among the plurality of UI element groups; generating, by the UI distribution control module, virtual display information by using the UI distribution information; transmitting the selected UI element to the client as a video stream; receiving UI control information according to a user interaction from the client; converting the received UI control information into a control signal of an IoT app and transmitting the control signal to an IoT device; and transmitting the IoT state information to the client when the IoT state information is received from the IoT device.
According to an exemplary embodiment of the present invention, the method may further include authorizing or delegating, by the UI distribution control module, an authority to divide the UI of the IoT application into the UI element groups to any one client among proximal clients.
Further, when the UI distribution control module transmits the selected UI element group to the client as the video stream, it is preferable to render and transmit the UI element group as the video stream to which a display size and resolution of the client are reflected.
According to another exemplary embodiment of the present invention, the method may further include storing the UI distribution information and the display size and resolution information of the proximal clients in a cloud server.
Further, the method may include receiving, by the UI distribution control module, the UI control information from the client and providing the UI control information to the IoT application after transmitting the selected UI element group to the client as the video stream.
In order to resolve the second problem, the present invention provides a function of a local server for transmitting IoT state information to a client, the function being supported in an OS level, the local server including: a UI element dividing module which divides a UI, the UI provided by an IoT application, into a plurality of UI element groups after the selected IoT application is executed in a client; a UI distribution information generating and storing module which generates UI distribution information by matching the client to a selected UI element among a plurality of UI elements; a virtual display generating module which generates virtual display information by using the UI distribution information; a UI element transmitting module which transmits the selected UI element group to the client as a video stream; a UI control information converting module which converts received UI control information to a control signal of an IoT app when the UI control information is received from the client; and a transmitting module which transmits the IoT state information to the client when the IoT state information is received from an IoT device.
According to an exemplary embodiment of the present invention, the local server may authorize or delegate an authority to divide the UI of the IoT application into the plurality of UI element groups to any one client among proximal clients as the function supported in the OS level.
Further, when the UI element transmitting module transmits the selected UI element group to the client as the video stream, it is preferable to render and transmit the UI element group as the video stream to which a display size and resolution of the client are reflected.
In order to solve another technical problem, the present invention provides a computer-readable recording medium in which a program for executing the method of transmitting the IoT state information to the client by the local server in a computer is recorded.
According to the present invention, the local server transmits a UI of an Internet of Things (IoT) application capable of controlling an IoT device to a proximal client by a video streaming, so that even a user device in which an IoT application is not installed is capable of monitoring and controlling the device.
Further, according to the present invention, UI display information of a content application is transceived in the unit of a UI element, so that it is possible to variously combining the UI elements and providing a plurality of devices with the combined UI elements.
Further, according to the present invention, it is possible to transmit UI display information independently from an OS by transmitting a video stream converted from UI element group included in UI distribution information.
Further, it is possible to monitor and control a client with minimizing the installation of an unnecessary app by installing a content app in one local server and transmitting UI distribution information to each client device as UI display information without a need to install the same content app in all of the multiple mobile devices participating as clients.
Further, it is possible to provide content services to which various UXs are employed by granting an authorities to forward an authority to all access a content to a third user, and to provide a new content-application to a digital content user with authorities to provide a new UI app, to divide a UI of the UI app, and to forward a authority to access a content of the UI app.
FIG. 1 is a conceptual diagram illustrating transmission of UI display information to proximal clients by a local server in a network consisting of a local server 100, a proximal client 200, a cloud server 300, and a content providing server 400.
FIG. 2 illustrates a concept in which the local server 100 divides a UI of an original content into UI element groups by using UI tree information and transmitting the UI element groups to proximal clients 21 and 22 according to an exemplary embodiment of the present invention.
FIG. 3 is a configuration diagram illustrating the local server and a client according to the exemplary embodiment of the present invention.
FIG. 4 is a flowchart illustrating a method of transmitting, by the local server, UI display information to proximal clients according to the exemplary embodiment of the present invention.
FIG. 5 is a flowchart illustrating a method of transmitting, by the local server, UI display information to proximal clients according to another exemplary embodiment of the present invention.
FIG. 6 is a flowchart illustrating a method of transmitting, by the local server, UI display information to proximal clients according to still another exemplary embodiment of the present invention.
FIG. 7 illustrates a concept of transmitting a UI element group by a relay method when the local server transmits UI display information to proximal clients according to the exemplary embodiment of the present invention.
FIG. 8 is a flowchart illustrating a method of dividing, by the local server, UI tree information illustrated in FIG. 7 and transmitting the divided UI tree information to the client according to the exemplary embodiment of the present invention.
FIG. 9 illustrates a flow of a method of editing a UI of a content app by a client according to the exemplary embodiment of the present invention.
FIG. 10 illustrates a flow of a method of transmitting, by the local server, Internet of Things (IoT) state information to the client according to another exemplary embodiment of the present invention.
FIG. 11 illustrates a flow of a method of transmitting, by the local server, IoT state information to the client according to still another exemplary embodiment of the present invention.
FIG. 12 illustrates a flow of a method of transmitting, by the local server, IoT state information to an unauthorized client according to still another exemplary embodiment of the present invention.
FIG. 13 illustrates a flow of a method of receiving an NFT content and a UI app, by the local server, and transmitting UI display information according to still another exemplary embodiment of the present invention.
According to the present invention, there is a provided a method of receiving and controlling state information of an Internet of Things (IoT) device in a user app of a user device, the method including: receiving UI element group information by a user app executed in a first user device or receiving the UI element group information as a video stream; and transmitting control information of a user in the user app, in which the UI element group information is state information of the IoT device and one UI element group information transmitted after dividing a single UI, the UI of an IoT app installed in a second user device, into a plurality of UI element groups for control, and the one UI element group information is determined by a distance between the second user device and the first user device or direction angle information of the first user device based on the second user device.
Prior to the description of the specific contents of the present invention, for the convenience of understanding, the outline of the solution to the problem to be solved by the present invention or the core of the technical spirit is first presented.
The present invention maximizes service scalability of a content application in consideration of a display size or resolution of a user display device in order to transmit free UI display information between display devices connected through a communication network.
The present invention presents an efficient interactive data transmitting method for improving user experience between devices of a user group in which a plurality of display devices registered by a user or display devices registered by multiple users are included.
A video, audio, image, touch, keyboard, and voice data may be shared or mirrored and forwarded between display devices connected through a communication network. In this case, a content UI of a content application executed in the display device is composed of UI element groups, and provides a function capable of transmitting the UI in the unit of one or more individual UI element groups to a proximal display device and providing a UI element management authority function to enable various UI changes and combinations.
A local server having an authority to divide and distribute a content UI may transmit a video stream of the UI element group constituting the UI to the proximal display device. As another method, when only UI element group information is transmitted to a proximal display device, each proximal display device generates a video stream by itself, and then each proximal display device transmits interaction information of a user to a local server. The local server may input user interaction information received from each proximal display device to a content application. In this case, a user app installed in the proximal display device for the present invention may virtualize a display by using the UI element group information received from the server and transmit UI information to a plurality of user devices. Accordingly, a function supported at an OS level in the local server may be minimized to and configured as a UI dividing function (UI virtualization), UI distribution information storage, and UI element group transmission function, and then the UI transmission function using a virtual display configuration and video streaming required in the present invention may be configured to be performed in a user app Installed in a user device.
Further, the local server may update the UI element group information to be transmitted to each proximal display device by reflecting UI distribution information of a content application received from a cloud server. In the case where the local server converts the UI element group information into a video stream and transmits the video stream to each proximal client device, there is an advantage in that it is possible to transmit the video stream independent to an OS of the client device in terms of a data communication aspect.
Further, the present invention presents a method of easily monitoring or controlling, by a user, a proximal client device and an IoT device.
In particular, the local server transmits a UI of an IoT application capable of controlling an IoT device to a proximal client by a video streaming method, so that it is possible to monitor state information of the IoT device even in a user’s device in which the IoT application is not installed, or control the IoT device by transmitting interaction information of the user from the user device to the local server.
Further, the present invention presents a method of managing a content authority which is capable of various user experience environments, and forwarding a content. The present invention may provide a platform environment providing a UI change authority to change a UI of a content, an authority to provide a new UI of an application, and a content access authority for a distribution function to a user group capable of accessing a digital content of a user.
Hereinafter, an exemplary embodiment in which the present invention can be easily carried out by those skilled in the art will be described in detail with reference to the accompanying drawings. However, the exemplary embodiments are for explaining the present invention in more detail, and it will be apparent to those skilled in the art that the scope of the present invention is not limited by the exemplary embodiments.
The configuration of the invention for clarifying the solution of the problem to be solved by the present invention will be described in detail with reference to the accompanying drawings based on the exemplary embodiment of the present invention, in assigning reference numbers to the components of the drawings, the same reference numbers are given to the same components even if they are in different drawings, and it should be noted in advance that components of other drawings can be cited when necessary in the description of the drawings. In addition, when it is determined that detailed descriptions of well-known functions or configurations related to the present invention and other matters may unnecessarily obscure the gist of the present invention in explaining the principle of operation of the exemplary embodiment of the present invention in detail, the detailed description is omitted.
FIG. 1 is a conceptual diagram illustrating transmission of UI display information to proximal clients by a local server in a network consisting of a local server 100, a proximal client 200, a cloud server 300, and a content providing server 400.
In order for the local server 100 to provide display information of a content application (CA) selected by a user to a proximal client 200 device, a content application may request a content from the content providing server 400. In this case, the content application installed in the local server 100 may directly access the content providing server 400 and request the corresponding content or request the content via the cloud server 300. In this case, when the content application required for receiving the content of the content providing server 400 is not installed in the local server 100, the local server 100 may install the content application by requesting the content application from the content providing server 400 and installing the received content application.
Referring to FIG. 1, the proximal client 200 includes various user devices, such as a smart display 21, smart phones 22 and 23, a smart light bulb 25, and a smart camera 26. The smart light bulb 25 and the smart camera 26 are an IoT device group.
The proximal client 200 may register its user device information in the local server 100.
Any one client in the proximal client 200 may request to register its device information with the local server 100, and the local server 100 may transmit the registered user device information to the cloud server 300 which stores the user device information.
That is, the devices of the proximal client 200, such as a set top box or a hub, may be registered in the local server 100 as a user device group, and the corresponding user device information may be transmitted to and re-stored in the cloud server 300. As the user device group information, a Service Set Identifier (SSID), a MAC address, a Universally Unique Identifier (UUID), or the like may be used, the user device group information may be differently configured according to a session from the execution of an app to the end of the app, or a time period from an installation of an app to a deletion of the app. The user device group information may further include display size and resolution information.
The proximal client 200 and the local server 100 may be connected by using near-field communication, such as Bluetooth, ultrasound, or Zigbee, and an initial communication connection and authentication may be performed. Thereafter, UI display information between the authorized devices can be transmitted by using a home network connected by Wi-Fi.
Further, in a connection of a Wi-Fi router between the proximal client 200 and the local server 100 for data communication, a new user device may be added to a Wi-Fi communication network by transmitting, by an initial smart phone or set top box connected with the Wi-Fi router, Wi-Fi connection information to another smart device or IoT device. When the Wi-Fi router serves as the local server 100, the set top box serves as the proximal client 200.
When the local server 100 transmits UI display information to the proximal clients 200, the divided UIs of the content application may be shared and viewed by multiple clients.
The local server 100 receives a content of the content providing server 400 and transmits the received content to the proximal clients 200. In this case, the UI display information of the content may be divided, matched, and transmitted for each client by a UI distribution control module 105 (see FIG. 3) implemented in the local server 100.
In particular, according to the exemplary embodiment of the present invention, the UI distribution control module 105 installed in the local server 100 does not divide UI display information of the content by modifying a source code of an application of the content, but has a function of dividing UI tree information, constituting UI display information at an Operating System (OS) level, into UI element groups; and rendering module is implemented, thereby dividing the UI display information and transmitting the divided UI display information to the plurality of user devices. The content application installed in the local server 100 is received from the content providing server 400, and the local server is implemented in the form of a microkernel providing a minimum function required at the OS level to divide single UI information of the content to a plurality of UI element groups. Further, the local server or an application, which is installed in the local server and includes a partial function of a UI editing control module, may match the plurality of UI element groups to proximal client devices and transmits the plurality of UI element groups by a video streaming method. Accordingly, the proximal clients 200 may receive a UI video stream without installing the content application installed in the local server. Further, the local server or the application installed in the local server may transmit the matched UI element groups to the proximal client 200 devices without the conversion to the video stream. In this case, the user application installed in the client for the present invention may convert the UI element group into a video stream and provide the UI display information to the client device.
In the present invention, in order for the local server to easily distribute and transmit the UI to the proximal clients 200, an authority or a function of dividing and virtualizing UI tree information in the content at the OS level is required. The local server 100 or the separate application installed in the local server may generate UI distribution information that matches the proximal clients 200 to the UI element group constituting the content UI display information by using the virtualized UI tree information.
For example, an image of a movie content may be matched and transmitted to the smart display 21, a chatting window may be matched and transmitted to the first smart phone 22, and the subtitles of the movie may be matched and transmitted to the second smart phone 23.
Therefore, the function of the present invention divides the single UI display information of the content application into each separable UI element group in the OS of the user device corresponding to the local server. Thereafter, the local server or the application installed in the local server may generate UI distribution information of the user device group by matching other user devices to the client devices and the UI element groups. The UI distribution information may be backed up to the cloud server 300, and the backed-up UI distribution information may be stored as related information of a device group for each user. When another client of the user is added to the user device group, a UI element to be matched to the added client may be updated in the UI distribution information.
When a client of another user is invited and included in a device group for each user, the client of another user may also be stored to the UI distribution information and updated. The UI distribution information stored in the cloud server 300 may be shared or copied to the client of the invited user, and the UI distribution information may be updated by reflecting only the client that needs to be added or changed to the UI distribution information or be stored as separate UI distribution information selectable by the user.
The user device group information and the UI distribution information stored in the cloud server 300 may be used when the local server 100 has a failure or a physical change of the local server is required. Further, the UI distribution information of the local server 100 and the proximal clients may be easily modified by copying or modifying existing UI distribution information of the user stored in the cloud server 300. In this case, the clients may be successfully authenticated to transmit the Wi-Fi network information to the proximal client and the local server to simplify local network setting.
The user generally monitors the state of the smart light 25 or the smart camera 26 registered in the device group for each user or removes the smart light 25 or the smart camera 26 through the content application. As an example of UI element group, when the content image is dark, a control signal for brightening the smart light may be included in the UI distribution information, and in the movie mode, a control signal for dimming the smart light may be included in the UI distribution information.
In general, the smart light 25 or the smart camera 26 among the proximal clients 200 does not have a separate display, so that it is necessary to download and install a necessary application (app) to the smart phones 22 and 23 or the smart display 21 of the user and sets the smart phones 22 and 23 or the smart display 21, or it is necessary to set the smart light 25 or the smart camera 26 by accessing a homepage of a manufacturing company.
In this case, it is necessary to install an app for each manufacturing company required for the smart light 25 or the smart camera 26, and the corresponding app may be installed in a smart device linked to a user account. Further, in order to control the smart light 25 or the smart camera 26, the inconvenience of having to perform the same operation by each user.
In the present invention, in the case where the user configuring the initial UI distribution information invites another user to the device group after a content app required is installed only in the local server, another user device may receive UI information for IoT monitoring and control, required content information of the IoT device (lighting state, camera information, and IoT device information), from the local server. In this case, the new user is capable of receiving the required UI display as a video stream only by installing the UI application of the present invention without installing the applications of various IoT devices, so that the new user is capable of receiving and controlling the video stream independent to the OS (Android, IOS, Window, Linux, and the like) of the user client device.
The IoT devices 25 and 26 are capable of transmitting the state change information to the local server 100 connected through the home network or transmitting the cloud server 300 connected through the Internet network. When the IoT app is installed in the cloud server 300, the cloud server 300 may provide the user who is allowed to access IoT state information with the state information of the IoT device through the Internet network corresponding to the present invention. In this case, when a context state of the IoT app installed in the local server or the cloud is updated, the two IoT apps are automatically synchronized.
As another exemplary embodiment, the IoT app required for monitoring and controlling the IOT device, such as the smart light 25 or the smart camera 26, may also be installed in the local server 100.
The local server 100 may allocate the UI element group for each proximal client 200 by using the UI tree information included in the content and matches the UI element group to the proximal client 200.
The local server 100 configures required UI display information corresponding to the UI distribution information and display sizes and resolution of the proximal clients 200, and converts the UI element groups, matched to the proximal clients 200, into video streams or images, and transmits the video streams or images to the user device.
The UI distribution information required for controlling the IoT device may be shared with other user devices invited to the device group for each user.
The invited user device may receive a video stream or image information for monitoring and UI control information of the IoT device that is the same as that included in the user device group. In this case, even when a OS systems used in the user devices are different, the user devices receive the UI control information as the video stream or the image information, so that there is an advantage in that it is possible to transmit the data independent to the OS systems.
FIG. 2 illustrates a concept in which the local server 100 divides a UI of an original content into UI element groups by using UI tree information and transmitting a UI element group to the proximal clients 21 and 22 by a video streaming according to an exemplary embodiment of the present invention.
Referring to FIG. 2, the original content is transmitted to the local server 100 from the content providing server 400 and is transmitted to the proximal clients 21 and 22. The local server 100 may modify a composition and a layout of the UI element group by using the UI tree information included in the original content.
The local server 100 may divide the original content into a first UI element group and a second UI element group, match the first UI element group and the second UI element group to the smart display 21 and the smart phone 22, respectively, and then transmit the matched first UI element group and second UI element group by pixel streaming. In this case, the data transmitted to the smart display is a video included in the original content, and is decoded in the local server and is transmitted to the smart display as a video stream, and when the second UI element is transmitted to the smart phone by encoded pixel streaming, and the smart phone decodes the received pixel stream. In this case, the reproduction and the stop of the video stream reproduced in the smart display are determined according to an interaction of the user in the smart phone. However, when it is necessary to synchronize the reproduction time of the video streams of two devices, the decoding time of the video stream of the smart display needs to be synchronized based on the time in which the video stream is decoded in the smart phone.
As illustrated in FIG. 3, the local server 100 includes the UI distribution control module 105, the smart display 21 includes a UI element display module 320 to a UI editing module 350, the smart phone 22 includes a UI element display module 321 or a UI editing module 351, and it is possible to provide an expanded UI function through a element group consisting of the UI displays of various content applications by generating and storing the UI distribution information which is using the UI element group information provided from the two or more content applications.
FIG. 3 is a configuration diagram illustrating the local server and a client according to the exemplary embodiment of the present invention.
Referring to FIG. 3, the local server 100 according to the present exemplary embodiment includes a receiving module 110, the UI distribution control module 105, and a transmitting module 190. The UI distribution control module 105 includes a content app executing module 120, a UI element dividing module 130, a UI distribution information generating module 140, a display requesting module 150, a UI element transmitting module 160, a virtual display generating module 170, and a UI control information converting module 180. The UI application installed in the local server 100 includes the entirety or a part of the configurations of the UI distribution control module 105. The local server 100 and clients A and B 21 and 22 may be expressed as user devices.
In the meantime, clients A and B 21 and 22 includes app selecting modules 370 and 301, transmitting modules 310 and 311, UI element display modules 320 and 321, virtual display generating modules 330 and 331, UI control information input modules 340 and 341, UI editing modules 350 and 351, and receiving modules 360 and 361.
The local server 100 transmits UI display information to clients A and B 21 and 22 by video streaming or transmits the UI element groups to clients A and B 21 and 22, and receives UI control information by a user interaction from the client and generates a control signal of the content app.
Client B 22 selects a content app for which UI distribution information is to be changed and transmits the selected content app to the local server 100, and displays UI display information according to the changed UI distribution information.
Client A 21 is the client added for changing the UI distribution information of client B 22, and receives the UI element group of the content app from the local server 100 and displays the received UI element group.
Hereinafter, the configuration of the local server 100 will be described in detail.
The receiving module 110 receives whether to select the content app for which the UI distribution information is to be changed from client B 22. In this case, the number of content apps selectable by client B 22 may be two or more.
The UI distribution control module 105 divides the UI display information of the content app into UI element groups, matches the divided UI element groups to each client, and generates UI distribution information.
The content app executing module 120 executes the content app selected by client B 22. The content app is not required in clients A 21 and B 22 when the UI information is transmitted from the local server 100 by pixel streaming.
The content app executing module 120 may be replaced with an IoT app executing module and a UI app executing module combined with a Non-Fungible Token (NFT) content.
The UI element dividing module 130 divides the UI provided in the content app executed in the content app executing module 120 into a plurality of UI element groups.
The UI distribution information generating module 140 matches the each proximal client to the each UI element group respectively, and generates UI distribution information.
The UI distribution information generating module 140 may determine the UI distribution information by a physical distance to the proximal client based on the local server 100 or a direction of the location of the proximal client. For example, the UI distribution information generating module 140 may determine to arrange the UI element group having a large display size in the client relatively far from the local server 100 and to arrange the UI element group having a small display size, such as a character input display, to the client relatively close to the local server 100.
The display requesting module 150 requests a UI element display to the client A 21 or the client B 22. When the local server 100 requests client A 21 to display the UI element, the local server 100 may request the UI element display module 320 to execute the UI element display through a push message. When the client receives the UI element group, the display requesting module 150 may make the client ready for the display of the received UI element group.
The UI element transmitting module 160, by using the UI distribution information, transmits one or more UI element groups, which is matched to the proximal each client respectively, or transmits video stream, converted from one or more UI element groups, to the matched each proximal client respectively.
When the local server 100 transmits the UI element group by the video stream, it is not necessary to install and execute the content application in client A 21 and client B 22.
The virtual display generating module 170 generates virtual display information by using the UI distribution information generated by the UI distribution information generating module 140.
When the UI control information converting module 180 receives the UI control information from clients A 21 and B 22, the UI control information converting module 180 converts the received UI control information to a control signal of the content app.
The transmitting module 190 performs the display request of the display requesting module 150, and the transmission of the UI element group of the UI element transmitting module 160, and the like.
Hereinafter, the configurations of clients A 21 and B 22 will be described in detail.
The configurations of client A 21 and client B 22 are the same, so that client A 21 will be mainly described.
The app selecting module 370 selects a content app of which UI display information needs to be changed. In this case, it is not necessary to execute the content app as a result of the selection.
The transmitting module 310 transmits user interaction information about which the app selecting module 370 of content app has selected to the local server 100, and the UI control information input module 340 transmits the received UI control information to the local server 100.
The UI element display module 320 is executed in client A 21 by the request of the display requesting module 150 of the local server 100. It is preferable to receive the UI element group or the UI element video streaming in the state where the UI element display module 320 is executed in client A 21.
The virtual display generating module 330 renders the UI element group received from the local server 100 and stores the UI element group in a graphic buffer.
The UI control information input module 340 receives the input of the UI control information through a touch screen, a keyboard, a voice, a picture, and the like. When a size of the data, such as pictures and voices, is relatively large, original data may be transmitted to a separate storage server, and information about the storage space may be changed to a web URL and transmitted.
The UI editing modules 350 and 351 may select at least one UI element group of the content app, and select a client to display the selected UI element group and change UI distribution information.
The receiving module 360 directly receives at least one UI element groups from the local server 100 or receives video stream of at least one of UI element groups.
FIG. 4 is a flowchart illustrating a method of transmitting, by the local server, UI display information to proximal clients according to the exemplary embodiment of the present invention.
Referring to FIG. 4, the method of transmitting UI display information according to the exemplary embodiment includes operations time-serially processed in the local server and the client illustrated in FIG. 3. Accordingly, even if omitted below, the contents described above with respect to the local server and the client illustrated in FIG. 3 are also applied to the method of transmitting the UI display information according to the exemplary embodiment.
Referring to FIG. 4, client A 21, client B 22, and the local server 100 are connected through the home network, and may be mutually called by using the HTTP-based web API (REST service or Remote Procedure Calls (RPC) mechanism), and the TCP/IP protocol. As a specific example, the REST API may be applied for synchronization of the content apps, the RPC may be applied for video streaming between devices, and the TCP/IP protocol may be applied to the transmit of the control information according to user interaction between devices.
In operation 400, the local server 100 receives UI control information according to a selection of a content app of a user from client B 22.
Once client B 22 transmits keyboard input and touch screen input information that the user reacts on the received video stream to the local server through the app selecting module 301 as UI control information of the user, the local server transmits the UI control information to a content application. The present invention is described based on one content application, but the app selecting module may select a plurality of content applications. In this case, the UI distribution control module 105 of the local server 100 may divide a plurality of content application UIs by matching the clients to each UI element group of the plurality of content application UIs. The content application is used for displaying the original content, and may be a video application, such as YouTube or Twitch.
In the meantime, even in the case where the content application is not installed in client B 22, client B 22 may receive the UI element group from the local server 100 by video streaming, or may receive UI element group information and then the client may render and display the UI element group.
In operation 410, the local server 100 executes the content app selected in operation 400.
In operation 420, the local server 100 divides the UI of the executed content app into a plurality of UI element groups. The UI provided from the content application executed in the local server 100 may be divided into a plurality of UI elements by using the UI distribution control module 105 installed in the local server 100. Herein, an authority for the functions of dividing the single UI of the content app into the plurality of UI element groups and generating and modifying a UI distribution information is granted only to the specific device of a user possessing the corresponding right, so that a security function of the system may be strengthened.
In operation 430, the local server 100 matches the proximal clients (21 and 22) to the plurality of UI element groups respectively, and generates UI distribution information.
When client B 22 selects the content app in operation 400, it is preferable to select which client among the proximal clients and which UI element group will be matched for displaying the UI element group.
In operation 440, the local server 100 requests client A (21) to display the UI element group.
The request to display in operation 440 may be made before operation 440. When the local server 100 requests client A (21) to display the UI element group, the local server 100 may make the UI element display module 320 to execute the UI element display by transmitting a push message to the user’s app.
In operation 450, the user’s app of client A (21) receiving a request to display the UI element group executes the UI element group display module 320.
When the UI element group display module 320 is executed, client A may display the UI element group by receiving the video streaming received from the local server 100 in operation 472.
In operation 460, the local server 100 generates virtual display information by using the UI distribution information generated in operation 430. The virtual display information means information stored to render UI element group information in a graphic buffer. Therefore, the graphic buffer may reflect the UI distribution information by using the virtual display information and transmit the video stream to each device.
In operation 471 and 472, the local server 100 transmits one or more UI element groups corresponding to client A (21) and client B (22), respectively, as a video stream by using the UI distribution information. Since the local server 100 transmits the UI element group as a video stream, it is not necessary to install and execute the content application in client A (21) and client B (22).
In operation S481 and 482, the local server 100 receives UI control information from client A (21) and client B (22).
In operation 490, the local server 100 converts the received UI control information to a control signal of the content app.
When the user inputs the UI control information through a touch screen, a keyboard, a voice, and the like of client A (21) and client B (22), the UI control information may be transmitted to the local server through the TCP/IP protocol or the web API.
The local server 100 may convert the UI control information received from client A (21) and client B (22) into the control signal of the content app in consideration of display size and resolution information of each client possessed by the local server 100. Further, the user control information may be transmitted to the local server by providing a separate UI together with the video stream received from the user’s app installed in the client device. In this case, the client device needs to convert the user control signal and transmit the converted user control signal to the local server.
For example, when the local server 100 receives a coordinate value input by the user on the display of client A (21), it is necessary to determine which element of the content app UI corresponds to the received coordinate value in consideration of the display size and resolution information. The determination may also be performed in client A (21) and may also be performed in the local server 100. Even though the local server 100 receives the coordinate value input by the user on the display of client A (21), the local server 100 has the display size and resolution information of client A (21), so that the local server 100 may determine which part of the content app may corresponds to the received coordinate value.
Further, the UI control information may include content control information and UI distribution editing information, and the content control and the UI distribution editing are performed by the following method.
As an example, when the same content app is installed in the local server 100 and the client 21 and 22, a context state value (or a parameter setting value of the content app) of the result of the UI distribution and its editing of selected UI by the client 21 and 22 may be transmitted to the local server 100.
The local server 100 performs with the authority to call the functions of the client 21 and 22 necessary for collecting the context state value.
In this case, it is not necessary for the local server 100 to transmit display data for the UI distribution editing to the client 21 and 22, and when the display information is generated and the UI editing is completed in the client 21 and 22, it is preferable that the client 21 and 22 change the context state value and provide the changed context state value to the content app of the local server 100.
As another example, when the content app is installed only in the local server 100, it is preferable that the local server 100 provides the information necessary for the information of UI distribution and its editing to the client 21 and 22 as a video stream. When the client 21 and 22 receives the UI distribution display transmitted by the local server 100 and transmits touch screen response results of the users in the client 21 and 22 to the local server 100, the local server 100 transmits the touch screen response as user interaction information to the content app. Thereafter, when the information of the single UI of the content app is updated, the local server applies the updated UI information of the content app to the UI distribution information and transmits the UI distribution information as a video stream. By repeating this process, the user is capable of performing UI distribution editing.
Further, the transmission of the video stream is possible independent to the OSs of the client 21 and 22, the client 21 and 22 may analyze the user’s control intention based on the response information of the touch screen or the touch pad and transmit a corresponding result to the local server 100.
FIG. 5 is a flowchart illustrating a method of transmitting, by the local server, UI display information to proximal clients according to another exemplary embodiment of the present invention.
Referring to FIG. 5, the method of transmitting UI display information according to the exemplary embodiment includes operations time-serially processed in the local server and the client illustrated in FIG. 3. Accordingly, even if omitted below, the contents described above with respect to the local server and the client illustrated in FIG. 3 are also applied to the method of transmitting the UI display information according to the exemplary embodiment.
The difference between the method of transmitting UI display information to the proximal clients 21 and 22 by the local server 100 illustrated in FIG. 5 and the method illustrated in FIG. 4 is that client A (21) receives the UI element group from the local server 100.
Client A (21) and client B (22) are connected with the local server 100 through the home network, and are capable of mutually calling through the HTTP-based web API (REST service or RPC mechanism).
Operation 400 to operation 460 are as illustrated in FIG. 4.
In operation 471, the local server 100 transmits a video stream for at least one UI element group corresponding to client B (22) by using the UI distribution information.
In the meantime, in operation 473, the local server 100 transmits at least one UI element group corresponding to client A 21 by using the UI distribution information.
In operation 475, since the local server 100 transmits the UI element group, the virtual display generating module 330 generates virtual display information by using the UI element group.
In the meantime, since client A (21) generates the virtual display information by using the received UI element group, client A (21) may have a function of a sub local server performing pixel streaming on another client. An example of the case where the client has the function of the sub local server will be described in detail with reference to FIG. 7.
In operation S481 and 482, the local server 100 receives UI control information from client A (21) and client B (22).
In operation 490, the local server 100 converts the received UI control information to a control signal of the content app. The local server 100 analyzes user interaction of the received UI control information, converts the user interaction into a control signal, and inputs the control signal to the content app that is being executed in the local server 100.
FIG. 6 is a flowchart illustrating a method of transmitting, by the local server, UI display information to proximal clients according to still another exemplary embodiment of the present invention.
The difference between the method of transmitting UI display information to the proximal clients by the local server 100 illustrated in FIG. 6 and the method illustrated in FIG. 5 is that both client A (21) and client B (22) receive the UI element groups from the local server 100.
Since both client A (21) and client B (22) receive the UI element groups and generate the virtual display information, clients 21 and 22 may have a function of a sub local server performing pixel streaming on another client.
Operation 400 to operation 440 are as illustrated in FIG. 4.
In operation 441 illustrated in FIG. 6, the local server 100 requests client B 22 to display the UI element group similar to operation 440.
In operations 450 and 451, client A (21) and client B (22) receiving the request to display the UI element group execute the UI element display module 320 and 321 respectively.
In operation 460, the local server 100 generates virtual display information by using the UI distribution information generated in operation 430.
In the meantime, when the client 21 and 22 execute the UI element display module 320 and 321 respectively, the client 21 and 22 may receive the UI element groups from the local server 100 in operations 473 and 474.
Similar to operation 460 in which the local server 100 generates the virtual display information, in operations 475 and 476, client A (21) and client B (22) generate virtual display information by using the UI element groups received from the local server 100. In this case, client A/B may transmit a first UI element to itself as a video stream, and may transmit a second UI element to another user device as a video stream by using the generated virtual display information.
In operation S481 and 482, the local server 100 receives UI control information from client A 21 and client B 22.
In operation 490, the local server 100 converts the received UI control information to a control signal of the content app.
In the meantime, when an error is generated in the operation of the local server 100, in order to build a resilient system, client A (21) or client B (22) may be defined as a local server candidate having the same authority as that of the local server 100.
In the situation where the local server 100 is operated, the UI distribution information generating module 140 of the local server is synchronized to the client defined as the local server candidate by the web RPC or REST service method, the content app installed in the local server needs to be installed even in the client, and the local server 100 and the content app of the client need to be synchronized by the web API method.
When an error is generated in the operation of the local server 100, the local server candidate client having the authority of the local server, the local server candidate client having the authority of the local server may serve as the local server by using the information of the synchronized elements.
In the meantime, in the case where the OS system of the local server 100 is different from the OS of the client, the matched UI element group information may be transmitted to the user’s app for the present invention of the client device by using a transport layer for each OS, and then, the client device may generate a virtual display and perform video streaming by using the UI element group.
FIG. 7 illustrates a concept of transmitting a UI element group by a relay method when the local server transmits UI display information to proximal clients according to the exemplary embodiment of the present invention.
Referring to FIG. 7, the local server extracts UI tree information included in the original content, virtualizes the single UI, and then divides the UI element groups constituting the UI tree information into group 1 and group 2.
The UI elements group divided into group 1 are stored as pixel data of a first graphic buffer, and the UI element groups divided into group 2 are stored without being converted to the pixel data. In this case, when the local server renders and transmits for the UI element group 1, among the UI element groups divided into group 1 and group 2, and transmits for the UI element group 2 without rending to the device of client A, the device of client A forwards the received UI element group 2 to devices of clients B and C. Clients B and C may forward and provide the UI display information between the devices by rendering the UI element group 2 by themselves and providing a UI display.
Further, clients B and C of group 2 may generate UI control information and transmit the generated UI control information to client A, and client A may transmit its UI control information and the control information received from clients B and C to the local server. Herein, when information discriminating the local server and the forwarding client is included in the UI element information, the user’s app of the present invention may be configured so that the UI control information of clients B and C is directly transmitted to the local server without passing through client A.
FIG. 8 is a flowchart illustrating a method of dividing, by the local server, UI tree information illustrated in FIG. 7 and transmitting the divided UI tree information to the client according to the exemplary embodiment of the present invention.
Referring to FIG. 8, the method of dividing UI tree information and transmitting the divided UI tree information to the client according to the exemplary embodiment includes operations time-serially processed in the local server and the client illustrated in FIG. 3. Accordingly, even if omitted below, the contents described above with respect to the local server and the client illustrated in FIG. 3 are also applied to the method of transmitting the UI display information according to the exemplary embodiment.
In operation 800, the local server 100 divides the UI tree information included in the content in consideration of the number of clients.
In operation 810, the local server 100 virtualizes the single UI tree. The virtualization means managing the UI display information of the user’s device as a single UI tree in the content application, and managing the single UI tree as a UI group divided into UI element groups at the OS level. Accordingly, the main function of the virtualization is to manage the divided UI display information to be transmitted to the plurality of user devices as UI group information, and it is possible to prevent a malfunction according to the UI tree virtualization by transmitting, to the content application for a single UI tree, collected user interaction information for the UI display information in which is divided into the plurality of UI groups and is transmitted to the plurality of devices.
Therefore, in order to prevent the malfunction, it is necessary to synchronize an update cycle of a main variable value of the divided UI element groups, the update of the user interaction information provides the user interaction information to the content application when a variable value of the main function of the UI element group is changed within a predetermined update cycle.
In operation 820, the local server 100 receives display resolution from client A (21).
In operation 830, the local server 100 generates a graphic buffer (for example, the virtual display) having the same resolution as the received display resolution.
In operation 840, the local server 100 renders the UI tree information divided in operation 800 to the graphic buffer corresponding to the client.
In operation 850, the local server 100 encodes pixel data stored in the graphic buffer and transmits the encoded pixel data to the corresponding client.
In operation 860, client A 21 decodes the received pixel data to a video stream and renders the video stream.
FIG. 9 illustrates a flow of a method of editing a UI of a content app by a client according to the exemplary embodiment of the present invention.
Referring to FIG. 9, the method of editing a UI of a content app by the client according to the exemplary embodiment includes operations time-serially processed in the local server and the client illustrated in FIG. 3. Accordingly, even if omitted below, the contents described above with respect to the local server and the client illustrated in FIG. 3 are also applied to the method of transmitting the UI display information according to the exemplary embodiment.
In operation 900, client A (21) transmits selected content app to the local server 100.
In operation 910, the local server 100 executes the selected content app. When there is no selected content app, the local server 100 may download the content app through the cloud server 300 and execute the downloaded content app.
In operation 920, client A (21) receives the UI of the content app from the local server 100 by video streaming.
In operation 930, client A (21) displays the received UI of the content app.
In operation 940, client A (21) executes a UI setting mode of the content app.
In operation 950, client A (21) selects at least one UI element group of the content app, and selects a client which is to display the selected UI element group.
In operation 960, client A (21) transmits UI distribution information matching a client information to the selected UI element group for display to the local server 100.
In operation 970, the local server 100 transmits the UI element group to the matched client that displays the UI element group with reference to the UI distribution information.
FIG. 10 illustrates a flow of a method of transmitting, by the local server, IoT state information to the client device in an OS level according to another exemplary embodiment of the present invention.
Referring to FIG. 10, the method of transmitting IoT state information to the client according to the exemplary embodiment includes operations time-serially processed in the local server and the client illustrated in FIG. 3. Accordingly, even if omitted below, the contents described above with respect to the local server and the client illustrated in FIG. 3 are also applied to the method of transmitting the IoT state information according to the exemplary embodiment.
In operation 1000, the local server 100 receives information about the IoT application selected for receiving IoT state information by a user from the client A (21). The selectable IoT applications are listed in client A (21), and when the user selects any one IoT application, the local server 100 receives which IoT application is selected as user UI control information.
In operation 1010, the local server 100 executes the selected IoT application. In the case where the IoT application selected in operation 1000 is not installed, the local server 100 may download the IoT application through the cloud server 300 of the IoT device.
In operation 1020, the local server 100 divides a UI of the executed IoT application into a plurality of UI element groups.
In operation 1030, the local server 100 generates UI distribution information by using the plurality of divided UI elements groups.
In operation 1040, the local server 100 generates virtual display information by using the generated UI distribution information.
In operation 1050, the local server 100 may transmit the virtual display information to client A (21) by video streaming or transmit the matched UI element group to the client device.
In operation 1060, the local server 100 receives UI control information from the user’s app of client A (21).
In operation 1070, the local server 100 converts the received UI control information to a control signal of the IoT application.
In operation 1071, the local server 100 transmits the IoT control signal to an IoT device 25 through the cloud or the home network. When the user device serves as a hub of the IoT device, the IoT device 25 may be controlled through the home network using ZigBee/Wi-Fi.
In the meantime, when the IoT app installed in the local server 100 transmits the IoT control signal to an IoT cloud server, the IoT cloud server transmits the IoT control signal to the IoT device through the IoT hub by the TCP/IP communication by referring to a IP address information uniquely mapped to the IoT device 25.
In operation 1072, the IoT app of the local server 100 receives IoT state information through the IoT device 25 or the IoT cloud server.
In operation 1073, the local server 100 transmits the IoT state information to the user’s app of client A (21) as the video stream or the UI element group information.
FIG. 11 illustrates a flow of a method of transmitting, by the cloud server, IoT state information to the client device according to another exemplary embodiment of the present invention.
Referring to FIG. 11, the method of transmitting IoT state information to the client according to the exemplary embodiment includes operations time-serially processed in the local server and the user of the client illustrated in FIG. 3. Accordingly, even if omitted below, the contents described above with respect to the local server and the user app of the client illustrated in FIG. 3 are also applied to the method of transmitting the IoT state information according to the exemplary embodiment.
FIG. 11 is different from FIG. 10 in that client A (21) located outside the home network receives the IoT state information through the cloud server 300.
In FIG. 11, for the remote control of the IoT device, it is preferable that the UI virtualization information of the cloud server 300 and the local server 100, the UI distribution information, the user device information matched to the UI distribution information, and the like are synchronized through the web API.
In operation 1100, the cloud server 300 receives user control information for selecting an IoT application by the user from client A (21).
In operation 1110, the cloud server 300 determines whether the device of client A (21) is a registered user device in the user device group information.
In operation 1120, when client A (21) is the registered user device as a result of the determination of the cloud server 300, the local server 100 receives information about the IoT application selected by the user of client A (21) from the cloud server 300.
In operation 1130, the local server 100 executes the selected IoT application.
In operation 1140, the local server 100 divides a UI of the executed IoT application into a plurality of UI element groups at an OS level.
In operation 1150, the local server 100 generates UI distribution information by using the plurality of divided UI elements groups. Further, the local server transmits an updated UI distribution information of the local server to the cloud server to synchronizes the UI distribution information between two devices.
In operation 1160, the local server 100 generates the UI of the IoT app of the local server synchronized with the IoT app of the cloud server as virtual display information by using the generated UI distribution information.
In operation 1170, the local server 100 transmits a UI display of the matched IoT app to the device of client A (21) by video streaming by using the virtual display information, or transmit the matched UI element group information.
In operation 1180, client A (21) transmits UI control information to the cloud server 300, and when client A (21) transmits the UI control information to the IoT app installed in the cloud server, context state information of the IoT app of the local server is updated by the cloud server of the IoT device (1181).
Therefore, the IoT app of the local server 100 transmits the received UI control information to the IoT device.
In operation 1191, the IoT app of the local server 100 transmits IoT control signal to the IoT cloud server through the Internet web, or when the local server serves as the IoT hub, the local server directly transmits the IoT control signal to the IoT device through the home network.
In operation 1192, the IoT app of the local server 100 receives the state information of the IoT device via the cloud of the IoT app and the IoT hub, or directly receives the state information from the IoT device through the home network when the local server serves as the IoT hub.
In operation 1193, when the context state information of the IoT app of the local server 100 is changed, the IoT app provides the cloud of the IoT device with the changed state information, or transmits the changed state information to the IoT app of the cloud server 300 when the local server serves as the IoT hub.
The hub for controlling the IoT device controls the IoT device on the home network by using ZigBee, Bluetooth, or Wi-Fi network.
FIG. 12 illustrates a flow of a method of transmitting, by the local server, IoT state information to an unauthorized client according to another exemplary embodiment of the present invention.
The method of transmitting, by the local server, the IoT state information to a non-authorized client is different from the methods illustrated in FIGS. 10 and 11 in that a non-authorized client 30 requests the local server 100 to transmit the IoT state information.
In operation 1200, the local server 100 sets IoT state information providing range according to a distance to the client device.
The local server 100 may differentiate and provide the IoT state information according to the location of the non-authorized client 30. There is a case where the non-authorized client 30 is physically close to the local server 100 and there is a case where the non-authorized client 30 requests the IoT state information through the home network or the Internet at a remote distance, and it is possible to differentiate and provide the IoT state information according to the physical distance between the local server 100 and the non-authorized client 30.
In this case, when the location of the non-authorized client 30 is physically present within a predetermined distance from the local 100, more IoT state information may be set to be provided.
Further, depending on whether the IoT state information is requested from inside or outside the place in which the local server 100 is installed, the local server 100 may differentiate and provide the state information of the IoT device 25.
In operation 1210, the local server 100 receives a transmission request of an IoT state information from the user’s app of the non-authorized client 30.
In operation 1220, the local server 100 calculates a distance to the non-authorized client 30 based on the transmission request of IoT state information received in operation 1210.
In order to accurately calculate the distance between the local server 100 and the non-authorized client 30, when an antenna of a wireless communication module (Bluetooth, Wi-Fi, and ultra-wide band) of the local server 100 is implemented with multi-antennas having a predetermined reference distance, some of the paired antennas may be implemented to receive horizontal polarization and some other antennas may be implemented to receive vertical polarization.
In this case, the local server 100 may receive a wireless communication signal from the non-authorized client 30 through the paired vertical polarization/horizontal polarization multi-antennas, and accurately calculate a location of the non-authorized client 30 by a arrival time difference according to a difference in the incident angles of the received signals between the paired multi-antennas.
Accordingly, the local server 100 may determine the IoT state information to be provided to the user’s app of the non-authorized client 30 by calculating the distance between the local server 100 and the non-authorized client 30.
In operation 1230, the local server 100 transmits the transmission request of the IoT state information to the IoT device 25.
In operation 1240, the local server 100 receives the IoT state information from the IoT device 25.
In operation 1250, the local server 100 determines the IoT state information and the UI distribution information to be transmitted to the non-authorized client 30 according to the distance calculated in operation 1220.
In operation 1260, the local server 100 generates virtual display information by using the UI distribution information.
In operation 1270, the local server 100 transmits the IoT state information as a video streaming to the user’s app of the non-authorized client device. In this case, the IoT state information is transmitted from the local server 100 to the non-authorized client 30 as a video stream and is displayed on a screen. Accordingly, the installation of the IoT application, required for receiving the IoT state information of plurality of IoT devices, is not required in the non-authorized client 30.
FIG. 13 illustrates a flow of a method of receiving an NFT content and a UI app, by the local server, and transmitting UI display information according to another exemplary embodiment of the present invention.
Referring to FIG. 13, the method of transmitting UI display information according to the exemplary embodiment includes operations time-serially processed in the local server and the client illustrated in FIG. 3. Accordingly, even if omitted below, the contents described above with respect to the local server and the client illustrated in FIG. 3 are also applied to the method of transmitting the UI display information according to the exemplary embodiment.
FIG. 13 illustrates a method in which a local server 20 or client A (21) uses a UI app including NFT content available for a predetermined period by transmitting the NFT content of an NFT content sever 500 and the UI app of a UI app server 600 to the local server 20.
In operation 1300, the user mutually concludes a smart contract with an NFT content provider 500 and a UI app provider 600.
Digital assets are managed by blockchain-based smart contracts, and digital asset tokens are largely divided into Fungible Tokens (FT) and Non-Fungible Tokens (NFT). In the Ethereum token, ERC-20 is defined as a fungible token and ERC-721 token is defined as a non-fungible token. In addition, ERC-1155 is a token that can manage ERC-20 and ERC-721 tokens through a single smart contract.
Accordingly, in the present invention, it is preferable that the NFT content provider/server 500, the UI app provider/server 600, and the user conclude a smart contract related to ownership/use of the NFT content and the UI app based on ERC-1155.
When the UI app provider/server 600, the NFT content provider/server 500, and the user/local server 20 conclude the smart contract effective for a specific period of time, the NFT content and the UI app may be transmitted to the local server 20. Accordingly, the local server 20 is granted to access the NFT contents for the predetermined period of time and is allowed to access as a sole user. In this case, the smart contract may include a condition of limiting display transmission of the NFT content transmitted to the plurality of user devices , which is a present invention limiting the number of authorized user devices included in the user device group.
In operation 1310, client A (21) selects the NFT content and the UI app to be used and transmits the selected NFT content and UI app to the local server 100.
In this case, it is possible to match the plurality of selectable UI apps stored in the UI app server 600 to a fungible token and provide the matched token to the local server 20 based on the contents of the smart contract, and the local server 20 may generate UI distribution information by dividing a single UI of the UI app into the units of UI element groups at the OS level and matching the divided UI element groups to the client devices of the user, and transmit one UI element group to the matched client A (21).
Client A (21) may select one of the plurality of selectable UI apps stored in the UI app server 600, combine the selected UI app with the NFT content, and display the combined NFT content.
In operation 1320, the local server 100 checks whether the smart contract is concluded between the UI app server 600 and the NFT content server 500, and then receives the UI app selected by client A (21) from the UI app server 600.
In operation 1321, the local server 100 checks whether the smart contract is concluded between the UI app server 600 and the NFT content server 500 and then receives the NFT content from the NFT content server 500.
In operation 1330, the local server 100 executes the selected UI app.
In operation 1340, the local server 100 divides a UI of the executed UI app into a plurality of UI element groups at an OS level.
In operation 1350, the local server 100 matches the divided each UI element group to each client device of the user respectively, and generates UI distribution information.
When the local server 100 is connected with the proximal clients through the home network, it is preferable that the UI distribution control module 105 transmits the generated UI distribution information to the cloud server 300.
In operation 1360, the local server 100 generates virtual display information by using the generated UI distribution information.
In operation 1370, the local server 100 transmits the UI element group to client A (21). Instead of transmitting the UI element group to client A (21), the local server 100 may transmit a video stream for the at least one UI element group corresponding to client A (21) in operation 1370.
In operation 1371, client A (21) generates virtual display information by using the UI element group received from the local server 100.
In operation 1380, the local server 100 receives UI control information from client A (21).
In operation 1381, the local server 100 converts the UI control information to a control signal of the UI app. The local server 100 may apply the control signal of the UI app and re-transmit the UI element group or the video stream.
In operation 1390, the local server 100 may delegate the authority to use the UI app and the NFT content to client A (21).
During the period of time in which the smart contract, between the NFT content server 500 and the UI app server 600 concluded, in operation 1300 is effective, a situation may arise in which the NFT content is transmitted to another client. In this case, when the authority to use is delegated to another client, the smart contract is updated, and each operation of FIG. 13 is executed so that the UI application is executable in a new client.
When the authority to use is delegated to client A (21), the smart contract between the UI app server 600 and the NFT content server 500 is updated including client A (21), instead of the local server 100, and the UI distribution control module 105 is executable in client A (21).
The cloud server 300 may manage information, such as the user’s authority to access to the NFT content, the authority to provide the UI by using the UI app, the authority to correct the UI distribution information, and the delegation of the authority to use. The NFT content user may be delegated all of the above authorities, use the NFT content for a predetermined period of time, and delegate the authority to another user to distribute the NFT content.
Further, the NFT content user may stream and use the NFT content in his/her display device by modifying the UI with a style that he/she wants for a predetermined period of time. Then, when the predetermined period of time elapses based on the contents of the smart contract, the authorities to use of all of the users, except for the content user may be reset.
The exemplary embodiments of the present invention may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer readable medium. The computer readable medium may include a program command, a data file, a data structure, and the like alone or in combination. The program command recorded in the medium may be specially designed and configured for the present invention, or may also be known and usable to those skilled in computer software. Examples of the computer readable recording medium include a magnetic medium, such as a hard disk, a floppy disk, or a magnetic tape, an optical recording medium, such as a CD-ROM or a DVD, a magneto-optical medium, such as a floptical disk, and a hardware device which is specifically configured to store and execute the program command such as a ROM, a RAM, and a flash memory. An example of the program command includes a high-level language code executable by a computer by using an interpreter, and the like, as well as a machine language code created by a compiler. The hardware device may be configured to be operated with one or more software modules in order to perform the operation of the present invention, and an opposite situation thereof is available.
As described above, the present invention has been described with reference to the specific matters, such as a specific component, limited embodiments, and drawings, but these are provided only for helping general understanding of the present invention, and the present invention is not limited to the aforementioned exemplary embodiments, and those skilled in the art will appreciate that various changes and modifications are possible from the description.
Accordingly, the spirit of the present invention shall not be determined while being limited to the foregoing exemplary embodiments, and it will be considered that all matters having equivalent or equivalent modifications to the claims are within the scope of the present invention.

Claims (12)

  1. A method of receiving and controlling state information of an Internet of Things (IoT) device in a user UI application of a user device, the method comprising:
    receiving, by a UI application executed in a first user device, UI element group information or receiving a video stream in association with the UI element group information;
    transmitting, by the UI application of the first device, control information of a user to a second user device,
    wherein the UI element group information corresponds to a UI element group of a plurality of UI element groups, the plurality of UI element groups divided from a UIs of an IoT application installed in the second user device, the IoT application to control state information of a IoT device, and
    wherein the UI element group information is determined by a distance between the second user device and the first user device or direction angle information of the first user device based on the second user device.
  2. The method of claim 1, further comprising:
    when the UI application receives the UI element group information, the UI application renders the UI element group information into a video stream.
  3. The method of claim 1, wherein when the UI application receives the video stream, the video stream is in association with a display size and resolution of the first user device.
  4. The method of claim 1, wherein the first user device is a user device that is registered in a group of the second user device.
  5. The method of claim 1, further comprising:
    converting, by a UI application of the second user device, the user control information into a user control information applicable to the IoT application of the second user device.
  6. The method of claim 1, wherein the reception of the video stream uses an RPC protocol, and the transmission of the user control information uses an TCT/IP protocol.
  7. A method of transceiving and controlling state information of an Internet of Things (IoT) device in a user device, the method comprising:
    receiving, by a first user device, UI element groups and generating virtual display information;
    rendering, by the first user device, a first UI element group of the UI element groups into a video stream;
    converting, by the first user device, user control information into a control signal applicable to an IoT application in the first user device; and
    transmitting, by the first user device, a second UI element group of the UI element groups or a video stream in association with the second UI element information to a second user device.
  8. The method of claim 7, wherein when the first user device transmits the second UI element information to the second user device as the video stream, the video stream in association with a display size and resolution of the second user device.
  9. The method of claim 7, wherein the transmission of the video stream uses an RPC protocol, and the transmission of the user control information uses an TCP/IP protocol.
  10. The method of claim 7, wherein the first user device and the second user device are included in a user device group of the first user device.
  11. The method of claim 7, wherein the UI element groups corresponds to a UI element groups of plurality of UI element groups, the plurality of UI element groups divided from a UIs of the IoT application.
  12. The method of claim 7, wherein the second UI element group is determined by a distance between the second user device and first user device or direction angle information of the second user device based on the first user device.
PCT/KR2023/002794 2022-02-28 2023-02-28 Method of transmitting state information of iot device by local server WO2023163567A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
KR10-2022-0026427 2022-02-28
KR1020220026427A KR20230128914A (en) 2022-02-28 2022-02-28 Method of transmitting IoT status information by a local server

Publications (1)

Publication Number Publication Date
WO2023163567A1 true WO2023163567A1 (en) 2023-08-31

Family

ID=87766469

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/KR2023/002794 WO2023163567A1 (en) 2022-02-28 2023-02-28 Method of transmitting state information of iot device by local server

Country Status (2)

Country Link
KR (1) KR20230128914A (en)
WO (1) WO2023163567A1 (en)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20200211546A1 (en) * 2018-03-14 2020-07-02 Google Llc Generating iot-based notification(s) and provisioning of command(s) to cause automatic rendering of the iot-based notification(s) by automated assistant client(s) of client device(s)
WO2021215547A1 (en) * 2020-04-21 2021-10-28 엘지전자 주식회사 Smart home device and method
US20210365681A1 (en) * 2018-03-02 2021-11-25 Purdue Research Foundation System and method for spatially mapping smart objects within augmented reality scenes

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20210365681A1 (en) * 2018-03-02 2021-11-25 Purdue Research Foundation System and method for spatially mapping smart objects within augmented reality scenes
US20200211546A1 (en) * 2018-03-14 2020-07-02 Google Llc Generating iot-based notification(s) and provisioning of command(s) to cause automatic rendering of the iot-based notification(s) by automated assistant client(s) of client device(s)
WO2021215547A1 (en) * 2020-04-21 2021-10-28 엘지전자 주식회사 Smart home device and method

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
KI TAEYEON; PARK CHANG MIN; DANTU KARTHIK; KO STEVEN Y.; ZIAREK LUKASZ: "Mimic: UI Compatibility Testing System for Android Apps", 2019 IEEE/ACM 41ST INTERNATIONAL CONFERENCE ON SOFTWARE ENGINEERING (ICSE), IEEE, 25 May 2019 (2019-05-25), pages 246 - 256, XP033602794, DOI: 10.1109/ICSE.2019.00040 *
PETITTI JOSEPH: "Appjudicator: Enhancing Android Network Analysis through U1 Monitoring", MASTER'S THESIS, WORCESTER POLYTECHNIC INSTITUTE, 1 May 2021 (2021-05-01), XP093087455 *

Also Published As

Publication number Publication date
KR20230128914A (en) 2023-09-05

Similar Documents

Publication Publication Date Title
WO2020190016A1 (en) Method and device for providing authentication in network-based media processing (nbmp) system
WO2016068622A1 (en) Terminal device and method of controlling same
WO2014014302A1 (en) Method and device for sharing content
WO2015041438A1 (en) Method for screen mirroring and source device thereof
WO2016027933A1 (en) Digital device and method for controlling same
WO2022050652A1 (en) Method, apparatus, and computer readable storage medium for controlling account
WO2013151374A1 (en) Method and system for transfering data between plurality of devices
WO2013085263A1 (en) Network tv and method for operating same
WO2012112011A2 (en) Method and apparatus for the seamless playback of content
WO2015016627A1 (en) Method and device for connecting single ap device among multiple ap devices on same network to terminal
WO2015137604A1 (en) Method for testing cloud streaming server, and apparatus and system therefor
WO2014157903A1 (en) Method and device for displaying service page for executing application
WO2013058423A1 (en) Electronic device and method for operating the electronic device
WO2012099378A2 (en) Method and apparatus for controlling the transceiving of content
WO2021133042A1 (en) Electronic device and method of operating the same
WO2021150060A1 (en) Method and apparatus for edge computing service
WO2022060175A1 (en) Method and device for transmission of image content using edge computing service
WO2017034298A1 (en) Digital device and method of processing data in said digital device
WO2022211465A1 (en) Apparatus and method for providing media streaming
WO2022075737A1 (en) Method and device for providing mec service
WO2023163567A1 (en) Method of transmitting state information of iot device by local server
WO2023163569A1 (en) Method of transceiving ui display between user devices
WO2023163568A1 (en) Method of transmitting ui display information based on smart contract
WO2023163566A1 (en) Method of transmitting ui display information to proximal clients by local server
WO2020149722A1 (en) Working environment provisioning method and apparatus for execution of application program between electronic device and external server

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23760441

Country of ref document: EP

Kind code of ref document: A1