KR20220148495A - Service server for providing screen and operating method thereof - Google Patents

Abstract

According to an embodiment, a screen providing method of a service server comprises the following operations of: receiving a key input from a user terminal; setting a capture area size based on the key input; capturing an image rendered in a browser based on the capture area size; and transmitting a captured image to the user terminal. In addition, various embodiments may be possible.

Description

A service server for providing a screen and an operating method thereof

Various embodiments of the present disclosure relate to a service server and an operating method thereof, and more particularly, to a service server providing a screen to a user terminal and an operating method thereof.

Recently, with the development of cloud streaming technology, a virtualization technology capable of running high-spec applications with a low-spec user terminal has been widely proposed mainly for game services.

As a related prior art, there is Korean Patent Publication No. 10-1795537 (Title of the Invention: Network Environment Adaptive Cloud Streaming Service Provision System and Method, Applicant: Kyungpook National University Industry-Academic Cooperation Foundation). In the corresponding registered patent publication, by setting the video quality to be transmitted to the user terminal in consideration of both the network state between the video server and the cloud server and the network state between the cloud server and the user terminal in the cloud server, the playback quality of the video output through the user terminal is determined. Disclosed are a network environment adaptive cloud streaming system and method for improving the network environment.

In the case of a cloud streaming-based service, since the service is provided to a large number of users, managing the sessions of the users who are accessing may have an important effect on the service quality.

If the user moves the screen quickly while receiving the image from the cloud server, he or she may experience some delay or lagging in operation.

However, the technical problems are not limited to the above-described technical problems, and other technical problems may exist.

According to an embodiment, a method for providing a screen of a service server includes: receiving a key input from a user terminal; setting a capture area size based on the key input; capturing an image rendered in a browser based on the size of the capture area; and transmitting the captured image to the user terminal.

According to an embodiment, the setting of the capture area size based on the key input may include: when the key input is a direction key input less than a threshold time, drag less than a threshold speed, and scroll less than the threshold speed ), setting the size of the capture area to be the same as the size of a user interface (UI) providing area displayed on the display of the user terminal.

According to an embodiment, the setting of the capture area size based on the key input may include any one of a direction key input when the key input exceeds a threshold time, drag exceeding a threshold speed, and scrolling more than a threshold speed. In the case of a single high-speed movement input, the size of the capture area is larger than the size of a user interface (UI) providing area displayed on the display of the user terminal based on at least one of the direction of the key input, the threshold time, and the threshold speed. It may include an operation of setting large.

According to an embodiment, the transmitting of the captured image to the user terminal includes down-scaling the image quality of the captured image according to the capture area size while the high-speed movement input is maintained to the user terminal. It may include an operation of transmitting to .

According to an embodiment, the operation of transmitting the captured image to the user terminal may further include the operation of restoring the image quality of the captured image to the original image quality and transmitting the image quality to the user terminal when the high-speed movement input is stopped. have.

A service server according to an embodiment includes a browser that executes and renders a web app; a memory in which computer-executable instructions are stored; and a processor that accesses the memory and executes the instructions, wherein the processor receives a key input from a user terminal and sets a capture area size based on the key input and capture the image rendered in the browser based on the capture area size, and transmit the captured image to the user terminal.

According to an embodiment, when the key input is any one of a direction key input less than a threshold time, a drag less than a threshold speed, and a scroll less than the threshold speed, the size of the capture area is determined by the processor. It may be set to be the same as the size of a user interface (UI) providing area displayed on the display of the user terminal.

According to an embodiment, when the key input is a high-speed movement input that is any one of a direction key input longer than a threshold time, a drag greater than a threshold speed, and a scroll greater than the threshold speed, the key input is The size of the capture area may be set to be larger than the size of a user interface (UI) providing area displayed on the display of the user terminal based on at least one of a direction, the threshold time, and the threshold speed.

According to an embodiment, the processor may down-scale the quality of the captured image according to the size of the capture area while the high-speed movement input is maintained and transmit the down-scaling to the user terminal.

According to an embodiment, when the high-speed movement input is stopped, the processor may restore the image quality of the captured image to the original image quality and transmit it to the user terminal.

A user terminal according to an embodiment includes a communication module for communicating with a service server; a display module for displaying the image received from the service server; a memory in which computer-executable instructions are stored; and a processor that accesses the memory and executes the instructions, wherein the processor transmits a key input to the service server, receives a captured image according to the key input from the service server, and the captured image may be captured according to a capture area size set based on the key input.

According to various embodiments of the present disclosure, in providing a screen to a user terminal, the service server may analyze a user's key input and set an extended capture area according to the key input to provide a screen to the user more quickly.

According to various embodiments, the service server may provide a service in consideration of the simultaneous access environment of multiple users by setting an extended capture area based on a key input and maintaining resource usage by down-scaling the image quality. .

In addition, various effects directly or indirectly identified through this document may be provided.

1 is a configuration diagram of a service server according to various embodiments of the present disclosure;
2 is a diagram for explaining an operation of connecting a session between a user terminal and a service server through a session server according to various embodiments of the present disclosure;
3A and 3B are diagrams for explaining screen information included in an access information database according to various embodiments of the present disclosure;
4 is a flowchart illustrating a method of providing a screen from a service server to a user terminal according to various embodiments of the present disclosure;
5 is a view for explaining a method of setting a capture area and providing a screen when a key input is not a high-speed movement input according to various embodiments of the present disclosure;
6 is a view for explaining a method of setting a capture area and providing a screen when a key input is a high-speed movement input according to various embodiments of the present disclosure;
7 is a flowchart illustrating a method of providing a screen of a service server according to various embodiments of the present disclosure.
8 is a flowchart illustrating a method of setting a capture area according to a key input according to various embodiments of the present disclosure.
9 is a configuration diagram of a user terminal according to various embodiments of the present disclosure;

Specific structural or functional descriptions of the embodiments are disclosed for purposes of illustration only, and may be changed and implemented in various forms. Accordingly, the actual implementation form is not limited to the specific embodiments disclosed, and the scope of the present specification includes changes, equivalents, or substitutes included in the technical spirit described in the embodiments.

Although terms such as first or second may be used to describe various elements, these terms should be interpreted only for the purpose of distinguishing one element from another. For example, a first component may be termed a second component, and similarly, a second component may also be termed a first component.

When a component is referred to as being “connected” to another component, it may be directly connected or connected to the other component, but it should be understood that another component may exist in between.

The singular expression includes the plural expression unless the context clearly dictates otherwise. In this specification, terms such as "comprise" or "have" are intended to designate that the described feature, number, step, operation, component, part, or combination thereof exists, and includes one or more other features or numbers, It should be understood that the possibility of the presence or addition of steps, operations, components, parts or combinations thereof is not precluded in advance.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present specification. does not

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. In the description with reference to the accompanying drawings, the same components are assigned the same reference numerals regardless of the reference numerals, and the overlapping description thereof will be omitted.

<service server>

1 is a configuration diagram of a service server according to various embodiments of the present disclosure;

Referring to FIG. 1 , a configuration diagram of a service server 100 that provides a service through a network is shown. According to an embodiment, the service server 100 may provide a screen virtualized with a user interface (UI) to the user terminal 180 through a network. According to an embodiment, the service server 100 includes an access information database 130 for storing access information for each user terminal and a memory 140 for storing computer-executable instructions. , it may include a browser 150 that executes and renders a web app, and a processor 120 that accesses the memory 140 and executes instructions.

According to an embodiment, the service server 100 provides a screen to the user terminal 180 in a cloud streaming method, and only inputs and outputs are made in the user terminal 180, so that the performance of the user terminal 180 is performed. Regardless, high-spec service can be provided. Specifically, by implementing an execution screen in the service server 100 connected to the user terminal 180 through a network, a user experience similar to that executed in the user terminal 180 may be provided. According to an embodiment, the service server 100 executes various applications such as a user interface (UI) application and a game application, captures the execution screen to generate a captured image, and encodes it in a streaming method. It can be transmitted to the user terminal 180 . The service server 100 may execute an application according to a user input transmitted from the user terminal 180 , and may capture the changed execution screen and transmit it to the user terminal 180 in a streaming manner.

Although not shown in FIG. 1 , the service server 100 may provide content to the user terminal 180 in conjunction with a media platform. The media platform may provide predetermined content separately from the service server 100 in a streaming manner. For example, content composed of moving images such as real-time broadcasting or VOD may be provided, and this media platform may be operated by a service provider different from the service server 100 .

According to an embodiment, through the interworking of the user terminal 180 and the service server 100 , the content provided by the media platform (not shown) is fused with the content provided by the service server 100 to the user terminal 180 . can be provided as For example, a user interface screen provided by the service server 100 may be fused with a video image of a real-time broadcast or VOD provided by a media platform. However, the present invention is not limited thereto, and various types of different contents may be fused according to the service server 100 according to various embodiments.

According to various embodiments, an application driven by the service server 100 according to a user input transmitted from the user terminal 180 may include a web app and a native app. There may be differences in that a web app is executed through the browser 150 , and a native app is developed in a language optimized for the user terminal 180 and executed on the user terminal 180 . .

According to an embodiment, when a web application execution request is received from the user terminal 180 , the processor 120 requests execution of the web application to the browser 150 , and the processor 120 for the image rendered in the browser 150 . ) may be captured, and the captured image may be provided to the user terminal 180 in a streaming manner.

According to an embodiment, when a request to execute a native app is received from the user terminal 180 , the processor 120 transmits information related to the execution of the native app to the user terminal 180 , and the user terminal 180 receives the native app execution request. This can be executed. For example, when a VOD content reproduction request is received from the user terminal 180 , the processor 120 receives the VOD content information from a media platform (not shown) and transmits it to the user terminal 180 , and the user terminal At 180 , the corresponding VOD content may be played through a VOD player that is a native app.

The network according to an embodiment is configured to connect between the user terminal 180 and the service server 100 to transmit data, and may be implemented as various types of communication networks. For example, wireless communication methods such as WLAN (Wireless LAN), Wi-Fi (Wi-Fi), Wibro, Wimax, HSDPA (High Speed Downlink Packet Access) or Ethernet (Ethernet), xDSL (ADSL, VDSL) , HFC (Hybrid Fiber Coaxial Cable), FTTC (Fiber to The Curb), FTTH (Fiber To The Home), such as a wired communication method may be used. In addition, in addition to the above-described communication methods, all other well-known or future communication methods may be included.

According to an embodiment, the user terminal 180 refers to a device capable of transmitting and receiving various information with the service server 100 through a network, and provides a cloud UI client, a program and a protocol for transmitting and receiving information. It may include a memory (not shown) for storing, a processor (not shown) for executing a program, and the like. According to an embodiment, the user terminal 180 is a communication module for communicating with the service server 100 or a media platform (not shown), an input module for receiving information from a user, and from the service server 100 or the media platform. It may further include a display module for displaying the provided screen or content.

According to an embodiment, when the user terminal 180 is a set-top box of a smart TV, a user input input through a remote control may be received and transmitted to the service server 100 . The user terminal 180 may transmit a request to the service server 100 through a user input, and may receive and reproduce streaming data from the service server 100 in response thereto. The user terminal 180 may receive and decode the encoded streaming data, and may render and output the decoded data. The user terminal 180 may be implemented in various forms, and the user terminal 180 will be described in detail with reference to FIG. 9 .

According to an embodiment, the processor 120 may request execution of a web app from the browser 150 based on the request received from the user terminal 180 , and the browser 150 may request the web app from the processor 120 ( web app) You can run a web app according to the execution request. According to an embodiment, the browser 150 may render and parse the data of the web application, for example, HTML-based web code, and the processor 120 captures the image rendered by the browser 150 in real time ( capture), the captured image may be encoded and provided to the user terminal 180 in a streaming manner.

According to an embodiment, the service server 100 may include an access information database 130 that stores access information for each of the user terminals 180 . Although the access information database 130 is illustrated as being included in the service server 100 in FIG. 1 , it is not limited thereto and may be implemented to be included in a server separate from the service server 100 .

According to an embodiment, when receiving a web app execution request from the user terminal 180 , the processor 120 may provide a screen to the user terminal 180 with reference to the access information database 130 . . According to an embodiment, in the access information database 130, the user terminal information 132, the information 134 on the web app previously provided to the user terminal, and the screen for the final screen in each of the app provided to the user terminal Information (or ID) 136 and focusing information (or ID) 138 on the position of the pointer in the final screen may be included. The screen information 136 according to various embodiments will be described in detail with reference to FIGS. 3A and 3B .

According to an embodiment, the user terminal information 132 , the app information 134 , the screen information 136 , and the focusing information 138 may be stored in association with each other. That is, a list of web apps previously accessed for one user terminal may be mapped, final screen information for each web app may be mapped, and focusing information for each final screen may be mapped and stored.

For example, when a home app execution request is received from the user terminal 180 , the processor 120 checks the app information 134 previously provided to the user terminal 180 with reference to the user terminal information 132 . If the home app has been previously provided to the user terminal 180, information 136 on the final screen (eg, the main screen of the home app) that was provided among the screens within the home app and the location of the pointer in the final screen (eg, : You can check the focusing information 138 for the coordinates of the arrow in the main screen of the home app). The processor 120 may request the browser 150 to execute the main screen of the home app based on this, and may provide the screen to the user terminal 180 by capturing an image rendered in the browser 150 .

By providing a screen based on access information according to various embodiments of the present disclosure, a user may more directly access a previously accessed screen, and the service server 100 may reduce resources.

According to an embodiment, the processor 120 may capture and transmit only the changed area, not all the rendered images of the web app executed in the browser 150 . Specifically, the processor 120 may compare the previous screen and the current screen by dividing the image rendered by the browser 150 (eg, divided into 40 tiles), and may transmit only the image of the changed area.

According to an embodiment, the processor 120 may operate differently depending on the type of request received from the user terminal 180 while providing the virtualized screen to the user terminal 180 . For example, a native app execution request and another web app execution request may be determined by the browser 150 . The service server 100 according to various embodiments may determine whether to maintain a session connection with the user terminal 180 according to an execution request to provide a service in consideration of the simultaneous access environment of multiple users.

According to an embodiment, when a native app execution request is received, the processor 120 stores access information corresponding to the user terminal 180 in the access information database 130 and sends it to the user terminal. A native app execution signal may be transmitted and a session connection with the user terminal 180 may be stopped. When the execution of the native app is terminated in the user terminal 180 and a session connection request is transmitted to the service server 100 again, the processor 120 connects the session with the user terminal and access information corresponding to the user terminal 180 is provided. Based on the screen, the screen may be provided to the user terminal 180 .

According to an embodiment, when an execution request for a web app different from the web app being provided from the service server 100 to the user terminal 180 is received, the processor 120 responds to the user terminal 180 . to store access information, transmit another web app execution signal to the user terminal, and stop a session connection with the user terminal 180 . After the session connection is stopped, when a request to execute another web app is received from the user terminal 180 back to the service server 100 , the processor 120 connects the session with the user terminal and accesses corresponding to the user terminal 180 . A screen may be provided to the user terminal 180 based on the information.

2 is a diagram for explaining an operation of connecting a session between a user terminal and a service server through a session server according to various embodiments of the present disclosure;

Referring to FIG. 2 , the service server 100 described in FIG. 1 may be connected to the user terminal 180 through the session server 200 . According to an embodiment, the session server 200 may update the node-related policy for the service server 100 from time to time or periodically, and may determine the service server based on the policy. According to an embodiment, when a server assignment request is received from the user terminal 180, the session server 200 selects a plurality (eg, three) of the service servers 100 determined based on the policy, and provides a service. The IP of the server 100 may be transmitted to the user terminal 180 . The user terminal 180 may request a session connection to one service server 100 among the received plurality of service server 100 IPs, and may receive a screen from the connected service server 100 in a streaming manner.

In other words, in FIG. 2 , the service servers 100 are connected to the session server 200 , and the session server 200 is illustrated only as being connected to the user terminal 180 , but the user terminal 180 is connected to the session server ( When receiving IPs of the service servers from 200 ) and requesting a session connection to one service server, the user terminal 180 and the corresponding service server may be connected.

According to an embodiment, the plurality of service servers 100 may each include an access information database 130 for each user terminal or may check access information through a separate server including the access information database 130 . . According to an embodiment, access information may be updated and managed in common among a plurality of service servers.

According to an embodiment, when the session connection between the user terminal 180 and the service server 100 is interrupted and the session is reconnected, the IP of the service server provided by the session server 200 is changed to the previous service server according to the policy. may be different from the IP of , and the user terminal 180 may be connected to a service server that has not been previously connected. Even when a session is connected to a new service server that is not previously connected, all service servers 100 can check access information for each user terminal commonly managed, so as described in FIG. 1 , the user terminal 180 provides access information A screen may be provided based on the .

For example, when a request to run a native app or a request to run a web app other than the web app being provided is received from the user terminal 180 , the processor 120 of the service server 100 corresponds to the user terminal 180 . access information may be stored and the session connection with the user terminal 180 may be stopped. Even if the user terminal 180 requests server assignment again and receives an IP of a service server different from the previously connected service server from the session server 200, the other service server refers to the access information and displays the screen to the user terminal 180. can provide

For the connection between the user terminal 180 and the service server 100 described below, the operation of the session server 200 may be omitted for a simple explanation, but when the user terminal 180 is connected to the service server again, the session server Based on the node policy of 200 , it may be connected to a service server different from the existing service server.

3A and 3B are diagrams for explaining screen information included in an access information database according to various embodiments of the present disclosure;

Referring to FIG. 3A , in order to describe the screen information 136 included in the access information database 130 , a hierarchy between screens is illustrated. According to an embodiment, the web app application executed through the browser 150 by receiving the web app execution request from the user terminal 180 may be divided into layers between screens.

According to an embodiment, the primary screen 330 may be provided by a user input (eg, selection) from the initial screen 310 of the application to the next screen.

According to an embodiment, the initial screen 310 may be provided as a user input returning to the previous screen from the primary screen 330 (eg, input to the upper screen, back, previous screen, etc.), and then to the next screen. The secondary screen may be provided by a user input (eg, selection) of .

According to an embodiment, the primary screen 330 may be provided as a user input returning to the previous screen on the secondary screen 350 (eg, input to the upper screen, back, previous screen, etc.), and the initial screen The initial screen 310 may be provided as a user input returning to (eg, an input for the top screen, the initial screen, the first time, etc.).

Referring to FIG. 3B , an initial screen, a first screen, and a second screen displayed through the user terminal 180 are illustrated. According to an embodiment, when a home app execution request (eg, a home app execution request through a remote control) is received from the user terminal 180 (eg, a set-top box of a smart TV), the service server 100 The processor 120 may provide the initial screen 360 to the user terminal 180 . According to an embodiment, when a user input for the movie tab 365 is received on the initial screen 360 , the processor 120 may provide the primary screen 370 to the user terminal 180 . According to an embodiment, when a user input for a specific movie content 375 is received on the primary screen 370 , the processor 120 may provide the secondary screen 380 to the user terminal 180 . .

According to an embodiment, although the first screen and the second screen are illustrated in FIGS. 3A and 3B , the present invention is not limited thereto, and screens of more layers may exist. The screen information 136 included in the access information database 130 may include identification information for the screens of each layer (310, 330, 350 in FIG. 3A or 360, 370, 380 in FIG. 3B), specifically In the access information database 130 , information on a web app previously accessed for each of the user terminals 180 and identification information on a final screen within the app may be mapped and stored.

4 is a flowchart illustrating a method of providing a screen from a service server to a user terminal according to various embodiments of the present disclosure;

As described with reference to FIG. 2 , an operation in which the user terminal 180 is connected to the service server 100 through the session server 200 is described with reference number 410 . The user terminal 180 may request the allocation of a service server to the session server 200 ( 415 ), and the session server 200 selects a plurality (eg, three) of service servers based on the node policy ( 415 ). 420) to transmit the IP of the server to the user terminal 180 (425).

Referring to FIG. 4 , an operation of transmitting an image from the service server 100 to the user terminal 180 in a streaming manner as described in FIG. 1 is described with reference to reference numerals 430 to 470 .

According to an embodiment, the user terminal 180 may request execution of a web app from the service server 100 based on the received IP of the service server ( 430 ). The processor 120 of the service server 100 checks the access information corresponding to the user terminal 180 ( 435 ), and requests execution of the web app to the browser 150 based on the received web app execution request and access information ( 435 ). may (440).

For example, the user terminal 180 previously selects a movie tab (eg, reference number 365 in FIG. 3B ) on the initial screen (eg, reference number 360 in FIG. 3B ) of the home app to display the home app. The session connection may be interrupted while the first screen (eg, reference number 370 of FIG. 3B ) is being provided. When the processor 120 receives a home app execution request from the user terminal 180 at reference number 430, the processor 120 may check access information at reference number 435 based on information on the user terminal 180 and information on the home app. have. As described with reference to FIG. 1 in relation to the access information database 130, the processor 120 is based on the terminal information 132, the app information 134, the screen information 136 and the focusing information 138 to the user terminal 180) may request the browser 150 to execute the first screen (eg, reference number 370 of FIG. 3B ), which is the final screen of the home app that has been previously provided.

As described for the operation of the session server 200 in FIG. 2 , the IP of the service server to which the user terminal 180 was previously connected and the service server 100 receiving the execution request at reference number 430 may be different, Since the access information database 130 is commonly managed among a plurality of service servers, the newly connected service server can access access information stored in the previously connected service server.

According to an embodiment, the browser 150 may execute a web app based on the access information and render an image ( 450 ), and the processor 120 captures a result rendered in the browser 150 ( 460 ). An image may be transmitted to the user terminal 180 ( 470 ). As described in FIG. 1 , the processor 120 divides the previous screen and the current screen (eg, divided into 40 tiles) and compares the image rendered in the browser 150, and transmits only the image of the changed area. can By providing a screen based on access information according to various embodiments of the present disclosure, a user may more directly access a previous access screen, and the service server 100 may reduce resources.

5 is a view for explaining a method of setting a capture area and providing a screen when a key input is not a high-speed movement input according to various embodiments of the present disclosure;

Referring to FIG. 5 , when the key input received from the user terminal 120 is not a high-speed movement input, the processor 120 sets the size of the capture area 520 for the image 510 rendered by the browser 150 . action is shown.

According to an embodiment, the image 510 rendered by the browser 150 may be all or at least a part of the screen of the web app being executed. The image 510 may be calculated at which location and in which size on the browser screen in the layout step of the rendering process in the browser 150, and the browser 150 may display the image 510 in the user terminal 180 during the image rendering process. It is possible to determine the relative position and size of each node within the browser area where the graphic is displayed according to the size of the display of the , the size of the browser window, and the like. For example, referring to FIG. 5 , when the user terminal 180 is a smart TV, the image 510 may be rendered in the browser 150 based on the size of the TV display on which the user UI is provided.

According to an embodiment, the processor 120 of the service server 100 may determine whether the key input received from the user terminal 180 is a high-speed movement input. Whether the input is a high-speed movement input may be determined in various embodiments based on the type of the key input, the duration of the key input, and the speed of the key input. For example, if the direction key input is less than a threshold time (eg, 3 seconds), if the drag is less than the threshold speed, if the scroll movement is less than the threshold speed, the processor 120 moves the received key input at a high speed It can decide that it is not an input. The type of each key input, the input duration threshold time and/or the input threshold speed according to the key input type may be different from each other.

5, when the received key input is not a high-speed movement input, the processor 120 sets the capture area size to be the same as the size of the UI providing area displayed on the display of the user terminal 180 (520), An image of a screen rendered according to a key input is transmitted to the user terminal 180 , and a screen as indicated by reference numeral 550 may be provided in the user terminal 180 . For example, if the user presses the down arrow key for less than a threshold time (eg, 3 seconds) while receiving the web app screen, the processor 120 determines that the key input received from the user terminal 180 is not a high-speed movement input. It is determined, and the screen of the web app moved downward according to the key input in the browser 150 may be rendered to have the same size as the size of the UI providing area displayed on the display of the user terminal 180 .

According to an embodiment, as described with reference to FIG. 1 , the processor 120 may capture and transmit only the changed area, not all the rendered images of the web app executed in the browser 150 . Specifically, the processor 120 may compare the previous screen and the current screen by dividing the image rendered by the browser 150 (eg, divided into 40 tiles), and may transmit only the image of the changed area.

6 is a view for explaining a method in which a capture area is set and a screen is provided when a key input is a high-speed movement input according to various embodiments of the present disclosure;

Referring to FIG. 6 , when the key input received from the user terminal 120 is a high-speed movement input, the processor 120 sets the capture area size for the image 610 rendered by the browser 150 is shown. have.

According to an embodiment, as described with reference to FIG. 5 , the image 610 rendered by the browser 150 may be all or at least a part of a screen of a web app being executed, and a layout during the rendering process in the browser 150 . ) step, it can be calculated at which location and at which size on the browser screen.

According to an embodiment, the processor 120 of the service server 100 may determine whether the key input received from the user terminal 180 is a high-speed movement input. Whether the input is a high-speed movement input may be determined in various embodiments based on the type of the key input, the duration of the key input, and the speed of the key input. For example, when a direction key input is greater than a threshold time (eg, 3 seconds), when a drag is greater than or equal to a threshold speed, and when a scroll movement is greater than or equal to a threshold speed, the processor 120 determines that the received key input moves at a high speed. can be determined as input. The type of each key input, the input duration threshold time and/or the input threshold speed according to the key input type may be different.

Referring to FIG. 6 , when the received key input is a high-speed movement input, the processor 120 sets the size of the capture area to be larger than the size 620 of the UI providing area displayed on the display of the user terminal 180 ( 630 ). can How the capture area size is set to be larger than the UI providing area size may be possible in various embodiments based on the type of key input, the key input duration threshold time, and the input threshold speed.

According to an embodiment, when the user presses the down arrow key for a threshold time (eg, 3 seconds) or more while receiving the web app screen, the processor 120 determines that the key input received from the user terminal 180 is a high-speed movement input. , and the capture area size can be set. As the key input time increases, the processor 120 may render the web app screen based on the size of the UI providing area displayed on the display of the user terminal 180 in the browser 150, which is further enlarged in the downward direction.

According to an embodiment, when the processor 120 sets the capture area size 630 larger than the UI providing area size 620, and based on this, the image rendered in the browser 150 is captured, user convenience will be increased. can For example, if the user long inputs the down direction key on the remote control while receiving the screen through the smart TV display, the area enlarged than before is captured and provided, so that the delay in user experience when providing the screen can be reduced.

According to an embodiment, as described with reference to FIG. 1 , the processor 120 may capture and transmit only the changed area, not all the rendered images of the web app executed in the browser 150 . Specifically, the processor 120 may compare the previous screen and the current screen by dividing the image rendered by the browser 150 (eg, divided into 40 tiles), and may transmit only the image of the changed area.

According to an embodiment, when the processor 120 determines that the key input received from the user terminal 180 is a high-speed movement input, and transmits the image to the user terminal 180 based on the enlarged capture area size 630 . , as shown in reference number 650, the image quality may be down-scaled and transmitted. When the key input is a high-speed movement input, there is a high possibility that the user does not focus on the screen, and the processor 120 can save resources by downscaling the image quality of the screen and transmitting it despite the enlarged capture area size 630. . According to an embodiment, when the user's high-speed movement input is stopped, the processor 120 may restore the original image quality again and normally transmit the image to the user terminal 180 .

<How to operate the service server>

7 and 8 are flowcharts for explaining a method of operating a service server according to various embodiments of the present disclosure, and are diagrams for explaining the screen providing method described with reference to FIGS. 1 to 6 from the service server side. An operation method will be described with a focus on the service server, but portions overlapping with those described in FIGS. 1 to 6 will be omitted for the sake of brevity.

7 is a flowchart illustrating a method of providing a screen of a service server according to various embodiments of the present disclosure.

Operations 710 to 740 may be performed by the processor 120 of the service server 100 described above with reference to FIG. 1 .

Referring to FIG. 7 , the processor 120 receives a key input from the user terminal 180 in operation 710 and sets the size of a capture area for capturing a screen to be provided to the user terminal based on the key input in operation 720 . can Before operation 710, the user terminal receives the IP of the service server through the session server 200 and requests a session connection to the service server 100 as described with reference to FIG. You may be provided with an app screen.

According to an embodiment, as described with reference to FIGS. 5 and 6 , in operation 720 , the processor 120 may determine whether the key input is a high-speed movement input. When the key input is not a high-speed movement input, the size of the capture region may be set to be the same as the size of the UI providing region displayed on the display of the user terminal 180 . When the key input is a high-speed movement input, the size of the capture region may be set larger than the size of the UI providing region displayed on the display of the user terminal 180, and the size of the capture region may be set based on the direction of the key input, the duration of the key input, the key input speed, etc. can The operation of setting the capture area size according to the key input in operation 720 will be described in detail with reference to FIG. 8 .

According to an embodiment, in operation 730 , the processor 120 may capture the image rendered in the browser 150 according to the capture area size set in operation 720 . As described with reference numerals 450 to 470 of FIG. 4 , in operation 730 , the processor 120 divides and compares the previous screen and the current screen, and captures only the image of the changed area.

According to an embodiment, in operation 740 , the processor 120 may transmit the captured image to the user terminal 180 . According to an embodiment, as described with reference to FIG. 6 , when it is determined that the key input is a high-speed movement input in operation 720 and the capture area size is set to be larger than the UI providing area, in operation 740, the processor 120 controls the capture image. It can be transmitted by down-scaling the image quality. According to an embodiment, since the image quality is downscaled, the image transmission speed may be maintained even if the size of the capture area is set to be larger than that of the existing UI providing area. When the high-speed movement input is stopped, the processor 120 may transmit the image again with the original image quality.

8 is a flowchart illustrating a method of setting a capture area according to a key input according to various embodiments of the present disclosure.

Operations 810 to 830 may be performed by the processor 120 of the service server 100 described above with reference to FIG. 1 . According to an embodiment, operations 810 to 830 may correspond to the operation of setting the capture area size based on the key input described with reference to FIG. 7 (eg, operation 720 of FIG. 7 ).

Referring to FIG. 8 , the processor 120 may determine whether the key input received from the user terminal 180 is a high-speed movement input in operation 810 . As described with reference to FIGS. 5 and 6 , the operation of determining whether a high-speed movement input is performed may be performed in various embodiments based on a type of a key input, a key input duration, and a key input speed.

According to an embodiment, in operation 820 , when the key input is a high-speed movement input, the processor 120 may set the size of the capture area to be larger than the size of the UI providing area displayed on the user display. For example, when a direction key input is greater than a threshold time (eg, 3 seconds), when a drag is greater than or equal to a threshold speed, and when a scroll movement is greater than or equal to a threshold speed, the processor 120 determines that the received key input moves at a high speed. It may be determined that it is an input, and the capture area including the UI providing area may be set to be larger than the UI providing area based on the direction of the key input, the key input duration threshold time, and the input threshold speed.

According to an embodiment, when the direction key input in the downward direction is longer than a threshold time (eg, 3 seconds), a capture area including the existing UI providing area and extending downward to a size corresponding to the input time may be set. .

According to an embodiment, in operation 830 , if the key input is not a high-speed movement input, the processor 120 may set the size of the capture area to be the same as the size of the UI providing area displayed on the user display. For example, if the direction key input is less than a threshold time (eg, 3 seconds), if the drag is less than the threshold speed, if the scroll movement is less than the threshold speed, the processor 120 moves the received key input at a high speed It may be determined that the input is not input, and the capture area may be set to the same size as the size of the UI providing area of the display of the user terminal 180 .

According to an embodiment, when the downward direction key input is less than a threshold time (eg, 3 seconds), the capture area may be set to the same size as the existing UI providing area size for the web app screen moved in the downward direction. .

<User terminal>

9 is a configuration diagram of a user terminal according to various embodiments of the present disclosure;

Referring to FIG. 9 , a configuration diagram of a user terminal 180 receiving a service from the service server 100 through a network is shown.

According to an embodiment, the user terminal 180 executes a web app and receives the communication module 910 for communication with the service server 100 that provides a web app screen, and the service server 100 It may include a display module 950 that displays an image, a memory 940 in which computer-executable instructions are stored, and a processor 920 that accesses the memory and executes the instructions.

According to an embodiment, the processor 920 may transmit a web app execution request to the service server 100 and receive a web app screen captured by the service server as described in FIG. 1 . have.

According to an embodiment, as described with reference to FIG. 1 , the user terminal 180 refers to a device capable of transmitting and receiving various information with the service server 100 through a network, and a cloud UI client for transmitting and receiving information. ), may further include an input module for receiving information from the user, a decoding module for receiving and decoding encoded streaming data, a rendering module for rendering the decoded data, and the like.

According to an embodiment, when the user terminal 180 is a set-top box of a smart TV, a user input input through a remote control may be received and transmitted to the service server 100 . The user terminal 180 may transmit a request to the service server 100 through a user input, and may receive and reproduce streaming data from the service server 100 in response thereto. The user terminal 180 may be implemented in various forms, for example, a smart phone, a tablet, a personal digital assistant (PDA), a portable multimedia player (PMP), or a smart TV of a smart TV. It may be a set-top box, a desktop computer, a laptop computer, and the like.

According to an embodiment, the processor 920 of the user terminal 180 receives a key input from the user while receiving the web app screen as described with reference to FIG. 4 from the service server 100 to the service server 100 . , and may receive a captured image according to a key input from the service server 100 and display it through the display module 950 . The captured image received from the service server 100 may be captured according to the capture area size set by the processor 120 of the service server 100 based on a key input.

According to an embodiment, the processor 120 of the service server 100 may determine whether the key input is a high-speed movement input using various methods based on the type of key input, the key input duration, and the key input speed. For example, if the direction key input is less than a threshold time (eg, 3 seconds), if the drag is less than the threshold speed, if the scroll movement is less than the threshold speed, the processor 120 moves the received key input at a high speed It can be determined that it is not an input, when the direction key input is more than a threshold time (eg, 3 seconds), when a drag is more than a threshold speed, when a scroll movement is more than a threshold speed, the processor 120 of the service server 100 ) may determine that the received key input is a high-speed movement input. The type of each key input, the input duration threshold time and/or the input threshold speed according to the key input type may be different from each other.

According to an embodiment, as described with reference to FIG. 5 , when the processor 120 of the service server 100 determines that the key input is not a high-speed movement input, the captured image received from the service server 100 is the user terminal ( 180) It may be captured according to the capture area size set to be the same as the size of the UI providing area displayed on the display.

According to an embodiment, as described with reference to FIG. 6 , when the key input is a high-speed movement input, the captured image received from the service server 100 is set larger than the size of the UI providing area displayed on the display of the user terminal 180 . It may be captured according to the size of the capture area. The size of the capture area including the UI providing area may be set large based on the direction of the key input, the duration of the key input, the key input speed, and the like.

According to an embodiment, as described with reference to FIG. 6 , the captured image received by the processor 920 of the user terminal 180 from the service server 100 determines that the key input is a high-speed movement input, so that the capture area size is reduced. If it is set to be larger than the UI providing area, the image quality may be down-scaled and transmitted. Even if the size of the capture area is set to be larger than that of the existing UI providing area, the image quality is downscaled, so that the image reception speed in the user terminal 180 can be maintained.

The embodiments described above may be implemented by a hardware component, a software component, and/or a combination of a hardware component and a software component. For example, the apparatus, methods and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate (FPGA). array), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions, may be implemented using a general purpose computer or special purpose computer. The processing device may execute an operating system (OS) and a software application running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that may include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

The software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or device, to be interpreted by or to provide instructions or data to the processing device. , or may be permanently or temporarily embody in a transmitted signal wave. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored in a computer-readable recording medium.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination, and the program instructions recorded on the medium are specially designed and configured for the embodiment or may be known and available to those skilled in the art of computer software. may be Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

The hardware devices described above may be configured to operate as one or a plurality of software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with reference to the limited drawings, a person skilled in the art may apply various technical modifications and variations based thereon. For example, the described techniques are performed in a different order than the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (12)

In the service server screen providing method,
receiving a key input from the user terminal;
setting a capture area size based on the key input;
capturing an image rendered in a browser based on the size of the capture area; and
Transmitting the captured image to the user terminal
comprising,
How to provide the screen of the service server.
According to claim 1,
The operation of setting the capture area size based on the key input includes:
When the key input is any one of a direction key input less than a threshold time, a drag less than a threshold speed, and a scroll less than the threshold speed,
Setting the capture area size to be the same as the UI (user interface) providing area size displayed on the display of the user terminal
comprising,
How to provide the screen of the service server.
According to claim 1,
The operation of setting the capture area size based on the key input includes:
When the key input is a high-speed movement input that is any one of a direction key input over a threshold time, a drag over a threshold speed, and a scroll over the threshold speed,
setting the size of the capture area to be larger than the size of a user interface (UI) providing area displayed on the display of the user terminal based on at least one of the direction of the key input, the threshold time, and the threshold speed
comprising,
How to provide the screen of the service server.
4. The method of claim 3,
The operation of transmitting the captured image to the user terminal is,
An operation of down-scaling the quality of the captured image according to the size of the capture area while the high-speed movement input is maintained and transmitting it to the user terminal
comprising,
How to provide the screen of the service server.
5. The method of claim 4,
The operation of transmitting the captured image to the user terminal is,
When the high-speed movement input is stopped, the image quality of the captured image is restored to the original image quality and transmitted to the user terminal.
further comprising,
How to provide the screen of the service server.
In the service server,
a browser that runs and renders a web app;
a memory in which computer-executable instructions are stored; and
a processor that accesses the memory and executes the instructions
including,
The processor receives a key input from a user terminal, sets a capture area size based on the key input, captures an image rendered in the browser based on the capture area size, and the capture sending an image to the user terminal,
service server.
7. The method of claim 6,
The processor is
When the key input is any one of a direction key input less than a threshold time, a drag less than a threshold speed, and a scroll less than the threshold speed,
Setting the capture area size to be the same as the UI (user interface) providing area size displayed on the display of the user terminal,
service server.
7. The method of claim 6,
The processor is
When the key input is a high-speed movement input that is any one of a direction key input over a threshold time, a drag over a threshold speed, and a scroll over the threshold speed,
setting the size of the capture area to be larger than the size of a user interface (UI) providing area displayed on the display of the user terminal based on at least one of the direction of the key input, the threshold time, and the threshold speed,
service server.
9. The method of claim 8,
The processor is
While the high-speed movement input is maintained, the image quality of the captured image is down-scaled according to the capture area size and transmitted to the user terminal,
service server.
10. The method of claim 9,
The processor is
When the high-speed movement input is stopped, the image quality of the captured image is restored to the original image quality and transmitted to the user terminal,
service server.
In the user terminal,
a communication module for communicating with the service server;
a display module for displaying the image received from the service server;
a memory in which computer-executable instructions are stored; and
a processor that accesses the memory and executes the instructions
including,
The processor transmits a key input to the service server, receives a captured image according to the key input from the service server,
The captured image is captured according to a capture area size set based on the key input,
user terminal.
A computer program stored in a computer-readable recording medium in combination with hardware to execute the method of any one of claims 1 to 5.

Images