KR20210133229A - User interface session recovery method in cloud streaming service and device therefor - Google Patents

Abstract

A user interface (UI) session recovery method performed by a cloud streaming server according to an embodiment of the present invention is a first user between the cloud streaming server and the client terminal to stream the user interface screen to the client terminal Establishing an interface session, establishing a video session between the cloud streaming server and the client terminal to stream video content to the client terminal, the first user interface in a state of maintaining the video session as a preset session release condition is satisfied The step of releasing the session, and the step of establishing a second user interface session between the cloud streaming server and the client terminal as a preset session recovery condition is satisfied.

Description

User interface session recovery method in cloud streaming service and device therefor

The present invention relates to a technique for recovering a user interface (UI) session in a cloud streaming service, and recovers a user interface session disconnected for reasons such as resource saving of a cloud streaming server based on a specific condition It's about technology.

The rapid development of the Internet has resulted in a rapid increase in personal communication speed, and this improvement in communication speed is like downloading or uploading large amounts of data by accessing a computer located at a remote location, or using a remote computer control program as if it were a remote computer. It provides an environment where you can use a remote computer as if you were logged in locally.

In addition, as applications driven in a mobile communication terminal such as a smart phone are developed in various ways, virtualization technologies for driving applications requiring high performance in a relatively low performance client terminal have been widely proposed.

Among them, a screen virtualization-based system that runs the application on the server, compresses the driving screen through video encoding, and transmits it to the client, and the client plays the transmitted video to have the same effect as if the application is running on its own terminal of cloud streaming services are in the spotlight.

The cloud streaming service basically applied the video codec-based cloud streaming technique. However, even in the case of servicing a static screen such as a menu display, the entire screen is unnecessarily captured and the video codec is used to operate, which is inefficient as a whole system.

Therefore, it is possible to capture only the changed image of the static part on the execution screen of the application executed according to the user's request and provide it to the user's terminal, and disconnect the browser when the connected client terminal is not actually using the application. An image-based cloud streaming service technology that can improve the simultaneous access rate of the cloud streaming service system by allowing other terminals to use it is urgently needed.

It is an object of the present invention to release the session established between the cloud streaming server and the client terminal according to a specific condition and to use the idle resources of the cloud streaming server more efficiently through the operation to restore the concurrent access rate of the cloud streaming service system is to improve

It is also an object of the present invention, by establishing a plurality of sessions divided by function and display area for one client terminal, and independently releasing and restoring these sessions, to build a more efficient cloud streaming service system.

Another object of the present invention is to provide an application execution screen that minimizes the sense of heterogeneity due to session release by restoring the user interface screen provided through the client terminal before the session release in the process of recovering the released user interface session.

Another object of the present invention is to provide a high-spec application execution screen even in a low-spec terminal by processing only the changed image on the application execution screen and providing it to the client terminal.

A user interface (UI) session recovery method performed by a cloud streaming server according to an embodiment of the present invention for achieving the above object is a cloud streaming server to stream a user interface screen to a client terminal and establishing a first user interface session between the client terminals, establishing a video session between the cloud streaming server and the client terminal to stream video content to the client terminal, and maintaining the video session as a preset session release condition is met It may include the steps of releasing the first user interface session in a state that is, and establishing a second user interface session between the cloud streaming server and the client terminal as a preset session recovery condition is satisfied.

The step of releasing the first user interface session includes generating session recovery data including backup data of the user interface screen being streamed to the client terminal through the first user interface session, and transmitting the session recovery data to the client terminal; , releasing the first user interface session, wherein the establishing of the second user interface session includes receiving session recovery data from the client terminal, and using the session recovery data to provide the client terminal through the second user interface session. and restoring the user interface screen to be streamed.

The session release condition may be satisfied when a preset time elapses from the reception time of the most recently received input signal from the client terminal.

The session release condition may be satisfied by receiving a session release request from the client terminal.

The session release request may be generated by the client terminal as a preset time elapses from the input time of the most recently inputted user input on the client terminal.

The session recovery condition may be satisfied by receiving an input signal from the client terminal.

The session recovery condition may be satisfied upon receiving a session recovery request from the client terminal.

The session recovery request may be generated by the client terminal as the first video content being streamed to the client terminal through the video session is switched to the second video content.

The session recovery condition may be satisfied when the first video content streaming to the client terminal through the video session is switched to the second video content.

The first area of the display screen of the client terminal is an area where the user interface screen is streamed through the first user interface session or the second user interface session, and the second area of the display screen of the client terminal is where video content is streamed through the video session. It may be an area where

Cloud streaming server according to an embodiment of the present invention for achieving the above object includes at least one program is recorded memory and a processor for executing the program, the program is a cloud to stream the user interface screen to the client terminal Establishing a first user interface session between the streaming server and the client terminal, establishing a video session between the cloud streaming server and the client terminal to stream video content to the client terminal, the video session as a preset session release condition is met may include instructions for performing the steps of releasing the first user interface session in a state of maintaining and establishing a second user interface session between the cloud streaming server and the client terminal as a preset session recovery condition is satisfied .

The step of releasing the first user interface session includes generating session recovery data including backup data of the user interface screen being streamed to the client terminal through the first user interface session, and transmitting the session recovery data to the client terminal; , releasing the first user interface session, wherein the establishing of the second user interface session includes receiving session recovery data from the client terminal, and using the session recovery data to provide the client terminal through the second user interface session. and restoring the user interface screen to be streamed.

The session release condition may be satisfied when a preset time elapses from the reception time of the most recently received input signal from the client terminal.

The session release condition may be satisfied by receiving a session release request from the client terminal.

The session release request may be generated by the client terminal as a preset time elapses from the input time of the most recently inputted user input on the client terminal.

The session recovery condition may be satisfied by receiving an input signal from the client terminal.

The session recovery condition may be satisfied upon receiving a session recovery request from the client terminal.

The session recovery request may be generated by the client terminal as the first video content being streamed to the client terminal through the video session is switched to the second video content.

The session recovery condition may be satisfied when the first video content streaming to the client terminal through the video session is switched to the second video content.

The first area of the display screen of the client terminal is an area where the user interface screen is streamed through the first user interface session or the second user interface session, and the second area of the display screen of the client terminal is where video content is streamed through the video session. It may be an area where

A user interface (UI) screen recovery method performed by a cloud streaming server according to an embodiment of the present invention for achieving the above object is a cloud streaming server and a client terminal to stream a user interface screen to a client terminal establishing a second user interface session between the client terminals, receiving session recovery data corresponding to the first user interface session established between the cloud streaming server and the client terminal before establishment of the second user interface session from the client terminal, and session recovery and restoring the user interface screen to be streamed to the client terminal through the second user interface session using the data.

Before establishing the second user interface session, establishing a first user interface session between the cloud streaming server and the client terminal to stream the user interface screen to the client terminal, the cloud streaming server to stream the video content to the client terminal and establishing a video session between client terminals, and releasing the first user interface session while maintaining the video session.

Before releasing the first user interface session, generating session recovery data including backup data of a user interface screen being streamed to the client terminal through the first user interface session, and session recovery data to the client terminal It may further include the step of transmitting.

The method may further include, before the step of releasing the first user interface session, replacing the user interface screen with a still screen.

The user interface screen corresponds to the execution screen of the application executed on the cloud streaming server, and the step of replacing the user interface screen with a still screen is a step of replacing the execution screen of the application with a still screen, and the client through a first user interface session It may include streaming the still screen as a user interface screen to the terminal.

Replacing the execution screen of the application with a still screen may include replacing at least one animation area included on the execution screen of the application with a still image.

The still image may be an image preset for at least one animation area.

The still image may be an image obtained by capturing at least one animation region.

The user interface screen streamed through the first user interface session corresponds to the execution screen of the first application executed on the cloud streaming server, and the user interface screen streamed through the second user interface session is a second screen executed on the cloud streaming server. Corresponding to the execution screen of the application, the backup data is data in which parameter values indicating the execution state of the first application are stored as a reference point, and the step of restoring the user interface screen is when the execution state of the second application is at a point in time. The method may include executing the second application using the backup data to correspond to the execution state of the first application.

The first area of the display screen of the client terminal is an area where the user interface screen is streamed through the first user interface session or the second user interface session, and the second area of the display screen of the client terminal is where video content is streamed through the video session. It may be an area where

Cloud streaming server according to an embodiment of the present invention for achieving the above object includes at least one program recorded memory, and a processor for executing the program, the program is to stream the user interface screen to the client terminal Establishing a second user interface session between the cloud streaming server and the client terminal, receiving session recovery data corresponding to the first user interface session established between the cloud streaming server and the client terminal before the establishment of the second user interface session from the client terminal and restoring the user interface screen to be streamed to the client terminal through the second user interface session using the session recovery data.

Before establishing the second user interface session, establishing a first user interface session between the cloud streaming server and the client terminal to stream the user interface screen to the client terminal, the cloud streaming server to stream the video content to the client terminal and instructions for performing the steps of establishing a video session between client terminals and releasing the first user interface session while maintaining the video session.

Before releasing the first user interface session, generating session recovery data including backup data of a user interface screen being streamed to the client terminal through the first user interface session, and session recovery data to the client terminal It may further include instructions for performing the step of transmitting.

The method may further include instructions for performing the step of replacing the user interface screen with a still screen before the step of releasing the first user interface session.

The user interface screen corresponds to the execution screen of the application executed on the cloud streaming server, and the step of replacing the user interface screen with a still screen is a step of replacing the execution screen of the application with a still screen, and the client through a first user interface session It may include streaming the still screen as a user interface screen to the terminal.

Replacing the execution screen of the application with a still screen may include replacing at least one animation area included on the execution screen of the application with a still image.

The still image may be an image preset for at least one animation area.

The still image may be an image obtained by capturing at least one animation region.

The user interface screen streamed through the first user interface session corresponds to the execution screen of the first application executed on the cloud streaming server, and the user interface screen streamed through the second user interface session is a second screen executed on the cloud streaming server. Corresponding to the execution screen of the application, the backup data is data in which parameter values indicating the execution state of the first application are stored as a reference point, and the step of restoring the user interface screen is when the execution state of the second application is at a point in time. The method may include executing the second application using the backup data to correspond to the execution state of the first application.

The first area of the display screen of the client terminal is an area where the user interface screen is streamed through the first user interface session or the second user interface session, and the second area of the display screen of the client terminal is where video content is streamed through the video session. It may be an area where

According to the present invention, the concurrent access rate of the cloud streaming service system is improved by enabling the idle resources of the cloud streaming server to be used more efficiently through the operation of releasing and restoring the session established between the cloud streaming server and the client terminal according to specific conditions. can be improved

In addition, according to the present invention, by establishing a plurality of sessions divided by function and display area for one client terminal and releasing and restoring these sessions independently, a more efficient cloud streaming service system can be built.

Also, according to the present invention, in the process of recovering the released user interface session, by restoring the user interface screen provided through the client terminal before the session release, it is possible to provide an application execution screen that minimizes the sense of heterogeneity caused by the session release.

Also, according to the present invention, by processing only the changed image on the application execution screen and providing it to the client terminal, it is possible to provide a high-spec application execution screen even in a low-spec terminal.

1 is a block diagram showing a cloud streaming service system according to an embodiment of the present invention.
2 is a view showing an example of the cloud streaming server shown in FIG.
3 is a diagram illustrating an example of a session connection method between a cloud streaming server and a client terminal.
4 is a diagram illustrating an example of a session recovery method after releasing a user interface session.
5 is a diagram illustrating an example of a method of providing a cloud streaming service using a web browser and a web server.
6 is a diagram illustrating an example of a method of capturing an application execution screen.
7 is a flowchart illustrating a user interface session recovery method according to an embodiment of the present invention.
8 is an operation flowchart illustrating a method for restoring a user interface screen according to an embodiment of the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, detailed descriptions of well-known functions or configurations that may obscure the gist of the present invention in the following description and accompanying drawings will be omitted. Also, it should be noted that throughout the drawings, the same components are denoted by the same reference numerals as much as possible. The embodiments of the present invention are provided in order to more completely explain the present invention to those of ordinary skill in the art. Accordingly, the shapes and sizes of elements in the drawings may be exaggerated for clearer description.

The terms or words used in the present specification and claims described below should not be construed as being limited to their ordinary or dictionary meanings, and the inventor is appropriate as a concept of terms for describing his invention in the best way. It should be interpreted as meaning and concept consistent with the technical idea of the present invention based on the principle that it can be defined in Accordingly, the embodiments described in this specification and the configurations shown in the drawings are only the most preferred embodiment of the present invention, and do not represent all of the technical spirit of the present invention. It should be understood that there may be equivalents and variations. In addition, terms such as first, second, etc. are used to describe various components, and are used only for the purpose of distinguishing one component from other components, and are not used to limit the components.

1 is a block diagram showing a cloud streaming service system according to an embodiment of the present invention.

Referring to FIG. 1 , the cloud streaming service system according to an embodiment of the present invention includes a cloud streaming server 110 , client terminals 120-1 to 120-N, and a communication network 130 .

The cloud streaming server 110 may receive an application execution request from the client terminals 120-1 to 120-N, and execute the application.

The application may be a native application, a web application, or a hybrid application corresponding to a specific operating system, but the application type of the present invention is not limited to the above example.

The cloud streaming server 110 may establish sessions for providing a cloud streaming service with the client terminals 120-1 to 120-N, and execute a plurality of applications as many as the number of sessions.

The cloud streaming server 110 may provide the execution screen of each application to the client terminals 120-1 to 120-N through a pipeline process corresponding to rendering, capturing, still image encoding, and sending.

In this case, the execution screen of each application may include a video playback area in which video content is provided and a user interface area in which a user interface (UI) for a user to operate the application is displayed.

The cloud streaming server 110 may determine whether a session established for any one client terminal corresponds to a release target, and if the determination result indicates that the session is a release target, the session may be released.

The cloud streaming server 110 may establish a plurality of sessions with respect to the client terminal in order to divide the screen to be displayed in any one client terminal into a plurality of regions for streaming.

For example, the cloud streaming server 110 performs a first session for performing streaming for a first area in which a user interface screen is displayed in the client terminal, and streaming for a second area in which video content is displayed in the client terminal. A second session may be established for each.

That is, the cloud streaming server 110 may establish a plurality of sessions divided by function or divided by display area for one client terminal.

Cloud streaming server 110 may independently establish or release each of a plurality of sessions for one client terminal.

For example, the cloud streaming server 110 may release the user interface session while maintaining the video content session established for the client terminal.

Cloud streaming server 110 may receive an input signal corresponding to an input for operating the application of the user corresponding to the client terminal from any one client terminal.

In addition, the cloud streaming server 110 is the identification information of the client terminals (120-1 ~ 120-N) in the case of releasing the session established between the client terminals (120-1 ~ 120-N), the current screen of the application Session recovery data corresponding to information and parameter values indicating an execution state of an application may be stored.

In addition, the cloud streaming server 110 checks the idle resource, and dynamically releases the session established with the client terminals 120-1 to 120-N in consideration of the idle resource, and the session is released client terminals 120 -1 to 120-N) and session can be re-established.

The client terminals 120-1 to 120-N receive the application execution screen corresponding to the cloud streaming service from the cloud streaming server 110 and provide it to the user.

In addition, each of the client terminals 120-1 to 120-N is a device that is connected to the communication network 130 to execute an application based on a cloud computing system, and is not limited to a mobile communication terminal, and is not limited to all information communication devices, multimedia It may be various terminals such as a terminal, a wired terminal, a fixed terminal, and an Internet Protocol (IP) terminal. In addition, the client terminals 120-1 to 120-N are, respectively, a mobile phone, a Portable Multimedia Played (PMP), a Mobile Internet Device (MID), a smart phone, a desktop, and a tablet computer. , a notebook (Notebook), a netbook (Net Book), personal portable information client terminals (PDA), a smart TV, and may be a mobile terminal having various mobile communication specifications, such as an information communication device.

In addition, the client terminals 120-1 to 120-N receive various information, such as number and character information, and set various functions and control functions of the client terminals 120-1 to 120-N. An input signal may be transmitted to the control unit through the input unit. In addition, the input units of the client terminals 120-1 to 120-N may include at least one of a keypad and a touchpad for generating an input signal according to a user's touch or manipulation. At this time, the input unit of the client terminals 120-1 to 120-N is in the form of one touch panel (or touch screen) together with the display unit of the client terminals 120-1 to 120-N. It can be configured to perform input and display functions at the same time. In addition, input units of the client terminals 120-1 to 120-N may use any type of input means that may be developed in the future in addition to input devices such as a keyboard, a keypad, a mouse, and a joystick. In particular, the input unit of the client terminals 120-1 to 120-N according to the present invention sends an input signal for uploading or downloading content based on cloud computing to the control unit of the client terminals 120-1 to 120-N. can transmit

In addition, the display unit of the client terminals 120-1 to 120-N may display information about a series of operation states and operation results, etc. occurring while the function of the client terminals 120-1 to 120-N is performed. . Also, the display units of the client terminals 120-1 to 120-N may display menus of the client terminals 120-1 to 120-N and user data input by the user. Here, the display units of the client terminals 120-1 to 120-N include a liquid crystal display (LCD), an ultra-thin liquid crystal display (TFT-LCD, Thin Film Transistor LCD), and a light emitting diode (LED, Light Emitting). Diode), organic light emitting diode (OLED, organic LED), active organic light emitting diode (AMOLED, Active Matrix OLED), retina display, flexible display and 3 dimensional display can be At this time, when the display units of the client terminals 120-1 to 120-N are configured in the form of a touch screen, the display units of the client terminals 120-1 to 120-N are the client terminals 120-1 to 120-N. Some or all of the functions of the input unit of N) may be performed. In particular, the display units of the client terminals 120-1 to 120-N according to the present invention may display information related to execution of content provided based on cloud computing on a screen.

In addition, the storage unit of the client terminals 120-1 to 120-N is a device for storing data, and includes a main memory and an auxiliary memory, and functions of the client terminals 120-1 to 120-N. Applications required for operation can be stored. The storage units of these client terminals 120-1 to 120-N may largely include a program area and a data area. Here, when each function is activated in response to a user's request, the client terminals 120-1 to 120-N execute corresponding application programs under the control of the controller to provide each function. In particular, the storage unit of the client terminals 120-1 to 120-N according to the present invention is an operating system for booting the client terminals 120-1 to 120-N, a program for uploading or downloading content based on cloud computing. etc. can be saved. Also, the storage unit of the client terminals 120-1 to 120-N may store a content DB for storing a plurality of contents and information of the client terminals 120-1 to 120-N. In this case, the content DB includes execution data for executing the content and attribute information on the content, and content usage information according to the content execution may be stored. And, the information of the client terminals 120-1 to 120-N may include terminal specification information.

In addition, the communication unit of the client terminals 120-1 to 120-N may perform a function for transmitting and receiving data through the cloud streaming server 110 and the communication network 130 . Here, the communication unit of the client terminals 120-1 to 120-N may include an RF transmitting means for up-converting and amplifying the frequency of the transmitted signal, and an RF receiving means for low-noise amplifying the received signal and down-converting the frequency. can The communication units of the client terminals 120-1 to 120-N may include at least one of a wireless communication module and a wired communication module. In addition, the wireless communication module is a configuration for transmitting and receiving data according to a wireless communication method, and when the client terminals 120-1 to 120-N use wireless communication, a wireless network communication module, a wireless LAN communication module, and a wireless fan Data may be transmitted/received to/from the cloud streaming server 110 using any one of the communication modules. In addition, the wired communication module is for transmitting and receiving data by wire. The wired communication module may be connected to the communication network 130 through a wire to transmit and receive data to the cloud streaming server 110 . That is, the client terminals 120-1 to 120-N may be connected to the communication network 130 using a wireless communication module or a wired communication module to transmit/receive data to and from the cloud streaming server 110 . In particular, the communication network 130 according to the present invention communicates with the cloud streaming server 110 or other client terminals 120-1 to 120-N to transmit and receive data required to upload or download content based on cloud computing. have.

In addition, the control unit of each of the client terminals 120-1 to 120-N may be an operating system (OS) and a process device for driving each component. For example, the controller may control the overall process of accessing the cloud streaming server 110 . When accessing the cloud streaming server 110 through a separate service application, it is possible to control the overall process of executing the service application according to the user's request, and the service use request is transmitted to the cloud streaming server 110 at the same time as execution. In this case, it is possible to control so that information of the client terminals 120-1 to 120-N required for user authentication is transmitted together.

In addition, the control unit of each of the client terminals 120-1 to 120-N may execute specific content stored in the storage unit of each of the client terminals 120-1 to 120-N according to a user's request. In this case, the controller may store the content use history according to the content execution as content use information.

In addition, the control unit of each of the client terminals 120-1 to 120-N receives execution data for executing the content, content information including attribute information on the content, and content use information, which is information according to the content use history. It can be transmitted and uploaded to the cloud streaming server 110 together. Thereafter, the control unit can be deleted from the storage unit of each of the client terminals 120-1 to 120-N according to the user's request for the uploaded content transmitted to the cloud streaming server 110, and the cloud streaming server 110 It is also possible to access and run the content through the cloud streaming server 110 to use.

In addition, the control unit of each of the client terminals 120-1 to 120-N may access the cloud streaming server 110 to download content from other client terminals and then control to be stored in the storage unit, and the cloud streaming server When executing content through 110 , it is also possible to control the content to be executed after receiving only data necessary for execution.

The communication network 130 provides a path for transferring data between the cloud streaming server 110 and the client terminals 120-1 to 120-N, and encompasses both existing networks and networks that can be developed in the future. Concept. For example, the communication network 130 is a wired/wireless local area network that provides communication of various information devices within a limited area, a mobile communication network that provides communication between mobile devices and between the mobile body and the outside of the mobile body, and communication between the earth station and the earth station using satellites. It may be a satellite communication network providing Meanwhile, the transmission method standard in the communication network 130 is not limited to the existing transmission method standard, and may include all transmission method standards to be developed in the future. In addition, the network used between the cloud streaming server 110 and the client terminals 120-1 to 120-N in FIG. 1 is different from the network used between the client terminals 120-1 to 120-N. may be or may be the same.

2 is a view showing an example of the cloud streaming server 110 shown in FIG.

Referring to FIG. 2 , the cloud streaming server 110 shown in FIG. 1 includes a terminal input receiving unit 210 , an application executing unit 220 , a streaming unit 230 , a video content session control unit 240 , and a user interface session control unit. 250 , a session recovery data storage unit 260 may be included.

The terminal input receiving unit 210 receives a key input from the client terminal.

In this case, the key input may be generated through an input of a menu button included in a user interface screen displayed through streaming in the client terminal or a button generated to perform a specific command.

For example, when a user receives shopping information through a client terminal and clicks a category information button to view information for each category of shopping items, a key input corresponding to providing category information may occur, which is transmitted to the cloud streaming server It can be received at (110).

The application execution unit 220 executes a plurality of applications by utilizing the idle resources of the cloud streaming server 110 . Here, the execution screen of each application may be streamed to each of the client terminals.

In this case, the application may correspond to a web browser application that allows all information to be viewed on the World Wide Web. A web browser may be a client program that uses HTTP to request data from a web server. In addition, the web browser may be a tool that assists in retrieving a hypertext document, rather than simply displaying the content of the document.

The basic function of such a web browser is to read and provide HTML, and it can also provide a function to freely transfer the read information to the initial screen of the back page of the front page. In addition, a bookmark and bookmark function for quickly navigating to a required page among various sites may be provided.

In this case, the application may be an application including a video as one element displayed in one area and a user interface area in the other area. That is, the application may be an application that provides a complex screen including a video, rather than an application that simply executes a video.

The streaming unit 230 provides the application execution screen to the client terminal through a pipeline process corresponding to rendering, capturing, still image encoding, and sending.

The video codec-based cloud streaming technique can perform encoding by capturing all frames corresponding to the screen on which the application operates. However, if the frame-to-frame change is not large on the screen on which the application is running, still image encoding is performed by capturing only the area of change of the frame that has changed compared to the previous frame, as in the image-based cloud streaming technique, and displayed on the user's terminal device. The display can be displayed in such a way that the parts except the change area are shown the same and only the change area is changed.

In this case, the application execution screen may be rendered, and an image may be captured from the rendered application execution screen. For example, by comparing a plurality of frames corresponding to the application execution screen, respectively, a changed portion between the previous frame and the current frame may be captured as an image.

In this case, it is possible to detect sections with little change between any two frames among the frames of the application execution screen, and capture an image in a section with little change between frames. For example, it may be inefficient to provide an image-based cloud streaming service because the number of images to be captured is large in a section where there is a lot of change between frames. Therefore, in such a section with many changes, the service is performed through the video codec-based cloud streaming technique, and the image of the changed capture area can be captured to perform the image-based cloud streaming service only in the section with little change between frames. .

In this case, the still image encoding method may be determined in consideration of the load generated by the cloud streaming server 110, service speed, image quality, or performance of a terminal that receives and renders the encoded image. For example, in order to improve the speed of a cloud streaming service, an image may be compressed using a palette PNG encoding method that has a high compression rate and a low load. In addition, if you want to provide a good image quality regardless of the load, you can compress the image using a PNG32bit encoding method or a JPEG encoding method.

The video content session controller 240 may establish a video content session with the client terminal to stream the video content to the client terminal, or release an already established video content session.

The user interface session controller 250 may establish a user interface session with the client terminal to stream the user interface screen to the client terminal, or release an already established user interface session.

The user interface session controller 250 may independently release only the user interface session while maintaining the video content session established with the specific client terminal.

The user interface session controller 250 may determine whether the user interface session is a release target for a specific client terminal, and release the user interface session for the client terminal that is the release target.

For example, in the case of multi-cloud streaming that simultaneously services a shopping product introduction video screen and a web page screen in a client terminal, the cloud streaming server 110 provides a video session and a web page for playing a shopping product introduction video to the client terminal. A user interface session for providing a (eg, user interface) screen may be established, respectively. However, even if the client terminal simultaneously displays the web page screen while playing the video content through two sessions, when the user is watching the video content, it does not pay attention to the web page screen and does not perform any operation other than the video player. may not be That is, even in such a case, maintaining both a video session and a user interface session for one user may correspond to a waste of resources from the point of view of the cloud streaming server 110 that provides a service with limited resources.

For example, in the case of driving a video player and a web page as an HTML5 application in the cloud streaming server 110, the cloud streaming server 110 may provide an application by executing a browser for each of the video player and the web page. . In this case, if the total number of browser instances that can be provided by the cloud streaming server 110 is limited to 100, there is no choice but to provide the service to up to 50 people.

However, when the connection of the browser allocated to the client terminal is released for an application that is not actually used by a user, the maximum number of service users may further increase. For example, it may be assumed that the web page application is not used for 5 minutes in the client terminal using the video player and the web page application at the same time. That is, when no input to the web page occurs for 5 minutes, the cloud streaming server 110 may release the connection of the browser allocated for the web page application so that it can be used in other client terminals.

In this case, when a key input different from the key input is not received from the client terminal for a preset input waiting time after the key input is received, any one of the browsers may be determined as the release target browser. That is, the criterion for disconnecting the connection between the client terminal and the browser executing the application is set to a period in which a key input corresponding to the application does not occur. It can determine the target browser and disconnect it.

The user interface session controller 250 may re-establish the user interface session when a specific condition is satisfied for the client terminal from which only the user interface session has been released. That is, even if the user interface session is canceled because a key input to the application has not been received for a certain period of time, the user of the terminal may be in a state where the application is not terminated without actually entering the key, and the video session is maintained. can Therefore, if the existing user interface session is released when a key input corresponding to the application is received again from the client terminal, the cloud streaming server 110 establishes a new user interface session for the client terminal using the currently idle resource. can do. In this case, the session recovery operation may be performed so that the operating state of the user interface provided to the client terminal through the existing session can be recovered.

The user interface session controller 250 may determine that the user interface session release condition is satisfied when an additional input following the recent input has not occurred for a preset time.

Here, the preset time may be dynamically set according to the amount of idle resources of the cloud streaming server 110 .

For example, when a video player and a web page are driven by an HTML5 application in the cloud streaming server 110, the cloud streaming server 110 creates browser instances and controls the application to be executed in the idle browser instance. .

In this case, the application may be executed with the idle browser, and the next application execution screen of the application execution screen may be generated using at least one of another key input and session recovery data.

For example, suppose that the client terminal was viewing the screen corresponding to the first page through the web page application, and the connection with the browser A running the application was disconnected because no key input was received during the preset input waiting time corresponding to 5 minutes. can do. If a specific menu is clicked on the first page from the client terminal at 7 minutes, the cloud streaming server 110 may have to connect the idle browser B and the client terminal to provide a screen for the specific menu.

The user interface session controller 250 checks the number of idle browsers among the plurality of browsers, and adjusts a preset input waiting time in consideration of the number of idle browsers. For example, it may be assumed that the total number of browser instances of the cloud streaming server 110 is 100 and the number of browsers currently in use is 10. In this case, since the number of idle browsers is 90 and there is room, if possible, the preset input waiting time can be set long so that the disconnection and reconnection of the browser do not occur frequently. That is, it is possible to adjust the preset input waiting time so as to save the resources used to perform the operation of releasing and reconnecting the browser in the cloud streaming server 110 .

However, if the total number of browser instances of the cloud streaming server 110 is 100, and it is assumed that all browsers are currently used, users may not be able to access the cloud streaming service even if they want to use it. Accordingly, in this case, by setting the preset input waiting time to be shorter, users who are already receiving the service can disconnect the browser that is not actually used more quickly and provide it to other users.

In addition, in the case where the preset input waiting time is set to be short because there are many users of the service as described above, when the browser is disconnected, a related guide may be provided to the user to inform the users that the current server usage is high.

When the session recovery data storage unit 260 releases a user interface session established with a specific client terminal, the session corresponding to at least one of identification information of the corresponding client terminal, current screen information of the application, and URL information corresponding to the application You can save the recovery data.

For example, in the cloud streaming server 110, before releasing the user interface session, the ID of the client terminal that can identify the client terminal or the application screen and application URL provided by the client terminal are stored as session recovery data, and the session recovery data is stored. It can be used to restore the user interface screen at the time of recovery.

At this time, the session recovery data may be provided to the client terminal and stored in the storage of the client terminal, and the cloud streaming server 110 receives the session recovery data together with another key input when another key input is received from the client terminal. can do.

Although not shown in Figure 2, the cloud streaming server 110 may further include a storage unit (not shown) for storing various information generated in the process of the cloud streaming service according to an embodiment of the present invention.

According to an embodiment, the storage unit may be configured independently of the cloud streaming server 110 to support a function for a cloud streaming service. In this case, the storage unit may operate as a separate mass storage and may include a control function for performing the operation.

In addition, the cloud streaming server 110 configured as described above may be implemented as one or more servers.

On the other hand, the cloud streaming server 110 is equipped with a memory can store information in the device. For one implementation, the memory is a computer-readable medium. In one implementation, the memory may be a volatile memory unit, and in another implementation, the memory may be a non-volatile memory unit. In one embodiment, the storage device is a computer-readable medium. In various different implementations, the storage device may include, for example, a hard disk device, an optical disk device, or some other mass storage device.

Through this, it is possible to improve the simultaneous access rate in the cloud streaming server having limited resources.

3 is a view showing an example of a session connection method between the cloud streaming server 310 and the client terminal 320.

Referring to FIG. 3 , it can be seen that a first session is established between the video content providing server 311 and the client terminal 320 , and a second session is established between the user interface providing server 312 and the client terminal 320 . .

The first session is a video session for the video content providing server 311 to stream the video content data so that the video content is displayed in the video content display area 321 , and the second session is the user interface providing server 312 using the user interface It may be a user interface session for providing user interface data to the user interface renderer 313 to display the user interface screen in the display area 322 and streaming the user interface screen rendered by the user interface renderer 313 .

Although the cloud streaming server 310 is shown as being session-connected with one client terminal 320 in FIG. 3 , the cloud streaming server 310 may be simultaneously session-connected with a plurality of client terminals.

At this time, the cloud streaming server 310 may run a plurality of applications, and each time the client terminals are session-connected, the applications may be connected to the client terminals one by one.

Also, a session once connected may be continuously maintained until the session connection is released.

Each of the video content providing server 311 and the user interface providing server 312 may be one of a plurality of servers included in the cloud streaming server 310 . A plurality of servers included in the cloud streaming server 310 may be physically or logically distinguished from each other.

In this case, the video content providing server 311 may establish a video session with the client terminal 320 in response to the control signal of the cloud streaming server 310 .

As an optional embodiment, the video content providing server 311 may be a server that is physically distinct from the cloud streaming server 310 as a server corresponding to a separate video content provider.

At this time, the video content providing server 311 establishes a video session with the client terminal 320 in response to the control signal of the cloud streaming server 310, or does not receive a separate control signal from the cloud streaming server 310. It is possible to independently establish a video session with the client terminal 320 without it. FIG. 4 is a diagram illustrating an example of a session recovery method after the user interface session is released.

Referring to FIG. 4 , first, according to a service provision request of the client terminal 420 , the cloud streaming server may establish a user interface session with the client terminal 420 .

At this time, the cloud streaming server may execute the application 411 corresponding to the client terminal 420 , and connect the application 411 to the client terminal 420 .

The user interface renderer 410 may render an execution screen of the application 411 and stream the rendered execution screen to the client terminal 420 connected to the application 411 .

That is, the client terminal 420 may receive streaming the execution screen of the application 411 as a user interface screen.

At this time, if an additional key input is not received until the preset input waiting time has expired based on the last key input time received from the client terminal 420, the cloud streaming server is established with the client terminal 420. The user interface session may be released, and the connection with the application 411 may also be released.

Thereafter, the application 411 may be connected to another client 421 making a service provision request to the cloud streaming server.

At this time, if a key input occurs in the disconnected client terminal 420, a new application 412 is executed on the cloud streaming server to provide an application execution screen corresponding to the key input of the client terminal 420, It can connect to the client terminal 420 .

Thereafter, the client terminal 420 may receive streaming the execution screen of the application 412 as a user interface screen.

5 is a diagram illustrating an example of a method of providing a cloud streaming service using a web browser 511 and a web server 530 .

Referring to FIG. 5 , in the cloud streaming service process according to an embodiment of the present invention, as a key input is first generated in the client terminal 520 , the web server 530 obtains a key input through the cloud streaming server 510 . can do.

At this time, the cloud streaming server 510 may execute an application corresponding to the request of the client terminal 520 in the web server 530 through the web browser 511 .

At this time, the cloud streaming server 510 may receive a request from the client terminal 520 and provide HTML or an object to the client terminal 520 using the HTTP protocol.

Accordingly, the cloud streaming server 510 that has obtained the key input of the client terminal 520 may execute an application through the web browser 511 to obtain a result corresponding to the key input.

In addition, according to the present invention, the cloud streaming server 510 may release the connection between the client terminal 520 and the web browser 511 when a key input is not received for a preset input waiting time from the client terminal 520 . have. In addition, the web browser 511 disconnected from the client terminal 520 may be connected to another terminal requesting execution of the application corresponding to the web server 530 to execute the application.

At this time, the web browser 511 may generate an application execution screen that is a result corresponding to the key input by accessing the web server 530 , and transmit the generated application execution screen through the cloud streaming server 510 to the client terminal ( 520) to perform a cloud streaming service.

In this case, the cloud streaming server 510 may capture and encode still images of only an image of a changed area in the received application execution screen to provide an image-based cloud streaming service, and transmit it to the client terminal 520 .

6 is a diagram illustrating an example of a method of capturing an application execution screen.

Referring to FIG. 6 , in the process of capturing an application execution screen according to an embodiment of the present invention, frames of a portion of the application execution screen executed according to a user's request with a small change in the screen may be acquired first.

In this case, it may be more efficient to provide an application execution screen to the user through a video codec-based cloud streaming service for a portion with a large number of screen changes.

Thereafter, the previous frame 610 and the current frame 620 may be compared among the frames of the small change portion. For example, if the previous frame 610 and the current frame 620 shown in FIG. 6 are compared, there is no change in the regions C and D of the two frames, but regions A and B of the previous frame 610 are currently In the frame 620, it can be seen that A' and B' are changed.

Accordingly, the cloud streaming server may check such a change, determine the changed A' area and B' area as the changed area, and capture it as a capture image 630 . That is, it is possible to capture an image corresponding to the changed area at the capture end of the cloud streaming server.

In addition, when the previous frame 610 and the current frame 620 are compared, if multiple regions are identified, one region including all of the changed regions may be captured as the capture image 630 . In addition, it is also possible to perform an image-based cloud streaming service by recognizing all of the changed regions as the captured images 630 , respectively, and capturing the plurality of captured images 630 .

7 is a flowchart illustrating a user interface session recovery method according to an embodiment of the present invention.

Referring to FIG. 7 , first, the cloud streaming server may establish a first user interface session between the cloud streaming server and the client terminal for streaming the user interface screen to the client terminal (S710).

Next, the cloud streaming server may establish a video session between the cloud streaming server and the client terminal to stream the video content to the client terminal (S720).

A first area of the display screen of the client terminal is an area in which a user interface screen is streamed through a first user interface session or a second user interface session to be described later, and a second area of the display screen of the client terminal is video content through a video session may be a streaming area.

In most cases, the user interface screen streamed through the first UI session or the second user interface session does not change significantly between frames on the screen on which the application is operated, and therefore, the first UI session or the second UI session An image-based cloud streaming technique may be used for streaming through .

According to the image-based cloud streaming technique, still image encoding is performed by comparing the current frame with the previous frame, capturing only the changed area of the changed frame, and displaying the same part except for the changed area when displaying on the user's terminal device. Streaming may be performed in such a way that only the change area changes.

On the other hand, the screen of video content streamed through the video session has a large frame-to-frame change in most cases, and therefore, the above-described image-based cloud streaming technique may not be used for streaming through the video session.

Although FIG. 7 shows that the video session establishment step (S720) is performed after the first user interface session establishment step (S710), the first user interface session establishment step (S720) is followed by the video session establishment step (S720). Step S710 may be performed.

Next, the cloud streaming server may release the first user interface session while maintaining the video session as a preset session release condition is satisfied ( S730 ).

At this time, the cloud streaming server generates session recovery data including backup data of the user interface screen being streamed to the client terminal through the first user interface session, and transmits the session recovery data to the client terminal, and the first A user interface session can be released.

The session release condition may be a first session release condition that is satisfied when a preset time elapses from a reception time of an input signal most recently received from the client terminal.

In this case, the input signal may be a signal generated in the client terminal when the user moves the cursor to a specific button or performs an operation to press the button on the user interface displayed on the client terminal.

For example, it may be assumed that a user is streaming an application screen providing shopping information through a terminal. At this time, when the user performs an operation of clicking a category information button to view information for each shopping product category among several menu buttons included in the user interface of the corresponding application, the client terminal inputs a key corresponding to providing the corresponding category information It is possible to detect this occurrence, generate an input signal corresponding thereto, and transmit it to the cloud streaming server.

The session release condition may be a second session release condition that is satisfied upon receiving a session release request from the client terminal.

In this case, the session release request may be generated by the client terminal as a preset time elapses from the input time of the most recently inputted user input on the client terminal.

The session release condition may be satisfied when the first session release condition or the second session release condition is satisfied.

Next, the cloud streaming server may establish a second user interface session between the cloud streaming server and the client terminal as a preset session recovery condition is satisfied (S740).

At this time, the cloud streaming server may receive the session recovery data from the client terminal, and restore the user interface screen to be streamed to the client terminal through the second user interface session using the session recovery data.

The session recovery condition may be a first session recovery condition that is satisfied when an input signal is received from the client terminal.

In an optional embodiment, the session recovery condition may be a second session recovery condition that is satisfied upon receiving a session recovery request from the client terminal.

In this case, the session recovery request may be generated by the client terminal as the first video content being streamed to the client terminal through the video session is switched to the second video content.

As another optional embodiment, the session recovery condition may be a third session recovery condition that is satisfied when the first video content streaming to the client terminal through the video session is switched to the second video content.

The session recovery condition may be satisfied when the first session recovery condition or the second session recovery condition is satisfied.

The session recovery condition may be satisfied when the first session recovery condition or the third session recovery condition is satisfied.

The session recovery condition may be satisfied when the second session recovery condition or the third session recovery condition is satisfied.

The session recovery condition may be satisfied when the first session recovery condition, the second session recovery condition, or the third session recovery condition is satisfied.

Accordingly, by establishing a plurality of sessions divided by function and display area according to specific conditions, and independently releasing and restoring these sessions, it is possible to more efficiently use the idle resources of the cloud streaming server.

8 is an operation flowchart illustrating a method for restoring a user interface screen according to an embodiment of the present invention.

Referring to FIG. 8 , first, the cloud streaming server may establish a second user interface session between the cloud streaming server and the client terminal in order to stream the user interface screen to the client terminal (S810).

Although not shown in FIG. 8, the cloud streaming server performs a first user interface session between the cloud streaming server and the client terminal to stream the user interface screen to the client terminal before the step (S810) of establishing the second user interface session (S810). Establish, establish a video session between the cloud streaming server and the client terminal to stream video content to the client terminal, and release the first user interface session while maintaining the video session.

The first area of the display screen of the client terminal is an area where the user interface screen is streamed through the first user interface session or the second user interface session, and the second area of the display screen of the client terminal is where video content is streamed through the video session. It may be an area where

In addition, before the step of releasing the first user interface session, the cloud streaming server generates session recovery data including backup data of the user interface screen being streamed to the client terminal through the first user interface session, and the client terminal You can send session recovery data to

The user interface screen streamed through the first user interface session corresponds to the execution screen of the first application executed on the cloud streaming server, and the user interface screen streamed through the second user interface session is a second screen executed on the cloud streaming server. Corresponding to the execution screen of the application, the backup data may be data in which parameter values indicating the execution state of the first application are stored based on a point in time.

As an optional embodiment, the cloud streaming server may replace the user interface screen with a still screen before the step of releasing the first user interface session.

Here, the user interface screen may correspond to the execution screen of the application executed in the cloud streaming server.

The cloud streaming server replaces the execution screen of the application with a still screen, and streams the still screen to the client terminal as a user interface screen through the first user interface session, thereby replacing the user interface screen with a still screen.

Here, the cloud streaming server may replace at least one animation area included on the execution screen of the application with a still image in order to replace the execution screen of the application with a still screen.

The still image may be an image preset for at least one animation area.

For example, the still image may be a representative image preset for each specific animation area. This may be a still image separately created by an animation producer or the like in order to well express the contents of animation reproduced in specific animation areas.

As an optional embodiment, the still image may be an image obtained by capturing at least one animation region.

For example, the still image may be an image obtained by capturing an animation being repeatedly reproduced in the animation area based on a point in time.

Next, the cloud streaming server may receive session recovery data corresponding to the first user interface session established between the cloud streaming server and the client terminal before establishment of the second user interface session from the client terminal (S820).

Next, the cloud streaming server may restore the user interface screen to be streamed to the client terminal through the second user interface session using the session recovery data (S830).

The cloud streaming server may execute the second application using the backup data so that the execution state of the second application corresponds to the execution state of the first application at a point in time.

Accordingly, by restoring the user interface screen provided through the client terminal before the release of the first user interface session so that it can be provided again through the second user interface session, it is possible to minimize the sense of heterogeneity that the user may feel following the session release. have.

The image cloud streaming service method through the browser occupancy management according to the present invention may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art of computer software. Examples of the computer readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floppy disks. - Includes magneto-optical media, and any form of hardware device specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions may include high-level language codes that can be executed by a computer using an interpreter or the like as well as machine language codes such as those generated by a compiler. Such hardware devices may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

Although embodiments of the present invention have been described above with reference to the accompanying drawings, those of ordinary skill in the art to which the present invention pertains can practice the present invention in other specific forms without changing its technical spirit or essential features. You will understand that there is Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive.

According to the present invention, the cloud streaming server releases the session established between the cloud streaming server and the client terminal according to a specific condition and then restores it again to more efficiently use the idle resources of the cloud streaming server. Since it can improve the simultaneous access rate of the streaming service system, it has industrial application potential.

Claims (20)

In a user interface (UI) session recovery method performed by a cloud streaming server,
establishing a first user interface session between the cloud streaming server and the client terminal to stream the user interface screen to the client terminal;
establishing a video session between the cloud streaming server and the client terminal to stream video content to the client terminal;
releasing the first user interface session while maintaining the video session as a preset session release condition is satisfied; and
Establishing a second user interface session between the cloud streaming server and the client terminal as a preset session recovery condition is satisfied
A user interface session recovery method in a cloud streaming service comprising a. The method according to claim 1,
The step of releasing the first user interface session includes:
generating session recovery data including backup data of a user interface screen being streamed to the client terminal through the first user interface session, and transmitting the session recovery data to the client terminal; releasing the session;
The step of establishing the second user interface session includes:
Receiving the session recovery data from the client terminal, and recovering a user interface screen to be streamed to the client terminal through the second user interface session using the session recovery data, a user in a cloud streaming service How to recover an interface session. 3. The method according to claim 2,
The session termination condition is
User interface session recovery method in the cloud streaming service, which is satisfied when a preset time elapses from the reception time of the most recently received input signal from the client terminal. 4. The method according to claim 3,
The session termination condition is
A user interface session recovery method in a cloud streaming service, which is satisfied by receiving a session release request from the client terminal. 5. The method according to claim 4,
The session release request is
User interface session recovery method in the cloud streaming service, which is generated by the client terminal as a preset time elapses from the input time of the most recently input user input on the client terminal. 3. The method according to claim 2,
The session recovery condition is
A user interface session recovery method in a cloud streaming service, which is satisfied by receiving an input signal from the client terminal. 3. The method according to claim 2,
The session recovery condition is
A user interface session recovery method in a cloud streaming service, which is satisfied by receiving a session recovery request from the client terminal. 8. The method of claim 7,
The session recovery request is
A user interface session recovery method in a cloud streaming service, which is generated by the client terminal as the first video content being streamed to the client terminal through the video session is switched to the second video content. 3. The method according to claim 2,
The session recovery condition is
A user interface session recovery method in a cloud streaming service, which is satisfied when the first video content being streamed to the client terminal through the video session is switched to the second video content. The method according to claim 1,
A first area of the display screen of the client terminal is an area in which the user interface screen is streamed through the first user interface session or the second user interface session, and a second area of the display screen of the client terminal displays the video session. A user interface screen recovery method in a cloud streaming service, which is an area through which the video content is streamed. a memory in which at least one program is recorded; and
a processor executing the program
includes,
the program is
establishing a first user interface session between the cloud streaming server and the client terminal to stream the user interface screen to the client terminal;
establishing a video session between the cloud streaming server and the client terminal to stream video content to the client terminal;
releasing the first user interface session while maintaining the video session as a preset session release condition is satisfied; and
Establishing a second user interface session between the cloud streaming server and the client terminal as a preset session recovery condition is satisfied
A cloud streaming server comprising instructions for performing 12. The method of claim 11,
The step of releasing the first user interface session includes:
generating session recovery data including backup data of a user interface screen being streamed to the client terminal through the first user interface session, and transmitting the session recovery data to the client terminal; releasing the session;
The step of establishing the second user interface session includes:
Receiving the session recovery data from the client terminal, and recovering a user interface screen to be streamed to the client terminal through the second user interface session using the session recovery data, Cloud streaming server. 13. The method of claim 12,
The session termination condition is
Cloud streaming server, which is satisfied when a preset time elapses from the reception time of the most recently received input signal from the client terminal. 14. The method of claim 13,
The session termination condition is
A cloud streaming server, which is satisfied upon receiving a session release request from the client terminal. 15. The method of claim 14,
The session release request is
Cloud streaming server, which is generated by the client terminal as a preset time elapses from the input time of the most recently input user input on the client terminal. 13. The method of claim 12,
The session recovery condition is
A cloud streaming server, which is satisfied by receiving an input signal from the client terminal. 13. The method of claim 12,
The session recovery condition is
A cloud streaming server that is satisfied upon receiving a session recovery request from the client terminal. 18. The method of claim 17,
The session recovery request is
The cloud streaming server, which is generated by the client terminal as the first video content being streamed to the client terminal through the video session is switched to the second video content. 13. The method of claim 12,
The session recovery condition is
The cloud streaming server, which is satisfied when the first video content streaming to the client terminal through the video session is switched to the second video content. 12. The method of claim 11,
A first area of the display screen of the client terminal is an area in which the user interface screen is streamed through the first user interface session or the second user interface session, and a second area of the display screen of the client terminal displays the video session. An area through which the video content is streamed, a cloud streaming server.

Images