KR102313529B1 - System for cloud streaming service, method of image cloud streaming service based on optimum rendering and apparatus for the same - Google Patents

Abstract

Disclosed are a cloud streaming service system, an image cloud streaming service method based on optimal rendering, and an apparatus therefor. Receives the application execution screen, renders the application execution screen in consideration of the configuration of the render tree corresponding to the rendering performance criteria, captures an image from the rendered application execution screen, and compresses the captured image with a still image encoding method to allow the user can be transmitted to the terminal of In the case of an image-based cloud streaming service, it is possible to use the resources of the cloud streaming server more efficiently by reducing the number of renderings, and to provide the service to more users by improving the concurrent access rate.

Description

Cloud streaming service system, image cloud streaming service method based on optimal rendering, and device therefor

The present invention relates to a cloud streaming service system capable of improving the number of concurrent users by performing optimal rendering based on a render tree, an image cloud streaming service method based on optimal rendering, and an apparatus therefor.

The rapid development of the Internet has resulted in a rapid increase in personal communication speed, and this improvement in communication speed is like downloading or uploading large amounts of data by accessing a computer located at a remote location, or using a remote computer control program as if it were a remote computer. It provides an environment where you can use a remote computer as if you were logged in locally.

In addition, as applications driven in mobile communication terminals such as smartphones have been developed in various ways, many virtualization technologies have been proposed for driving applications requiring high performance in user terminals with relatively low performance.

Among them, a screen virtualization base that runs an application on the server, compresses the driving screen through video encoding, and transmits it to the client, and the client plays the transmitted video to have the same effect as if the application is running on its own terminal. of cloud streaming services are in the spotlight.

The cloud streaming service basically applied the video codec-based cloud streaming technique. However, even in the case of servicing a static screen such as a menu display, the entire screen is unnecessarily captured and the video codec is used to operate, which is inefficient as a whole system.

In addition, in the case of a web application, a large number of screens are rendered in the browser, and the rendered screen is captured and encoded and delivered, so resource consumption is high, which can affect the maximum number of concurrent users.

Therefore, when servicing a static screen as described above, cloud streaming service is provided by capturing only the changed part of the static screen, but rendering is performed only after the actual meaningful change is completed and the pipe corresponding to rendering, capturing, encoding and sending An image-based cloud streaming service technology that can optimize the number of executions of the line process is urgently emerging.

Korean Patent No. 10-1134807, registered on April 2, 2012 (Title: Early Rendering for Fast Channel Switching)

An object of the present invention is to reduce the load of the cloud streaming server by reducing the number of renderings in the case of an image-based cloud streaming service.

In addition, an object of the present invention is to improve the simultaneous access rate by more efficiently utilizing the limited resources of the cloud streaming server.

In addition, an object of the present invention is to provide a high-spec application execution screen even in a low-spec terminal by processing only the changed image on the application execution screen and providing it to the user terminal.

Cloud streaming server according to the present invention for achieving the above object, the receiving unit for receiving the application execution screen; a rendering unit for rendering the application execution screen in consideration of the configuration of a render tree corresponding to a rendering performance criterion; and a streaming unit that captures an image from the rendered application execution screen, compresses the captured image using a still image encoding method, and transmits it to the user's terminal.

In this case, the rendering performance criterion may correspond to any one of the maximum number of rendering delays and rendering waiting time.

In this case, the rendering unit is the render tree in any one of a case where the number of rendering delays of the render tree corresponds to the maximum number of rendering delays and when at least one change occurs in the configuration of the render tree during the rendering waiting time. The application execution screen may be rendered according to the configuration of .

In this case, the rendering unit may increase the number of rendering delays whenever a change occurs in the configuration of the render tree.

In this case, the number of rendering delays may be initialized when the rendering is performed.

In this case, when the configuration of the render tree is completed, the rendering unit may render the application execution screen according to the finished render tree without considering the rendering performance criteria.

In this case, the streaming unit may capture the changed area as the image by comparing the rendered frames of the application execution screen.

In addition, the image cloud streaming service method based on the optimal rendering according to the present invention, receiving an application execution screen; rendering the application execution screen in consideration of a configuration of a render tree corresponding to a rendering performance criterion; and capturing an image from the rendered application execution screen, compressing the captured image using a still image encoding method, and transmitting the captured image to a user's terminal.

In this case, the rendering performance criterion may correspond to any one of the maximum number of rendering delays and rendering waiting time.

In this case, the rendering is performed when the number of rendering delays of the render tree corresponds to the maximum number of rendering delays and when at least one change in the configuration of the render tree occurs during the rendering waiting time. The application execution screen may be rendered according to the configuration of the render tree.

In this case, the rendering may increase the number of rendering delays whenever a change occurs in the configuration of the render tree.

In this case, the number of rendering delays may be initialized when the rendering is performed.

In this case, the rendering may render the application execution screen according to the completed render tree without considering the rendering performance criteria when the rendering of the render tree is completed.

In this case, the transmitting may capture the changed region as the image by comparing the rendered frames of the application execution screen.

According to the present invention, it is possible to reduce the load of the cloud streaming server by reducing the number of renderings in the case of an image-based cloud streaming service.

In addition, the present invention can improve the simultaneous access rate by more efficiently utilizing the limited resources of the cloud streaming server.

In addition, the present invention can provide a high-spec application execution screen even in a low-spec terminal by processing only the changed image on the application execution screen and providing it to the user terminal.

1 is a block diagram showing a cloud streaming service system according to an embodiment of the present invention.
FIG. 2 is a block diagram illustrating an example of the cloud streaming server shown in FIG. 1 .
3 is a diagram illustrating a general operational flow of a browser rendering engine.
4 is a diagram illustrating a process of generating a render tree by combining a DOM tree and a CSSOM tree in a browser.
5A is a diagram illustrating a process of performing rendering based on the maximum number of rendering delays according to an embodiment of the present invention.
5B is a diagram illustrating a process of performing rendering based on a rendering waiting time according to an embodiment of the present invention.
6 is a diagram illustrating a process of detecting an image to be captured by comparing frames according to an embodiment of the present invention.
7 is a diagram illustrating an image cloud streaming service method based on optimal rendering according to an embodiment of the present invention.
8 is a diagram illustrating in detail an image cloud streaming service method based on optimal rendering according to an embodiment of the present invention.
9 is a diagram illustrating in detail an image cloud streaming service method based on optimal rendering according to another embodiment of the present invention.
10 is a diagram illustrating an image cloud streaming service process based on optimal rendering according to an embodiment of the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, detailed descriptions of well-known functions or configurations that may obscure the gist of the present invention in the following description and accompanying drawings will be omitted. Also, it should be noted that throughout the drawings, the same components are denoted by the same reference numerals as much as possible. The embodiments of the present invention are provided in order to more completely explain the present invention to those of ordinary skill in the art. Accordingly, the shapes and sizes of elements in the drawings may be exaggerated for clearer description.

The terms or words used in the present specification and claims described below should not be construed as being limited to their ordinary or dictionary meanings, and the inventor is appropriate as a concept of terms for describing his invention in the best way. It should be interpreted as meaning and concept consistent with the technical idea of the present invention based on the principle that it can be defined in Accordingly, the embodiments described in this specification and the configurations shown in the drawings are only the most preferred embodiment of the present invention, and do not represent all of the technical spirit of the present invention. It should be understood that there may be equivalents and variations. In addition, terms such as first, second, etc. are used to describe various components, and are used only for the purpose of distinguishing one component from other components, and are not used to limit the components.

1 is a block diagram showing a cloud streaming service system according to an embodiment of the present invention.

Referring to FIG. 1 , the cloud streaming service system according to an embodiment of the present invention includes a cloud streaming server 110 , terminals 120-1 to 120-N, and a network 130 .

Cloud streaming server 110 receives the application execution screen.

In addition, the cloud streaming server 110 renders the application execution screen in consideration of the configuration of the render tree corresponding to the rendering performance criteria.

In this case, the rendering performance criterion may correspond to any one of the maximum number of rendering delays and rendering waiting time.

At this time, when the number of rendering delays of the render tree corresponds to the maximum number of rendering delays and when at least one change occurs in the configuration of the render tree during the rendering waiting time, the application is executed according to the configuration of the render tree The screen can be rendered.

In this case, the number of rendering delays may be increased whenever a change occurs in the configuration of the render tree.

In this case, the number of rendering delays may be initialized when rendering is performed.

In this case, when the configuration of the render tree is completed, the application execution screen may be rendered according to the configured render tree without considering the rendering performance criteria.

In addition, the cloud streaming server 110 captures an image from the rendered application execution screen, compresses the captured image by a still image encoding method, and transmits it to the user's terminals 120-1 to 120-N.

In this case, the changed area may be captured as an image by comparing the rendered frames of the application execution screen.

The terminals 120-1 to 120-N receive the application execution screen corresponding to the cloud streaming service from the cloud streaming server 110 and provide it to the user.

In addition, each of the terminals 120-1 to 120-N is a device that is connected to a communication network and can execute an application based on a cloud computing system, and is not limited to a mobile communication terminal, and includes all information and communication devices, multimedia terminals, wired terminals, It may be various terminals such as a fixed terminal and an IP (Internet Protocol) terminal. In addition, each of the terminals 120-1 to 120-N is a mobile phone, PMP (Portable Multimedia Played), MID (Mobile Internet Device), smartphone (Smart Phone), desktop (Desktop), tablet computer (Tablet PC), notebook It may be a mobile terminal having various mobile communication specifications, such as a note book, a net book, a personal digital assistant (PDA), a smart TV, and an information communication device.

In addition, the terminals 120-1 to 120-N receive various information, such as number and character information, and receive signals input in connection with setting various functions and controlling functions of the terminals 120-1 to 120-N. It can be transmitted to the control unit through the input unit. In addition, the input unit of the terminals 120-1 to 120-N may be configured to include at least one of a keypad and a touchpad that generate an input signal according to a user's touch or manipulation. At this time, the input unit of the terminals 120-1 to 120-N is configured in the form of a single touch panel (or touch screen) together with the display unit of the terminals 120-1 to 120-N, so that input and The display function can be performed at the same time. In addition, as the input unit of the terminals 120-1 to 120-N, all types of input means that may be developed in the future may be used in addition to input devices such as a keyboard, a keypad, a mouse, and a joystick. In particular, the input unit of the terminals 120-1 to 120-N according to the present invention may transmit an input signal for uploading or downloading content based on cloud computing to the control unit of the terminals 120-1 to 120-N.

In addition, the display unit of the terminals 120-1 to 120-N may display information about a series of operation states and operation results that occur while the functions of the terminals 120-1 to 120-N are performed. Also, the display unit of the terminals 120-1 to 120-N may display menus of the terminals 120-1 to 120-N and user data input by the user. Here, the display unit of the terminals 120-1 to 120-N includes a liquid crystal display (LCD), an ultra-thin liquid crystal display (TFT-LCD, Thin Film Transistor LCD), and a light emitting diode (LED, Light Emitting Diode). , organic light emitting diode (OLED, Organic LED), active organic light emitting diode (AMOLED, Active Matrix OLED), retina display (Retina display), flexible display (Flexible display) and three-dimensional (3 Dimension) display, etc. have. At this time, when the display unit of the terminals 120-1 to 120-N is configured in the form of a touch screen, the display unit of the terminals 120-1 to 120-N functions as an input unit of the terminals 120-1 to 120-N. You can do some or all of them. In particular, the display unit of the terminals 120-1 to 120-N according to the present invention may display information related to the execution of content provided based on cloud computing on a screen.

In addition, the storage unit of the terminals 120-1 to 120-N is a device for storing data, and includes a main memory and an auxiliary memory, and is an application necessary for the functional operation of the terminals 120-1 to 120-N. program can be saved. The storage unit of the terminals 120-1 to 120-N may largely include a program area and a data area. Here, when each function is activated in response to a user's request, the terminals 120-1 to 120-N execute corresponding application programs under the control of the control unit to provide each function. In particular, the storage unit of the terminals 120-1 to 120-N according to the present invention can store an operating system for booting the terminals 120-1 to 120-N, a program for uploading or downloading content based on cloud computing, etc. have. In addition, the storage unit of the terminals 120-1 to 120-N may store a content DB for storing a plurality of contents and information of the terminals 120-1 to 120-N. In this case, the content DB includes execution data for executing the content and attribute information on the content, and content usage information according to the content execution may be stored. And, the information of the terminals 120-1 to 120-N may include terminal specification information.

In addition, the communication unit of the terminals 120-1 to 120-N may perform a function for transmitting and receiving data through the cloud streaming server 110 and the network 130 . Here, the communication unit of the terminals 120-1 to 120-N may include an RF transmitting means for up-converting and amplifying the frequency of the transmitted signal, and an RF receiving means for low-noise amplifying the received signal and down-converting the frequency. . The communication unit of the terminals 120-1 to 120-N may include at least one of a wireless communication module and a wired communication module. In addition, the wireless communication module is a configuration for transmitting and receiving data according to a wireless communication method, and when the terminals 120-1 to 120-N use wireless communication, a wireless network communication module, a wireless LAN communication module, and a wireless fan communication module Using any one of the data can be transmitted and received to the cloud streaming server (110). In addition, the wired communication module is for transmitting and receiving data by wire. The wired communication module may connect to the network 130 through a wire and transmit/receive data to/from the cloud streaming server 110 . That is, the terminals 120-1 to 120-N may connect to the network 130 using a wireless communication module or a wired communication module, and may transmit/receive data to and from the cloud streaming server 110 through the network 130 . In particular, the network 130 according to the present invention may communicate with the cloud streaming server 110 or other terminals 120-1 to 120-N to transmit and receive data required to upload or download content based on cloud computing.

In addition, the control unit of the terminal devices 120-1 to 120-N may be a process device for driving an operating system (OS, Operation System) and each configuration. For example, the control unit is a cloud streaming server 110 . When accessing the cloud streaming server 110 through a separate service application, it is possible to control the overall process of executing the service application according to the user's request, and simultaneously with the execution of the cloud It is possible to control the request to use the service to be transmitted to the streaming server 110, and at this time, it is possible to control so that information of the terminals 120-1 to 120-N required for user authentication is transmitted together.

In addition, the controller of the terminals 120-1 to 120-N may execute specific content stored in the storage unit of the terminals 120-1 to 120-N according to a user's request. In this case, the controller may store the content use history according to the content execution as content use information.

In addition, the control unit of the terminals 120-1 to 120-N performs cloud streaming together with execution data for executing the content, content information including attribute information about the content, and content use information, which is information according to the content use history. It can be uploaded by sending it to the server 110 . Thereafter, the control unit can delete the uploaded content by transmitting to the cloud streaming server 110 from the storage unit of the terminals 120-1 to 120-N according to the user's request, and accessing the cloud streaming server 110 to Content may be executed and used through the cloud streaming server 110 .

In addition, the control unit of the terminals (120-1 to 120-N) may be controlled to be stored in the storage after accessing the cloud streaming server 110 to download the content from the other terminals (120-1 to 120-N), , when executing content through the cloud streaming server 110, it is also possible to control the content to be executed after receiving only the data necessary for execution.

The network 130 is a concept that provides a path for transferring data between the cloud streaming server 110 and the terminals 120-1 to 120-N, and encompasses both existing networks and networks that can be developed in the future. . For example, the network 130 is a wired/wireless local area network that provides communication of various information devices within a limited area, a mobile communication network that provides communication between mobile bodies and between the mobile body and the outside of the mobile body, and communication between the earth station and the earth station using satellites. It may be a satellite communication network providing Meanwhile, the transmission method standard of the network 130 is not limited to the existing transmission method standard, and may include all transmission method standards to be developed in the future. In addition, the network used between the cloud streaming server 110 and the terminals 120-1 to 120-N in FIG. 1 may be different from the network used between the terminals 120-1 to 120-N, It may be the same.

FIG. 2 is a block diagram illustrating an example of the cloud streaming server shown in FIG. 1 .

Referring to FIG. 2 , the cloud streaming server 110 shown in FIG. 1 includes a receiving unit 210 , a rendering unit 220 , a streaming unit 230 , and a storage unit 240 .

The receiver 210 receives an application execution screen.

In this case, the application execution screen may be received through a communication network such as the network shown in FIG. 1 . In addition, a request for a cloud streaming service may be received from the terminal.

At this time, it is possible to receive an application execution screen from the application server for the cloud streaming service requested by the terminal.

The rendering unit 220 renders the application execution screen in consideration of the configuration of the render tree corresponding to the rendering performance criterion.

In this case, the render tree may be configured by selecting only data actually displayed on the screen among the data of the application execution screen received by the cloud streaming server. Therefore, all elements of the render tree can be rendered and become one of the images that can be captured by the cloud streaming server.

In the case of a web application, rendering is performed through a browser, and in the browser, a part constituting a render tree and a part performing actual rendering may exist separately. Therefore, rendering is performed even before the render tree is actually constructed and all necessary data is received.

At this time, the browser can know when necessary data is completely received and all elements of the render tree are configured, so that optimal rendering can be performed based on this information.

For example, if rendering is performed after all the configuration of the render tree is completed, rendering is delayed from the point of view of the cloud streaming server, which may affect the speed of subsequent operations. Accordingly, rendering may be performed an optimal number of times in consideration of the configuration of the render tree corresponding to the rendering performance criterion.

In this case, the rendering performance criterion may correspond to any one of the maximum number of rendering delays and rendering waiting time.

At this time, when the number of rendering delays of the render tree corresponds to the maximum number of rendering delays and when at least one change occurs in the configuration of the render tree during the rendering waiting time, the application is executed according to the configuration of the render tree The screen can be rendered.

For example, it may be assumed that the rendering performance criterion is the maximum number of rendering delays, and the maximum number of rendering delays is three. Also, initially, the number of rendering delays of the render tree may be initialized to 0. At this time, the number of rendering delays can be increased by one whenever there is a change in the configuration of the render tree, and rendering can be performed according to the configuration of the render tree only when the number of rendering delays corresponds to the maximum number of rendering delays. have. That is, in the case of a general rendering operation, there is an effect of reducing the number of renderings by shortening a situation in which three renderings have been performed to one rendering.

As another example, if it is assumed that the rendering waiting time is 0.5 seconds, it may be determined whether a change has occurred in the configuration of the render tree every 0.5 seconds. After that, if the configuration of the render tree is changed for 0.5 seconds, rendering is performed, and if no change occurs, it is waited for the rendering waiting time again to determine the change in the configuration of the render tree.

In other words, if the number of rendering delays is a means of reducing the number of renderings based on the number of changes in the configuration of the render tree, the rendering waiting time divides the configuration of the render tree by time and renders the changes occurring within the distributed time all at once to increase the number of renderings. It may be a means to reduce.

Therefore, when the configuration change of the render tree occurs three times in 0.5 seconds, rendering is performed three times, but only one rendering is performed using the rendering latency, so that the number of renderings can be significantly reduced.

In this case, the number of rendering delays may be increased whenever a change occurs in the configuration of the render tree. In addition, by increasing only the number of rendering delays and not performing rendering, rendering is performed whenever a change in the configuration of the render tree occurs, thereby preventing unnecessary consumption of a large amount of resources of the cloud streaming server. In addition, the cloud streaming service can be provided to more users at the same time by increasing the simultaneous access rate by the amount of resources saved in the cloud streaming server.

In this case, the number of rendering delays may be initialized when rendering is performed.

For example, when the configuration of the render tree is completed, the rendering process for the execution screen of the corresponding application may be terminated after rendering. Accordingly, the number of rendering delays may be initialized in order to prepare for rendering execution on an application execution screen to be received next.

In addition, the number of rendering delays may be initialized even after rendering is performed by reaching the maximum number of rendering delays. If the number of rendering delays reaches the maximum number of rendering delays and the number of rendering delays is not initialized even after rendering is performed, rendering is not performed until the render tree is completed, resulting in a slow service speed problem. .

For example, if it is assumed that the maximum number of rendering delays is 3, rendering may be performed when the number of rendering delays reaches 3 according to a change in the render tree. After that, if the number of rendering delays is not initialized, the number of rendering delays gradually increases to 4 and 5, and eventually, a case corresponding to the maximum number of rendering delays may not occur until the construction of the render tree is completed. After all, in the application execution screen, rendering is performed when the change of the render tree appears 3 times, and final rendering is performed when the configuration of the render tree is completed. It may not be possible, so the user may not be provided with the screen quickly.

So, by initializing the number of rendering delays after rendering. Rendering can be performed whenever a change in the configuration of the render tree occurs as much as the maximum number of rendering delays, which can have the effect of quickly providing the application execution screen to the user.

In this case, when the configuration of the render tree is completed, the application execution screen may be rendered according to the configured render tree without considering the rendering performance criteria. That is, if the configuration of the render tree is completed and all elements to be displayed on the screen in the application execution screen are ready to be rendered, rendering may be performed without considering the rendering performance criteria. Accordingly, the cloud streaming server can complete the rendering of the application execution screen as quickly as possible.

The streaming unit 230 captures an image from the rendered application execution screen, compresses the captured image using a still image encoding method, and transmits it to the user's terminal.

The video codec-based cloud streaming technique can perform encoding by capturing all frames corresponding to the screen on which the application operates. However, if the frame-to-frame change is not large on the screen on which the application is running, still image encoding is performed by capturing only the area of change of the frame that has changed compared to the previous frame, as in the image-based cloud streaming technique, and displayed on the user's terminal device. The display can be displayed in such a way that the parts except the change area are shown the same and only the change area is changed.

In this case, the changed area may be captured as an image by comparing the rendered frames of the application execution screen. For example, if the current frame is changed compared to the previous frame when the current frame and the previous frame are compared, the changed area among the screen areas of the current frame may be captured as an image. In addition, the size or properties of the captured image may vary according to an input signal input from the user's terminal.

Also, it is possible to detect sections in which a change between frames is small among frames, and capture an image in a section in which a change between frames is small. For example, it may be inefficient to provide an image-based cloud streaming service because the number of images to be captured is large in a section where there is a lot of change between frames. Therefore, in such a section with many changes, a service is performed through a video codec-based cloud streaming technique, and an image of a changed area can be captured in order to perform an image-based cloud streaming service only in a section with little change between frames.

In this case, the still image encoding method may be determined in consideration of the load generated by the cloud streaming server 110, service speed, image quality, or performance of a terminal that receives and renders the encoded image. For example, in order to improve the speed of a cloud streaming service, an image may be compressed using a palette PNG encoding method that has a high compression rate and a low load. In addition, if you want to provide a good image quality regardless of the load, you can compress the image using a PNG32bit encoding method or a JPEG encoding method.

The storage unit 240 stores various information generated in the process of the cloud streaming service according to the embodiment of the present invention as described above.

According to an embodiment, the storage unit 240 may be configured independently of the cloud streaming server 110 to support a function for a cloud streaming service. In this case, the storage unit 240 may operate as a separate mass storage and may include a control function for performing an operation.

In addition, the cloud streaming server 110 configured as described above may be implemented as one or more servers.

On the other hand, the cloud streaming server 110 is equipped with a memory can store information in the device. For one implementation, the memory is a computer-readable medium. In one implementation, the memory may be a volatile memory unit, and in another implementation, the memory may be a non-volatile memory unit. In one embodiment, the storage device is a computer-readable medium. In various different implementations, the storage device may include, for example, a hard disk device, an optical disk device, or some other mass storage device.

By using such a cloud streaming server 110 to provide an image-based cloud streaming service, it is possible to significantly reduce the number of renderings performed during the cloud streaming service. In addition, since the load of the cloud streaming server 110 may be reduced by the reduced number of renderings, it is possible to provide a more smooth service to users using the service.

In addition, since it is possible to save the resources of the cloud streaming server 110 used for rendering, it is possible to provide an image-based cloud streaming service to more users at the same time.

In addition, by providing an image-based cloud streaming service, there is an effect that high-spec applications and high-spec content can be performed in real time even in the portable terminals of individual users who do not have high specifications.

3 is a diagram illustrating a general operational flow of a browser rendering engine.

Referring to FIG. 3 , the browser rendering engine may operate in processes corresponding to document object model (DOM) tree generation, render tree generation, render tree layout, and render tree painting.

For example, it may be assumed that a component of a browser receives an HTML document from an application server. At this time, the HTML document goes through an analysis process. This process is HTML document parsing, and the work of drawing the parsed result on the screen is rendering.

At this time, since HTML is data having a structure, a DOM tree having a tree structure suitable for expressing the structure of HTML can be created in memory. At this time, the object created in memory through parsing is called document, and the completed structure can be called DOM (Document Object Model).

Also, while HTML is parsed, the browser creates a render tree corresponding to the structure displayed on the screen through the rendering engine that actually composes the screen. In other words, when the DOM tree is completed, instead of displaying the HTML document immediately on the screen, the rendering engine may need to select and display only those that need to be displayed on the current screen in consideration of HTML and CSS in order to quickly compose the screen. At this time, the render tree may be created by selecting only the elements shown on the screen.

After that, a render tree layout that determines where objects belonging to the render tree should be displayed on the screen may be configured, and a painting process of drawing the configured layout on the screen may be performed.

At this time, the rendering engine can display the screen even a little faster by performing the render tree creation, render tree layout configuration, and painting processes almost simultaneously with the creation of the DOM tree.

As described above, it may be possible to configure a screen more quickly by performing rendering using a render tree generated using only elements to be displayed on the screen.

In addition, as in the present invention, by adjusting the number of renderings according to a change in the configuration of the render tree, resources of the cloud streaming server that may be wasted unnecessarily can be saved to provide a cloud streaming service to more users.

4 is a diagram illustrating a process of generating a render tree by combining a DOM tree and a CSSOM tree in a browser.

Referring to FIG. 4 , the browser generates a render tree 430 by combining the DOM tree 410 and the Cascading Style Sheets Object Model (CSSOM) tree 420 .

At this time, the DOM tree 410 and the CSSOM tree 420 are mutually separate objects in charge of different parts of the document. The DOM tree 410 is in charge of content and the CSSOM tree 420 is the style rule to be applied to the document. can be responsible for

First, the browser may start from the root of the DOM tree 410 and search for nodes to be drawn on the screen one by one. At this time, some nodes that the browser does not output to the screen, such as script tags and meta tags, are not reflected when rendering, so they can be skipped.

Also, a node hidden by CSS may not be included in the render tree 430 . For example, a span node to which a rule corresponding to 'display: noon' is specified among span nodes in the CSSOM tree 420 may correspond thereto.

Thereafter, a CSSOM rule suitable for a node to be drawn on the screen found in the DOM tree 410 may be found in the CSSOM tree 420 and each rule may be applied.

After that, you can configure the render tree by putting the nodes to which the CSSOM rules are applied.

Accordingly, the generated render tree has the content and style information of all content displayed on the screen, and data can be output to the screen through the layout and painting process.

5A is a diagram illustrating a process of performing rendering based on the maximum number of rendering delays according to an embodiment of the present invention.

Referring to FIG. 5A , in order to explain the process of rendering based on the maximum number of rendering delays according to an embodiment of the present invention, there is a page that receives a date rendering element and a corresponding picture from another server and displays it on the screen. can be assumed

In this case, the date data can be immediately rendered because the data size is small. After that, picture data received from another server can be rendered according to the maximum number of rendering delays.

For example, if you check the configuration states 510 , 520 , 530 of the render tree shown in FIG. 5A , first, it is confirmed that the date data is configured like the configuration state 510 of the render tree, and the picture data is not yet configured. can

At this time, if it is assumed that the maximum number of rendering delays is 2, there is a change in the composition of date data according to the configuration state 510 of the render tree, but the number of rendering delays is still 1, so the configuration state 510 of the render tree is Rendering may not be performed.

After that, if about half of the picture data is configured as in the configuration state 520 of the render tree and a change in the render tree occurs, the number of rendering delays of the render tree also corresponds to two times, so the configuration state of the render tree 520 is Rendering can be performed accordingly.

In addition, after the rendering delay number corresponds to the maximum number of rendering delays and rendering is performed, the rendering delay number may be initialized. For example, after the maximum number of rendering delays is two and the number of rendering delays reaches two and rendering is performed, the number of rendering delays may be initialized to 0 and then the next rendering process may be performed.

At this time, the maximum number of rendering delays may be set or modified by a cloud streaming service system developer.

Thereafter, when the configuration of the render tree is completed by completely configuring the picture data as in the configuration state 530 of the render tree, rendering may be performed without considering the number of rendering delays. For example, even if the maximum number of rendering delays is 3 and the number of rendering delays is 1 at the time when the render tree configuration is completed, it is necessary to provide a screen corresponding to the already fully configured render tree, so can provide

In this way, when the configuration of the render tree is completed, rendering is performed without delay, so that the completed screen can be provided to the user using the cloud streaming service as quickly as possible.

5B is a diagram illustrating a process of performing rendering based on a rendering waiting time according to an embodiment of the present invention.

Referring to FIG. 5B , in the same manner as in FIG. 5A, a date rendering element and a corresponding picture are received from another server and displayed on the screen in the same manner as in FIG. 5A to explain a process of rendering based on the rendering latency according to an embodiment of the present invention. Let's assume we have a page that shows.

At this time, in FIG. 5A , the picture data received from another server was rendered according to the maximum number of rendering delays, but in FIG. 5B , rendering may be performed in consideration of the rendering latency.

Referring to FIG. 5B , it can be seen that the rendering waiting time is set to 1 second.

At this time, it can be seen that the configuration of the render tree is changed as shown in the configuration state 540 of the render tree while the date data is configured at the change point 541 of the rendering waiting time between 0 and 1 second.

At this time, with respect to the change in the configuration of the render tree occurring at the change point 541 , rendering may be performed in 1 second, which is the rendering waiting time. For example, even if another data is configured in the render tree and a change occurs between the change point 541 and 1 second, all of the changes are rendered in 1 second corresponding to the rendering waiting time. can As described above, the overall number of renderings can be reduced by collectively rendering the changes in the configuration of the render tree that occurred within the rendering waiting time.

In addition, it can be seen that the configuration change of the render tree does not occur during the rendering waiting time corresponding to 1 second to 2 seconds. Therefore, in this case, since the configuration state 550 of the render tree is the same as the configuration state 540 of the previously rendered render tree, rendering may not be performed on the configuration of the render tree even after the rendering waiting time has elapsed. .

Also, like the configuration state 560 of the render tree, when the tree configuration is completed at the change point 561 before the rendering waiting time ends, rendering may be performed irrespective of the rendering waiting time.

In this way, even when optimal rendering is performed in consideration of the rendering latency, the application execution screen can be provided to the user as quickly as possible by rendering without delay when the configuration of the render tree is completed.

6 is a diagram illustrating a process of detecting an image to be captured by comparing frames according to an embodiment of the present invention.

Referring to FIG. 6 , in the process of detecting a captured image by comparing frames according to the present invention, frames of a portion of an application execution screen in response to a user's request with little change in the screen may be acquired first.

In this case, it may be more efficient to provide an application execution screen to the user through a video codec-based cloud streaming service for a portion with a large number of screen changes.

Thereafter, the current frame 620 and the previous frame 610 may be compared among the frames of the small change portion. For example, if the current frame 620 and the previous frame 610 shown in FIG. 6 are compared, there is no change in regions A and B of the two frames, but regions C and D of the previous frame 610 are the current frame. In 620, it can be seen that C' and D' are changed.

Therefore, the cloud streaming server may check such a change, and capture the changed C' region and D' region as the changed region 630 . That is, it is possible to capture an image corresponding to the changed area 630 at the capture end of the cloud streaming server.

In addition, when comparing the current frame 620 with the previous frame 610 , when multiple regions are identified, such as the changed region 630 , one region including all the changed regions is converted into the changed region 630 . ) can be recognized and captured. In addition, it is also possible to perform an image-based cloud streaming service by recognizing all of the changed regions as the changed regions 630 , and capturing the plurality of changed regions 630 .

7 is a diagram illustrating an image cloud streaming service method based on optimal rendering according to an embodiment of the present invention.

Referring to FIG. 7 , the image cloud streaming service method based on optimal rendering according to an embodiment of the present invention receives an application execution screen (S710).

In this case, the application execution screen may be received through a communication network such as the network shown in FIG. 1 . In addition, a request for a cloud streaming service may be received from the terminal.

At this time, it is possible to receive an application execution screen from the application server for the cloud streaming service requested by the terminal.

In addition, the image cloud streaming service method based on optimal rendering according to an embodiment of the present invention renders an application execution screen in consideration of the configuration of a render tree corresponding to a rendering performance criterion (S720).

In this case, the render tree may be configured by selecting only data actually displayed on the screen among the data of the application execution screen received by the cloud streaming server. Therefore, all elements of the render tree can be rendered and become one of the images that can be captured by the cloud streaming server.

In the case of a web application, rendering is performed through a browser, and in the browser, a part constituting a render tree and a part performing actual rendering may exist separately. Therefore, rendering is performed even before the render tree is actually constructed and all necessary data is received.

At this time, the browser can know when necessary data is completely received and all elements of the render tree are configured, so that optimal rendering can be performed based on this information.

For example, if rendering is performed after all the configuration of the render tree is completed, rendering is delayed from the point of view of the cloud streaming server, which may affect the speed of subsequent operations. Accordingly, rendering may be performed an optimal number of times in consideration of the configuration of the render tree corresponding to the rendering performance criterion.

In this case, the rendering performance criterion may correspond to any one of the maximum number of rendering delays and rendering waiting time.

At this time, when the number of rendering delays of the render tree corresponds to the maximum number of rendering delays and when at least one change occurs in the configuration of the render tree during the rendering waiting time, the application is executed according to the configuration of the render tree The screen can be rendered.

For example, it may be assumed that the rendering performance criterion is the maximum number of rendering delays, and the maximum number of rendering delays is three. Also, initially, the number of rendering delays of the render tree may be initialized to 0. At this time, the number of rendering delays can be increased by one whenever there is a change in the configuration of the render tree, and rendering can be performed according to the configuration of the render tree only when the number of rendering delays corresponds to the maximum number of rendering delays. have. That is, in the case of a general rendering operation, there is an effect of reducing the number of renderings by shortening a situation in which three renderings have been performed to one rendering.

As another example, if it is assumed that the rendering waiting time is 0.5 seconds, it may be determined whether a change has occurred in the configuration of the render tree every 0.5 seconds. After that, if the configuration of the render tree is changed for 0.5 seconds, rendering is performed, and if no change occurs, it is waited for the rendering waiting time again to determine the change in the configuration of the render tree.

In other words, if the number of rendering delays is a means of reducing the number of renderings based on the number of changes in the configuration of the render tree, the rendering waiting time divides the configuration of the render tree by time and renders the changes occurring within the distributed time all at once to increase the number of renderings. It may be a means to reduce.

Therefore, when the configuration change of the render tree occurs three times in 0.5 seconds, rendering is performed three times, but only one rendering is performed using the rendering latency, so that the number of renderings can be significantly reduced.

In this case, the number of rendering delays may be increased whenever a change occurs in the configuration of the render tree. In addition, by increasing only the number of rendering delays and not performing rendering, rendering is performed whenever a change in the configuration of the render tree occurs, thereby preventing unnecessary consumption of a large amount of resources of the cloud streaming server. In addition, the cloud streaming service can be provided to more users at the same time by increasing the simultaneous access rate by the amount of resources saved in the cloud streaming server.

In this case, the number of rendering delays may be initialized when rendering is performed.

For example, when the configuration of the render tree is completed, the rendering process for the execution screen of the corresponding application may be terminated after rendering. Accordingly, the number of rendering delays may be initialized in order to prepare for rendering execution on an application execution screen to be received next.

In addition, the number of rendering delays may be initialized even after rendering is performed by reaching the maximum number of rendering delays. If the number of rendering delays reaches the maximum number of rendering delays and the number of rendering delays is not initialized even after rendering is performed, rendering is not performed until the render tree is completed, resulting in a slow service speed problem. .

For example, if it is assumed that the maximum number of rendering delays is 3, rendering may be performed when the number of rendering delays reaches 3 according to a change in the render tree. After that, if the number of rendering delays is not initialized, the number of rendering delays gradually increases to 4 and 5, and eventually, a case corresponding to the maximum number of rendering delays may not occur until the construction of the render tree is completed. After all, in the application execution screen, rendering is performed when the change of the render tree appears 3 times, and final rendering is performed when the configuration of the render tree is completed. It may not be possible, so the user may not be provided with the screen quickly.

So, by initializing the number of rendering delays after rendering. Rendering can be performed whenever a change in the configuration of the render tree occurs as much as the maximum number of rendering delays, which can have the effect of quickly providing the application execution screen to the user.

In this case, when the configuration of the render tree is completed, the application execution screen may be rendered according to the configured render tree without considering the rendering performance criteria. That is, if the configuration of the render tree is completed and all elements to be displayed on the screen in the application execution screen are ready to be rendered, rendering may be performed without considering the rendering performance criteria. Accordingly, the cloud streaming server can complete the rendering of the application execution screen as quickly as possible.

In addition, the image cloud streaming service method based on optimal rendering according to an embodiment of the present invention captures an image from a rendered application execution screen, compresses the captured image using a still image encoding method, and transmits it to the user's terminal (S730) ).

The video codec-based cloud streaming technique can perform encoding by capturing all frames corresponding to the screen on which the application operates. However, if the frame-to-frame change is not large on the screen on which the application is running, still image encoding is performed by capturing only the area of change of the frame that has changed compared to the previous frame, as in the image-based cloud streaming technique, and displayed on the user's terminal device. The display can be displayed in such a way that the parts except the change area are shown the same and only the change area is changed.

In this case, the changed area may be captured as an image by comparing the rendered frames of the application execution screen. For example, if the current frame is changed compared to the previous frame when the current frame and the previous frame are compared, the changed area among the screen areas of the current frame may be captured as an image. In addition, the size or properties of the captured image may vary according to an input signal input from the user's terminal.

Also, it is possible to detect sections in which a change between frames is small among frames, and capture an image in a section in which a change between frames is small. For example, it may be inefficient to provide an image-based cloud streaming service because the number of images to be captured is large in a section where there is a lot of change between frames. Therefore, in such a section with many changes, a service is performed through a video codec-based cloud streaming technique, and an image of a changed area can be captured in order to perform an image-based cloud streaming service only in a section with little change between frames.

In this case, the still image encoding method may be determined in consideration of the amount of load generated by the cloud streaming server, service speed, image quality, or performance of a terminal that receives and renders the encoded image. For example, in order to improve the speed of a cloud streaming service, an image may be compressed using a palette PNG encoding method that has a high compression rate and a low load. In addition, if you want to provide a good image quality regardless of the load, you can compress the image using a PNG32bit encoding method or a JPEG encoding method.

In addition, although not shown in FIG. 7, the image cloud streaming service method based on optimal rendering according to an embodiment of the present invention stores various information generated in the process of the cloud streaming service according to the embodiment of the present invention as described above. do.

According to an embodiment, a storage module, storage, and database for storing information may be configured independently of a cloud streaming server to support a function for a cloud streaming service. In this case, the storage module, the storage, and the database may operate as separate mass storage and may include a control function for performing the operation.

By providing an image-based cloud streaming service through the image cloud streaming service method based on such an optimal rendering, the number of renderings performed during the cloud streaming service can be significantly reduced. In addition, since the load of the cloud streaming server may be reduced by the reduced number of renderings, it is possible to provide a smoother service to users who use the service.

In addition, since it is possible to save the resources of the cloud streaming server used for rendering, it is possible to provide an image-based cloud streaming service to more users at the same time.

In addition, by providing an image-based cloud streaming service, there is an effect that high-spec applications and high-spec content can be performed in real time even in the portable terminals of individual users who do not have high specifications.

8 is a diagram illustrating in detail an image cloud streaming service method based on optimal rendering according to an embodiment of the present invention.

Referring to FIG. 8 , the image cloud streaming service method based on optimal rendering according to an embodiment of the present invention first receives an application execution screen (S810). In this case, the application execution screen may be an execution screen of an application executed according to a user's request.

Thereafter, in order to perform optimal rendering, a render tree corresponding to the application execution screen is configured (S820). In this case, the render tree may be composed of only data to be actually displayed on the screen among the data of the application execution screen. In addition, Render Tree can help provide image-based cloud streaming services faster and more efficiently.

Thereafter, it is determined whether a change occurs in the configuration of the render tree (S815).

If no change occurs as a result of the determination in step S815, the render tree is continuously constructed.

If a change occurs as a result of the determination in step S815, it is determined whether the configuration of the render tree is complete (S825).

If it is determined in step S825 that the construction of the render tree is not completed, it is determined whether the number of rendering delays corresponds to the maximum number of rendering delays (S835).

If it is determined in step S835 that does not correspond to the maximum number of rendering delays, the number of rendering delays is increased by 1 (S840) and a change in the configuration of the render tree is checked again.

In this case, only the number of rendering delays is increased and actual rendering is not performed, thereby reducing the number of renderings.

If the determination result of step S835 corresponds to the maximum number of rendering delays, the rendering is performed according to the configuration of the current render tree (S850).

Thereafter, the image of the application execution screen is captured through the capture end of the cloud streaming server (S852).

In this case, the changed area may be captured as an image by comparing the frames of the application execution screen.

Thereafter, the captured image is encoded using a still image encoding method (S854), and the still image-encoded image is transmitted to the terminal (S856).

In this case, the still image encoding method may be determined in consideration of the amount of load generated by the cloud streaming server, service speed, image quality, or performance of a terminal that receives and renders the encoded image.

Then, after the number of rendering delays is initialized to correspond to the initial value again (S858), a change in the configuration of the render tree is checked again.

If, as a result of the determination in step S825, the configuration of the render tree is completed, the rendering is performed according to the configuration of the completed render tree (S860). In this case, after rendering, the number of rendering delays may be initialized to correspond to an initial value, and the initial value may correspond to zero.

Thereafter, the image of the application execution screen is captured through the capture end of the cloud streaming server (S862), and the captured image is encoded as a still image (S864) and transmitted to the terminal (S866).

9 is a diagram illustrating in detail an image cloud streaming service method based on optimal rendering according to another embodiment of the present invention.

Referring to FIG. 9 , the image cloud streaming service method based on optimal rendering according to another embodiment of the present invention first receives an application execution screen (S910).

After that, in order to perform optimal rendering, a render tree corresponding to the application execution screen is configured (S920).

Thereafter, a render tree is configured during the rendering waiting time (S930).

Thereafter, it is determined whether there is a change in the configuration of the render tree within the rendering waiting time (S935).

If there is no change in the configuration of the render tree as a result of the determination in step S935, the render tree is reconfigured during the rendering waiting time without performing rendering.

If there is a change in the configuration of the render tree as a result of the determination in step S935, it is determined whether the configuration of the render tree is complete (S945).

If it is determined in step S945 that the configuration of the render tree is not completed, rendering is performed according to the configuration of the current render tree (S950).

Thereafter, an image of the rendered application execution screen is captured (S952), a still image is encoded (S954), and the still image-encoded image is transmitted to the terminal (S956), and then a render tree is reconfigured during the rendering waiting time.

If it is determined in step S945 that the configuration of the render tree is completed, rendering is performed according to the configuration of the completed render tree (S960).

Thereafter, the image of the rendered application execution screen is captured (S962), the still image is encoded (S964), and the still image-encoded image is transmitted to the terminal (S966).

10 is a diagram illustrating an image cloud streaming service process based on optimal rendering according to an embodiment of the present invention.

Referring to FIG. 10 , in the image cloud streaming service process based on optimal rendering according to an embodiment of the present invention, the application is executed in the application server according to the user's request (S1002). In this case, the application server may receive a user request from the user's terminal 120-1.

Thereafter, the receiving unit of the cloud streaming server 110 receives the application execution screen according to the user's request from the application server (S1004).

The rendering unit of the cloud streaming server 110 configures the render tree corresponding to the application execution screen (S1006)

In this case, the render tree may be composed of only elements to be actually displayed on the screen in the application execution screen.

At this time, it is determined whether rendering is performed in consideration of the configuration of the render tree corresponding to the rendering performance criterion (S1008).

For example, it may be determined that rendering is performed when the configuration of the render tree is completed or the number of rendering delays corresponds to the maximum number of rendering delays. At this time, the maximum number of rendering delays may be set by the administrator or developer of the cloud streaming service system, and it may be possible to freely change it.

In addition, it may be determined whether rendering is performed based on a change in the configuration of the render tree during the rendering waiting time. For example, the number of renderings can be reduced by continuously performing the configuration of the render tree during the rendering waiting time and then rendering the changes in the configuration of the render tree that occurred during the rendering waiting time. Also, if the configuration of the render tree does not change during the rendering waiting time, rendering may not be performed.

In this way, by adjusting the number of rendering executions according to the configuration of the render tree, it is possible to bring about the effect of saving resources that could have been wasted by performing frequent rendering in the cloud streaming service.

If it is determined that rendering is not performed as a result of the determination in step S1008, the construction of the render tree may be continued. In this case, if it is determined whether or not rendering is performed based on the number of rendering delays, the number of rendering delays may be increased by one.

If it is determined that rendering is performed as a result of the determination in step S1008, the rendering unit renders according to the configuration of the render tree (S1010).

Thereafter, the rendered application execution screen is transmitted to the streaming unit (S1012), and an image of the application execution screen is captured (S1014). In this case, an image may be captured by comparing frames corresponding to the application execution screen.

Thereafter, the captured image is encoded as a still image (S1016), and the still image-encoded image is transmitted to the terminal 120-1 (S1018).

In this case, the captured image may be compressed by selecting any one of still image encoding methods.

At this time, in consideration of the load generated in the cloud streaming server 110, the service transmission speed, the quality level of the transmitted image, and the performance of the terminal 120-1, one of the various still image encoding methods is determined. can

The terminal 120-1 renders the still image encoded image received from the cloud streaming server 110 (S920), and displays it to the user of the terminal 120-1 (S1022). In this case, the application execution screen displayed to the user may be a screen before the configuration of the render tree is completed.

Accordingly, it is determined whether or not rendering is completed based on the configuration of the render tree ( S1024 ), and if not completed, the configuration of the render tree may be continuously performed.

In addition, when the configuration of the render tree of the corresponding application execution screen is completed and rendering corresponding to the already completed render tree is performed, it is determined that the rendering is complete and waits for the next operation (S1026).

The image cloud streaming service method based on the optimal rendering according to the present invention may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - Includes magneto-optical media, and any form of hardware device specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions may include high-level language codes that can be executed by a computer using an interpreter or the like as well as machine language codes such as those generated by a compiler. Such hardware devices may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

As described above, in the cloud streaming service system according to the present invention, the image cloud streaming service method based on the optimal rendering, and the apparatus therefor, the configuration and method of the embodiments described above are not limitedly applicable, but the above embodiment Examples may be configured by selectively combining all or part of each embodiment so that various modifications can be made.

According to the present invention, an application execution screen is received, an application execution screen is rendered in consideration of the configuration of a render tree corresponding to a rendering performance criterion, an image is captured from the rendered application execution screen, and the captured image is encoded as a still image. can be compressed and transmitted to the user's terminal. Furthermore, by saving the amount of resources used in the image-based cloud streaming service, the service can be provided to more users at the same time.

110: cloud streaming server 120-1~ 120-N: terminal
130: network 210: receiver
220: rendering unit 230: streaming unit
240: storage 410: DOM (Document Object Model)
420: Cascading Style Sheets Object Model (CSSOM)
430: render tree 510 to 560: render tree configuration status
541, 561: change point 610: previous frame
620: current frame 630: changed area

Claims (10)

a receiver for receiving an application execution screen;
a rendering unit for rendering the application execution screen in consideration of the configuration of a render tree corresponding to a rendering performance criterion; and
A streaming unit that captures an image from the rendered application execution screen, compresses the captured image using a still image encoding method, and transmits it to the user's terminal
including,
The rendering performance criterion is
corresponding to any one of the maximum number of rendering delays and rendering latency;
The rendering unit
In either case, when the number of rendering delays of the render tree corresponds to the maximum number of rendering delays and when at least one change occurs in the configuration of the render tree during the rendering waiting time, corresponding to the configuration of the render tree Cloud streaming server, characterized in that for rendering the application execution screen. delete delete delete delete delete delete receiving an application execution screen;
rendering the application execution screen in consideration of a configuration of a render tree corresponding to a rendering performance criterion; and
Capturing an image from the rendered application execution screen, compressing the captured image using a still image encoding method and transmitting it to a user's terminal
including,
The rendering performance criterion is
corresponding to any one of the maximum number of rendering delays and rendering latency;
The rendering step
In either case, when the number of rendering delays of the render tree corresponds to the maximum number of rendering delays and when at least one change occurs in the configuration of the render tree during the rendering waiting time, corresponding to the configuration of the render tree Image cloud streaming service method based on optimal rendering, characterized in that rendering the application execution screen. delete delete

Images