KR102409033B1 - System for cloud streaming service, method of image cloud streaming service using alpha level of color bit and apparatus for the same - Google Patents

Abstract

A cloud streaming service system, an image cloud streaming service method using an alpha level, and an apparatus therefor are disclosed. The cloud streaming server according to the present invention renders the application execution screen, divides the frame screen corresponding to the application execution screen based on the alpha level into at least one divided area, and provides each of the user terminals in the at least one divided area At least one image to be used may be captured, a still image encoding technique may be selected based on the alpha level, and the at least one image may be still image-encoded using the selected still image encoding technique and transmitted to the user's terminal. Accordingly, it is possible to encode the captured image more naturally and efficiently in the case of an image-based cloud streaming service.

Description

Cloud streaming service system, image cloud streaming service method using alpha level, and apparatus therefor

The present invention relates to an image-based cloud streaming service technology, and in particular, a cloud streaming service system that can perform efficient encoding by separating the screen according to the alpha level in the application execution screen and processing the service, and image cloud streaming using the alpha level It relates to a service method and an apparatus therefor.

The rapid development of the Internet has resulted in a rapid increase in individual communication speed, and this communication speed improvement is like downloading or uploading large amounts of data by accessing a computer located at a remote location, or using a remote computer control program as if it were a remote computer. It provides an environment where you can use a remote computer as if you were logged in locally.

In addition, as applications driven in a mobile communication terminal such as a smart phone are developed in various ways, virtualization technologies for driving applications requiring high performance in a user terminal with relatively low performance have been widely proposed.

Among them, a screen virtualization-based system that runs the application on the server, compresses the driving screen through video encoding, and transmits it to the client, and the client plays the transmitted video to have the same effect as if the application is running on its own terminal of cloud streaming services are in the spotlight.

The cloud streaming service basically applied the video codec-based cloud streaming technique. However, even when a static screen such as a menu display is provided, the entire screen is unnecessarily captured and the video codec is used to operate, which is inefficient as a whole system.

Therefore, only the changed image of the static part on the execution screen of the application executed according to the user's request is captured and provided to the user's terminal. An image-based cloud streaming service technology that can perform efficient still image encoding for each level is urgently needed.

Korean Patent Laid-Open Patent No. 2015-0065069, published on June 12, 2015 (Title: Display device, image display method of display device, and computer-readable recording medium)

It is an object of the present invention to naturally and efficiently process images captured for each alpha level in an image-based cloud streaming service using a suitable still image encoding technique.

Another object of the present invention is to omit additional processing such as counting the number of colors in order to understand the properties of the image during encoding by dividing the image according to the encoding method in the capture step.

In addition, it is an object of the present invention to provide a cloud streaming service smoothly even if the performance of the user terminal is low by rendering and displaying only the changed image of the frame in the user terminal.

A cloud streaming server according to the present invention for achieving the above object is a frame dividing unit for rendering an application execution screen, and dividing the frame screen corresponding to the application execution screen into at least one divided area based on the alpha level; a capture unit configured to capture at least one image to be provided to a user's terminal in each of the at least one divided area; an encoding unit that selects a still image encoding method based on the alpha level and encodes the at least one image as a still image by using the selected still image encoding method; and a streaming unit for transmitting the still image-encoded at least one image to a user's terminal.

In this case, the at least one divided region includes a first region having the alpha level corresponding to a first level, a second region having the alpha level corresponding to a second level, and a third region having the alpha level corresponding to a third level. may correspond to at least one of

In this case, the alpha level is at least one of the first level having an alpha value of 255, the second level including the alpha value within a preset range based on 128, and a third level having the alpha value corresponding to 0 can correspond to one.

In this case, the first area may correspond to an opaque area of the frame screen, the second area may correspond to a translucent area of the frame screen, and the third area may correspond to a transparent area of the frame screen.

In this case, the encoding unit performs still image encoding using a JPEG still image encoding technique when the at least one image is captured in the first area, and a PNG still image when the at least one image is captured in the second area. Still image encoding may be performed using an image encoding technique.

In this case, the encoding unit may not perform still image encoding when the at least one image is captured in the third area.

In this case, the capture unit may compare the previous frame screen of the frame screen with the at least one divided area, and capture the changed part for each of the at least one divided area as the at least one image.

In addition, the image cloud streaming service method using an alpha level according to the present invention comprises the steps of rendering an application execution screen, and dividing a frame screen corresponding to the application execution screen into at least one divided area based on the alpha level; capturing at least one image to be provided to each user's terminal in the at least one divided area; selecting a still image encoding scheme based on the alpha level, and still image encoding the at least one image using the selected still image encoding scheme; and transmitting the still image-encoded at least one image to a terminal of a user.

In this case, the at least one divided region includes a first region having the alpha level corresponding to a first level, a second region having the alpha level corresponding to a second level, and a third region having the alpha level corresponding to a third level. may correspond to at least one of

In this case, the alpha level is at least one of the first level having an alpha value of 255, the second level including the alpha value within a preset range based on 128, and a third level having the alpha value corresponding to 0 can correspond to one.

In this case, the first area may correspond to an opaque area of the frame screen, the second area may correspond to a translucent area of the frame screen, and the third area may correspond to a transparent area of the frame screen.

In this case, the step of encoding the still image may include performing still image encoding using a JPEG still image encoding technique when the at least one image is captured in the first area, and capturing the at least one image in the second area. In this case, still image encoding may be performed using the PNG still image encoding technique.

In this case, the encoding of the still image may not perform encoding of the still image when the at least one image is captured in the third area.

In this case, the capturing may include comparing a previous frame screen of the frame screen with the at least one divided area, and capturing a changed portion for each of the at least one divided area as the at least one image.

In addition, as another means for solving the problems of the present invention, there is provided a computer program stored in a medium for executing the above-described method.

In addition, the cloud streaming service system according to the present invention renders an application execution screen, divides a frame screen corresponding to the application execution screen based on the alpha level into at least one divided area, and each user in the at least one divided area a cloud streaming server that captures at least one image to be provided to a terminal of , encodes the at least one image using a still image encoding technique selected based on the alpha level, and transmits the still image to the terminal; and a terminal that receives and renders the at least one still image-encoded image from the cloud streaming server, and displays the rendered at least one image to the user.

According to the present invention, in the case of an image-based cloud streaming service, images captured for each alpha level can be naturally and efficiently processed using a suitable still image encoding technique.

In addition, in the present invention, since the image is divided according to the encoding method in the capture step, additional processing such as counting the number of colors in order to grasp the properties of the image during encoding may be omitted.

In addition, the present invention allows the user terminal to render and display only the changed image of the frame, so that the cloud streaming service can be smoothly provided even if the performance of the user terminal is low.

1 is a block diagram illustrating a cloud streaming service system according to an embodiment of the present invention.
2 is a block diagram showing an example of the cloud streaming server shown in FIG.
3 is a diagram illustrating a frame screen for each alpha level according to an embodiment of the present invention.
4 is a diagram illustrating a frame screen of an application execution screen according to an embodiment of the present invention.
FIG. 5 is a diagram illustrating the frame screen shown in FIG. 4 at an alpha level.
6 is a diagram illustrating a process of capturing an image for each region according to an alpha level according to an embodiment of the present invention.
7 is an operation flowchart illustrating an image cloud streaming service method using an alpha level according to an embodiment of the present invention.
8 is a detailed operation flowchart illustrating a method of dividing a frame screen in an image cloud streaming service method using an alpha level according to an embodiment of the present invention.
9 is a detailed operation flowchart illustrating a method of performing still image encoding in an image cloud streaming service method using an alpha level according to an embodiment of the present invention.
10 is a detailed operation flowchart illustrating a method of performing still image encoding in an image cloud streaming service method using an alpha level according to another embodiment of the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, detailed descriptions of well-known functions or configurations that may obscure the gist of the present invention in the following description and accompanying drawings will be omitted. Also, it should be noted that, throughout the drawings, the same components are denoted by the same reference numerals as much as possible. The embodiments of the present invention are provided in order to more completely explain the present invention to those of ordinary skill in the art. Accordingly, the shapes and sizes of elements in the drawings may be exaggerated for clearer description.

The terms or words used in the present specification and claims described below should not be construed as being limited to their ordinary or dictionary meanings, and the inventor is appropriate as a concept of terms for describing his invention in the best way. It should be interpreted as meaning and concept consistent with the technical idea of the present invention based on the principle that it can be defined in Therefore, the embodiments described in this specification and the configurations shown in the drawings are only the most preferred embodiment of the present invention, and do not represent all the technical spirit of the present invention, so at the time of the present application, various It should be understood that there may be equivalents and variations. In addition, terms such as 1st, 2nd, etc. are used to describe various components, and are used only for the purpose of distinguishing one component from other components, and are not used to limit the components.

1 is a block diagram illustrating a cloud streaming service system according to an embodiment of the present invention.

Referring to FIG. 1 , the cloud streaming service system according to an embodiment of the present invention includes a cloud streaming server 110 , terminals 120-1 to 120-N, and a network 130 .

The cloud streaming server 110 renders the application execution screen, and divides the frame screen corresponding to the application execution screen into at least one divided area based on the alpha level.

In this case, the at least one divided region is at least one of a first region having an alpha level corresponding to a first level, a second region having an alpha level corresponding to a second level, and a third region having an alpha level corresponding to a third level. can correspond to

In this case, the alpha level may correspond to at least one of a first level having an alpha value of 255, a second level having an alpha value within a preset range based on 128, and a third level having an alpha value of 0. can

In this case, the first area may correspond to an opaque area of the frame screen, the second area may correspond to a translucent area of the frame screen, and the third area may correspond to a transparent area of the frame screen.

In addition, the cloud streaming server 110 captures at least one image to be provided to each user's terminals 120-1 to 120-N in at least one divided area.

In this case, by comparing the previous frame screen of the frame screen with at least one divided area, the changed portion for each at least one divided area may be captured as at least one image.

In addition, the cloud streaming server 110 selects a still image encoding method based on the alpha level, and encodes at least one image as a still image encoding method using the selected still image encoding method.

At this time, still image encoding is performed using the JPEG still image encoding technique when at least one image is captured in the first area, and still image encoding is performed using the PNG still image encoding technique when at least one image is captured in the second area. encoding can be performed.

In this case, when at least one image is captured in the third area, still image encoding may not be performed.

In addition, the cloud streaming server 110 transmits at least one still image-encoded image to the terminal.

The terminals 120-1 to 120-N receive the application execution screen corresponding to the cloud streaming service from the cloud streaming server 110 and provide it to the user.

In addition, each of the terminals 120-1 to 120-N is a device that is connected to a communication network and can execute an application based on a cloud computing system, and is not limited to a mobile communication terminal, and includes all information and communication devices, multimedia terminals, wired terminals, It may be various terminals such as a fixed terminal and an IP (Internet Protocol) terminal. In addition, each of the terminals 120-1 to 120-N is a mobile phone, PMP (Portable Multimedia Played), MID (Mobile Internet Device), smartphone (Smart Phone), desktop (Desktop), tablet computer (Tablet PC), notebook It may be a mobile terminal having various mobile communication specifications such as a note book, a net book, a personal digital assistant (PDA), a smart TV, and an information communication device.

In addition, the terminals 120-1 to 120-N receive various information such as number and character information, set various functions, and receive signals input in relation to function control of the terminals 120-1 to 120-N. It can be transmitted to the control unit through the input unit. Also, the input unit of the terminals 120-1 to 120-N may include at least one of a keypad and a touchpad that generate an input signal according to a user's touch or manipulation. At this time, the input unit of the terminals 120-1 to 120-N is configured in the form of a single touch panel (or touch screen) together with the display unit of the terminals 120-1 to 120-N, so that input and The display function can be performed at the same time. In addition, as the input unit of the terminals 120-1 to 120-N, all types of input means that may be developed in the future may be used in addition to input devices such as a keyboard, a keypad, a mouse, and a joystick. In particular, the input unit of the terminals 120-1 to 120-N according to the present invention may transmit an input signal for uploading or downloading content based on cloud computing to the control unit of the terminals 120-1 to 120-N.

In addition, the display unit of the terminals 120-1 to 120-N may display information about a series of operation states and operation results that occur while the functions of the terminals 120-1 to 120-N are performed. In addition, the display unit of the terminals 120-1 to 120-N may display menus of the terminals 120-1 to 120-N and user data input by the user. Here, the display unit of the terminals 120-1 to 120-N includes a liquid crystal display (LCD), an ultra-thin liquid crystal display (TFT-LCD, Thin Film Transistor LCD), and a light emitting diode (LED, Light Emitting Diode). , organic light emitting diodes (OLED, Organic LED), active organic light emitting diodes (AMOLED, Active Matrix OLED), retina display (Retina Display), flexible display (Flexible display) and three-dimensional (3 Dimension) display, etc. have. At this time, when the display unit of the terminals 120-1 to 120-N is configured in the form of a touch screen, the display unit of the terminals 120-1 to 120-N functions as an input unit of the terminals 120-1 to 120-N. You can do some or all of them. In particular, the display unit of the terminals 120-1 to 120-N according to the present invention may display information related to execution of content provided based on cloud computing on a screen.

In addition, the storage unit of the terminals 120-1 to 120-N is a device for storing data, and includes a main memory and an auxiliary memory, and is an application necessary for the functional operation of the terminals 120-1 to 120-N. program can be saved. The storage unit of the terminals 120-1 to 120-N may largely include a program area and a data area. Here, when each function is activated in response to a user's request, the terminals 120-1 to 120-N execute corresponding application programs under the control of the controller to provide each function. In particular, the storage unit of the terminals 120-1 to 120-N according to the present invention can store an operating system for booting the terminals 120-1 to 120-N, a program for uploading or downloading content based on cloud computing, etc. have. In addition, the storage unit of the terminals 120-1 to 120-N may store a content DB for storing a plurality of contents and information of the terminals 120-1 to 120-N. In this case, the content DB includes execution data for executing the content and attribute information on the content, and content usage information according to the content execution may be stored. And, the information of the terminals 120-1 to 120-N may include terminal specification information.

In addition, the communication unit of the terminals 120-1 to 120-N may perform a function for transmitting and receiving data through the cloud streaming server 110 and the network 130 . Here, the communication unit of the terminals 120-1 to 120-N may include an RF transmitting means for up-converting and amplifying the frequency of the transmitted signal, and an RF receiving means for low-noise amplifying the received signal and down-converting the frequency. . The communication unit of the terminals 120-1 to 120-N may include at least one of a wireless communication module and a wired communication module. In addition, the wireless communication module is a configuration for transmitting and receiving data according to a wireless communication method, and when the terminals 120-1 to 120-N use wireless communication, a wireless network communication module, a wireless LAN communication module, and a wireless fan communication module Using any one of the data can be transmitted and received to the cloud streaming server (110). In addition, the wired communication module is for transmitting and receiving data by wire. The wired communication module may connect to the network 130 through a wire and transmit/receive data to/from the cloud streaming server 110 . That is, the terminals 120-1 to 120-N may connect to the network 130 using a wireless communication module or a wired communication module, and may transmit/receive data to and from the cloud streaming server 110 through the network 130 . In particular, the network 130 according to the present invention may communicate with the cloud streaming server 110 or other terminals 120-1 to 120-N to transmit and receive data required to upload or download content based on cloud computing.

In addition, the control unit of the terminals 120-1 to 120-N may be a process device for driving an operating system (OS, Operation System) and each configuration. For example, the control unit is configured to be connected to the cloud streaming server 110 . It is possible to control the entire process of accessing When accessing the cloud streaming server 110 through a separate service application, the entire process of executing the service application can be controlled according to the user's request, and at the same time as the execution of the cloud streaming It is possible to control so that a service use request is transmitted to the server 110, and at this time, it is possible to control so that information of the terminals 120-1 to 120-N required for user authentication is transmitted together.

Also, the controller of the terminals 120-1 to 120-N may execute specific content stored in the storage unit of the terminals 120-1 to 120-N according to a user's request. In this case, the controller may store the content use history according to the content execution as content use information.

In addition, the control unit of the terminals 120-1 to 120-N performs cloud streaming together with execution data for executing the content, content information including attribute information about the content, and content use information that is information according to the content use history. It can be uploaded by sending it to the server 110 . Thereafter, the control unit can delete the uploaded content transmitted to the cloud streaming server 110 from the storage unit of the terminals 120-1 to 120-N according to the user's request, and access the cloud streaming server 110 to Content may be executed and used through the cloud streaming server 110 .

In addition, the control unit of the terminals 120-1 to 120-N may access the cloud streaming server 110 to download content from other terminals 120-1 to 120-N and then control it to be stored in the storage unit, , when executing content through the cloud streaming server 110, it is also possible to control so that the content is executed after receiving only the data necessary for execution.

The network 130 is a concept that provides a path for transferring data between the cloud streaming server 110 and the terminals 120-1 to 120-N, and encompasses both existing networks and networks that can be developed in the future. . For example, the network 130 may include a wired/wireless local area network that provides communication of various information devices within a limited area, a mobile communication network that provides communication between mobile devices and between the mobile body and the outside of the mobile device, and communication between the earth station and the earth station using satellites. It may be a satellite communication network that provides Meanwhile, the transmission method standard of the network 130 is not limited to the existing transmission method standard, and may include all transmission method standards to be developed in the future. In addition, the network used between the cloud streaming server 110 and the terminals 120-1 to 120-N in FIG. 1 may be different from the network used between the terminals 120-1 to 120-N, It may be the same.

By providing a cloud streaming service through such a cloud streaming service system, more natural and efficient encoding can be performed according to the alpha level of an image captured during the cloud streaming service.

In addition, since the image is divided according to the encoding method in the capture step, additional processing such as counting the number of colors in order to understand the properties of the image during encoding may be omitted.

In addition, by rendering and displaying only the changed image of the frame in the user terminal, the cloud streaming service can be smoothly provided even if the performance of the user terminal is low.

2 is a block diagram showing an example of the cloud streaming server shown in FIG.

Referring to FIG. 2 , the cloud streaming server 110 shown in FIG. 1 includes a communication unit 210 , a frame division unit 220 , a capture unit 230 , an encoding unit 240 , a streaming unit 250 and a storage unit. (260).

The communication unit 210 serves to transmit and receive information related to a plurality of terminals through a communication network such as the network shown in FIG. 1 . In particular, the communication unit 210 according to an embodiment of the present invention may receive a request for a cloud streaming service from a terminal, and provide an execution screen of an application corresponding to the cloud streaming service requested by the terminal to the terminal.

At this time, it is possible to receive the application execution screen from the cloud streaming server for the cloud streaming service requested by the terminal.

The frame divider 220 renders the application execution screen and divides the frame screen corresponding to the application execution screen into at least one divided area based on the alpha level.

At this time, the web application server may execute the application according to the user's request, and transmit the application execution screen corresponding to the result of executing the application to the cloud streaming server 110 . That is, the cloud streaming server 110 may provide an image-based cloud streaming service to the user using only the result screen of executing the application.

In this case, the still image encoder used for the cloud streaming service may include an encoder such as PNG that may include an alpha channel indicating transparency, or an encoder without an alpha channel such as JPEG. In addition, since there is a high possibility that the characteristics of the corresponding part are different for each alpha level in the screen configuration, the classification of objects during encoding for each alpha level is natural and the encoding performance may be improved.

Accordingly, in order to perform capture and still image encoding for each alpha level in the image-based cloud streaming service, a frame screen corresponding to the rendered application execution screen may be divided based on the alpha level. This may be more efficient than encoding by separating the regions for each alpha level by checking the characteristics of each image region when performing still image encoding.

In this case, the at least one divided region is at least one of a first region having an alpha level corresponding to a first level, a second region having an alpha level corresponding to a second level, and a third region having an alpha level corresponding to a third level. can correspond to

In this case, at least one divided area may be generated by checking the alpha level for each pixel constituting the frame screen and determining one of the first area, the second area, and the third area. In this case, during the rendering process, the alpha level may be checked while checking the color for each pixel constituting the frame screen.

In this case, the alpha level may correspond to at least one of a first level having an alpha value of 255, a second level having an alpha value within a preset range based on 128, and a third level having an alpha value of 0. can

In addition, when the alpha value corresponds to 128 through the setting of the cloud streaming service system, it may be determined that the alpha level corresponds to the second level.

In this case, the first area may correspond to an opaque area of the frame screen, the second area may correspond to a translucent area of the frame screen, and the third area may correspond to a transparent area of the frame screen. For example, in the application execution screen, main images such as a poster or picture may correspond to an opaque area, a text part including a description of the poster or picture may correspond to a translucent area, and a live channel in the background may correspond to an opaque area. Since it should always be visible, the background can correspond to a transparent area.

In addition, since at least one divided area is an area according to the alpha level, even if they are spaced apart from each other in location, they may be recognized as the same area. For example, if it is assumed that regions A, B, and C corresponding to the first alpha level exist, all regions corresponding to A, B, and C may be recognized as the first region.

The capture unit 230 captures at least one image to be provided to each user's terminal in at least one divided area. That is, images may be captured in each of the first region, the second region, and the third region.

Since the cloud streaming service basically applies the video codec-based cloud streaming technique, even when the cloud streaming technique is applied to a static screen such as a menu display, it is an inefficient way to unnecessarily capture the entire screen and encode it with a video codec. A cloud streaming service was performed. In order to solve this problem, in the present invention, when the change between frames is not large, such as a menu display screen, the cloud streaming server detects the change between frames, captures the changed area as an image, and then encodes it with a still image compression technique. A server 110 is provided.

In addition, the video codec-based cloud streaming technique can perform encoding by capturing all frames corresponding to the screen on which the application operates. However, if the frame-to-frame change is not significant on the screen on which the application is running, the still image encoding is performed by capturing only the change area of the frame that has changed compared to the previous frame, and the portion except the change area is the same when displayed on the user's terminal device. It can be displayed in such a way that only the change area changes.

In this case, by comparing the previous frame screen of the frame screen with at least one divided area, the changed portion for each at least one divided area may be captured as at least one image.

For example, it may be assumed that the frame screen is divided into a first area and a second area according to the alpha level. In this case, if there is a changed part in the first region when the first region is compared with the previous frame screen, the corresponding part may be captured as an image of the first region. Also, if there is a changed part in the second region when the second region is compared with the previous frame screen, the corresponding part may be captured as an image of the second region.

Also, it is possible to detect a section in which the frame change is small among the entire frame to be transmitted to the user's terminal, and compare the frames in the section in which the frame change is small to capture an image.

In this case, it is possible to detect a section in which the frame change is small by detecting the event type of the application that occurs in the user's terminal. For example, an event type of an application may be determined according to an input signal generated from a user's terminal. Depending on the input signal, when an application is executed, such as a menu, an option, or a guide window, a static event with a small frame change or a dynamic event with a frequent frame change such as video playback and game play may occur. Accordingly, if the user detects an event type with an input signal generated through the terminal and the event type is a static event, it is possible to detect a section in which the frame change is small among the entire frame section corresponding to the event.

The encoding unit 240 selects a still image encoding method based on the alpha level, and encodes at least one still image using the selected still image encoding method.

At this time, still image encoding is performed using the JPEG still image encoding technique when at least one image is captured in the first area, and still image encoding is performed using the PNG still image encoding technique when at least one image is captured in the second area. encoding can be performed.

That is, when the alpha level of the image corresponds to the first level and the alpha value corresponds to 255, the image may be encoded as a still image using the JPEG still image encoding technique to preserve natural image characteristics.

In this case, since the JPEG still image encoding technique has good compression efficiency, it is possible to significantly reduce the amount of transmitted data during encoding.

Also, when the alpha level of the image is the second level and the alpha value is within a preset range based on 128, the still image may be encoded using the PNG still image encoding technique.

In this case, the PNG still image encoding technique has good image quality during encoding, but the transmission speed may be lowered when the bandwidth of the transmission network is low due to the large data size.

In this case, when at least one image is captured in the third area, still image encoding may not be performed. That is, when the alpha level of the image is the third level and the alpha value is a transparent area corresponding to 0, it is possible to individually process the still image without separately encoding the still image.

The streaming unit 250 transmits at least one still image-encoded image to the terminal.

At this time, by completing still image encoding and transmitting the still image encoded image to the user's terminal, an image-based cloud streaming service can be provided so that the user can feel that the application is running on the terminal.

The storage unit 260 stores various information generated in the process of the cloud streaming service according to the embodiment of the present invention as described above.

According to an embodiment, the storage unit 260 may be configured independently of the cloud streaming server 110 to support a function for a cloud streaming service. In this case, the storage unit 260 may operate as a separate mass storage and may include a control function for performing an operation.

In addition, the cloud streaming server 110 configured as described above may be implemented as one or more servers.

On the other hand, the cloud streaming server 110 is equipped with a memory can store information in the device. In one implementation, the memory is a computer-readable medium. In one implementation, the memory may be a volatile memory unit, and in another implementation, the memory may be a non-volatile memory unit. In one embodiment, the storage device is a computer-readable medium. In various different implementations, the storage device may include, for example, a hard disk device, an optical disk device, or some other mass storage device.

By providing the cloud streaming service through the cloud streaming server 110 as described above, images captured for each alpha level in the image-based cloud streaming service can be naturally and efficiently processed using a suitable still image encoding technique.

In addition, it is an object of the present invention to omit additional processing such as counting the number of colors in order to understand the properties of the image during encoding by dividing the image according to the encoding method in the capture step.

In addition, an object of the present invention is to render and display only the changed image of the frame in the user terminal, so that the cloud streaming service can be smoothly provided even if the performance of the user terminal is low.

3 is a diagram illustrating a frame screen for each alpha level according to an embodiment of the present invention.

Referring to FIG. 3 , it can be confirmed that the regions of the current frame 310 and the next frame 320 have changed.

For example, regions A and B of the current frame 310 correspond to first regions 311 and 312 having an alpha level corresponding to the first level, and region C corresponds to a second region having an alpha level corresponding to the second level. 2 may be assumed to correspond to the region 313 .

At this time, in the next frame 320 , it can be seen that the region A is changed to the region A′, the region B is changed to the region B′, and the region C is changed to the region C′.

At this time, if an image of a changed place is captured for an image-based cloud streaming service according to an embodiment of the present invention, the image may be captured by dividing the first region and the second region according to the alpha level.

That is, in FIG. 3 , the first regions 311 and 312 of the current frame 310 and the first regions 321 and 322 of the next frame 320 are compared, and regions A′ and B′ of the next frame 320 are compared. Regions may be captured as images in the first regions 321 and 322 .

In addition, in FIG. 3 , the second region 313 of the current frame 310 and the second region 323 of the next frame 320 are compared, and region C′ of the next frame 320 is used as the second region 323 . It can be captured as an image from

At this time, if the pipeline process for the cloud streaming service is performed without dividing the region by alpha level before performing capture, the color information of the image is checked again to determine which encoder is suitable for still image encoding. Additional processing may be required.

Accordingly, by performing an encoding procedure by capturing images in regions divided according to the alpha level, it is possible to more efficiently encode an image by easily selecting a still image encoding technique suitable for the alpha level of the image.

4 is a diagram illustrating a frame screen of an application execution screen according to an embodiment of the present invention.

Referring to FIG. 4 , the frame screen of the application execution screen according to an embodiment of the present invention may be divided into an opaque area 410 , a semi-transparent area 420 , and a transparent area.

In this case, in the case of a broadcast-related UI as shown in FIG. 4 , there may be a requirement to always show a live channel in the background. Accordingly, the background portion becomes the transparent area, the general description portion becomes the translucent area 420 , and important objects such as drawings, photos, and posters are highly likely to become the opaque area 410 .

In the case of a frame screen in which the opaque area 410, the semi-transparent area 420, and the transparent area are mixed as described above, rather than simply comparing the area of the previous frame screen to obtain a difference, the area of the previous frame screen is divided by alpha level. Finding the region and difference can improve the efficiency in terms of encoding.

FIG. 5 is a diagram illustrating the frame screen shown in FIG. 4 at an alpha level.

Referring to FIG. 5 , when the frame screen shown in FIG. 4 is displayed for each alpha level, the opaque area 510 , the semi-transparent area 520 , and all areas corresponding to the background may correspond to the transparent area 530 .

That is, since the opaque region 510 has an alpha value of 255 or an alpha value of 250 or greater, the opaque region 510 may correspond to an image in which a background region is not visible, such as a poster or a picture.

Also, since the translucent area 520 has an alpha value of 128 or an alpha value between 50 and 250, it may correspond to a text image such as an amplified description of a main object.

Also, since the alpha value corresponds to 0 or the alpha value corresponds to 50 or less, the transparent region 530 may correspond to a transparent background to show an image of a background background.

In this case, it can be assumed that the change is detected at the encoding stage without dividing the region by alpha level as shown in FIG. 5 . In this case, since the image and background for the change area are mixed, additional processing such as having to count the number of colors in the image again in the process of checking what is the appropriate encoder for the captured image may be necessarily accompanied.

Therefore, before capturing an image in the application execution screen, a service can be provided through more efficient still image encoding by dividing a region according to the alpha level and performing a streaming pipeline process based on the divided regions.

6 is a diagram illustrating a process of capturing an image for each region according to an alpha level according to an embodiment of the present invention.

Referring to FIG. 6 , the process of capturing an image for each region according to the alpha level according to an embodiment of the present invention does not compare changes based on the entire frame region in the previous frame 610 and the current frame 620 . , the change may be compared based on at least one divided region divided by alpha level.

For example, regions corresponding to A1 and A2 of the previous frame 610 and A3 and A4 of the current frame 620 all correspond to the first region, and B1 of the previous frame 610 and the regions of the current frame 620 . It may be assumed that all regions corresponding to B2 correspond to the second region.

Also, in the previous frame 610 and the current frame 620 , it may be assumed that an area that does not correspond to any of the first area and the second area corresponds to the third area, that is, the background.

In this case, in order to compare the change regions of the previous frame 610 and the current frame 620 , regions corresponding to the first region may be compared. That is, it is possible to compare whether the first area corresponding to A1 and A2 of the previous frame 610 has changed in the current frame 620 .

At this time, if it is assumed that A1 and A2 of the previous frame 610 are changed to A3 and A4 in the current frame 620 , in the first area of the current frame 620 , the areas corresponding to A3 and A4 are divided into the first area. You can capture it as a capture image.

Also, after capturing the image of the changed region in the first region, it may be compared whether there is a portion to be captured in the second region and the third region.

That is, it is possible to compare whether the second region corresponding to B1 of the previous frame 610 has changed in the current frame 620 .

At this time, assuming that B1 of the previous frame 610 is changed to B2 in the current frame 620, the area corresponding to B2 in the second area of the current frame 620 can be captured as the capture image of the second area. have.

Also, after capturing the image of the changed area in the second area, it may be compared whether there is a portion to be captured in the third area.

In this way, by capturing an image according to each divided region and generating a captured image for each divided region, still image encoding can be performed by selecting a still image encoding technique suitable for each captured image for each divided region when performing still image encoding. . In addition, encoding efficiency may be improved by performing still image encoding using a still image encoding technique suitable for captured images for each divided region.

7 is an operation flowchart illustrating an image cloud streaming service method using an alpha level according to an embodiment of the present invention.

Referring to FIG. 7 , the image cloud streaming service method using an alpha level according to an embodiment of the present invention renders an application execution screen, and based on the alpha level, a frame screen corresponding to the application execution screen is displayed in at least one divided area. is divided into (S710).

At this time, the web application server may execute the application according to the user's request, and transmit the application execution screen corresponding to the result of executing the application to the cloud streaming server. That is, the cloud streaming server may provide an image-based cloud streaming service to the user using only the result screen of executing the application.

In this case, the still image encoder used for the cloud streaming service may include an encoder such as PNG that may include an alpha channel indicating transparency, or an encoder without an alpha channel such as JPEG.

In addition, since there is a high possibility that the characteristics of the corresponding part are different for each alpha level in the screen configuration, the classification of objects during encoding for each alpha level is natural and the encoding performance may be improved.

Accordingly, in order to perform capture and still image encoding for each alpha level in the image-based cloud streaming service, a frame screen corresponding to the rendered application execution screen may be divided based on the alpha level. This may be more efficient than encoding by separating the regions for each alpha level by checking the characteristics of each image region when performing still image encoding.

In this case, the at least one divided region is at least one of a first region having an alpha level corresponding to a first level, a second region having an alpha level corresponding to a second level, and a third region having an alpha level corresponding to a third level. can correspond to

In this case, at least one divided area may be generated by checking the alpha level for each pixel constituting the frame screen and determining one of the first area, the second area, and the third area. In this case, during the rendering process, the alpha level may be checked while checking the color for each pixel constituting the frame screen.

In this case, the alpha level may correspond to at least one of a first level having an alpha value of 255, a second level having an alpha value within a preset range based on 128, and a third level having an alpha value of 0. can

In addition, when the alpha value corresponds to 128 through the setting of the cloud streaming service system, it may be determined that the alpha level corresponds to the second level.

In this case, the first area may correspond to an opaque area of the frame screen, the second area may correspond to a translucent area of the frame screen, and the third area may correspond to a transparent area of the frame screen. For example, in the application execution screen, main images such as a poster or picture may correspond to an opaque area, a text part including a description of the poster or picture may correspond to a translucent area, and a live channel in the background may correspond to an opaque area. Since it should always be visible, the background can correspond to a transparent area.

In addition, since at least one divided area is an area according to the alpha level, even if they are spaced apart from each other in location, they may be recognized as the same area. For example, if it is assumed that regions A, B, and C corresponding to the first alpha level exist, all regions corresponding to A, B, and C may be recognized as the first region.

In addition, the image cloud streaming service method using an alpha level according to an embodiment of the present invention captures at least one image to be provided to each user's terminal in at least one divided area (S720). That is, images may be captured in each of the first region, the second region, and the third region.

Since the cloud streaming service basically applies the video codec-based cloud streaming technique, even when the cloud streaming technique is applied to a static screen such as a menu display, it is an inefficient way to unnecessarily capture the entire screen and encode it with a video codec. A cloud streaming service was performed. In order to solve this problem, in the present invention, when the change between frames is not large, such as a menu display screen, the cloud streaming server detects the change between frames, captures the changed area as an image, and then encodes it with a still image compression technique. server is provided.

In addition, the video codec-based cloud streaming technique can perform encoding by capturing all frames corresponding to the screen on which the application operates. However, if the frame-to-frame change is not significant on the screen on which the application is running, the still image encoding is performed by capturing only the change area of the frame that has changed compared to the previous frame, and the portion except the change area is the same when displayed on the user's terminal device. It can be displayed in such a way that only the change area changes.

In this case, by comparing the previous frame screen of the frame screen with at least one divided area, the changed portion for each at least one divided area may be captured as at least one image.

For example, it may be assumed that the frame screen is divided into a first area and a second area according to the alpha level. In this case, if there is a changed part in the first region when the first region is compared with the previous frame screen, the corresponding part may be captured as an image of the first region. Also, if there is a changed part in the second region when the second region is compared with the previous frame screen, the corresponding part may be captured as an image of the second region.

Also, it is possible to detect a section in which the frame change is small among the entire frame to be transmitted to the user's terminal, and compare the frames in the section in which the frame change is small to capture an image.

In this case, it is possible to detect a section in which the frame change is small by detecting the event type of the application that occurs in the user's terminal. For example, an event type of an application may be determined according to an input signal generated from a user's terminal. Depending on the input signal, when an application is executed, such as a menu, an option, or a guide window, a static event with a small frame change or a dynamic event with a frequent frame change such as video playback and game play may occur. Accordingly, if the user detects an event type with an input signal generated through the terminal and the event type is a static event, it is possible to detect a section in which the frame change is small among the entire frame section corresponding to the event.

In addition, the image cloud streaming service method using an alpha level according to an embodiment of the present invention selects a still image encoding method based on the alpha level and encodes at least one image using the selected still image encoding method (S730). ).

At this time, still image encoding is performed using the JPEG still image encoding technique when at least one image is captured in the first area, and still image encoding is performed using the PNG still image encoding technique when at least one image is captured in the second area. encoding can be performed.

That is, when the alpha level of the image corresponds to the first level and the alpha value corresponds to 255, the image may be encoded as a still image using the JPEG still image encoding technique to preserve natural image characteristics.

In this case, since the JPEG still image encoding technique has good compression efficiency, it is possible to significantly reduce the amount of transmitted data during encoding.

Also, when the alpha level of the image is the second level and the alpha value is within a preset range based on 128, the still image may be encoded using the PNG still image encoding technique.

In this case, the PNG still image encoding technique has good image quality during encoding, but the transmission speed may be lowered when the bandwidth of the transmission network is low due to the large data size.

In this case, when at least one image is captured in the third area, still image encoding may not be performed. That is, when the alpha level of the image is the third level and the alpha value is a transparent area corresponding to 0, it is possible to individually process the still image without separately encoding the still image.

In addition, the image cloud streaming service method using the alpha level according to an embodiment of the present invention transmits at least one still image-encoded image to the terminal (S740).

In this case, by completing still image encoding and transmitting the still image encoded image to the user's terminal, an image-based cloud streaming service can be provided so that the user can feel that the application is running on the terminal.

In addition, although not shown in FIG. 7 , the image cloud streaming service method using the alpha level according to an embodiment of the present invention transmits and receives information related to a plurality of terminals through a communication network such as the network shown in FIG. 1 . In particular, it is possible to receive a request for a cloud streaming service from the terminal, and provide an execution screen of an application corresponding to the cloud streaming service requested by the terminal to the terminal.

At this time, it is possible to receive the application execution screen from the cloud streaming server for the cloud streaming service requested by the terminal.

In addition, although not shown in FIG. 7, the image cloud streaming service method using the alpha level according to an embodiment of the present invention stores various information generated in the process of the cloud streaming service according to the embodiment of the present invention as described above. do.

According to an embodiment, the storage module may be configured independently of a cloud streaming server to support a function for a cloud streaming service. In this case, the storage module may operate as a separate mass storage and may include a control function for performing the operation.

In addition, the cloud streaming server performing the above-described image cloud streaming service method may be implemented as one or more servers.

On the other hand, the cloud streaming server is equipped with a memory can store information in the device. In one implementation, the memory is a computer-readable medium. In one implementation, the memory may be a volatile memory unit, and in another implementation, the memory may be a non-volatile memory unit. In one embodiment, the storage device is a computer-readable medium. In various different implementations, the storage device may include, for example, a hard disk device, an optical disk device, or some other mass storage device.

By providing a cloud streaming service through such an image-based cloud streaming service method, images captured for each alpha level in an image-based cloud streaming service can be naturally and efficiently processed with an appropriate still image encoding technique.

In addition, it is an object of the present invention to omit additional processing such as counting the number of colors in order to understand the properties of the image during encoding by dividing the image according to the encoding method in the capture step.

In addition, an object of the present invention is to render and display only the changed image of the frame in the user terminal, so that the cloud streaming service can be smoothly provided even if the performance of the user terminal is low.

8 is a detailed operation flowchart illustrating a method of dividing a frame screen in an image cloud streaming service method using an alpha level according to an embodiment of the present invention.

Referring to FIG. 8 , in the method of dividing a frame screen in the image cloud streaming service method using an alpha level according to an embodiment of the present invention, first, an alpha level of a region corresponding to the frame screen is checked ( S810 ).

In this case, the alpha level may correspond to a level indicating transparency of a corresponding region, image, or pixel. For example, if an alpha value is 0 to 255, an area having an alpha value of 0 may correspond to a transparent area, and an area having an alpha value of 255 may correspond to an opaque area. Also, when the alpha value is between 0 and 255, it corresponds to between the transparent area and the opaque area, and when the alpha value is close to 128, it can be viewed as a semi-transparent area.

In this case, the alpha level may correspond to at least one of a first level having an alpha value of 255, a second level having an alpha value within a preset range based on 128, and a third level having an alpha value of 0. can

Thereafter, it is determined whether the alpha level of the checked area corresponds to the first level (S815).

In this case, the alpha level may be checked in units of pixels of each area.

If it is determined in step S815 that the alpha level corresponds to the first level, the corresponding area is divided into the first area (S820).

In this case, all regions divided into the first region may be combined and recognized as the first region.

Also, if it is determined in step S815 that the alpha level does not correspond to the first level, it is determined whether the alpha level of the checked area corresponds to the second level (S825).

If it is determined in step S825 that the alpha level corresponds to the second level, the corresponding area is divided into a second area (S830).

Also, if it is determined in step S825 that the alpha level does not correspond to the second level, the corresponding area is divided into a third area (S840).

In this case, the at least one divided region is at least one of a first region having an alpha level corresponding to a first level, a second region having an alpha level corresponding to a second level, and a third region having an alpha level corresponding to a third level. can correspond to

In this case, at least one divided area may be generated by checking the alpha level for each pixel constituting the frame screen and determining one of the first area, the second area, and the third area. In this case, during the rendering process, the alpha level may be checked while checking the color for each pixel constituting the frame screen.

In this case, the first area may correspond to an opaque area of the frame screen, the second area may correspond to a translucent area of the frame screen, and the third area may correspond to a transparent area of the frame screen. For example, in the application execution screen, main images such as a poster or picture may correspond to an opaque area, a text part including a description of the poster or picture may correspond to a translucent area, and a live channel in the background may correspond to an opaque area. Since it should always be visible, the background can correspond to a transparent area.

9 is an operation flowchart illustrating in detail a method of performing still image encoding in an image cloud streaming service method using an alpha level according to an embodiment of the present invention.

Referring to FIG. 9 , in the method of performing still image encoding in the image cloud streaming service method using the alpha level according to an embodiment of the present invention, first, the alpha level of at least one image is checked ( S910 ).

In this case, a still image encoding technique for encoding at least one image as a still image may be selected according to the alpha level.

In this case, it may be possible to encode the still image more efficiently and naturally by performing still image encoding using the still image encoding technique according to the alpha level.

Thereafter, it is determined whether the alpha level of the at least one image corresponds to the first level (S915).

In this case, the first level may have an alpha value of 255.

If it is determined in step S915 that the alpha level of the at least one image is the first level, the at least one image is still image-encoded using the JPEG still image encoding technique (S920).

In this case, since the JPEG still image encoding technique has good compression efficiency, it is possible to significantly reduce the amount of transmitted data during encoding.

Also, if it is determined in step S915 that the alpha level of the at least one image is not the first level, it is determined whether the alpha level of the at least one image corresponds to the second level (S925).

In this case, the second level may include an alpha value within a preset range based on 128.

If it is determined in step S925 that the alpha level of the at least one image is the second level, the still image is encoded using the PNG still image encoding technique (S930).

In this case, the PNG still image encoding technique has good image quality during encoding, but the transmission speed may be lowered when the bandwidth of the transmission network is low due to the large data size.

In addition, if it is determined in step S925 that the alpha level of at least one image is not the second level, it is determined that the alpha level of at least one image corresponds to the third level and the at least one image is individually processed (S940). ).

In this case, the third level may have an alpha value of 0. That is, the background image corresponding to the transparent area can be individually processed and serviced without using a still image encoding technique.

10 is a detailed operation flowchart illustrating a method of performing still image encoding in an image cloud streaming service method using an alpha level according to another embodiment of the present invention.

Referring to FIG. 10 , in the method of performing still image encoding in the image cloud streaming service method using the alpha level according to another embodiment of the present invention, first, a divided region in which at least one image is captured is checked ( S1010 ).

In this case, since the alpha level can be naturally checked by checking the divided region, a still image encoding technique for encoding at least one image as a still image can be selected using only the divided region.

Accordingly, it may be possible to more efficiently and naturally encode an image by performing still image encoding using a still image encoding technique according to a divided region.

Thereafter, it is determined whether the divided area in which at least one image is captured corresponds to the first area (S1015).

In this case, the alpha level of the first region may correspond to the first level.

If it is determined in step S1015 that the divided region in which the at least one image is captured is the first region, the at least one image is still image-encoded using the JPEG still image encoding technique (S1020).

In this case, since the JPEG still image encoding technique has good compression efficiency, it is possible to significantly reduce the amount of transmitted data during encoding.

Also, if it is determined in step S1015 that the divided area in which at least one image is captured is not the first area, it is determined whether the area in which at least one image is captured corresponds to the second area (S1025).

In this case, the alpha level of the second region may correspond to the second level.

If it is determined in step S1025 that the divided region in which the at least one image is captured is the second region, the at least one image is still image-encoded using the PNG still image encoding technique (S1030).

In this case, the PNG still image encoding technique has good image quality during encoding, but the transmission speed may be lowered when the bandwidth of the transmission network is low due to the large data size.

In addition, if it is determined in step S1025 that the divided region in which at least one image is captured is not the second region, it is determined that the divided region in which at least one image is captured corresponds to the third region, and the at least one image is individually separated. processing (S1040).

In this case, the alpha level of the third region may correspond to the third level. That is, the background image corresponding to the transparent area can be individually processed and serviced without using a still image encoding technique.

The image cloud streaming service method using the alpha level according to the present invention may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - Includes magneto-optical media, and any form of hardware device specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions may include high-level language codes that can be executed by a computer using an interpreter or the like as well as machine language codes such as those generated by a compiler. Such hardware devices may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

As described above, in the cloud streaming service system according to the present invention, the image cloud streaming service method using the alpha level, and the apparatus for the same, the configuration and method of the embodiments described above are not limitedly applicable, but the above embodiment Examples may be configured by selectively combining all or part of each embodiment so that various modifications can be made.

According to the present invention, the application execution screen is rendered, the frame screen corresponding to the application execution screen is divided into at least one divided area based on the alpha level, and at least one or more to be provided to the user's terminal in the at least one divided area, respectively. The image may be captured, and at least one still image may be encoded by selecting a still image encoding technique based on the alpha level, and the at least one still image-encoded image may be transmitted to the user's terminal. Furthermore, in the case of an image-based cloud streaming service, it is possible to provide a more effective and natural image to the user by performing still image encoding according to the alpha level.

110: cloud streaming server 120-1~ 120-N: terminal
130: network 210: communication unit
220: frame division unit 230: capture unit
240: encoding unit 250: streaming unit
260: storage unit 310, 620: current frame
320: next frame 311, 312, 321, 322: first area
313. 323: second area 410, 510: opaque area
420, 520: semi-transparent area 530: transparent area
610: previous frame

Claims (10)

a frame divider for rendering the application execution screen and dividing a frame screen corresponding to the application execution screen into at least one divided area based on the alpha level;
a capture unit configured to capture at least one image to be provided to a user's terminal in each of the at least one divided area;
an encoding unit that selects a still image encoding method based on the alpha level and encodes the at least one image using the selected still image encoding method; and
Streaming unit for transmitting the still image-encoded at least one image to the terminal
including,
The at least one partitioned area is
Cloud streaming server, characterized in that it corresponds to at least one of a first region corresponding to an opaque region, a second region corresponding to a translucent region, and a third region corresponding to a transparent region. The method according to claim 1,
The at least one partitioned area is
wherein the alpha level corresponds to at least one of a first region corresponding to a first level, a second region having the alpha level corresponding to a second level, and a third region having the alpha level corresponding to a third level. cloud streaming server. 3. The method according to claim 2,
The alpha level is
The alpha value corresponds to at least one of the first level corresponding to 255, the second level including the alpha value within a preset range based on 128, and a third level corresponding to 0 Cloud streaming server with delete delete delete delete rendering the application execution screen, and dividing a frame screen corresponding to the application execution screen into at least one divided area based on the alpha level;
capturing at least one image to be provided to each user's terminal in the at least one divided area;
selecting a still image encoding technique based on the alpha level, and still image encoding the at least one image using the selected still image encoding technique; and
Transmitting the at least one still image-encoded image to the terminal
including,
The at least one partitioned area is
A cloud streaming service method using an alpha level, characterized in that it corresponds to at least one of a first region corresponding to an opaque region, a second region corresponding to a translucent region, and a third region corresponding to a transparent region. delete delete

Images