KR102272357B1 - System for cloud streaming service, method of image cloud streaming service using drawing layer separation and apparatus for the same - Google Patents

Abstract

Disclosed are a cloud streaming service system, an image cloud streaming service method using drawing layer separation, and an apparatus therefor. When the cloud streaming server transmits a still image encoded captured image and an animation message corresponding to the application execution screen to the user terminal device, the terminal device uses the animation information and the captured image included in the animation message to display an animation intermediate screen And by creating an application execution screen including an animation end screen, and displaying the animation intermediate screen and the animation ending screen through different screen layers, unnecessary animation processing in the cloud streaming server is reduced, and the number of web containers that can be driven in the server It can be increased and returned to the normal ending screen even if an error occurs during animation playback.

Description

Cloud streaming service system, image cloud streaming service method using drawing layer separation, and device therefor

The present invention relates to an image cloud streaming service technology using drawing layer separation, and in particular, a cloud streaming service system that displays an intermediate screen and an end screen on a separate drawing layer when an animation is played in a user terminal for image cloud streaming, drawing It relates to an image cloud streaming service using layer separation and an apparatus therefor.

Due to the recent development of the Internet, the speed of communication using personal mobile communication devices has rapidly improved, and based on this, an Internet environment such as downloading and uploading large amounts of data through remote computer access or logging into a remote computer locally is created. became In addition, as applications driven in a mobile communication terminal such as a smart phone are developed in various ways, virtualization technologies for driving applications requiring high performance in a user terminal with relatively low performance have been widely proposed. Among them, a screen virtualization base that runs an application on the server, compresses the driving screen through video encoding, and transmits it to the client, and the client plays the transmitted video to have the same effect as if the application is running on its own terminal. of cloud streaming services are in the spotlight.

This cloud streaming service basically applies a video codec-based cloud streaming technique, but even when a service such as a menu display is provided, the entire screen is unnecessarily captured and operated using the video codec, resulting in inefficiency of the system as a whole. exist.

Therefore, by generating a captured image for a part that requires a dynamic effect such as animation among the execution result screen of the application executed by the user's request, and transmitting the captured image and animation related information to the user's set-top box, animation in the set-top box A technology that can create a service screen with effects applied is urgently emerging.

Korean Patent Laid-Open Patent No. 10-2012-0096909, published on August 31, 2012 (Name: Merge gateway/set-top box image for delivery to a serviced client device)

An object of the present invention is to capture only a partial area of the web application execution screen in the cloud streaming server, generate a message having animation information to be applied to the area, and transmit the captured image and message to the set-top box, using the message in the set-top box By applying animation to the captured image, unnecessary animation processing is reduced and the number of web containers that can be run on the server is increased.

It is also an object of the present invention, when the cloud streaming server transmits the captured image and message to the set-top box, by applying an animation to the captured image using the message in the set-top box, and dividing the animation middle screen and the end screen by layering, It is to provide a more stable streaming service that returns to the normal ending screen even if an error occurs during animation playback.

An image cloud streaming service apparatus using drawing layer separation according to the present invention for achieving the above object is a receiver that receives a still image encoded captured image and an animation message corresponding to an application execution screen from a cloud streaming server ; a screen generator for generating the application execution screen including an animation intermediate screen and an animation end screen by using the animation information included in the animation message and the captured image; and a display unit for displaying the animation intermediate screen and the animation ending screen through different screen layers.

In this case, the screen layer is divided into a top layer for displaying the animation middle screen and a base layer for displaying the animation ending screen, and the display unit is masked when animation playback starts. The animation intermediate screen may be generated in a base layer, and when the animation reproduction is finished, the masking setting may be canceled to display the animation intermediate screen.

In this case, the captured image may be an image obtained by capturing a change region of the current frame changed from the previous frame with respect to the previous frame and the current frame included in the application execution screen.

In this case, the captured image may be an image corresponding to the last frame among the frames in the change region.

In this case, the animation message may include an animation type for determining the type of the animation.

In this case, the animation type may be any one of flying up, brightness change, rising, and wiping.

In this case, the animation message may include the animation information corresponding to at least one of an animation type, animation application time, animation repetition number, start coordinates, end coordinates, start size, and end size.

In addition, the image cloud streaming service method using the drawing layer separation according to the present invention comprises the steps of: receiving a still image encoded captured image and an animation message corresponding to an application execution screen from a cloud streaming server; generating the application execution screen including an animation intermediate screen and an animation end screen by using the animation information included in the animation message and the captured image; and displaying the animation intermediate screen and the animation ending screen through different screen layers.

In this case, the screen layer is divided into a top layer for displaying the animation middle screen and a base layer for displaying the animation end screen, and the displaying is masked when animation playback starts The animation intermediate screen may be generated in the set base layer, and when the animation reproduction is finished, the masking setting may be canceled to display the animation intermediate screen.

In this case, the captured image may be an image obtained by capturing a change region of the current frame changed from the previous frame with respect to the previous frame and the current frame included in the application execution screen.

In this case, the captured image may be an image corresponding to the last frame among the frames in the change region.

In this case, the animation message may include an animation type for determining the type of the animation.

In this case, the animation type may be any one of flying up, brightness change, rising, and wiping.

In this case, the animation message may include the animation information corresponding to at least one of an animation type, animation application time, animation repetition number, start coordinates, end coordinates, start size, and end size.

According to the present invention, when a cloud streaming server captures only a partial area of the web application execution screen, generates a message having animation information to be applied to the corresponding area, and transmits the captured image and message to the set-top box, the set-top box uses the message By animating the captured image, it is possible to reduce unnecessary animation processing and increase the number of web containers that can be run on the server.

In addition, the present invention, when the cloud streaming server transmits the captured image and message to the set-top box, by using the message in the set-top box to apply animation to the captured image, and display the animation middle screen and the end screen in layers, animation playback It is possible to provide a more stable streaming service that returns to the normal ending screen even when an error occurs.

1 is a block diagram showing a cloud streaming service system according to an embodiment of the present invention.
2 is a block diagram illustrating an example of the terminal device shown in FIG. 1 of the present invention.
3 is a diagram illustrating a process of performing a cloud streaming service based on a still image according to an embodiment of the present invention.
4 is a diagram illustrating a method of displaying a web application execution screen by separating a layer according to an embodiment of the present invention.
5 is a flowchart illustrating an image cloud streaming service method using drawing layer separation according to an embodiment of the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, detailed descriptions of well-known functions or configurations that may obscure the gist of the present invention in the following description and accompanying drawings will be omitted. Also, it should be noted that throughout the drawings, the same components are denoted by the same reference numerals as much as possible.

The terms or words used in the present specification and claims described below should not be construed as being limited to their ordinary or dictionary meanings, and the inventor is appropriate as a concept of terms for describing his invention in the best way. It should be interpreted as meaning and concept consistent with the technical idea of the present invention based on the principle that it can be defined in Accordingly, the embodiments described in this specification and the configurations shown in the drawings are only the most preferred embodiment of the present invention, and do not represent all of the technical spirit of the present invention. It should be understood that there may be equivalents and variations. In addition, terms such as first, second, etc. are used to describe various components, and are used only for the purpose of distinguishing one component from other components, and are not used to limit the components.

1 is a block diagram showing a cloud streaming service system according to an embodiment of the present invention.

Referring to FIG. 1 , the cloud streaming service system according to an embodiment of the present invention includes a cloud streaming server 110 , terminal devices 120-1, ..., 120-N, and a network 130 .

Cloud streaming server 110 receives the application result execution screen from the web application server.

In addition, the cloud streaming server 110 determines a change region of the current frame changed from the previous frame by comparing the current frame with the previous frame included in the web application execution screen. In this case, the change area may be a menu area overlappingly displayed on the main screen among the web application execution screens.

In addition, the cloud streaming server 110 generates an animation message including animation information applied to the change area. In this case, the animation message may include an animation type for determining the type of animation. For example, the animation message may include an animation type corresponding to any one of flying up, brightness change, rising, and wiping. Also, the animation message may include animation information corresponding to at least one of an animation type, an animation application time, an animation repetition number, start coordinates, end coordinates, a start size, and an end size.

In addition, the cloud streaming server 110 generates a captured image from any one of the frames in the change area. In this case, the captured image may be an image corresponding to the first frame or the last frame among the frames in the change region.

Also, the cloud streaming server 110 encodes the captured image as a still image using a still image compression technique. In this case, the captured image may be encoded as a still image by using a still image compression technique corresponding to any one of portable network graphics (PNG), paletted portable network graphics (PNG), and joint photography experts group (JPEG).

In addition, the cloud streaming server 110 transmits the still image-encoded captured image and animation message to the user terminal so as to generate a web application execution screen in the terminal devices 120-1, ..., 120-N, and transmits the still image to the user terminal. based cloud streaming service.

The terminal devices 120-1, ..., 120-N receive a still image-encoded captured image and animation message corresponding to an application execution screen from the cloud streaming server 110 .

In addition, the terminal devices 120-1, ..., 120-N generate an application execution screen including an animation intermediate screen and an animation end screen by using the animation information and the captured image included in the animation message.

Also, the terminal devices 120-1, ..., 120-N display the animation intermediate screen and the animation ending screen through different screen layers. In this case, the screen layer is divided into a top layer for displaying an animation middle screen and a base layer for displaying an animation end screen. At this time, the terminal devices 120-1, ..., 120-N start animation playback by generating and displaying the animation intermediate screen on the top layer, and separately generate the animation intermediate screen on the base layer masked separately, and perform the animation. When playback is finished, the animation intermediate screen is displayed by releasing the masking setting. As a result, even if an error occurs during animation playback, it is possible to return to the normal end screen.

Each of the terminal devices 120-1, ..., 120-N is connected to a communication network and is a device capable of executing an application based on a cloud computing system, and is not limited to a mobile communication terminal, and all information and communication devices, multimedia terminals, and wired It may be various terminals such as a terminal, a fixed terminal, and an IP (Internet Protocol) terminal. In addition, the terminal devices 120-1, ..., 120-N are, respectively, a mobile phone, a Portable Multimedia Played (PMP), a Mobile Internet Device (MID), a smart phone, a desktop, and a tablet computer. PC), a notebook (Notebook), a netbook (Net Book), a personal digital assistant (PDA), may be a mobile (Mobile) terminal having various mobile communication specifications, such as a smart TV and information communication device.

The web application server is a server that receives and processes requests from the terminal devices 120-1, ..., 120-N. Rather than processing all the requests received from the terminal devices 120-1, ..., 120-N in the web application server, since the functions of the web application server can be structurally separated and processed through web containers, the terminal devices ( As the number of service requests from 120-1, ..., 120-N) increases, a service method using a web application server may be more effective. For example, BEA's WEB LOGIC, IBM's Web Spere, T-max's JEUS, Tomcat, RedHot's Jboss, etc. may correspond to the web application server.

This application server may transmit a web application execution screen to the cloud streaming server 110 based on the service request received from the terminal devices 120-1, ..., 120-N.

The network 130 provides a path for transferring data between the cloud streaming server 110 and the terminal devices 120-1, ..., 120-N, and includes both an existing network and a network that can be developed in the future. It is an inclusive concept. For example, the network 130 is a wired/wireless local area network that provides communication of various information devices within a limited area, a mobile communication network that provides communication between mobile devices and between the mobile body and the outside of the mobile body, and communication between the earth station and the earth station using satellites. It may be any one of a satellite communication network or a wired/wireless communication network that provides Meanwhile, the transmission method standard of the network 130 is not limited to the existing transmission method standard, and may include all transmission method standards to be developed in the future. In addition, in FIG. 1 , the network used between the cloud streaming server 110 and the terminal devices 120-1, ..., 120-N is used between the terminal devices 120-1, ..., 120-N. It may be different from the network, or it may be the same.

2 is a block diagram illustrating an example of the terminal device shown in FIG. 1 of the present invention.

Referring to FIG. 2 , the terminal device shown in FIG. 1 includes a receiving unit 210 , a screen generating unit 220 , a display unit 230 , and a storage unit 240 .

The receiver 210 receives the still image-encoded captured image and animation message corresponding to the application execution screen from the cloud streaming server. Such an application execution screen may include a main screen and a menu area superimposed on the main screen.

In this case, the animation message may include an animation type for determining the type of animation. For example, the animation message may include an animation type corresponding to any one of flying up, brightness change, rising, and wiping. Also, the animation message may include animation information corresponding to at least one of an animation type, an animation application time, an animation repetition number, start coordinates, end coordinates, a start size, and an end size. Also, the animation message may include an animation application time.

At this time, the start coordinate is the animation start coordinate of the video object to which the animation is applied, the end coordinate is the animation end coordinate of the video object to which the animation is applied, the start size is the animation start size of the video object to which the animation is applied, and the end size is the animation end size of the video object to which the animation is applied.

For example, the animation type is 'effect type', the animation application time is 'duration', the number of animation repetitions is 'isRepeat', the start coordinate is 'start X, start Y', the end coordinate is 'end X, end Y', The start size may be expressed as 'start W, start H' and the end size may be expressed as 'end W, end H'.

In this case, the captured image may be an image corresponding to the first frame or the last frame among the frames in the change region.

In this case, the captured image may be a still image-encoded image using a still image compression technique corresponding to any one of portable network graphics (PNG), paletted portable network graphics (PNG), and joint photography experts group (JPEG). .

The PNG still image compression technique has good image quality during encoding, but the transmission speed may be lowered when the bandwidth of the transmission network is low due to the large data size.

The paletted PNG still image compression technique reduces the data size of the image during encoding, so the transmission speed can be guaranteed, but since the color is expressed in 8 bits, it can cause deterioration in image quality.

Since the JPEG still image compression technique has good compression efficiency, the amount of transmitted data can be greatly reduced during encoding. However, since the compression efficiency is high, it can generate a lot of system load when encoding and decoding.

The screen generator 220 generates an application execution screen including an animation intermediate screen and an animation end screen by using the animation information and the captured image included in the animation message. For example, the application execution screen may include a main screen and a menu area superimposed on the main screen. When the menu area is displayed as an animation, the animation in which the menu area is gradually displayed on the main screen is displayed in the middle of the animation. A screen may be generated as a screen, and a screen in which the menu area is completely displayed on the main screen may be generated as an animation end screen for the menu area.

The display unit 230 displays the animation intermediate screen and the animation ending screen through different screen layers. In this case, the screen layer is divided into a top layer for displaying an animation middle screen and a base layer for displaying an animation end screen. At this time, the display unit 230 starts animation playback by creating and displaying an animation intermediate screen on the top layer, creates an animation intermediate screen on a base layer that has been masked separately, and cancels the masking setting when animation playback is finished. The middle screen will be displayed. As a result, even if an error occurs during animation playback, it is possible to return to the normal end screen.

In addition, the cloud streaming server can completely create and stream the application result screen, but by transmitting to generate the application result screen in the user terminal in the same way as above, the load generated in the cloud streaming server can be significantly reduced. As such, when the load of the cloud streaming server is reduced, the number of services that can be simultaneously processed by the cloud streaming server increases, so that more users can receive the service at the same time.

The storage unit 240 stores various information generated in the process of the cloud streaming service according to the embodiment of the present invention as described above.

According to an embodiment, the storage unit 280 may be configured independently of the terminal device 120-1 to support a function for a cloud streaming service. In this case, the storage unit 280 may operate as a separate mass storage and may include a control function for performing an operation.

Meanwhile, the terminal device 120 - 1 may have a memory mounted therein to store information in the device. For one implementation, the memory is a computer-readable medium. In one implementation, the memory may be a volatile memory unit, and in another implementation, the memory may be a non-volatile memory unit. In one embodiment, the storage device is a computer-readable medium. In various different implementations, the storage device may include, for example, a hard disk device, an optical disk device, or some other mass storage device.

3 is a diagram illustrating a process of performing a cloud streaming service based on a still image according to an embodiment of the present invention.

Referring to FIG. 3 , the process of performing a cloud streaming service based on a still image according to an embodiment of the present invention can be described by classifying it into a cloud streaming server area 300 , a terminal area 305 , and an application server area 310 . have.

First, in the cloud streaming server area 300 , based on a user's request, a web application execution screen generated by executing the web application server may be received 320 .

At this time, the cloud streaming server area 300 may detect a change area on the web application execution screen, generate a captured image for the change area, or generate an animation message (340).

In addition, in the cloud streaming server area 300 , the captured image and the animation message on which the encoding 330 is performed may be streamed 350 together to the terminal area 305 .

In the terminal area 305 , a web application execution screen may be generated using the captured image and the animation message received from the cloud streaming server area 300 . At this time, since the animation information includes information such as animation type, animation application time, animation repetition number, start coordinates, end coordinates, start size, and end size, it is possible to accurately and easily apply animation to the captured image and execute the web application. You can create a screen.

In this case, the terminal area 305 may generate a web application execution screen including an animation intermediate screen and an animation end screen by using the animation information and the captured image. For example, the web application execution screen may include a main screen and a menu area superimposed on the main screen. When the menu area is displayed as an animation, an animation in which the menu area is gradually displayed on the main screen is animated. An intermediate screen may be generated, and a screen in which the menu area is completely displayed on the main screen may be generated as an animation end screen for the menu area.

In the terminal area 305 , the web application execution screen generated as described above may be output to the user by performing image rendering 370 .

In this case, the terminal area 305 displays the animation intermediate screen and the animation end screen through different screen layers. In this case, the screen layer is divided into a top layer for displaying an animation middle screen and a base layer for displaying an animation end screen. At this time, the terminal area 305 starts animation playback by creating and displaying an animation intermediate screen on the top layer, creates an animation intermediate screen on a base layer that has been masked separately, and cancels the masking setting when animation playback is finished. An animation middle screen will be displayed.

The application server area 310 receives the user's input signal, and transmits an output screen corresponding to the input signal to the cloud streaming server.

4 is a diagram illustrating a method of displaying a web application execution screen by separating a layer according to an embodiment of the present invention.

The method of displaying the web application execution screen by separating the layers of the present invention is a method of displaying the application execution screen by dividing the application execution screen in a top layer and a base layer. More specifically, an animation intermediate screen among application execution screens is generated in the top layer, and an animation end screen among application execution screens is generated in the base layer. In this case, each layer may not be displayed to the user in the masking setting state, but may be displayed to the user in the masking unmasking state.

For example, the top layer may not display the animation intermediate screen to the user by maintaining the unmasking state until just before the end of the animation to display the animation intermediate screen to the user, and then switch to the masking setting state at the moment the animation ends. At this time, the base layer maintains the masking setting state until just before the end of the animation, and does not display the animation end screen generated in advance to the user, but switches to the unmasking state only at the animation end moment to display the animation end screen to the user. .

Referring to FIG. 4 , it can be seen that the method of displaying the web application execution screen by separating the layers is explained using an example of an animation in which the menu screens 425 , 435 , 445 are slid from the right end of the full screen. .

At this time, like the screens 410, 420, and 430 displayed on the top layer of FIG. 4, if there is a delay in the animation due to an unexpected error, even when the animation ends, the menu screen is displayed on the top layer. (425, 435) may not all be displayed. In this case, like the screen 440 displayed on the base layer, when the animation ends, the base layer is displayed instead of the top layer to completely display the menu screen 445 .

In this way, according to the present invention, it is possible to provide a more stable streaming service by returning to the normal ending screen even if an error occurs during animation playback.

5 is a flowchart illustrating an image cloud streaming service method using drawing layer separation according to an embodiment of the present invention.

5 , the image cloud streaming service method using drawing layer separation according to an embodiment of the present invention receives a still image encoded captured image and an animation message corresponding to an application execution screen from a cloud streaming server. do (S510).

In addition, the image cloud streaming service method using drawing layer separation according to an embodiment of the present invention generates an application execution screen including an animation intermediate screen and an animation end screen using animation information and a captured image included in an animation message. (S520).

In addition, the image cloud streaming service method using drawing layer separation according to an embodiment of the present invention displays an animation intermediate screen and an animation end screen through different screen layers ( S530 ).

The image cloud streaming service method using drawing layer separation according to the present invention may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - Includes magneto-optical media, and any form of hardware device specially configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions may include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. Such hardware devices may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

As described above, in the cloud streaming service system according to the present invention, the image cloud streaming service method using the drawing layer separation, and the apparatus for the same, the configuration and method of the embodiments described above are not limitedly applicable, but the above Embodiments may be configured by selectively combining all or part of each embodiment so that various modifications can be made.

According to the present invention, a captured image and an animation message are generated for an area where video codec-based cloud streaming is not appropriate and transmitted to the user, so that the user terminal can generate a web application execution result screen to which the animation is applied. Furthermore, by reducing unnecessary animation processing in the cloud streaming server, it is possible to increase the number of web containers that can be run in the cloud streaming server, thereby helping the convenience of the service provider.

110, 300: cloud streaming server 120-1~120-N: terminal device
130: network 210: receiver
220: screen generating unit 230: display unit
240: storage 305: user terminal
410, 420, 430: the screen displayed in the top layer
440: Screen displayed in the base layer
425, 435, 445: menu screen

Claims (16)

a receiver for receiving a still image-encoded captured image and animation message corresponding to the application execution screen from the cloud streaming server;
a screen generating unit for generating the application execution screen including an animation intermediate screen and an animation end screen by using the animation information included in the animation message and the captured image; and
A display unit for displaying the animation intermediate screen and the animation ending screen through different screen layers
including,
The animation information is
Corresponds to at least one of an animation type, an animation application time, the number of animation repetitions, a start coordinate, an end coordinate, a start size, and an end size,
The screen layer is
It is divided into a top layer for displaying the animation middle screen and a base layer for displaying the animation end screen from the start of the animation to just before the end of the animation,
the display unit
Drawing layer separation, characterized in that when animation playback starts, the animation end screen is created on the base layer that is masked, and when the animation playback ends, the masking setting is canceled to display the animation end screen Image cloud streaming service device used. delete The method according to claim 1,
The captured image is
Image cloud streaming service apparatus using drawing layer separation, characterized in that it is an image captured by a change region of the current frame changed from the previous frame with respect to the previous frame and the current frame included in the application execution screen. delete delete delete delete Receiving a still image (still image) encoded captured image and animation message corresponding to the application execution screen from the cloud streaming server;
generating the application execution screen including an animation intermediate screen and an animation end screen by using the animation information included in the animation message and the captured image; and
Displaying the animation intermediate screen and the animation ending screen through different screen layers
including,
The animation information is
Corresponds to at least one of an animation type, an animation application time, the number of animation repetitions, a start coordinate, an end coordinate, a start size, and an end size,
The screen layer is
It is divided into a top layer for displaying the animation middle screen and a base layer for displaying the animation end screen from the start of the animation to just before the end of the animation,
The step of displaying
Drawing layer separation, characterized in that when animation playback starts, the animation end screen is created on the base layer that is masked, and when the animation playback ends, the masking setting is canceled to display the animation end screen How to use image cloud streaming service. delete delete delete delete delete delete delete Receives an application execution screen from the application server, compares a previous frame included in the application execution screen with a current frame to determine a change region of the current frame changed from the previous frame, A captured image is generated, an animation message including animation information applied to the change region is generated, the captured image is still image-encoded using a still image compression technique, and the still image-encoded captured image and the animation message are transmitted to the user terminal. Cloud streaming server that transmits to; and
A user terminal that generates an animation intermediate screen and an animation end screen by using the animation information included in the animation message and the captured image, and displays the animation intermediate screen and the animation end screen through different screen layers
including,
The animation information is
Corresponds to at least one of an animation type, an animation application time, the number of animation repetitions, a start coordinate, an end coordinate, a start size, and an end size,
The screen layer is
It is divided into a top layer for displaying the animation middle screen and a base layer for displaying the animation end screen from the start of the animation to just before the end of the animation,
The user terminal is
Drawing layer separation, characterized in that when animation playback starts, the animation end screen is created on the base layer that is masked, and when the animation playback ends, the masking setting is canceled to display the animation end screen Image cloud streaming service system used.

Images