KR20210027342A - System for cloud streaming service, method of message-based image cloud streaming service and apparatus for the same - Google Patents

Abstract

Disclosed are a cloud streaming service system, a message-based image cloud streaming service method, and a device for the same. A cloud streaming server detects a change area from a web application execution screen to generate an animation message and a captured image. When the cloud streaming server transmits the still image-encoded captured image and animation message to a terminal device of a user using a still image compression technique, the terminal device can perform a still image-based cloud streaming service by applying animation to the captured image, thereby reducing unnecessary animation processing in the cloud streaming server and increasing the number of web containers which can be driven in the server.

Description

Cloud streaming service system, message-based image cloud streaming service method and device therefor {SYSTEM FOR CLOUD STREAMING SERVICE, METHOD OF MESSAGE-BASED IMAGE CLOUD STREAMING SERVICE AND APPARATUS FOR THE SAME}

The present invention relates to a message-based image cloud streaming service technology, and in particular, a web application execution result screen to which animation is applied in a user terminal by generating and transmitting a captured image and an animation message to a user for an area where cloud streaming based on a video codec is not appropriate. It relates to a cloud streaming service system for generating a message, a message-based image cloud streaming service method, and an apparatus therefor.

Due to the recent development of the Internet, the speed of communication using personal mobile communication devices has rapidly improved, and based on this, an Internet environment such as downloading and uploading large amounts of data through a remote computer connection, or local login to a remote computer is created. Became. In addition, as applications driven in a mobile communication terminal such as a smart phone have been developed in various ways, many virtualization technologies for driving applications requiring high performance in a user terminal having a relatively low performance have been proposed. Among them, the application is run on the server, the driving screen is compressed through video encoding and transmitted to the client, and the client plays the transmitted video to create the same effect as running the application on its own terminal based on screen virtualization. The cloud streaming service of is in the spotlight.

The cloud streaming service like this basically applies a cloud streaming technique based on a video codec, but even when a service such as menu display is provided, the entire screen is unnecessarily captured and operated using the video codec, resulting in an inefficient aspect of the system as a whole. exist.

Therefore, it creates a capture image for a part that needs dynamic effects such as animation among the execution result screens of the application executed by the user's request, and transmits the captured image and information related to the animation to the user's set-top box. A technology that can create a service screen to which the effect is applied is urgently emerging.

Korean Patent Publication No. 10-2012-0096909, published on August 31, 2012 (Name: Gateway/set-top box image merge for delivery to serviced client devices)

An object of the present invention is to capture only a partial area of a web application execution screen in a cloud streaming server, generate a message with animation information to be applied to the corresponding area, and transmit the captured image and message to the set-top box, the set-top box uses the message. Therefore, by applying animation to the captured image, unnecessary animation processing is reduced and the number of web containers that can be run on the server is increased.

A cloud streaming server according to the present invention for achieving the above object comprises: a screen receiving unit for receiving a web application execution screen from a web application server; A change area detection unit comparing a previous frame included in the web application execution screen with a current frame to determine a change area of the current frame changed from the previous frame; An animation message generator that generates an animation message including animation information applied to the change area; A capture unit generating a captured image from one of the frames in the change area; An encoding unit encoding the captured image as a still image using a still image compression technique; And a sending unit to perform a still image-based cloud streaming service by transmitting the animation message to the user terminal so that the user terminal can generate the web application execution screen.

In this case, the sending unit transmits the animation message and the still image encoded captured image to the user terminal so that the user terminal applies the animation information included in the animation message to the captured image to generate the web application execution screen. Can be transferred to.

In this case, the change area may be a menu area overlappingly displayed on the main screen among the web application execution screens.

In this case, the captured image may be an image corresponding to the last frame among the frames in the change area.

In this case, the animation message may include an animation type for determining the type of animation.

In this case, the animation type may be any one of flying, changing brightness, rising, and wiping.

In this case, the animation message may include the animation information corresponding to at least one of an animation type, an animation application time, an animation repetition number, a start coordinate, an end coordinate, a start size, and an end size.

In addition, the message-based image cloud streaming service method according to the present invention includes: receiving a web application execution screen from a web application server; Determining a change area of the current frame changed from the previous frame by comparing the previous frame and the current frame included in the web application execution screen; Generating an animation message including animation information applied to the change area; Generating a captured image from one of the frames in the change area; Encoding the captured image as a still image using a still image compression technique; And performing a still image-based cloud streaming service by transmitting the animation message to the user terminal so that the user terminal can generate the web application execution screen.

In this case, the performing of the cloud streaming service includes the animation message and the still image encoded so that the user terminal applies the animation information included in the animation message to the captured image to generate the web application execution screen. The captured image can be transmitted to the user terminal.

In this case, the change area may be a menu area overlappingly displayed on the main screen among the web application execution screens.

In this case, the captured image may be an image corresponding to the last frame among the frames in the change area.

In this case, the animation message may include an animation type for determining the type of animation.

In this case, the animation type may be any one of flying, changing brightness, rising, and wiping.

In this case, the animation message may include the animation information corresponding to at least one of an animation type, an animation application time, an animation repetition number, a start coordinate, an end coordinate, a start size, and an end size.

According to the present invention, when a cloud streaming server captures only a partial area of the web application execution screen, generates a message with animation information to be applied to the corresponding area, and transmits the captured image and message to the set-top box, the set-top box uses the message. By applying animation to the captured image, it is possible to increase the number of web containers that can be run on the server by reducing unnecessary animation processing.

1 is a block diagram showing a cloud streaming service system according to an embodiment of the present invention.
FIG. 2 is a block diagram illustrating an example of a cloud streaming server shown in FIG. 1.
3 is a diagram illustrating a process of performing a still image-based cloud streaming service according to an embodiment of the present invention.
4A to 4C are diagrams illustrating a web application execution screen according to an embodiment of the present invention.
5 is a flowchart illustrating a message-based image cloud streaming service method according to an embodiment of the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, in the following description and the accompanying drawings, detailed descriptions of known functions or configurations that may obscure the subject matter of the present invention will be omitted. In addition, it should be noted that the same components are denoted by the same reference numerals as much as possible throughout the drawings.

The terms or words used in the present specification and claims described below should not be construed as being limited to their usual or dictionary meanings, and the inventors are appropriate as the concept of terms for describing their own invention in the best way. It should be interpreted as a meaning and concept consistent with the technical idea of the present invention on the basis of the principle that it can be defined. Therefore, the embodiments described in the present specification and the configurations shown in the drawings are only the most preferred embodiments of the present invention, and do not represent all of the technical spirit of the present invention. It should be understood that there may be equivalents and variations. In addition, terms such as first and second are used to describe various elements, and are only used for the purpose of distinguishing one element from other elements, and are not used to limit the elements.

1 is a block diagram showing a cloud streaming service system according to an embodiment of the present invention.

Referring to FIG. 1, a cloud streaming service system according to an embodiment of the present invention includes a web application server 105, a cloud streaming server 110, terminal devices 120-1, ..., 120-N, and a network 130. ).

The cloud streaming server 110 receives an application result execution screen from the web application server 105.

* In addition, the cloud streaming server 110 determines a change area of the current frame changed from the previous frame by comparing the previous frame and the current frame included in the web application execution screen. In this case, the change area may be a menu area overlappingly displayed on the main screen among the web application execution screens.

In addition, the cloud streaming server 110 generates an animation message including animation information applied to the change area. In this case, the animation message may include an animation type for determining the type of animation. For example, the animation message may include an animation type corresponding to any one of flying, changing brightness, rising, and wiping. In addition, the animation message may include animation information corresponding to at least one of an animation type, an animation application time, an animation repetition number, a start coordinate, an end coordinate, a start size, and an end size.

In addition, the cloud streaming server 110 generates a captured image from any one of the frames within the change area. In this case, the captured image may be an image corresponding to the first frame or the last frame among the frames in the change area.

In addition, the cloud streaming server 110 encodes a still image using a still image compression technique. In this case, the captured image may be encoded using a still image compression technique corresponding to any one of portable network graphics (PNG), portable network graphics (PNG), and joint photography experts group (JPEG).

In addition, the cloud streaming server 110 transmits a still image encoded captured image and animation message to the user terminal so that the web application execution screen can be generated by the terminal devices 120-1, …, 120-N, and the still image Performs a cloud-based streaming service.

The terminal devices 120-1, …, 120-N receive an application execution result screen corresponding to the cloud streaming service from the cloud streaming server 110 and provide it to the user.

Terminal devices 120-1, …, 120-N are devices that are connected to a communication network and can execute applications based on a cloud computing system, and are not limited to mobile communication terminals, and all information and communication devices, multimedia terminals, and wired It may be various terminals such as a terminal, a fixed terminal, and an Internet Protocol (IP) terminal. In addition, the terminal devices 120-1, …, 120-N are mobile phones, portable multimedia played (PMP), mobile internet devices (MID), smart phones, desktops, and tablet computers, respectively. PC), a notebook (Note book), a net book (Net Book), a personal digital assistant (PDA), a smart TV, it may be a mobile terminal having various mobile communication specifications such as information communication device.

The web application server 105 is a server that receives and processes requests from the terminal devices 120-1, …, 120-N. Rather than processing all requests received from the terminal devices 120-1, …, 120-N in the web application server 105, the functions of the web application server 105 can be structurally separated and processed through web containers. Therefore, as the number of service requests from the terminal devices 120-1, …, 120-N increases, a method of providing a service using the web application server 105 may be more effective. For example, BEA's WEB LOGIC, IBM's Web Spere, T-max's JEUS, Tomcat, RedHot's Jboss, etc. may correspond to the web application server 105.

The application server 105 may transmit a web application execution screen to the cloud streaming server 110 based on a service request received from the terminal devices 120-1, …, 120-N.

The network 130 provides a path for transferring data between the cloud streaming server 110 and the terminal devices 120-1, …, 120-N, and provides both an existing network and a network that can be developed in the future. It is an inclusive concept. For example, the network 130 is a wired/wireless local area communication network that provides communication of various information devices within a limited area, a mobile communication network that provides communication between mobiles and the mobile and the outside of the mobile, and communicates between the earth station and the earth station using satellites. It may be a satellite communication network, a wired or wireless communication network, or a combination of two or more. Meanwhile, the transmission method standard of the network 130 is not limited to the existing transmission method standard, and may include all transmission method standards to be developed in the future. In addition, in FIG. 1, a network used between the cloud streaming server 110 and the terminal devices 120-1, …, 120-N is used between the terminal devices 120-1, …, and 120-N. It may be different from the network, or it may be the same.

FIG. 2 is a block diagram illustrating an example of a cloud streaming server shown in FIG. 1.

Referring to FIG. 2, the cloud streaming server illustrated in FIG. 1 includes a screen receiving unit 210, a change area detection unit 220, an animation message generation unit 230, a capture unit 250, an encoding unit 260, and a sending unit. It includes 270 and a storage unit 280.

The screen receiving unit 210 receives a web application execution screen from a web application server. Such a web application execution screen may include a main screen and a menu area that is superimposed and displayed on the main screen.

The change area detection unit 220 determines a change area of the current frame changed from the previous frame by comparing the previous frame and the current frame included in the web application execution screen. In this case, the change area may be a menu area overlappingly displayed on the main screen among the web application execution screens.

The animation message generator 230 generates an animation message including animation information applied to the change area. In this case, the animation message may include an animation type for determining the type of animation. For example, the animation message may include an animation type corresponding to any one of flying, changing brightness, rising, and wiping. In addition, the animation message may include animation information corresponding to at least one of an animation type, an animation application time, an animation repetition number, a start coordinate, an end coordinate, a start size, and an end size. In addition, the animation message may include an animation application time.

In this case, the start coordinate is the animation start coordinate of the video object to which the animation is applied, the end coordinate is the animation end coordinate of the video object to which the animation is applied, the start size is the animation start size of the video object to which the animation is applied, and the end size Is the animation end size of the video object to which the animation is applied.

For example, the animation type is'effect type', the animation application time is'duration', the animation repeat count is'isRepeat', the start coordinates are'start X, start Y', the end coordinates are'end X, end Y', The start size may be expressed as'start W, start H'and the end size may be expressed as'end W, end H'.

The capture unit 250 generates a captured image from any one of the frames within the change area. In this case, the captured image may be an image corresponding to the first frame or the last frame among the frames in the change area.

The encoding unit 260 encodes a still image using a still image compression technique. In this case, the captured image may be encoded using a still image compression technique corresponding to any one of portable network graphics (PNG), portable network graphics (PNG), and joint photography experts group (JPEG).

In the PNG still image compression technique, the image quality is good when encoding, but the transmission speed may decrease when the bandwidth of the transmission network is low due to an increase in the size of the data.

The palletized PNG still image compression technique reduces the data size of the image during encoding, so the transmission speed can be guaranteed, but since it expresses colors in 8 bits, it may deteriorate the image quality.

The JPEG still image compression technique has good compression efficiency and can greatly reduce the amount of transmitted data during encoding, but as the compression efficiency is high, it can generate a lot of system load when encoding and decoding are performed.

The sending unit 270 performs a still image-based cloud streaming service by transmitting a still image-encoded captured image and an animation message to the user terminal to generate a web application execution screen in the user terminal. For example, although the cloud streaming server 110 may perfectly generate and stream the application result screen, the load generated by the cloud streaming server 110 is transmitted to the user terminal to generate the application result screen in the same way as above. Can be significantly reduced. When the load on the cloud streaming server 110 is reduced in this way, the number of services that can be simultaneously processed by the cloud streaming server 110 increases, so that more users can receive the service at the same time.

As described above, the storage unit 280 stores various types of information generated in the process of a cloud streaming service according to an embodiment of the present invention.

According to an embodiment, the storage unit 280 may be configured independently from the cloud streaming server 110 to support a function for a cloud streaming service. In this case, the storage unit 280 may operate as a separate mass storage, and may include a control function for performing the operation.

In addition, the cloud streaming server 110 configured as described above may be implemented as one or more servers.

Meanwhile, the cloud streaming server 110 is equipped with a memory to store information in the device. In one implementation, the memory is a computer-readable medium. In one implementation, the memory may be a volatile memory unit, and in another implementation, the memory may be a non-volatile memory unit. In one implementation, the storage device is a computer-readable medium. In various different implementations, the storage device may include, for example, a hard disk device, an optical disk device, or some other mass storage device.

3 is a diagram illustrating a process of performing a still image-based cloud streaming service according to an embodiment of the present invention.

Referring to FIG. 3, a process of performing a still image-based cloud streaming service according to an embodiment of the present invention can be described by classifying it into a cloud streaming server area 300, a terminal area 305, and an application server area 310. have.

First, in the cloud streaming server area 300, a web application execution screen generated by executing in a web application server may be received (320) based on a user's request.

In this case, the cloud streaming server area 300 may detect a change area on the web application execution screen, generate a capture image for the change area, or generate an animation message (340).

In addition, in the cloud streaming server area 300, the captured image and the animation message performed by the encoding 330 may be streamed 350 to the terminal area 305 together.

In the terminal area 305, a web application execution screen may be generated by using the captured image and animation message received from the cloud streaming server area 300. At this time, animation information includes information such as animation type, animation application time, number of animation repetitions, start coordinates, end coordinates, start size and end size, so that the web application is executed by accurately and easily applying animation to the captured image. You can create a screen.

In the terminal area 305, the generated web application execution screen may be rendered to a user by performing an image rendering 370.

The application server area 310 receives a user's input signal and transmits an output screen corresponding to the input signal to the cloud streaming server.

4A to 4C are diagrams illustrating a web application execution screen according to an embodiment of the present invention.

Referring to FIGS. 4A to 4C, it can be seen that the web application execution screen according to an embodiment of the present invention shows a'raising' animation of a menu area in time series.

4A shows a state in which the menu area 420 does not appear on the main screen 410.

In response to a user's request, when a menu display command is executed in the web application server, the menu area 420 is overlapped from the right end of the main screen 410, and the screen is displayed as shown in FIG. 4A through FIG. 4B. Changes.

In this case, there may be a plurality of frames representing intermediate steps between FIGS. 4A and 4B or between FIGS. 4B and 4A. For example, there may be 10 frames representing intermediate steps between FIGS. 4A and 4B.

In this case, the cloud streaming server according to the present invention uses an image-based cloud streaming method instead of performing video codec-based streaming by capturing each of 10 frames representing these intermediate steps.

More specifically, the cloud streaming server according to the present invention captures only the image of the menu area 420 of FIG. 4C, generates an animation message including information for applying the'raise' animation to the image from the right side, and generates a set-top box. By transmitting to, the'raising' animation is applied to the image of the menu area 420 in the set-top box to generate an intermediate frame between FIGS. 4A and 4B.

5 is a flowchart illustrating a message-based image cloud streaming service method according to an embodiment of the present invention.

5, the message-based image cloud streaming service method according to an embodiment of the present invention receives a web application execution screen from a web application server (S510).

In addition, the message-based image cloud streaming service method according to an embodiment of the present invention determines a change area of the current frame changed from the previous frame by comparing the previous frame included in the web application execution screen with the current frame (S520).

In addition, the message-based image cloud streaming service method according to an embodiment of the present invention generates an animation message including animation information applied to a change area (S530).

In addition, the message-based image cloud streaming service method according to an embodiment of the present invention generates a captured image from any one of the frames in the change area (S550).

In addition, the message-based image cloud streaming service method according to an embodiment of the present invention encodes a still image using a still image compression technique (S560).

In addition, the message-based image cloud streaming service method according to an embodiment of the present invention performs a still image-based cloud streaming service by transmitting an animation message to the user terminal to generate a web application execution screen in the user terminal (S570). .

The message-based image cloud streaming service method according to the present invention may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the present invention, or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -Includes magneto-optical media, and any type of hardware device specially configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of the program instructions may include not only machine language codes such as those produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. Such a hardware device may be configured to operate as one or more software modules to perform the operation of the present invention, and vice versa.

As described above, the cloud streaming service system, the message-based image cloud streaming service method, and the apparatus therefor according to the present invention are not limited to the configuration and method of the embodiments described as described above, but the embodiments are All or part of each of the embodiments may be selectively combined and configured so that various modifications may be made.

According to the present invention, a captured image and an animation message are generated for an area where cloud streaming based on a video codec is not appropriate, and transmitted to the user, thereby allowing the user terminal to generate an animation-applied web application execution result screen. Furthermore, by reducing unnecessary animation processing in the cloud streaming server, the number of web containers that can be run in the cloud streaming server can be increased, thereby helping the service provider's convenience.

105, 310: web application server 110, 300: cloud streaming server
120-1~120-N: terminal device 130: network
210: screen receiving unit 220: change area detection unit
230: animation message generation unit 250: capture unit
260: encoding unit 270: sending unit
280: storage unit 305: user terminal
410: main screen 420: menu area

Claims (7)

A screen receiving unit receiving a web application execution screen from a web application server;
An animation message generator that generates an animation message including animation information corresponding to frames included in the web application execution screen;
A capture unit generating a captured image corresponding to the frames;
An encoding unit encoding the captured image as a still image using a still image compression technique; And
A sending unit that transmits the animation message to the user terminal so that the user terminal can generate an animation-applied execution screen
Cloud streaming server comprising a. The method according to claim 1,
The sending unit
The animation message and the still image encoded captured image are transmitted to the user terminal so that the user terminal applies the animation information included in the animation message to the captured image to generate the web application execution screen. Cloud streaming server. The method according to claim 2,
The captured image is
Cloud streaming server, characterized in that the image corresponding to the last frame among the frames corresponding to the animation. The method according to claim 1,
The animation message is
A cloud streaming server comprising an animation type for determining the type of animation. The method of claim 4,
The animation type above is
Cloud streaming server, characterized in that any one of flying, changing brightness, rising and wiping. The method according to claim 1,
The animation message is
A cloud streaming server comprising the animation information corresponding to at least one of an animation type, an animation application time, an animation repetition number, a start coordinate, an end coordinate, a start size, and an end size. Receiving a web application execution screen from a web application server;
Generating an animation message including animation information corresponding to frames included in the web application execution screen;
Generating a captured image corresponding to the frames;
Encoding the captured image as a still image using a still image compression technique; And
Transmitting the animation message to the user terminal so that the user terminal can generate an animation-applied execution screen
Message-based image cloud streaming service method comprising a.

Images