KR102003044B1 - Method and apparatus for improving QoE in sharing screen between plural devices and recording medium thereof - Google Patents

Abstract

The present invention relates to a technology for improving QoE when a screen is shared between a plurality of terminals, and it relates to a method for determining a QoE policy suitable for a content type determined by discriminating a content type displayed on a current screen, The frame rate and the resolution are adjusted to encode the screen, so that a plurality of terminals can maintain an optimal quality of view per content type when sharing a screen.

Description

[0001] The present invention relates to a method, apparatus, and recording medium for improving QoE (Quality of Experience) in screen sharing between a plurality of terminals,

The present invention relates to a method and apparatus for sharing screen data between a plurality of terminals via a network.

A bit rate of a stream to be transmitted is determined in consideration of a network state when a screen is shared between a plurality of terminals. It is possible to transmit a stream having a high bit rate when the network state is smooth (unloaded) and a stream having a low bit rate when the network state is not smooth (loaded or congested).

However, since the conventional method considers only the network state, there is a problem that the screen is disconnected or the resolution is lowered when the screen is shared.

The present invention provides a method and apparatus for improving QoE for transmitting screen data in consideration of a type of content displayed on a current screen when a screen is shared between a plurality of terminals.

According to an embodiment of the present invention, there is provided a method of improving QoE in a terminal sharing a screen with another terminal, the method comprising: determining a type of content displayed on a current screen; Determining a QoE policy according to the determination result; Encoding an image of the screen according to the determined QoE policy; And transmitting the encoded screen image to the another terminal.

In the method for enhancing QoE, the encoding step is preferably performed using a frame rate and resolution adjusted according to the determined QoE policy.

In the method for improving QoE, the content type preferably includes graphic contents, real-life contents, and text contents.

In the method for improving QoE, it is preferable that the QoE policy is set to have a higher frame rate as the change in the movement of the content is greater.

In the method for improving QoE, it is preferable that the step of determining the type of content displayed on the screen is performed based on the name of the application being executed.

In the method for improving QoE, the determining step may include selecting a QoE policy corresponding to a content type having the highest priority according to a preset priority for each content type when a plurality of types of content displayed on the current screen are selected desirable.

In the method for improving QoE, the adjusting step may include: determining an allowable bit rate based on a network state between terminals sharing the screen; And adjusting a frame rate and a resolution applied to the encoding of the shared picture according to the bit rate determination result.

Another embodiment of the present invention includes a memory in which at least one program is stored; And a processor for enhancing QoE in a terminal sharing the screen with another terminal by executing the at least one program, wherein the at least one program includes: determining a type of an application displayed on a current screen; Determining a QoE policy according to the determination result; Encoding an image of the screen according to the determined QoE policy; And transmitting the encoded screen image to the other terminal.

In the transmitting apparatus, the encoding step is preferably performed using a frame rate and a resolution adjusted according to the determined QoE policy.

In the transmitting apparatus, the content type may include graphic content, actual content, and text content.

In the transmitting apparatus, the QoE policy is preferably set to have a higher frame rate as the movement of the content changes more.

The step of determining the type of the content displayed on the screen in the transmitting apparatus may be performed based on the name of the application being executed.

Wherein the determining step comprises a step of determining whether a content type of the content displayed on the current screen is a QoE corresponding to a content type having a highest priority according to a preset priority for the graphic content, It is desirable to determine the policy.

Wherein the adjusting step comprises: determining an allowable bit rate based on a network state between terminals sharing the screen; And adjusting a frame rate and a resolution applied to the encoding of the shared picture according to the bit rate determination result.

Another embodiment of the present invention provides a computer-readable recording medium on which a computer program for performing the method for improving QoE is recorded.

1 is a conceptual diagram for explaining screen sharing among a plurality of terminals according to an embodiment of the present invention.
2 is a flowchart illustrating a method of improving QoE according to a content type according to an embodiment of the present invention.
3 is a view for explaining a method for determining a content type according to an embodiment of the present invention.
FIG. 4 is a table showing a QoE policy according to content type according to an embodiment of the present invention.
5 is a diagram illustrating a screen displayed when a plurality of content types exist in a screen according to an embodiment of the present invention.
FIG. 6 is a flowchart illustrating a method for improving QoE by analyzing a network state according to an exemplary embodiment of the present invention. Referring to FIG.
7 is a diagram illustrating a structure of a UE capable of operating as a transmitter for improving QoE according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. The following examples of the present invention are intended only to illustrate the present invention and do not limit or limit the scope of the present invention. It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims.

1 is a conceptual diagram for explaining screen sharing among a plurality of terminals according to an embodiment of the present invention. Hereinafter, the transmitting terminal 102 is a terminal for transmitting a screen image to be shared when a screen is shared, and the receiving terminal 106 is a terminal for receiving a screen image to be shared. A terminal according to an embodiment of the present invention includes a PC, a smart phone, a tablet, and the like.

 The transmitting terminal 102 transmits the screen image currently displayed through the network to the receiving terminal 106. For example, when a photo album application is being executed in the transmitting terminal 102, a list of photos may be displayed as a screen image 104 of the transmitting terminal 102 at present. When the screen is shared, the transmitting terminal 102 can capture the current screen image 104 and transmit it to the receiving terminal 106 via the network.

When the receiving terminal 106 receives the captured screen image 104, the receiving terminal 106 shares the same screen image 108 as the transmitting terminal 102. [

 Meanwhile, in transmitting the screen image 104 from the transmitting terminal 102, it is necessary to consider the network state. This is because the allowable bit rate in transmission of the screen image 104 is determined according to the network conditions. It is also necessary to consider QoE (quality of experience) along with the network status. QoE means the total tolerance of the application or service perceived by the service user.

 For example, in the case of a map application, most screen images are frozen at the time of application execution or simply moved according to the scroll control. On the other hand, readability is a very important factor in map applications. Therefore, when the map application shares the screen on which the map application is executed, when the allowable bit rate is determined through the network state analysis, the resolution is set to a higher level than the frame rate within the determined bit rate.

 On the other hand, even if the allowable bit rate is determined to be the same value due to the same network status, when the characteristics of the application are as high as the game application, the frame rate is maintained at a higher level than the resolution within the determined bit rate.

 That is, even if the allowable bit rate values are the same, the resolution and the frame rate must be determined according to the characteristics of the application. Therefore, the resolution and the frame rate should be adjusted according to the characteristics to maintain the optimal QoE .

2 is a flowchart illustrating a method of improving QoE according to a content type according to an embodiment of the present invention. According to an embodiment of the present invention, improvement of QoE when a screen is shared between a plurality of terminals is performed in a transmitter.

 In step 202, the transmitting apparatus determines the type of contents of the screen image to be transmitted to the receiving terminal when the screen is shared.

The type of content can be classified into graphic content, real image content, and text content. In addition, the extent to which the screen image moves can also be a classification criterion of the content type. That is, even in the case of the same graphic content, the content type can be classified in detail according to the movement of the screen image is small and small. However, the content type in the present invention is not limited to this.

 An example of graphics content is a game application. Because the image of the game application is motion-intensive, the frame rate can be set to a high level in order to keep the QoE optimal.

On the other hand, if the main screen is running, the icons displayed on the main screen correspond to the graphic contents as in the game application, but in the case of the main screen, the screen image is mostly stopped or the movement is very simple. Therefore, the resolution can be determined to be higher with a lower frame rate than the game application.

On the other hand, camera applications and photo album applications are examples of actual content. In the case of the camera application, the motion of the screen image is large because it performs functions such as moving picture or photographing. On the other hand, in the case of photobook applications, there is not much movement. Therefore, the frame rate of the photo album application is determined to be higher than the frame rate of the photo album application.

For reference, the resolution of the actual content is generally set higher than the resolution of the graphic content.

An example of text content is an email application. For text content, readability is an important factor, so the resolution is determined to be higher than the frame rate.

According to an embodiment of the present invention, the type of content displayed on the current screen can be determined based on the name of the application. Here, the application name may correspond to all the identification information that can refer to the application by OS.

A method of determining a content type using an application name will be described with reference to FIG.

 Referring to FIG. 3, an instruction 302 is input to detect an application currently running in the terminal. When a command 302 for detecting a running application is inputted, a process of confirming the name of the application is performed according to the input. The information for detecting the running application in the confirmation process is not limited to the name of the application.

For example, information for detecting an application can be determined using header information in a bitstream. Specifically, after setting header information unique to each application in a program of the terminal in a memory, information matching with header information obtained by parsing a bitstream of a screen image at the time of screen sharing is retrieved from the memory. The matching application can be identified by the application displayed on the current screen.

In order to prevent unauthorized copying of the application, the application can be discriminated by using the watermarking information inserted for each frame. Even when watermarking information is used, if watermarking information unique to each application is set in advance, an application to be displayed on the current screen can be determined by searching matching applications.

 When the currently activated application is confirmed, it inputs an instruction 304 that determines which content type the application belongs to. According to an embodiment of the present invention, a list in which a type of an application and a content type are mapped is stored in a memory, and if a type of an application currently being executed is identified, a content type mapped in the list can be found.

The criteria for mapping the type of application and the content type are based on the characteristics such as the type of image such as graphic and real image, and the amount of motion information for each application.

 According to an embodiment of the present invention, whether the type of content displayed on the current screen is text content can be determined using a text detection algorithm. The text detection algorithm in the present invention is not specified, and various text detection algorithms can be used.

According to an embodiment of the present invention, whether the type of the content displayed on the current screen is real-world content or not can be determined by using whether or not the codec is used. For example, if it is confirmed that the video decoder is being used through the iomx and the live video is being reproduced on the current screen, the content type displayed on the current screen can be discriminated as the live video content.

According to an embodiment of the present invention, a content type can be determined using a motion vector in a currently displayed picture. After setting a range in advance and mapping a content type according to the setting range, if the motion vector of the current screen corresponds to the set range, the content type mapped to the setting range is detected to determine the content type displayed on the current screen have.

 The content type may be determined using a text detection algorithm together with the sum of motion vectors. After obtaining the ratio information that the text occupies in the screen using the text detection algorithm, the content type classification based on the sum of the motion vectors can be classified and classified according to the resolution.

Specifically, if the ratio occupied by the text in the screen is less than a predetermined value, the importance of the resolution is lowered. If the ratio is higher than the predetermined value, the importance of resolution is increased and the weight value can be set differently in determining the resolution accordingly.

In step 204, the transmitting apparatus determines the QoE policy according to the determination result of the content type. The QoE policy according to an embodiment of the present invention is to adjust the resolution and the frame rate according to the type of the content within the allowable bit rate range when the currently acceptable bit rate is determined as a result of the network analysis. Here, the determination of the allowable bit rate according to the network analysis will be described later with reference to FIG.

In step 206, the transmitting apparatus encodes the current screen according to the determined QoE policy. For example, when the application displayed on the current screen is a game, the user sets the frame rate to be high because the user perceives the deterioration of the image quality relatively with a large amount of motion and graphic contents.

In step 208, the transmitting apparatus transmits the encoded picture to the receiving apparatus. One or more receiving apparatuses according to an embodiment of the present invention may be one or more.

FIG. 4 is a table showing a QoE policy according to content type according to an embodiment of the present invention.

According to an embodiment of the present invention, when the content type displayed on the current screen is graphic content or real content, the resolution varies between VGA and QVGA, and the frame rate can be variably changed between 5 fps and 30 fps. Also, if the content type displayed on the current screen is text content, the resolution may vary between HD and VGA, and the frame rate may be variably changed at 1 fps and 15 fps.

The frame rate can be set to the highest frame rate of the graphic content with a small amount of motion due to data loss and the resolution of the text content with the highest readability can be set to the highest. Here, the setting of the frame rate and the resolution is not limited to the above setting.

For example, when the bit rate of the allowable stream is 100 kbits, the frame rate is 15 fps when the content type displayed on the current screen is graphic content, the frame rate is 10 fps when the content is real content, The frame rate can be set to 1 fps.

According to an embodiment of the present invention, since the allowable bit rate changes as the network status changes, the frame rate and resolution also change according to the changed bit rate. For example, if the bit rate increases, the frame rate increase rate may be set differently depending on the content type.

Specifically, the bit rate required for increasing the frame rate by 1 can be set to 10 kbits in the case of graphic contents, 15 kbits in case of real contents, and 30 kbits in case of text contents.

For example, when the bit rate of the allowable stream is 100 kbits, it is assumed that the frame rate is 15 fps in the case of graphic contents, 10 fps in case of real contents, and 1 fps in case of text contents. When the allowable bit rate increases to 280 kbits according to the change of the network state, the graphic content has a bit rate of 33 fps, the real content has a bit rate of 26 fps, and the text content has a bit rate of 7 fps.

According to an embodiment of the present invention, the content type displayed on the current screen can be subdivided based on the degree of motion. For example, the content type can be subdivided when the content is graphical but the movement is small or when the content is graphical and moving. Actual contents can also be subdivided into small and large movements. In the case of the text content, since there is no motion or is very small, it is not separately classified on the basis of motion in the present embodiment.

The QoE policy according to the content type sets the frame rate to be high when the graphic content is small but the motion is small (GRAPHIC_SMALLMOTION), but the resolution can be set lower than the actual content or text. An application corresponding to a case where the graphic content is small in movement may be exemplified as a main screen of the terminal.

  In the case of graphic contents and large motion (GRAPHIC_LARGEMOTION), it is difficult to recognize the deterioration of image quality when the motion is large, so the frame rate can be maintained at a higher level than the resolution. An application corresponding to a case in which the graphic content is large in motion may be a game application.

If the application is REAL_SMALLMOTION, the application may be a gallery application. If the motion is small, it is mostly a stop or a simple motion. Therefore, the resolution can be set to a higher level than the frame rate.

If the motion is large but the motion is large (REAL_LARGEMOTION), the frame rate can be set to a higher level than the resolution because it is difficult to recognize the image deterioration relatively when the motion is large. At this time, the resolution is generally set to a higher level than the graphic content.

For text content (TEXT_SMALLMOTION), readability is important, so you can set the resolution to a higher level than the frame rate. Examples of applications corresponding to the text content include an e-mail application, a web page, and a map application.

5 is a diagram illustrating a screen displayed when a plurality of content types exist in a screen according to an embodiment of the present invention. When there are a plurality of content types in the screen, the QoE policy corresponding to the content type having the highest priority can be determined according to a preset priority for the graphic content, the real content, and the text content.

For example, referring to FIG. 5, the A area 502 corresponds to actual content, the B area 504 corresponds to text content, and the C area 506 corresponds to graphic content. If the preset priority is the actual content, the graphic content, or the text content, the QoE policy suitable for the real content is determined, and the parameter applied to the encoding of the current screen is adjusted according to the QoE policy determined.

According to an embodiment of the present invention, when there are a plurality of content types in a screen, the QoE policy can be determined according to the ratio of content types existing in the screen. For example, referring to FIG. 5, since the C area 506, which is graphic content, occupies the largest portion in the screen, a QoE policy suitable for the graphics content is determined and parameters to be applied to the encoding of the current screen are adjusted according to the QoE policy determined .

According to an embodiment of the present invention, when a plurality of content types exist, the criteria for determining the QoE policy is not limited to a preset priority or a ratio of content types existing in the screen.

FIG. 6 is a flowchart illustrating a method for improving QoE by analyzing a network state according to an exemplary embodiment of the present invention. Referring to FIG.

In step 602, the transmitting apparatus analyzes a network state between a plurality of terminals sharing a screen. For example, RTCP (Realtime Transport Control Protocol) is used to analyze network status. RTCP performs network monitoring from data obtained by periodically transmitting control packets to terminals participating in stream transmission / reception on the network. In particular, it is used to collect information about a stream receiving end and calculate a receiving side transmission rate.

The network state can be classified into an unload state in which stream transmission and reception are smooth, a load state in which a load starts to occur in stream transmission and reception, and a congested state in which packet loss starts to occur.

According to an embodiment of the present invention, the network status analysis can be performed by analyzing the current network status and the latest n network statuses. Recently, n network states can be obtained by reading the network state stored in the buffer for a certain period of time.

In step 604, the transmitting apparatus determines an allowable bit rate for stream transmission based on the network state analysis result.

In step 606, the transmitting device adjusts the frame rate and resolution based on the bit rate determined in step 604. Specifically, when the allowable bit rate is determined according to the network state analysis, the QoE policy suitable for the type of the content displayed on the current screen can be determined based on the determined bit rate. And adjusts the frame rate and resolution based on the determined QoE policy.

For example, at time t, the transmitting terminal starts transmitting the image of the screen to be shared to the receiving terminal. Assume that the network state at time t is in the load state and the allowable bit rate is 100 kbps. The application displayed on the current screen corresponds to a case where the game application is graphic content and has a large movement.

Assume that the network state is unloaded and the allowable bit rate is 280 kbps after analyzing the current network state and n network states at the end of n network state analysis from the time t. As the allowable bit rate increases, the frame rate and resolution values used to encode the scene also change.

The application displayed on the current screen should be maintained at a higher frame rate than the resolution by the game application. Assuming that the bit rate required to increase the frame rate by 1 in the case of graphic contents is 10 kbits, the frame rate of the current screen image is increased by 18.

7 is a diagram illustrating a structure of a UE capable of operating as a transmitter for improving QoE according to an embodiment of the present invention. In Fig. 7, an example of a case where a transmitting apparatus for improving QoE is a smartphone is shown. The terminal of the present invention is not limited to a smart phone as described above, but includes various devices such as a PC and a tablet.

The mobile communication unit 701 performs call setup, data communication, and the like with the base station through a cellular network such as 3G / 4G. The sub communication unit 702 performs functions for local communication such as Bluetooth or NFC. The broadcast receiving unit 703 receives the DMB broadcasting signal.

The camera unit 704 includes lenses and optical elements for photographing or photographing.

The sensor unit 705 may include a gravity sensor for sensing movement of the terminal 700, an illuminance sensor for sensing the brightness of light, a proximity sensor for sensing proximity of a person, a motion sensor for sensing movement of a person, have.

The GPS receiver 706 receives GPS signals from satellites. Various services can be provided to the user using such GPS signals.

The input / output unit 710 provides an interface with an external device or a person and includes a button 711, a microphone 712, a speaker 713, a vibration motor 714, a connector 715, and a keypad 716 .

The touch screen 718 receives the user's touch input. Here, the touch input may be a drag gesture or a tap gesture. The touch screen controller 717 transmits the touch input inputted through the touch screen 718 to the controller 750. A power supply 719 is connected to a battery or an external power source to supply the power required by the multivision controller.

The control unit 750 executes the programs stored in the memory 760 to allow the terminal 700 to operate as a device for improving QoE according to the present invention.

The programs stored in the memory 760 can be classified into a plurality of modules according to their functions. The mobile communication module 761, the Wi-Fi module 762, the Bluetooth module 763, the DMB module 762, Module 765, a sensor module 766, a GPS module 767, a moving picture playback module 768, an audio playback module 769, a power module 770, a touch screen module 771, a UI module 772, A QoE control module 773, and the like.

The function of each module can be intuitively deduced from the name by a person skilled in the art. Therefore, only the QoE control module 773 will be described here. The QoE control module 773 controls the parameters applied to the encoding of the screen according to the type of the content displayed on the current screen.

That is, the QoE control module 773 includes instructions for determining a content type to which an application displayed on the current screen belongs, determining a QoE policy suitable for the determined content type, and determining a frame rate and resolution according to the determined QoE policy do.

According to an embodiment of the present invention, the determination of the application displayed on the current screen is performed based on the name of the application. The QoE control module 773 may include an extracting unit for extracting the name information of the application. If the name of the application displayed on the current screen is extracted by the extracting unit, it can be determined which content type the application belongs by using the list stored in the memory.

In addition, the QoE control module 773 analyzes a network state when a plurality of terminals share a screen through a network, determines a bitstream according to an analysis result, and determines a content type suitable for a content type displayed on the screen based on the determined bitstream Determine the frame rate and resolution.

One embodiment of the present invention may also be embodied in the form of a recording medium including instructions executable by a computer, such as program modules, being executed by a computer. Computer readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. In addition, the computer-readable medium may include both computer storage media and communication media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Communication media typically includes any information delivery media, including computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave, or other transport mechanism.

It will be understood by those skilled in the art that the foregoing description of the present invention is for illustrative purposes only and that those of ordinary skill in the art can readily understand that various changes and modifications may be made without departing from the spirit or essential characteristics of the present invention. will be. It is therefore to be understood that the above-described embodiments are illustrative in all aspects and not restrictive. For example, each component described as a single entity may be distributed and implemented, and components described as being distributed may also be implemented in a combined form.

The scope of the present invention is defined by the appended claims rather than the detailed description and all changes or modifications derived from the meaning and scope of the claims and their equivalents are to be construed as being included within the scope of the present invention do.

102: transmitting terminal
104: current screen image
106: receiving terminal
108: Same screen image as the transmitting terminal

Claims (15)

A method for improving QoE in a terminal sharing a screen with another terminal,
Determining a type of content displayed on the screen;
Determining a QoE policy related to a frame rate and a resolution based on a ratio of each content type when a plurality of types of content are displayed on the screen as a result of the determination;
Encoding an image of the screen according to the determined QoE policy; And
And transmitting the encoded screen image to the another terminal. 2. The method of claim 1, wherein the encoding step is performed using a frame rate and a resolution that are adjusted according to the determined QoE policy. The method of claim 1, wherein the type of the content includes graphic content, real content, and text content. The method of claim 1, wherein the QoE policy is set such that the higher the frame rate is, the more the movement of the content changes. The method of claim 1, wherein the step of determining the type of content displayed on the screen is performed based on the name of the application being executed. [2] The method of claim 1, wherein the determining step comprises selecting a QoE policy corresponding to a content type having a highest priority according to a preset priority for each content type when a plurality of types of content displayed on the current screen are selected Lt; / RTI > The method of claim 1, wherein the encoding step comprises: determining an allowable bit rate based on a network state between terminals sharing the screen; And
And adjusting a frame rate and a resolution applied to the encoding of the shared picture according to the determination result of the bit rate. A memory in which at least one program is stored; And
And a processor for enhancing QoE in a terminal sharing the screen with another terminal by executing the at least one program,
Wherein the at least one program comprises:
Determining a type of content displayed on the screen;
Determining a QoE policy related to a frame rate and a resolution based on a ratio of each content type when a plurality of types of content are displayed on the screen as a result of the determination;
Encoding an image of the screen according to the determined QoE policy; And
And transmitting the encoded screen image to the another terminal. 9. The transmitter as claimed in claim 8, wherein the encoding step is performed using a frame rate and a resolution adjusted according to the determined QoE policy. 9. The transmission apparatus of claim 8, wherein the type of the content includes graphic contents, real contents, and text contents. 9. The transmitter as claimed in claim 8, wherein the QoE policy is set such that the higher the frame rate is, the more the movement of the content changes. The transmitting apparatus of claim 8, wherein the step of determining the type of the content displayed on the screen is performed based on the name of the application being executed. 9. The method according to claim 8, wherein, in the determining step, when a plurality of content types are displayed on the current screen, a QoE policy corresponding to a content type having the highest priority is selected according to a preset priority for each content type . 9. The method of claim 8, wherein the encoding comprises: determining an allowable bit rate based on a network state between terminals sharing the screen; And
And adjusting a frame rate and a resolution applied to the encoding of the shared picture according to a result of the determination of the bit rate. A computer-readable recording medium on which a computer program for executing the method according to any one of claims 1 to 5 and 7 is recorded.

Images