KR102118535B1 - Method and apparatus for providing multimedia service - Google Patents

Abstract

The present invention provides a method and apparatus for changing the quality of service of multimedia data using a sensor. According to the present invention, a process for receiving multimedia data provided from a server, a process for filtering the received multimedia data, a process for decoding and rendering the filtered data, and for the terminal through one or more sensors connected to the terminal The process of measuring the user's perception, determining whether it is necessary to change the service quality of the multimedia data according to the measured perception, and when it is determined that the service quality needs to be changed, it is necessary to change the service quality. And transmitting information to the server.

Description

Method and device for providing multimedia service {METHOD AND APPARATUS FOR PROVIDING MULTIMEDIA SERVICE}

The present invention relates to a method for providing a multimedia service in a broadcasting and communication system, and more particularly, to a method and apparatus for adjusting the quality of multimedia data.

With the widespread use of smart phones, various methods for optimizing the use of smart phones in a mobile environment are being sought. In particular, since a smart phone can receive various data services compared to a conventional mobile phone, power consumption is relatively higher than that of a conventional mobile phone, so attention is focused on a method for increasing the capacity of the battery or minimizing the power consumption. have. In addition, since resources are limited in a wireless network environment, it is necessary to use radio resources as efficiently as possible.

In order to efficiently use radio resources while minimizing power consumption, it is necessary to properly adjust service quality according to the situation of the terminal. There are two methods for adjusting the quality of service: a method for adjusting at the server side and a method for adjusting at the terminal side. The method of adjusting on the server side allows the server to adjust the quality of service according to the network conditions of the terminal, and the method of adjusting on the terminal side is the quality of service of multimedia data received from the server according to the terminal specifications such as display resolution and processing power. It is common to adjust.

However, this conventional service quality adjustment method does not take into account the movement of the terminal or the concentration of the user's terminal, and thus, in order to minimize the power consumption, the service by considering the movement of the terminal or the concentration of the user's terminal is additionally considered. We need a way to adjust quality.

The problem to be solved by the present invention is to provide a method and apparatus for efficiently using radio resources and minimizing power consumption by adjusting service quality in consideration of a terminal use environment.

In addition, a problem to be solved by the present invention is to provide a method and apparatus for efficiently using radio resources and minimizing power consumption by adjusting a service quality by using a sensor embedded in a terminal.

According to an embodiment of the present invention, in a method for providing a multimedia service, a process of receiving multimedia data provided from a server, a process of filtering the received multimedia data, and a process of decoding and rendering the filtered data And, the process of measuring the user's awareness of the terminal through one or more sensors connected to the terminal, and determining whether it is necessary to change the service quality of the multimedia data according to the measured recognition, and the need to change the service quality If it is determined that there is, it includes the step of transmitting the information necessary for the change in the quality of service to the server.

In addition, according to an embodiment of the present invention, in an apparatus for providing a multimedia service, a receiving unit for receiving multimedia data provided from a server, a data filter for filtering the received multimedia data, and a decoder for decoding the filtered data Wow, a renderer that renders the decoded data, at least one sensor connected to a terminal to measure a user's perception of the terminal, and whether there is a need to change the quality of service of the multimedia data according to the recognition measured by the sensor. If it is determined, and it is determined that it is necessary to change the quality of service, it includes a state analysis unit that transmits information necessary for changing the quality of service to the server.

According to the present invention, power consumption can be reduced by appropriately adjusting the quality of service using a sensor embedded in the terminal when servicing multimedia data.

In addition, according to the present invention, power consumption can be reduced by appropriately adjusting service quality according to a user's state or a terminal's state when servicing multimedia data.

1 is a view showing an example of sensors that can be mounted on a terminal.
2 is a diagram briefly showing the visualization process of the human eye
3 is a diagram showing CFS corrected by Daly.
4 is a diagram showing the structure of an application program for providing a multimedia service according to an embodiment of the present invention
5 is a diagram showing the types of RTCP messages.
6 is a view showing the operation of an application program for providing a multimedia service according to an embodiment of the present invention
7 and 8 are views comparing the sizes of videos displayed on the screen according to the direction of the terminal.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. And, in the description of the present invention, if it is determined that a detailed description of related known functions or configurations may unnecessarily obscure the subject matter of the present invention, the detailed description will be omitted. In addition, terms to be described later are terms defined in consideration of functions in the present invention, which may vary according to a user's or operator's intention or practice. Therefore, the definition should be made based on the contents throughout this specification.

The present invention proposes a method for adjusting the service quality of a multimedia service using a sensor embedded in a terminal. Currently, sensors built into the terminal include an acceleration sensor and a gyroscope, and in the future, an EEG measurement sensor and a pulse measurement sensor may be incorporated.

1 shows an example of sensors that can be mounted on a terminal.

As illustrated in FIG. 1, there may be various sensors capable of measuring a user's state in a terminal. In the present invention, it is determined that the concentration of the user's terminal is reduced by using these sensors, and a method for controlling the quality of service accordingly is proposed. Therefore, in order to minimize the power consumption, which is the object of the present invention, it is not preferable to use a sensor with high power consumption, such as a camera or GPS, and sensors with less power consumption and less cost than when transmitting/receiving data using a CPU operation or a network. That is, it is preferable to use an acceleration sensor or a gyroscope with low power consumption.

In more detail, the degree of movement of the terminal is measured using an acceleration sensor. For example, it can measure when the user is stationary and when walking at a normal speed or when running or walking at a high speed. In some of these cases, the values measured by the accelerometer have different ranges, so it can be seen that the user's movement state has changed when it is out of a predetermined range. The power consumption can be reduced by transmitting low-quality video by using the fact that if the movement is high, the concentration of the user's terminal is poor. Similarly, when the resolution of multimedia data received through the gyroscope and the aspect ratio displayed on the current terminal do not match, it is possible to reduce power consumption by transmitting a low-quality image. In addition, the EEG measurement sensor can measure the concentration level of the user's terminal through the EEG measurement, and can properly adjust the service quality of the multimedia data according to the measured concentration level.

In the case of using the acceleration sensor, in an environment in which the user uses the multimedia service while walking or running, the user has less concentration on the terminal than when using the terminal in a stationary state. That is, even if the user focuses on the direction of movement than the terminal, or even if the user concentrates on the screen viewed from the terminal, the terminal moves by the user's movement, so even if a high quality image is provided, image recognition is lower than that of the still state. If the video being viewed by the user is encoded with a multi-layered codec according to quality, such as SVC (Scalable Video Codec), the quality can be adjusted according to the user's perception. That is, it is possible to determine whether the degree of movement of the terminal is such that the user can focus on the terminal by using the acceleration sensor, and accordingly, an image of a quality adjusted to the extent that the user cannot recognize the quality change can be provided. Power consumption can be reduced by adjusting the service quality.

In addition, the gyroscope is a sensor that measures the rotation of an object, and through the gyroscope, it is possible to determine whether the user is viewing the current screen in the horizontal direction or the vertical direction. If the image transmitted by the server is a horizontal video image and the user is using the terminal in the vertical direction, the service may be reduced by reducing the size of the image to fit the vertical direction, thereby reducing power consumption.

It is also expected that glasses-type mobile devices will be developed in the future, and it is highly likely that low-power BCI (Brain Computer Interface) technology will be installed in these mobile devices. BCI technology is an interface technology that directly connects the human brain and a computer to control the computer through brain waves. This technology recognizes EEG stimulation with an EEG sensor, converts it into a signal, analyzes it, and commands it. Therefore, it is highly likely to be used as a next-generation input device because the user can input without thinking. Electroencephalography is a measurement of a signal on the surface of a microscopic brain where electrical signals generated from numerous nerves in the brain are synthesized. It changes spatio-temporally as the brain functions according to various functions. It shows an amplitude of 200 μV. EEG is divided into delta wave, theta wave, alpha wave, beta wave, and gamma wave according to the range of frequency and voltage. Among them, beta wave shows the frequency of 12~30Hz and the amplitude of 2~20μV and can perform awake or conscious activity. When it appears. Among them, if a medium frequency band of 16 to 20 Hz is used, the user's concentration or activity status can be measured. And theta wave shows frequency of 4~7Hz and amplitude of 20~100 μV and appears in emotionally stable or sleeping state. Analyzing these two waveforms can determine a person's concentration and degree of awakening. As the beta wave increases, the concentration increases, and when the theta wave increases, the concentration decreases. Therefore, in the present invention, it is possible to adjust the quality of the multimedia data service according to the concentration of the user.

In adjusting the quality of service using such a sensor, first, the principle of the change in the quality of the image perceived by the user is examined.

2 briefly illustrates the visualization process of the human eye.

As shown in FIG. 2, the final image is recognized in the brain through three steps of image formation, exposure control, and detection. The image delivered to the brain is a high-quality, high-resolution image, but the region that is shown to be of interest in the brain is part of the image. In the graphic image, the same high-quality simulated image is partially changed according to the state of the camera, the characteristics of the object being drawn, or the viewpoint of the user viewing the graphic image.

In addition, there is a limit to the accuracy of the image made by the human eye. The number of rods and cones present in the center of the eye is about 7 million and 150 million, and images formed at angles of less than 1/120 degrees of recognition are difficult to recognize, and the viewing angle that produces the best image quality is 4 ~5 degrees. Also, when viewed from the image plane, movements of less than 1 degree are difficult to detect. Therefore, if the quality of service is adjusted using the limits of visual recognition, unnecessary waste of resources can be minimized.

Since the human eye does not recognize images above 60 Hz in time frequency, providing an image above 60 Hz is a waste of resources. It has similar characteristics with spatial frequency. To analyze this, the contrast sensitivity function CSF (Contrast Sensitivity Function) is used. CSF was modeled by Kelly, and Daly corrected the CSF modeled by Kelly using the constant c.

3 shows CFS corrected by Daly.

As shown in FIG. 3, in a certain section, CSF (ie, recognition power) increases, and then rapidly falls around 8 cpp (cycle per degree). That is, it can be seen that an image of 8 cpd or more has poor recognition ability.

These results can also be used for human movement. When a user watches a video on a mobile device while walking, the screen itself shakes according to the user's movement even if the user moves while looking around or in the direction of movement, or moves while looking at only the screen of the mobile device. In addition, when the movement speed increases, the movement of the user's viewpoint changes more rapidly or rarely looks at the screen. Moving the viewpoint means that multimedia is being serviced at a location of 8cpd or above. Therefore, in such a case, it is possible to reduce the amount of data received and the amount of decoding operation while reducing the image quality through various methods.

Hereinafter, a specific method of adjusting service quality using a sensor according to the present invention will be described.

When the service quality principal is the terminal (hereinafter referred to as Client-driven QoS), when the terminal receives the multimedia service from the server, the terminal selects the quality of service using the built-in sensor, and the service quality principal is the server (hereinafter Server-driven QoS), when the terminal delivers the measured value through the sensor to the server, the server selects the quality of service based on this value. Therefore, an application program that receives multimedia data from a server and provides service to a user should be able to analyze video data and audio, and analyze measured values from various sensors embedded in the terminal. Power consumed in operations such as decoding or receiving a large amount of data required for multimedia services is very large. Also, when a user uses a mobile terminal, most of them are used while performing other tasks than when using fixed devices such as a TV or a PC monitor. For example, an action such as using the terminal while moving it or changing the direction of the terminal is performed. In this way, the moving user has less concentration than usual and the perception of the media is also reduced. Providing high-quality media in the case of low media perception can be considered as overhead, so the quality and service required to receive data while maintaining the quality of experience (QoE) by adjusting the quality of service according to the user's perception, And it is important to reduce data usage cost and power consumption by reducing the amount of computation required for decoding.

In using the existing multimedia service, many studies have been conducted on adjusting the service quality using various parameters. Methods for adjusting the quality of service include a media filtering method that does not transmit a video when video and audio are transmitted together, and a media scaling stepwise controlling the quality of a specific media. In order to use these methods, the present invention proposes a method for generating a parameter for adjusting service quality using sensors embedded in a terminal without directly inputting a quality of service that a user can consume. The sensors measure the user's perception or the service status of the terminal and analyze it to adjust the quality of service according to the situation. That is, it does not always maintain the same quality, but reduces the power consumption by reducing the quality in an appropriate situation.

4 shows the structure of an application program for providing a multimedia service according to an embodiment of the present invention.

Referring to FIG. 4, the multimedia service application program simultaneously receives multimedia data from the server at the multimedia data receiving unit 410 and the sensor measuring unit 450 measures and measures signals through one or more sensors embedded in the terminal. The used data is transmitted to the state analysis unit 460 for use in analyzing the user's state. The state analysis unit 460 determines whether the user's perception of the user's terminal or the quality of the currently serviced multimedia data is appropriate by using data measured by various sensors through a state analysis algorithm. As a result of the determination, if it is determined that the quality of the multimedia data currently being serviced is higher than that of the user's terminal, the service quality appropriate to the user's awareness of the terminal is determined and a value indicating this is transmitted to the data filter 420. The data filter 420 filters the multimedia data received from the multimedia data receiving unit 410 and passes it to the decoder 430. At this time, the data filter 420 appropriately filters the multimedia data according to the service quality change information transmitted from the state analysis unit 460. The decoder 430 decodes data transmitted from the data filter 420, and the decoded data is rendered through the renderer 440 and serviced to the user.

The analysis result from the state analysis unit 460 is also reported to the server. In the case of using server-driven QoS, the current state is processed for each sensor and reported to the server to adjust the quality of service based on the information received from the terminal by the server. Make it possible. In addition, when using client-driven QoS, the terminal selects the quality of service according to the user's state and requests the server to change the quality of service. This procedure is repeated until the multimedia service is terminated.

When reporting information related to the change in quality of service to the server, the value measured by the sensor using RTCP (Real Time Control Protocol), especially in multimedia services using Real-time Transport Protocol (RTP) and User Datagram Protocol (UDP) Can be sent to the server. 5 shows the types of RTCP messages. Among the messages shown in FIG. 5, an application-designated message is a packet for an application that wants to use a new application, and allows a new message type to be defined. Therefore, in the embodiment of the present invention, data related to the change in the quality of service can be transmitted to the server using the application designation message. Alternatively, the measured value from the sensor may be transmitted to a server using an extended header of RTSP (Real Time Streaming Protocol).

6 shows an operation sequence of an application program for providing a multimedia service according to an embodiment of the present invention.

Referring to FIG. 6, a server and a socket are created through a call setup process 611, and after performing a connection operation, data is received (612) and filtering is performed (613). At the same time, the sensor provided in the terminal is driven (621) to measure the current state of the user (622). At this time, each sensor measures the current state at a predetermined cycle. The measured values extracted from one or more sensors confirm the user's perception or state of the terminal through predetermined algorithms, and determine whether the service quality of the currently serviced multimedia is appropriate (623). If the result of the determination is that the service quality is appropriate, the process returns to the process of measuring the current state through the sensor (622), and if it is determined that the service quality needs to be increased or decreased, the QoS control mode of the currently connected server is checked (624). In the case of client-driven QoS, the quality of service suitable for the current state is determined (625) and the determined content is transmitted to a data filter to be applied to the filtering process 613. On the other hand, in the case of Server-driven QoS, the value measured by the sensor is processed (626) and reported to the server (627) so that the server can determine the quality of service. Also, in the case of client-driven QoS, the quality of service value determined in step 625 is reported to the server (627). Thereafter, it is checked whether the service has ended (629), and if not, the process returns to measuring the current state through the sensor (622).

On the other hand, the data filter filters the received data according to the quality of service updated frequently by the sensor (613) and passes it to the decoder. The decoder decodes the filtered data according to the quality of service (614) and the decoded data is rendered (615) through a renderer. After rendering, it is determined whether the service has ended (617), and if not, the process returns to receiving data (611).

On the other hand, adjusting the service quality of multimedia data can be applied in various ways depending on the type of sensor. When using the acceleration sensor to measure the perception through the user's movement speed, it provides the highest quality when the user is stationary, provides intermediate quality when moving at a normal speed within a predetermined range, and is faster than a predetermined value. When moving at speed, it can provide low quality. As another example, when adjusting the quality of service through a gyroscope, most mobile terminals and video contents generate data in a 16:9 ratio, so that when a user views the video while holding the terminal in a horizontal direction, The video size when viewing it while holding it vertically is significantly different. This difference is illustrated in FIGS. 7 and 8. That is, since the video size in the case of holding and viewing the terminal in the vertical direction is smaller than in the case of holding and viewing in the horizontal direction, it is overhead to service the video of the same quality as when holding and viewing in the horizontal direction. Therefore, in this case, receiving and servicing low-quality video reduces power consumption. In addition, in the case of using the EEG sensor, it is possible to reduce power consumption by providing a high-quality service if the user is concentrating and using a multimedia service and providing a low-quality service when the concentration falls below a predetermined value.

In addition, adjusting the service quality of multimedia data can be applied in various ways depending on the type of data. In the case of data combining audio and video, only audio minus video may be received and processed to reduce service quality. When receiving the encoded video, the quality of service can be adjusted by adjusting the amount of each frame in the I-frame, P-frame, and B-frame constituting the GOP (Group of Picture). That is, the amount of B-frames and P-frames can be increased to change to low quality, and I-frames are frequently received to provide high quality.

In addition, if the data is encoded with a codec that can be divided and combined for each quality such as SVC, service quality can be adjusted more flexibly. When the video quality is reduced in SVC, the amount of received data is also reduced. Therefore, if the quality of the video is reduced while maintaining QoE when the multimedia service provides SVC-encoded video, it is possible to benefit from the amount of power required for receiving data and the amount of power required for decoding. In an environment in which a user receives and views SVC-encoded video in real time from a server using a mobile device, when it is determined that the user is walking, the server may request the server to lower the level of the video currently received and decoded. At this time, the user's movement speed could be determined by measuring the acceleration sensor. If the user's motion is not detected and all the images up to enhance2 are received and played back, the power consumption can be reduced by lowering the step to enhance1 when normal speed motion is detected and to base when fast motion is detected. have. In addition, since an image with a low level is received, the amount of computation required for decoding is reduced and the amount of power consumption is also reduced.

On the other hand, in the detailed description of the present invention, specific embodiments have been described, but various modifications are possible without departing from the scope of the present invention. Therefore, the scope of the present invention should not be limited to the described embodiments, but should be determined not only by the scope of the claims to be described later, but also by the scope and equivalents of the claims.

Claims (20)

In the method of providing a multimedia service,
The process of receiving the multimedia data provided from the server,
Filtering the received multimedia data;
Decoding and rendering the filtered data;
Measuring a user's awareness of the terminal through one or more sensors connected to the terminal,
Determining whether it is necessary to change the quality of service of the multimedia data according to the measured recognition;
And when it is determined that the service quality needs to be changed, transmitting information required to change the service quality to the server,
The service quality of the multimedia data includes a resolution for the service of the multimedia data,
A method of providing a multimedia service, wherein the user's awareness is measured based on the rotation of the terminal. According to claim 1,
The process of transmitting to the server,
If it is determined that it is necessary to change the quality of service, the process of identifying the subject to change the quality of service,
A method of providing a multimedia service, comprising transmitting information necessary for changing the quality of service to a server with different values depending on the identified subject. According to claim 2,
When the subject is a server, the result value measured through the sensor is transmitted to the server as information necessary to change the quality of service,
A method of providing a multimedia service that receives multimedia data having a service quality changed by the server based on information necessary for changing the service quality. According to claim 2,
When the subject is a terminal, the quality of service is changed based on a result value measured through the sensor,
A method for providing a multimedia service, transmitting the changed quality of service as information necessary for changing the quality of service to the server. The method of claim 4,
The filtering process is a multimedia service providing method of filtering the received multimedia data according to the changed service quality. delete delete The method according to any one of claims 1 to 5,
If the sensor is a gyroscope sensor,
The process of measuring the user's perception,
A method of providing a multimedia service for determining that the user's perception is different depending on whether the terminal is rotated. delete The method according to any one of claims 1 to 5,
The process of transmitting to the server,
When the multimedia service uses a Real Time Control Protocol (RTCP), information required to change the quality of service is transmitted to the server using an application designation message,
When the multimedia service uses a Real Time Streaming Protocol (RTSP), a method for providing a multimedia service that transmits information necessary for changing the quality of service to the server using an extended header. In the device for providing a multimedia service,
And a receiving unit for receiving the multimedia data provided from the server,
A data filter for filtering the received multimedia data,
A decoder for decoding the filtered data,
A renderer that renders the decoded data,
At least one sensor connected to the terminal to measure the user's perception of the terminal,
It is determined whether it is necessary to change the quality of service of the multimedia data according to the recognition measured by the sensor, and when it is determined that the quality of service needs to be changed, a state analysis of transmitting information necessary for changing the quality of service to the server Including wealth,
The service quality of the multimedia data includes a resolution for the service of the multimedia data,
The apparatus for providing multimedia service, wherein the recognition of the user is measured based on the rotation of the terminal. The method of claim 11,
The state analysis unit,
If it is determined that it is necessary to change the quality of service, identify a subject who changes the quality of service,
Device for providing a multimedia service that transmits information necessary for changing the quality of service to different values according to the identified subject. The method of claim 12,
When the subject is a server, the status analysis unit transmits a result value measured through the sensor as information necessary for changing the quality of service to the server,
The receiving unit, the multimedia service providing apparatus for receiving the multimedia data having the service quality changed by the server on the basis of the information required to change the service quality. The method of claim 12,
The state analysis unit,
When the subject is a terminal, the quality of service is changed based on a result value measured through the sensor,
A multimedia service providing apparatus for transmitting the changed quality of service as information necessary for changing the quality of service to the server. The method of claim 14,
The data filter is a multimedia service providing apparatus for filtering the received multimedia data according to the changed service quality. delete delete The method according to any one of claims 11 to 15,
If the sensor is a gyroscope sensor,
The sensor,
Device for providing a multimedia service for determining that the user's perception is different depending on whether the terminal is rotated. delete The method according to any one of claims 11 to 15,
The state analysis unit,
When the multimedia service uses a Real Time Control Protocol (RTCP), information required to change the quality of service is transmitted to the server using an application designation message,
When the multimedia service uses a Real Time Streaming Protocol (RTSP), a multimedia service providing apparatus that transmits information necessary for changing the quality of service to the server using an extended header.

Images