GB2568690A

GB2568690A - Method for adaptive displaying of video content

Info

Publication number: GB2568690A
Application number: GB1719425.9A
Authority: GB
Inventors: Cricri Francesco; Roimela Kimmo; Pystynen Johannes
Original assignee: Nokia Technologies Oy
Current assignee: Nokia Technologies Oy
Priority date: 2017-11-23
Filing date: 2017-11-23
Publication date: 2019-05-29
Also published as: GB201719425D0

Abstract

A method of displaying video content comprises: receiving video content comprising an area of interest 410, wherein the area of interest is received with a first quality and the area outside of the area of interest is received with a second lower quality; displaying the video content 420; receiving viewing direction of a user 430; determining a current focus area of the user based on their viewing direction 440; and decreasing the quality of a video region corresponding to the current focus area of the user if it is outside of the area of interest 450. The viewing direction data may comprise gaze direction data of the user. The video content may be 360 video content and may be displayed in a head mounted display (HMD). Decreasing the quality of a video region may comprise decreasing resolution or bitrate, or removing an object. A method of providing video for transmission comprises: obtaining information on area(s) of interest within video content and providing the video content from transmission, wherein the area(s) of interest are provided with a first quality and the area outside of the area(s) of interest is provided with a second lower quality.

Description

(57) A method of displaying video content comprises: receiving video content comprising an area of interest 410, wherein the area of interest is received with a first quality and the area outside of the area of interest is received with a second lower quality; displaying the video content 420; receiving viewing direction of a user 430; determining a current focus area of the user based on their viewing direction 440; and decreasing the quality of a video region corresponding to the current focus area of the user if it is outside of the area of interest 450. The viewing direction data may comprise gaze direction data of the user. The video content may be 360 video content and may be displayed in a head mounted display (HMD). Decreasing the quality of a video region may comprise decreasing resolution or bitrate, or removing an object. A method of providing video for transmission comprises: obtaining information on area(s) of interest within video content and providing the video content from transmission, wherein the area(s) of interest are provided with a first quality and the area outside of the area(s) of interest is provided with a second lower quality.

400

410 /420

430 j 450

Fig. 4

440 /7

106

Fig. 1a

100

102g

Fig. 1b

2/7

306

Fig. 3

3/7

400

Fig. 4

4/7

Fig. 5a

Fig. 5c

5/7

Fig. 6b

6/7

706

Fig. 7

7/7

Fig. 8a

800

Fig. 8b

Method for adaptive displaying of video content

Background

360 degrees video content, also known e.g. as immersive video or spherical video, is video content captured using an omnidirectional camera or a collection of cameras. 360 degrees video content may be viewed e.g. using a head mounted display (HMD) or a 360 degrees screen. A HMD is a display device, worn on the head of a user. HMD has a display in front of the eyes. When the 360 degrees video content is streamed over a network, e.g. the Internet, to the display device, the streaming consumes a lot of bandwidth.

There is, therefore, a need for improved method for streaming and displaying video content.

Summary

Various aspects of the invention include a method, an apparatus, and a computer readable medium comprising a computer program stored therein, which are characterized by what is stated in the independent claims. Various embodiments of the invention are disclosed in the dependent claims.

According to a first aspect, a method comprises receiving a first video content comprising an area of interest, wherein the area of interest is received with a first quality and the area outside of the area of interest is received with a second quality, wherein the first quality is higher than the second quality; displaying the first video content in a video playback apparatus; receiving viewing direction data of a user of the video playback apparatus; determining, in the first video content, a current focus area of the user based on the viewing direction data; and decreasing quality of a video region in the first video content corresponding to the current focus area of the user, when the current focus area of the user is outside of the area of interest.

According to an embodiment, the method according further comprises determining a path in the first video content from the current focus area of the user towards the area of interest, when the current focus area is outside of the area of interest; and increasing quality of a video region corresponding to the path.

According to an embodiment, decreasing quality of the video region corresponding to the current focus area of the user, when the current focus area of the user is outside of the area of interest, comprises decreasing resolution.

According to an embodiment, decreasing quality of the video region corresponding to the current focus area of the user, when the current focus area of the user is outside of the area of interest, comprises decreasing bitrate.

According to an embodiment, decreasing quality of the video region corresponding to the current focus area of the user, when the current focus area of the user is outside of the area of interest, comprises removing an object present in the current focus area of the user.

According to an embodiment, the method further comprises determining, based on the viewing direction data, that the user’s current focus area remains outside of the area of interest; evaluating conditions for adding the current focus area to a group of areas of interest; and adding the current focus area to the group of areas of interest when the conditions are fulfilled.

According to an embodiment, the first video content is 360 degrees video content.

According to an embodiment, the video playback apparatus is a head mounted display.

According to an embodiment, the viewing direction data comprises gaze direction data of the user of the video playback apparatus.

According to a second aspect, a method, comprises obtaining information on at least one area of interest within a first video content; and providing the first video content for transmission, wherein the at least one area of interest is provided with a first quality and the area outside of the at least one area of interest is provided with a second quality, wherein the first quality is higher than the second quality.

According to an embodiment, the method further comprises determining the at least one area of interest in the first video content based on saliency detection.

According to an embodiment, the method further comprises receiving viewing history data of other users relating to the first video content; and determining the area of interest in the first video content based on the viewing history data of other users.

According to an embodiment, the method further comprises determining a first feature-viewing pair based on visual features of the first video content and actualized viewing patterns derived from the viewing history data of other users relating to the first video content; receiving a second feature-viewing pair relating to a second video content which has similarities with the first video content, wherein the second feature-viewing pair has been confirmed as correct; calculating similarity between the first feature-viewing pair and the second feature-viewing pair; and determining the area of interest in the first video content based on the visual features of the first feature-viewing pair when the similarity exceeds a pre-determined threshold value.

According to a third aspect, there is provided an apparatus comprising at least one processor, memory including computer program code, the memory and the computer program code configured to, with the at least one processor, cause the apparatus to receive a first video content comprising an area of interest, wherein the area of interest is received with a first quality and the area outside of the area of interest is received with a second quality, wherein the first quality is higher than the second quality; display the first video content in a video playback apparatus; receive viewing direction data of a user ofthe video playback apparatus;

determine, in the first video content, a current focus area of the user based on the viewing direction data; and decrease quality of a video region in the first video content corresponding to the current focus area of the user, when the current focus area of the user is outside of the area of interest.

According to an embodiment, the apparatus is further caused to determine a path in the first video content from the current focus area of the user towards the area of interest, when the current focus area is outside of the area of interest; and increase quality of a video region corresponding to the path.

According to an embodiment, the apparatus is further caused to determine, based on the viewing direction data, that the user’s current focus area remains outside of the area of interest; evaluate conditions for adding the current focus area to a group of areas of interest; and add the current focus area to the group of areas of interest when the conditions are fulfilled.

According to a fourth aspect, there is provided an apparatus comprising means for receiving a first video content comprising an area of interest, wherein the area of interest is received with a first quality and the area outside of the area of interest is received with a second quality, wherein the first quality is higher than the second quality; displaying the first video content in a video playback apparatus; receiving viewing direction data of a user of the video playback apparatus; determining, in the first video content, a current focus area of the user based on the viewing direction data; and decreasing quality of a video region in the first video content corresponding to the current focus area of the user, when the current focus area of the user is outside of the area of interest.

According to an embodiment, the apparatus further comprises means for determining a path in the first video content from the current focus area of the user towards the area of interest, when the current focus area is outside of the area of interest; and increasing quality of a video region corresponding to the path.

According to an embodiment, the apparatus further comprises means for determining, based on the viewing direction data, that the user’s current focus area remains outside of the area of interest; evaluating conditions for adding the current focus area to a group of areas of interest; and adding the current focus area to the group of areas of interest when the conditions are fulfilled.

According to fifth aspect, there is provided a computer program product embodied on a non-transitory computer readable medium, comprising computer program code configured to, when executed on at least one processor, cause an apparatus to perform at least the method according to any of the above embodiments.

According to sixth aspect, there is provided an apparatus comprising at least one processor, memory including computer program code, the memory and the computer program code configured to, with the at least one processor, cause the apparatus to obtain information on at least one area of interest within a first video content; and provide the first video content for transmission, the first video content comprising at least one area of interest, wherein the at least one area of interest is provided with a first quality and the area outside of the at least one area of interest is provided with a second quality, wherein the first quality is higher than the second quality.

According to an embodiment, the apparatus is further caused to determine the at least one area of interest in the first video content based on saliency detection.

According to an embodiment, the apparatus is further caused to receive viewing history data of other users relating to the first video content; and determine the area of interest in the first video content based on the viewing history data of other users.

According to an embodiment, the apparatus is further caused to determine a first feature-viewing pair based on visual features of the first video content and actualized viewing patterns derived from the viewing history data of other users relating to the first video content; receive a second feature-viewing pair relating to a second video content which has similarities with the first video content, wherein the second feature-viewing pair has been confirmed as correct; calculate similarity between the first feature-viewing pair and the second feature-viewing pair; and determine the area of interest in the first video content based on the visual features of the first feature-viewing pair when the similarity exceeds a pre-determined threshold value.

According to seventh embodiment, there is provided an apparatus comprising means for obtaining information on at least one area of interest within a first video content; and providing the first video content for transmission, the first video content comprising at least one area of interest, wherein the at least one area of interest is provided with a first quality and the area outside of the at least one area of interest is provided with a second quality, wherein the first quality is higher than the second quality.

According to an embodiment, the apparatus further comprises means for determining the at least one area of interest in the first video content based on saliency detection.

According to an embodiment, the apparatus further comprises means for receiving viewing history data of other users relating to the first video content; and determining the area of interest in the first video content based on the viewing history data of other users.

According to an embodiment, the apparatus further comprises means for determining a first feature-viewing pair based on visual features of the first video content and actualized viewing patterns derived from the viewing history data of other users relating to the first video content; receiving a second feature-viewing pair relating to a second video content which has similarities with the first video content, wherein the second feature-viewing pair has been confirmed as correct; calculating similarity between the first feature-viewing pair and the second feature-viewing pair; and determining the area of interest in the first video content based on the visual features of the first feature-viewing pair when the similarity exceeds a pre-determined threshold value.

Description of the Drawings

In the following, various embodiments of the invention will be described in more detail with reference to the appended drawings, in which

Fig. 1a shows, by way of an example, a multi-camera system;

Fig. 1b shows, by way of an example, a perspective view of a multicamera system;

Fig. 2 shows, by way of an example, a system and devices for video image capturing and video streaming and displaying;

Fig. 3 shows, by way of an example, a video playback apparatus as a simplified block diagram;

Fig. 4 shows, by way of an example, a flow diagram of a method for displaying video content;

Fig. 5a, 5b and 5c show, by way of examples, an adaptive displaying method;

Fig. 6a, 6b show, by way of examples, an adaptive displaying method;

Fig. 7 shows, by way of an example, a streaming server as a simplified block diagram; and

Fig. 8a and 8b show, by way of examples, flow diagrams of a method for streaming video content.

Description of Example Embodiments

In the following, several embodiments of the invention will be described in the context of streaming and displaying 360 degrees video content. It is to be noted, however, that the invention is not limited to 360 degrees video content. The invention may be applied to other video contents as well, such as 2D video content or panoramic video content. In fact, the different embodiments have applications in any environment where streaming and displaying of video content is required.

Figure 1a illustrates an example of a multi-camera system 100, which may be able to capture and produce 360 degree video. The multi-camera system 100 comprises camera units 102. For example, the number of camera units 102 may be eight, but may also be less than eight or more than eight. Each camera unit 102 is located at a different location in the multi-camera system, and may have a different orientation with respect to other camera units 102, so that they may capture a part of the 360 degree scene from different viewpoints substantially simultaneously. The camera units 102 may have an omnidirectional constellation, so that it has a 360° viewing angle in a 3Dspace. In other words, such multi-camera system 100 may be able to see each direction of a scene so that each spot of the scene around the multicamera system 100 can be viewed by at least one camera unit 102. The multi camera system may comprise mirrors, e.g. curved mirrors, to reflect the light from the environment towards camera sensor(s) in order to increase the viewing angle.

The multi-camera system 100 of Figure 1a may also comprise a processor 104 for controlling operations of the multi-camera system 100. There may also be a memory 106 for storing data and computer code to be executed by the processor 104, and communication interface (transceiver) 108 for communicating with, for example, a communication network and/or other devices in a wireless and/or wired manner. The multi-camera system 100 may further comprise a user interface (Ul) 110 for displaying information to the user, for generating audio signals, and/or for receiving user inputs. However, the multi-camera system 100 need not comprise each feature mentioned above, or may comprise other features as well. For example, there may be electric and/or mechanical elements for adjusting and/or controlling optics of the camera units 102.

Figure 1 b shows, by way of an example, a perspective view of the multicamera system 100. In Figure 1b seven camera units 102a—102g can be seen, but the multi-camera system 100 may comprise even more camera units which are not visible from this perspective view. Figure 1b also shows two microphones 112a, 112b, but the apparatus may also comprise one microphone or more than two microphones.

In accordance with an embodiment, the multi-camera system 100 may be controlled by another device, wherein the multi-camera system 100 and the other device may communicate with each other and a user may use a user interface ofthe other device for entering commands, parameters, etc. and the user may be provided with information from the multi-camera system 100 via the user interface of the other device.

Figure 2 shows, by way of an example, a system and devices for video image capturing and video streaming and displaying. The different devices may be connected to each other via a communication connection 210, for example via Internet, a mobile communication network, Wireless Local Area Network (WLAN), Bluetooth®, or other contemporary and future networks. Different networks may be connected to each other by means of a communication interface. The multi-camera device 100 may be used to capture 360 degrees video content. The captured video content may be stored in a video database 220. The video database may be a server or a group of servers. The servers may form a server system, e.g. a cloud. Streaming server 230 may receive the video content from the video database 220 or from the multi-camera device 100. The streaming server may stream the video content for viewing to a video playback apparatus. The video playback apparatus may be e.g. a head-mounted display (HMD) 300, a 360 degrees screen or other suitable immersive display unit.

Figure 3 shows, by way of an example, a video playback apparatus as a simplified block diagram. The video playback apparatus 300 may be e.g. a head mounted display (HMD) or a 360 degrees screen. The video playback apparatus may receive the video stream via a communication interface 308. The video playback apparatus may comprise a processor 304 for performing control operations for the apparatus and operations to modify video streams. The video playback apparatus may comprise a decoder 314 for decoding received data streams. The video playback apparatus may comprise memory 306 for storing data and computer code which can be executed by the processor to perform the desired function, e.g. decoding. The video playback apparatus may comprise a user interface 310 for receiving e.g. user instructions or inputs. The video playback device may comprise these elements or fewer or more elements. An element, for example decoder 314, may be a separate device which may perform decoding operations and provide decoded video stream to the video playback apparatus 300 for further processing and displaying decoded video streams.

The video playback apparatus may comprise or may be connected to a gaze tracker 312 comprising means for tracking user’s viewing direction. More specifically, the gaze tracker may comprise means for tracking user’s gaze. In addition, the gaze tracker may comprise means for measuring depth layer attention of the user. The gaze tracking means may comprise a camera or any suitable optical sensor for sensing infrared light reflected from the user’s eye(s). The gaze tracking means may comprise communication means for sending gaze direction data to e.g. the video playback apparatus. For example, when the 360 degrees video content is viewed using a 360 degrees screen, the gaze tracker may be a separate device communicating with the screen.

The video playback apparatus may comprise a head tracking system comprising means for tracking user’s viewing direction. A head-tracker may be integrated in to a head-mounted display. Alternatively, the head-tracker may be a separate device worn by the user. Head orientation and head movement may be tracked, for example, using a combination of 3-axis rotational tracking and 3-axis positional tracking. The head-tracker may comprise gyroscopes, accelerometers, and magnetometers which may be used in head-tracking. Alternatively, a camera may analyse the rotation and other movements of the user’s head.

The video playback apparatus may comprise an image modifier 316. The image modifier may perform image modification as will be described later in this specification. The image modifier may be implemented, for example, as a software code, which may be executed by the processor to perform the desired function, e.g. image modification. The image modifier 316 may be a separate device which may perform image modification operations and provide modified video stream to the video playback apparatus 300 for displaying.

Size (in bytes) of 360 degrees video content is usually big. Big contents may consume a lot of bandwidth when streaming the content over a communication network.

In the following, adaptive displaying method will be described in more detail with reference to the flow diagram of Figure 4. The method 400 comprises receiving 410 a first video content comprising an area of interest, wherein the area of interest is received with a first quality and the area outside of the area of interest is received with a second quality. The first quality is higher than the second quality. The method comprises displaying 420 the first video content in a video playback apparatus. The method comprises receiving 430 viewing direction data of a user of the video playback apparatus. The method comprises determining 440, in the first video content, a current focus area of the user based on the viewing direction data. The method comprises decreasing 450 quality of a video region in the first video content corresponding to the current focus area of the user, when the current focus area of the user is outside of the area of interest.

Hence, it is provided an adaptive displaying method which reduces bandwidth requirements when streaming 360 degrees video content to a user for viewing. The image modification may be implemented in the client side, e.g. in the video playback device. Hence, latency between transmitting, from the video playback device, user’s pose measurement data to a streaming server and receiving, in the video playback device, the adapted stream from a streaming server may be avoided.

In the following, the adaptive displaying method will be described by way of an example relating to the 360 degrees video content captured from a theatre play. The video content may be received and viewed by the user via the video playback device, e.g. the HMD worn by the user or a 360 degrees screen. The video content may comprise areas comprising targets that are meant to be seen by the user. In other words, the video content may comprise areas having different degrees of importance for the user. The video content may comprise e.g. areas that are determined to be important for the user and areas that are determined to be less important for the user. The important areas may be determined in different ways.

Figure 5a shows, by way of an example, adaptive displaying method. A scene 500 is viewed by the user with the video playback device. Important areas,

i.e. areas of interest, may be determined by content producers and/or authors of e.g. the theatre play. For example, it may be decided that a part of the scene 500, wherein actors have a discussion, is important. This area of the video may be determined as an area of interest 510. The area of interest may be provided with a first quality to the video playback device such as the HMD. The area outside of the area of interest may be provide with a second quality to the video playback device such as the HMD. The first quality is higher than the second quality. The first quality may be high quality. The second quality may be low quality. The low quality is lower than the high quality. As another example, authors may decide that an area corresponding to a part of a scene, where next act is going to take place, e.g. in near future (few seconds), is an area of interest. Thus, the user may be guided to watch a part of scene before an important event happens in that part of the scene by determining that area as the area of interest and by providing the area of interest with high quality to the video playback device.

The high quality of the video content may be determined as e.g. high resolution and/or high bitrate. The low quality of the video content may be determined as e.g. low resolution and/or low bitrate. High quality may mean e.g. the original quality of the video content. Resolution and bitrate are examples of quality measures. Quality may also be understood as some other suitable measure of quality. For example, another quality metric may be the quality of depth data, if applicable. Examples of high resolution may be resolutions in the range of e.g. from 8K to 16K. In the context of 360 degree video content, examples of high resolution may be e.g. from 20 to 60 pixels per degree. Example of low resolution may be e.g. less than 10 pixels per degree. Examples of high bitrate may be bitrates over 20 Mbps, e.g. 25 Mbps. Examples of low bitrates may be bitrates less than 10 Mbps, e.g. 8 Mbps. When decreasing or increasing the quality, e.g. resolution and bitrate may be both decreased or increased, or only one of them may be decreased or increased. Resolution may be decreased e.g. by reducing the number of pixels transmitted to the user. For example, before transmission, few neighboring pixels may be interpolated into a single pixel, and then only the resulting pixels are transmitted. As another example, only one pixel may be chosen from the neighboring pixels. Decreasing quality may further comprise e.g. blurring of the image.

There may be one or more other areas in the scene that may catch the interest of the user, in addition or alternatively to the areas of interest already determined before transmission. However, those areas may be determined as less important for the user from the point of view of the proposed system. The reason for this may be that targets or events e.g. in the background of the scene are not so important with respect to a storyline of the theatre play. Targets may be props, e.g. painted trees 501 or other actors. Therefore, content producers may decide that those targets or events are of less importance for the user. These areas, which are outside of the area of interest 510 may be streamed with a second quality to the video playback device. The second quality is lower than the first quality.

Figure 5b shows, by way of an example, adaptive displaying method. The viewing direction of the user may be tracked using the gaze tracking and/or head tracking means described earlier. The viewing direction data may comprise gaze direction data. Gaze direction data may be received e.g. by the image modifier from the gaze tracker. Gaze direction data may comprise

e.g. image coordinates of those video regions on which the user has focused one’s gaze. In addition, gaze direction data may comprise information on user’s depth layer attention. A current focus area, e.g. area 512 or area 514, of the user may be determined based on the gaze direction data. If the user is watching the previously described areas of less importance, the current focus area is outside of the area of interest. When the current focus area of the user is outside of the area of interest, the quality of a video region corresponding to the current focus area of the user may be decreased. When the current focus area of the user is in the area or interest, it may be continued to display the video content, wherein the area of interest is provided with high quality.

According to an embodiment, decreasing quality of the video region corresponding to the current focus area of the user, when the current focus area of the user is outside of the area of interest, comprises decreasing resolution. For example, neighboring pixels may be interpolated into a single pixel, and then only the resulting pixels are displayed. As another example, only one pixel may be chosen from the neighboring pixels. User attention may be guided towards the areas of interest by decreasing resolution of less important areas.

According to an embodiment, decreasing quality of the video region corresponding to the current focus area of the user, when the current focus area of the user is outside of the area of interest, comprises decreasing bitrate. User attention may be guided towards the areas of interest by decreasing bitrate of less important areas.

Figure 5c shows, by way of an example, adaptive displaying method. Decreasing quality may comprise modifying the semantics of the video content. According to an embodiment, decreasing quality of the video corresponding to the current focus area of the user may comprise removing an object present in the current focus area of the user, when the current focus area of the user is outside of the area of interest. The object may be removed e.g. by running a neural network generator of visual data, and the content may be re-imagined in order to remove the object or an event which triggered attention of the user. In the example of Figure 5c, the trees inside the current focus area 514 are removed. As an implementation example, the generator neural network may receive the whole video frame and a binary mask which may indicate the area currently being watched by the user. For example, the binary mask may be a matrix of same dimensions as the video frame, and its values may be one for the pixels the user is currently watching, and zero elsewhere. The neural network may then re-imagine only the content indicated by the binary mask. User attention may be guided towards the areas of interest by modifying the semantics of less important areas, e.g. by removing an object present in the less important areas.

Figure 6a shows, by way of an example, adaptive displaying method. According to an embodiment, the method comprises determining a path in the first video content from the current focus area of the user towards the area of interest, when the current focus area is outside of the area of interest; and increasing quality of a video region corresponding to the path. The gaze tracker and head tracker may comprise or may be connected to means for determining for how long time the current focus area of the user is outside of the area of interest. The user may persistently watch the areas outside of the area of interest, i.e. the user’s current focus area 514 remains outside of the area of interest 510. A threshold time period may be predefined such that when the user watches the areas outside of the area of interest longer than the predefined threshold time period, a high quality path may be determined to guide the user towards the area of interest. When the current focus area 514 is outside of the area of interest 510, a path 610 may be determined in the video content from the current focus area 514 of the user towards the area of interest 510. Quality of the video region corresponding to the path may be increased. Quality of the path 610 may be increased at once or it may be increased gradually, as shown schematically in Fig. 6b. The increasing of quality may happen gradually in a way that in a first frame, quality of the pixels or voxels of a first path region 621 close to the current focus area which is outside of the area of interest is increased. Then, in a second frame, quality of next portion of pixels or voxels of a second path region 622 along the path is increased. In a third frame, quality of next portion of pixels or voxels of a third path region 623 along the path is increased. In a fourth frame, quality of next portion of pixels or voxels of a fourth path region 624 is increased. The frames may be considered as image layers in the example of Fig. 6b. By creating a high quality path towards the area of interest, the user attention may be guided towards desired areas.

There may be several areas of interest visible at the same time for the user. Several paths with increased quality may be created from the current focus area to different areas of interest. The increased quality of each ofthe several paths may be the same. Alternatively, the qualities of the paths may be different from each other. For example, the qualities ofthe several paths may be determined such that the qualities are inversely proportional to the distances between the current focus area and the areas of interest. For example, a short distance may correspond to higher quality path with respect to another path with longer distance. Distances may be calculated e.g. based on differences between the image coordinates ofthe current focus area and the areas of interest. The areas of interest may be ordered according to the calculated distances. The area of interest having the shortest distance to the current focus area may be determined as the closest area of interest. The area of interest having the longest distance to the current focus area may be determined as the farthest area of interest. For example, quality of a path to the closest area ofthe interest may be determined as the highest, and quality of a path to the farthest area of interest may be determined as the lowest.

As an alternative to several paths, the closest area of the interest may be determined and a high quality path may be created from the current area of interest to the closest area of interest.

According to an embodiment, the method comprises determining, based on the viewing direction data, that the user’s current focus area remains outside ofthe area of interest; evaluating conditions for adding the current focus area to a group of areas of interest; and adding the current focus area to the group of areas of interest when the conditions are fulfilled. It may be determined, based on the gaze direction data, that the user’s current focus area remains outside of the area of interest, even after creating the paths to areas of interest. As described above, a threshold time period may be predefined. Conditions for adding the current focus area to a group of areas of interest may be evaluated. If the conditions are fulfilled, the current focus area may be added to the group of areas of interest. Evaluation may be carried out e.g. based on the gaze direction data of many users. The gaze direction data of users may be sent and saved to a database. A threshold number may be determined such that when enough (i.e. number above the predetermined threshold) users persist on watching a certain area which is not included in the areas of interest, that area is added to the areas of interest. By adding new areas of interest based on viewing direction data, the video content may be improved.

Referring back to the streaming server 230 of Figure 2, the streaming server may perform determination of areas of interest. The determination of areas of interest may be implemented, for example, as a software code, which may be executed by the processor to perform the desired function.

Figure 7 shows, by way of an example, a streaming server as a simplified block diagram. The streaming server may receive video content e.g. from the video database or from the multi-camera system via a communication interface 708. The streaming server may stream the video content for viewing to a video playback apparatus. The video playback apparatus may be e.g. a head-mounted display (HMD) 300, a 360 degrees screen or other suitable immersive display unit. The streaming server may comprise a processor 704 for performing e.g. streaming and operations to determine areas of interest in the video content. The streaming server may comprise an encoder 714 for encoding received data streams. The streaming server may comprise memory 706 for storing data and computer code which can be executed by the processor to perform the desired function, e.g. encoding. The streaming server may comprise a user interface 710 for receiving e.g. user instructions or inputs. The streaming server may comprise these elements or fewer or more elements.

For example, the streaming server may comprise an element for determining area of interest, an area of interest definer 716. The area of interest definer may be implemented for example, as a software code, which may be executed by the processor to perform the desired function, e.g. determination of area of interest.

In the following, adaptive streaming method will be described in more detail with reference to the flow diagram of Figure 8a. The method 800 comprises obtaining 810 information on at least one area of interest within a first video content. Information may comprise e.g. image coordinates of the at least one area of interest. Information may be obtained from a database. For example, content producers and/or authors may have determined the areas of interest and saved the information on the areas of interest within a first video content in a database. The method comprises providing 820 the first video content for transmission, the first video content comprising at least one area of interest, wherein the at least one area of interest is provided with a first quality and the area outside of the at least one area of interest is provided with a second quality, wherein the first quality is higher than the second quality. The first video content may be provided for transmission to a video playback apparatus. By the adaptive streaming method 800 communication bandwidth is reduced, since the whole video content is not provided with full quality.

According to an embodiment, the method 800 further comprises determining the at least one area of interest in the first video content based on saliency detection. Referring back to different ways to determine the importance of the areas, another way to determine the areas of interest may be automatic saliency detection. Automatic saliency detection consists of automatically determining the areas or objects in an image or video which are salient for humans, based on for example a human-annotated dataset from which the system may learn. Automatic saliency detection may be performed by known saliency detection algorithms or by using a trained artificial neural network. As an alternative to applying saliency detection, similar video content may be used as a reference content for transferring areas of interest. Known targets, e.g. humans, which are determined as the areas of interest in the reference content, may be determined as the areas of interest in that video content the areas of interest of which are being determined.

Figure 8b shows, by way of an example, a flow diagram of a method for streaming video content. According to an embodiment, the method 800 further comprises receiving 830 viewing history data of other users relating to the first video content; and determining 840 the area of interest in the first video content based on the viewing history data of other users. Still another way to determine the areas of interest is determination based on viewing history data of other users. Viewing history data of other users may be received e.g. from a database. The viewing history data may comprise gaze direction data comprising e.g. image coordinates of those video regions on which the users or most of the users have focused ones’ gaze. Other users may be e.g. test users which are professional video watchers or conventional consumers.

According to an embodiment, the method 800 further comprises determining a first feature-viewing pair based on visual features of the first video content and actualized viewing patterns derived from the viewing history data of other users relating to the first video content; receiving a second feature-viewing pair relating to a second video content which has similarities with the first video content, wherein the second feature-viewing pair has been confirmed as correct; calculating similarity between the first feature-viewing pair and the second feature-viewing pair; and determining the area of interest in the first video content based on the visual features of the first feature-viewing pair when the similarity exceeds a pre-determined threshold value. It may be beneficial to validate the viewing history data of other users, especially in the case of conventional (non-professional) users, since some users may watch for long time areas which are likely to be non-interesting for other users. The viewing history data may comprise the viewing direction and/or gaze direction data and visual features present in the video regions corresponding to the viewing direction and/or gaze direction data, i.e. the visual features that have caught the attention of the users. Let us denote a user whose viewing history data is validated as a first user. The first user may be e.g. a conventional consumer. The first user views a first video content (the 360 degrees video content) with the video playback apparatus, wherein the first video content is the content whose viewing history data is validated. A first feature-viewing pair may be determined from the viewing history data, based on visual features of the first video content and actualized viewing/gaze patterns. The gaze of the first user may follow an actor running through the scene of the theatre play. The first feature-viewing pair may be determined as a “running human - actualized viewing”. The first user is interested in beautiful props, e.g. painted trees, visible at the scene, and thus the first user focuses one’s gaze on the painted trees. Another feature-viewing pair may be determined as “trees - actualized viewing”.

A second video content has similarities with the first video content. For example, the second video content may also comprise a theatre play of the same genre. A second feature-viewing pair relating to the second video content may be received e.g. from a database. The second feature-viewing pair may comprise also a running human and actualized viewing. The featureviewing pairs relating to the second video content may have been confirmed as correct, e.g. by the content producers and/or the professional test users. The similarity between the first feature-viewing pair and the second featureviewing pair may be calculated. The feature-viewing pairs may be inputted to a neural network. The neural network may embed the feature-viewing pairs into an embedding space, where similarity may be computed. For example, by using a neural network pre-trained on a large visual dataset such as ImageNet, it is possible to extract visual features which represent a compressed information or an embedding, which are suitable for similarity computation. The actual similarity may be computed for example by cosine similarity, or by dot product between the embeddings (feature vectors) extracted from the two contents to be compared. When the similarity exceeds a pre-determined threshold value, the first feature-viewing pair may be classified as correct. Then, the visual features of the feature-viewing pair may be determined as the area of interest in the first video content. In this example, pixels or voxels representing the running human may be determined as the area of interest.

On the other hand, another feature-viewing pair determined as “trees actualized viewing” may be classified as incorrect. The props may be created in a way that they are attractive, but probably they are not so important for a storyline of the theatre play. Probably they are not meant to be followed instead of e.g. the actors. Thus, it may be that corresponding feature-viewing pair for “trees - actualized viewing” is not found from feature-viewing pairs relating to the second video content. It may be determined that the video region corresponding to the trees is outside of the area of interest, and its quality may be decreased.

Embodiments of a method for adaptive displaying of video content may be implemented by an apparatus comprising means for performing the embodiments of the method. The means of the apparatus may comprise at least one processor, a memory, and a computer program being stored in the memory. The apparatus may comprise means for receiving a first video content comprising an area of interest, wherein the area of interest is received with a first quality and the area outside of the area of interest is received with a second quality, wherein the first quality is higher than the second quality; means for displaying the first video content in a video playback apparatus; means for receiving viewing direction data of a user of the video playback apparatus; means for determining, in the first video content, a current focus area of the user based on the viewing direction data; and means for decreasing quality of a video region in the first video content corresponding to the current focus area of the user, when the current focus area of the user is outside of the area of interest.

Embodiments of a method for adaptive streaming of video content may be implemented by an apparatus comprising means for performing the embodiments of the method. The means of the apparatus may comprise at least one processor, a memory, and a computer program being stored in the memory. The apparatus may comprise means for obtaining information on at least one area of interest within a first video content. The apparatus may comprise means for providing the first video content for transmission, the first video content comprising at least one area of interest, wherein the at least one area of interest is provided with a first quality and the area outside ofthe at least one area of interest is provided with a second quality, wherein the first quality is higher than the second quality.

The various embodiments may provide advantages. The adaptive streaming method reduces communication bandwidth requirements. Further, problems relating to latency between transmitting, from the video playback device, user’s orientation measurement data to a streaming server and receiving, in the video playback device, the adapted stream from a streaming server may be avoided by the adaptive displaying method. The adaptive displaying method may guide user’s attention towards desired areas.

It is obvious that the present invention is not limited solely to the abovepresented embodiments, but it can be modified within the scope of the appended claims.

Claims

It is obvious that the present invention is not limited solely to the abovepresented embodiments, but it can be modified within the scope of the appended claims. Claims:

1. A method, comprising:

receiving a first video content comprising an area of interest, wherein the area of interest is received with a first quality and the area outside of the area of interest is received with a second quality, wherein the first quality is higher than the second quality;

displaying the first video content in a video playback apparatus; receiving viewing direction data of a user of the video playback apparatus; determining, in the first video content, a current focus area of the user based on the viewing direction data; and decreasing quality of a video region in the first video content corresponding to the current focus area of the user, when the current focus area of the user is outside of the area of interest.

2. The method according to claim 1, further comprising determining a path in the first video content from the current focus area of the user towards the area of interest, when the current focus area is outside of the area of interest; and increasing quality of a video region corresponding to the path.

3. The method according to claim 1 or 2, wherein decreasing quality of the video region corresponding to the current focus area of the user, when the current focus area of the user is outside of the area of interest, comprises decreasing resolution.

4. The method according to any of the claims 1 to 3, wherein decreasing quality of the video region corresponding to the current focus area of the user, when the current focus area of the user is outside of the area of interest, comprises decreasing bitrate.

5. The method according to any of the claims 1 to 4, wherein decreasing quality of the video region corresponding to the current focus area of the user, when the current focus area of the user is outside of the area of interest, comprises removing an object present in the current focus area of the user.

6. The method according to any of the claims 1 to 5, further comprising determining, based on the viewing direction data, that the user’s current focus area remains outside of the area of interest;

evaluating conditions for adding the current focus area to a group of areas of interest; and adding the current focus area to the group of areas of interest when the conditions are fulfilled.

7. The method according to any of the claims 1 to 6, wherein the first video content is 360 degrees video content.

8. The method according to any of the claims 1 to 7, wherein the video playback apparatus is a head mounted display.

9. The method according to any of the claims 1 to 8, wherein the viewing direction data comprises gaze direction data of the user of the video playback apparatus.

10. A method, comprising obtaining information on at least one area of interest within a first video content; and providing the first video content for transmission, wherein the at least one area of interest is provided with a first quality and the area outside of the at least one area of interest is provided with a second quality, wherein the first quality is higher than the second quality.

11. The method according to claim 10, further comprising determining the at least one area of interest in the first video content based on saliency detection.

12. The method according to claim 10, further comprising receiving viewing history data of other users relating to the first video content; and determining the area of interest in the first video content based on the viewing history data of other users.

13. The method according to claim 12, further comprising determining a first feature-viewing pair based on visual features of the first video content and actualized viewing patterns derived from the viewing history data of other users relating to the first video content;

receiving a second feature-viewing pair relating to a second video content which has similarities with the first video content, wherein the second featureviewing pair has been confirmed as correct;

calculating similarity between the first feature-viewing pair and the second feature-viewing pair; and determining the area of interest in the first video content based on the visual features of the first feature-viewing pair when the similarity exceeds a predetermined threshold value.

14. An apparatus comprising at least one processor, memory including computer program code, the memory and the computer program code configured to, with the at least one processor, cause the apparatus to receive a first video content comprising an area of interest, wherein the area of interest is received with a first quality and the area outside of the area of interest is received with a second quality, wherein the first quality is higher than the second quality;

display the first video content in a video playback apparatus; receive viewing direction data of a user of the video playback apparatus; determine, in the first video content, a current focus area of the user based on the viewing direction data; and decrease quality of a video region in the first video content corresponding to the current focus area of the user, when the current focus area of the user is outside of the area of interest.

15. The apparatus according to claim 14, further caused to determine a path in the first video content from the current focus area of the user towards the area of interest, when the current focus area is outside of the area of interest; and increase quality of a video region corresponding to the path.

16. The apparatus according to claim 14 or 15, wherein decreasing quality of the video region corresponding to the current focus area of the user, when the current focus area of the user is outside of the area of interest, comprises decreasing resolution.

17. The apparatus according to any of the claims 14 to 16, wherein decreasing quality of the video region corresponding to the current focus area of the user, when the current focus area of the user is outside of the area of interest, comprises decreasing bitrate.

18. The apparatus according to any of the claims 14 to 17, wherein decreasing quality of the video region corresponding to the current focus area of the user, when the current focus area of the user is outside of the area of interest, comprises removing an object present in the current focus area of the user.

19. The apparatus according to any of the claims 14 to 18, further caused to determine, based on the viewing direction data, that the user’s current focus area remains outside of the area of interest;

evaluate conditions for adding the current focus area to a group of areas of interest; and add the current focus area to the group of areas of interest when the conditions are fulfilled.

20. The apparatus according to any of the claims 14 to 19, wherein the first video content is 360 degrees video content.

21. The apparatus according to any of the claims 14 to 20, wherein the video playback apparatus is a head mounted display.

22. The apparatus according to any of the claims 14 to 21, wherein the viewing direction data comprises gaze direction data of the user of the video playback apparatus.

23. An apparatus comprising means for receiving a first video content comprising an area of interest, wherein the area of interest is received with a first quality and the area outside of the area of interest is received with a second quality, wherein the first quality is higher than the second quality;

24. The apparatus according to claim 23, further comprising means for determining a path in the first video content from the current focus area of the user towards the area of interest, when the current focus area is outside of the area of interest; and increasing quality of a video region corresponding to the path.

25. The apparatus according to claim 23 or 24, wherein decreasing quality of the video region corresponding to the current focus area of the user, when the current focus area of the user is outside of the area of interest, comprises decreasing resolution.

26. The apparatus according to any of the claims 23 to 25, wherein decreasing quality of the video region corresponding to the current focus area of the user, when the current focus area of the user is outside of the area of interest, comprises decreasing bitrate.

27. The apparatus according to any of the claims 23 to 26, wherein decreasing quality of the video region corresponding to the current focus area of the user, when the current focus area of the user is outside of the area of interest, comprises removing an object present in the current focus area of the user.

28. The apparatus according to any of the claims 23 to 27, further comprising means for determining, based on the viewing direction data, that the user’s current focus area remains outside of the area of interest;

29. The apparatus according to any of the claims 23 to 28, wherein the first video content is 360 degrees video content.

30. The apparatus according to any of the claims 23 to 29, wherein the video playback apparatus is a head mounted display.

31. The apparatus according to any of the claims 23 to 30, wherein the viewing direction data comprises gaze direction data of the user of the video playback apparatus.

32. A computer program product embodied on a non-transitory computer readable medium, comprising computer program code configured to, when executed on at least one processor, cause an apparatus to perform at least the method according to any of the claims 1 to 9.

33. An apparatus comprising at least one processor, memory including computer program code, the memory and the computer program code configured to, with the at least one processor, cause the apparatus to obtain information on at least one area of interest within a first video content; and provide the first video content for transmission, the first video content comprising at least one area of interest, wherein the at least one area of interest is provided with a first quality and the area outside of the at least one area of interest is provided with a second quality, wherein the first quality is higher than the second quality.

34. The apparatus according to claim 33, further caused to determine the at least one area of interest in the first video content based on saliency detection.

35. The apparatus according to claim 33, further caused to receive viewing history data of other users relating to the first video content; and determine the area of interest in the first video content based on the viewing history data of other users.

36. The apparatus according to claim 35, further caused to determine a first feature-viewing pair based on visual features of the first video content and actualized viewing patterns derived from the viewing history data of other users relating to the first video content;

receive a second feature-viewing pair relating to a second video content which has similarities with the first video content, wherein the second featureviewing pair has been confirmed as correct;

calculate similarity between the first feature-viewing pair and the second feature-viewing pair; and determine the area of interest in the first video content based on the visual features of the first feature-viewing pair when the similarity exceeds a predetermined threshold value.

37. An apparatus comprising means for