KR102284913B1 - Users Field of View selection method and apparatus in 360 degree videos - Google Patents

Abstract

Disclosed are a method for selecting a user's viewing angle in a 360-degree image and a device thereof. A method of selecting the user's viewing angle in a 360-degree image includes the steps of: (a) preprocessing the 360-degree image and dividing each into three FoV (Field of View) images having a 120-degree horizontal view; (b) calculating a salience score for each of the FoV images; and (c) selecting and transmitting any one of the three FoV images using the salience score for each FoV image. Therefore, bandwidth waste can be reduced by allowing only interesting FoVs to be transmitted in the 360-degree image.

Description

User Field of View selection method and apparatus in 360 degree videos}

The present invention relates to a method and apparatus for selecting a user's viewing angle in a 360-degree image.

The 360-degree camera provides a complete view of the immersive world, making it more immersive than traditional cameras. These 360-degree cameras are being adopted in the latest technologies and applications such as virtual reality (VR) and augmented reality (AR) because they provide an enjoyable experience for users.

Due to high demand, it has recently been created and supported by tech and social media giants such as Facebook, YouTube, and Google. Recently, more than 1 million 360-degree video content and 25 million 360-degree images have been uploaded to Facebook.

Compared to traditional 2D video, these 360-degree videos provide users with an exciting experience through the illusion of being in virtual content. It has attracted users, giants and researchers to the field of 360-degree video, but at the same time, it faces new challenges while exploring various applications.

360-degree video content is problematic due to the high resolution of the image and the limited human field of view (FoV) for visual content. First of all, the wide range of 360-degree video makes it very difficult for users to choose "where to watch". For this reason, in the conventional case, as shown in FIG. 1 , a method of manually selecting an FoV by searching for a FoV in a 360-degree video using a wearable device such as a head-mounted display (HMD) is used.

An object of the present invention is to provide a method and an apparatus for selecting a user's viewing angle in a 360-degree image.

Another object of the present invention is to provide a method and apparatus for selecting a user's viewing angle in a 360-degree image that can automatically select the FoV most interesting to the user in the 360-degree image.

In addition, the present invention detects a saliency object existing in a 360-degree image using a deep learning technique, and based on this, a user in a 360-degree image that can automatically select the most interesting and memorable FoV of the user. An object of the present invention is to provide a viewing angle selection method and an apparatus therefor.

Another object of the present invention is to provide a method and an apparatus for selecting a user's viewing angle in a 360-degree image that can reduce bandwidth waste by transmitting only interesting FoV from a 360-degree image.

According to one aspect of the present invention, a method for selecting a user's viewing angle in a 360-degree image is provided.

According to an embodiment of the present invention, (a) pre-processing a 360-degree image and dividing each of the three FoV (Field of View) images having a 120-degree horizontal view (view); (b) calculating a salience score for each FoV image; and (c) selecting and transmitting any one of the three FoV images using the saliency score for each FoV image.

The step (b) may include: applying each FoV image to an object detection model to detect an object, and then deriving an object class and an accuracy value of each detected object; calculating an ease of memory score for each FoV image by applying each of the FoV images to a previously learned memorability calculation model; and calculating a saliency score for each FoV image using the object class and accuracy value of each object and the memorization score.

The saliency score for each FoV image is calculated using the following equation,

는 밸런싱 가중치를 나타낸다. Here, X, Y, and Z represent the total number of objects included in each object class, P, A, and V represent each object included in each object class, acc represents the accuracy value of each object, and M represents the memory recall score,

denotes a balancing weight.

In step (c), the FoV image having the highest saliency score among the three FoV images may be selected and transmitted as a viewport.

Each of the FoV images may be generated by dividing the number of pixels in the entire column of the 360-degree image into three so that the horizontal field of view has the same 120 degrees.

The object class of each detected object is classified into any one of the classified classes, and after deriving the accuracy values of the previously classified class and each of the detected objects, each detected as the class having the highest accuracy value The object class of the object may be determined.

According to another aspect of the present invention, an apparatus for selecting a user's viewing angle in a 360-degree image is provided.

According to an embodiment of the present invention, there is provided an apparatus comprising: a memory for storing at least one instruction; and a processor executing the instructions stored in the memory, wherein the instructions executed by the processor include: (a) three Field of View (FoV) images having a 120-degree horizontal view by preprocessing a 360-degree image dividing each into (b) calculating a salience score for each FoV image; and (c) selecting and transmitting any one of the three FoV images using the saliency score for each FoV image.

There is an advantage in that the FoV most interesting to the user can be automatically selected from the 360-degree image according to an embodiment of the present invention.

In addition, the present invention has the advantage of automatically selecting the most interesting and memorable FoV of the user based on the detection of a saliency object existing in a 360-degree image using a deep learning technique.

In addition, the present invention can reduce bandwidth waste by allowing only interesting FoVs to be transmitted in a 360-degree image.

1 is a diagram illustrating a method of manually searching for a FoV in a 360-degree video using a conventional head-mounted displays (HMD);
2 is a flowchart illustrating a method for selecting a user's viewing angle in a 360-degree image according to an embodiment of the present invention.
3 is a view for explaining the resolution of a 360-degree video frame according to an embodiment of the present invention.
4 is a view for explaining the resolution of the divided FoV according to an embodiment of the present invention.
5 is a diagram illustrating equally divided FoV to have the same horizontal field of view according to an embodiment of the present invention;
6 is a diagram illustrating a saliency table according to an embodiment of the present invention.
7 is a block diagram schematically illustrating an internal configuration of an apparatus capable of automatically selecting a user's viewing angle in a 360-degree image according to an embodiment of the present invention.

As used herein, the singular expression includes the plural expression unless the context clearly dictates otherwise. In this specification, terms such as “consisting of” or “comprising” should not be construed as necessarily including all of the various components or various steps described in the specification, some of which components or some steps are It should be construed that it may not include, or may further include additional components or steps. In addition, terms such as "...unit" and "module" described in the specification mean a unit that processes at least one function or operation, which may be implemented as hardware or software, or a combination of hardware and software. .

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

2 is a flowchart illustrating a method for selecting a user's viewing angle in a 360-degree image according to an embodiment of the present invention, and FIG. 3 is a diagram illustrating the resolution of a 360-degree video frame according to an embodiment of the present invention. , FIG. 4 is a diagram illustrating the resolution of a divided FoV according to an embodiment of the present invention, and FIG. 5 illustrates an equally divided FoV to have the same horizontal field of view according to an embodiment of the present invention 6 is a diagram illustrating a saliency table according to an embodiment of the present invention.

In step 210 , the device 100 preprocesses the input immersive video frame and divides it into three FoV images each having the same horizontal view.

This will be described in more detail.

The input immersive video frame is a 360 degree image. In the case of a general 2D image, the resolution is determined based on the horizontal and vertical dimensions of the image. However, in the case of a 360-degree image, it is somewhat complicated due to the panorama. In the case of such an immersive video (hereinafter referred to as a 360-degree video frame), the content is stretched by 360 degrees horizontally and 180 degrees vertically, and the entire scene can be divided by the viewer's two eyes. . Thus, the user's FoV is limited to 120 degrees rather than a full 360 degree video frame. Accordingly, according to an embodiment of the present invention, three Field of View (FoV) images having the same horizontal field of view 120 as a 360-degree video frame may be divided into 120-degree respectively.

360 degree video frames are provided in various resolutions from 2k to 16k, as shown in FIG. 3 . Accordingly, the equation of Equation 1 may be used to automatically adjust the resolution of the user's FoV image.

Here, r and c represent the number of pixels in rows and columns, respectively, and pixels represent the total number of pixels in a 360 degree video frame.

The device 100 may split the 360-degree video frame into three FoVs with the same horizontal field of view at 120 degrees based on the total number of columns of the 360-degree video frame.

For example, if this is expressed as an equation, it is equal to Equation 2.

는 120도의 수평 시야(view)이다. 수학식 2에 의해 장치는 360도 비디오 프레임으로부터 120도 수평 시야를 가지는 FoV 영상으로 각각 분할될 수 있다. here,

is a horizontal view of 120 degrees. By Equation 2, the device can be divided into FoV images each having a 120-degree horizontal field of view from a 360-degree video frame.

An example of dividing a 360-degree video frame into three FoV images having the same 120-degree horizontal field of view is shown in FIG. 4 .

The apparatus 100 may divide the input 360-degree video frame into three FoVs each having a 120-degree field of view, and adjust the resolution of each FoV based on the size of the input 360-degree video frame.

For example, a 360-degree video frame having 12k FoV composed of 12000 pixels has a 120-degree field of view and may be divided into 4k FoV composed of 4000 pixels, respectively. The resolution of UFoV of a 360-degree video frame based on various input resolutions is shown in FIG. 5 .

In step 215, the device 100 detects an object by applying each FoV image to a deep learning-based object detection model, and derives an object class and accuracy value for the detected object.

The deep learning-based object detection model may be an SSDLite (Single Shot Detector)-based model.

The object detection model assumes that classes for various objects are learned in advance. Here, the class may be a person, a car, an animal, and the like. It goes without saying that the types of classes may be more diverse than others.

That is, the device 100 detects an object in each FoV image through a deep learning-based object detection model, and then derives an accuracy value with the class (object class) from the sensed object to detect the highest accuracy class It can be derived by determining the object class of the object.

Through this, the apparatus 100 may store the object class and accuracy value for each FoV image in a table, as shown in FIG. 6 .

In step 220 , the device 100 derives a memorization score by applying each FoV image to the pre-learned memorability calculation model.

Here, the memorability computational model is a hybrid-AlexNet model, which can be fine-tuned with many annotated image-based databases. In the memory recall calculation model, memory recall scores of various object classes including humans and beautiful natural landscapes may be pre-trained.

Accordingly, the apparatus 100 may calculate a memorability score for each FoV image by applying the learned memorization calculation model to each FoV image.

The memorability calculation model is based on an image-based database, in which memory scores for images containing various objects are learned. Accordingly, the apparatus 100 may derive a memorability score for the corresponding FoV image by applying each FoV image to the learned memory ease calculation model.

The hybrid-AlexNet model for calculating the memory recall score is a known model, and various methods for calculating the known memory score can be equally applied.

In step 225 , the device 100 calculates a saliency score for each FoV image by using the object class and accuracy and memorability scores of each FoV image.

For example, the apparatus 100 may calculate a saliency score for each FoV image by using Equation 3 below.

는 밸런싱 가중치를 나타낸다. Here, X, Y, and Z represent the total number of objects included in each object class, P, A, and V represent each object included in each object class, acc represents the accuracy value of each object, and M represents the memory recall score,

denotes a balancing weight.

In step 230 , the device 100 selects any one of the three FoV images by using the saliency scores for each FoV image.

For example, the device 100 may select any one of the FoV images using Equation 4 below.

That is, the device 100 may select the FoV image having the highest saliency score for the FoV image as the viewport and transmit only the selected viewport image.

Through this, the device 100 automatically selects the most interesting and memorable FoV image to the user and selectively transmits only the corresponding FoV image, thereby reducing bandwidth waste.

Also, according to an implementation method, after a specific FoV image is selected as a viewport, a FoV image may be automatically selected from a 360-degree video frame after the specific FoV image to track a saliency object in the corresponding FoV image.

7 is a block diagram schematically illustrating an internal configuration of an apparatus capable of automatically selecting a user's viewing angle in a 360-degree image according to an embodiment of the present invention.

Referring to FIG. 7 , the apparatus 100 according to an embodiment of the present invention includes a preprocessor 710 , a calculator 715 , a viewport selector 720 , a memory 725 , and a processor 730 . is composed

The preprocessor 710 is a means for preprocessing a 360-degree video frame and dividing it into three FoV images to have the same 120-degree horizontal field of view.

The calculator 715 is a means for calculating a saliency score for each FoV image.

In order to calculate the saliency score, the calculator 715 first detects an object for each FoV image, derives an object class and accuracy value of the detected object, and derives a memorization score for each FoV image. can

Next, the calculator 715 may calculate a saliency score for each FoV image by using the object class, accuracy value, and memorization score of the object detected in each FoV image.

As already described above, the calculator 715 detects each object by applying each FoV image to the deep learning-based object detection model, and classifies each detected object into any one of the subclassified classes, and the accuracy value thereof can be derived.

In addition, the calculator 715 may derive an ease of memory score by applying each FoV image to the previously-learned calculation model for ease of memory.

The number of objects detected for each class classified in each table and an accuracy value for each object may be stored corresponding to each FoV image. Of course, the table may store a memorability score derived for each FoV image.

Accordingly, the calculator 715 may calculate a saliency score for each FoV image using an object class, an accuracy value, and a memorization score for each FoV based on the table.

The viewport selection unit 720 may select any one of three FoV images for a 360-degree video frame as a viewport based on a saliency score for each FoV image.

For example, the viewport selector 720 may select, as a viewport, a FoV image having the highest saliency score among FoV images based on the saliency score.

In addition, when selecting any one of the FoV images as the viewport using the saliency score, the viewport selection unit 720 converts the FoV image including the main object (saliency object) included in the previously selected FoV image to the viewport. You can also choose At this time, when the FoV image including the main object included in the previously selected FoV image is selected as the viewport, if the saliency score of the corresponding FoV image is less than or equal to the reference value, the FoV image with the highest saliency score does not follow the main object. can be selected as the viewport.

The memory 725 stores instructions necessary for performing the method for selecting a user's viewing angle in a 360-degree image according to an embodiment of the present invention.

The processor 730 includes internal components (eg, a preprocessor 710 , a calculator 715 , a viewport selector 720 , and a memory 725 ) of the device 100 according to an embodiment of the present invention. etc.) is a means to control. In addition, although not described separately, the processor 730 may execute instructions stored in the memory, and the instructions executed by the processor may include (perform) each step as described with reference to FIGS. 1 to 6 . there is.

The apparatus and method according to an embodiment of the present invention may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions recorded on the computer readable medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the computer software field. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floppy disks. - Includes magneto-optical media and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

So far, the present invention has been looked at focusing on the embodiments thereof. Those of ordinary skill in the art to which the present invention pertains will understand that the present invention can be implemented in modified forms without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments should be considered in an illustrative rather than a restrictive sense. The scope of the present invention is indicated in the claims rather than the foregoing description, and all differences within the scope equivalent thereto should be construed as being included in the present invention.

100: device
710: preprocessor
715: calculator
720: viewport selection
725: memory
730: processor

Claims (9)

(a) pre-processing a 360-degree image and dividing each into three FoV (Field of View) images having a 120-degree horizontal view;
(b) calculating a salience score for each FoV image; and
(c) selecting and transmitting any one of the three FoV images using the saliency score for each FoV image,
Step (b) is,
applying each FoV image to an object detection model to detect an object, and then deriving an object class and an accuracy value of each detected object;
calculating an ease of memory score for each FoV image by applying each of the FoV images to a previously learned memorability calculation model; and
calculating a saliency score for each FoV image using the object class and accuracy value of each object and the memorability score,
The object class of each detected object is classified into any one of the classified classes,
User viewing angle selection in a 360-degree image, characterized in that the object class of each detected object is determined as the class having the highest accuracy value after deriving the accuracy values of the previously classified class and each of the detected objects method.
delete According to claim 1,
A saliency score for each FoV image is calculated using the following equation.

Here, X, Y, and Z represent the total number of objects included in each object class, P, A, and V represent each object included in each object class, acc represents the accuracy value of each object, and M represents the memory recall score,

represents the balancing weight.
According to claim 1,
The step (c) is,
A method for selecting a user's viewing angle in a 360-degree image, characterized in that the FoV image having the highest saliency score among the three FoV images is selected and transmitted as a viewport.
According to claim 1,
Each of the FoV images is generated by dividing the number of pixels in the entire column of the 360-degree image into thirds so that the horizontal field of view is the same 120 degrees.
delete A computer-readable recording medium in which a program code for performing the method of any one of claims 1 to 5 is recorded.
In the device,
a memory storing at least one instruction; and
A processor for executing instructions stored in the memory,
The instructions executed by the processor are
(a) pre-processing a 360-degree image and dividing each into three FoV (Field of View) images having a 120-degree horizontal view;
(b) calculating a salience score for each FoV image; and
(c) selecting and transmitting any one of the three FoV images using the saliency score for each FoV image,
Step (b) is,
applying each FoV image to an object detection model to detect an object, and then deriving an object class and an accuracy value of each detected object;
calculating an ease of memory score for each FoV image by applying each of the FoV images to a previously learned memorability calculation model; and
Calculating a saliency score for each FoV image using the object class and accuracy value of each object and the memorability score,
The object class of each detected object is classified into any one of the classified classes,
After deriving an accuracy value between the previously classified class and each of the detected objects, the object class of each detected object is determined as a class having the highest accuracy value.

delete

Images