KR102034826B1 - Apparatus and method for providing virtual reality video - Google Patents

Abstract

A virtual reality video providing method performed by a server according to an embodiment of the present invention, the method comprising: (a) converting a first video frame into a plurality of frame pieces based on a reference viewpoint; (b) cropping the remaining frame pieces of the frame piece corresponding to the reference time point among the frame pieces; And (c) providing a second video frame including cropped frame pieces, wherein the first video frame corresponds to a 360 degree image, and a reference point is a point of view viewed by the user on the first video frame. Can be.

Description

Virtual Reality Video Providing Device and Provision Method {APPARATUS AND METHOD FOR PROVIDING VIRTUAL REALITY VIDEO}

The present invention relates to an apparatus and method for providing a virtual reality video.

In the conventional case, when the entire frame of the virtual reality video is sent to the user terminal, the user terminal displays only a portion of the image corresponding to the entire frame corresponding to the direction that the user views. In this case, even though each frame of the virtual reality video transmitted from the server has a resolution of 4k, only the resolution of 540p is shown in the user terminal. In other words, only a part of the video frames transmitted from the server is displayed.

Korean Laid-Open Patent Publication No. 10-2003-0040808 (name of the invention: an imaging device capable of simultaneously capturing 360 degrees)

An object of the present invention is to increase the ratio of the displayed portion of the entire frame of the virtual reality video to be transmitted by excluding the portion that is less likely to be displayed on the user terminal in the frame of the virtual reality video.

In the virtual reality video providing method performed by the server according to an embodiment of the present invention for achieving the above technical problem, (a) based on a reference time point the first video frame into a plurality of frame pieces Converting; (b) cropping the remaining frame pieces of the frame piece corresponding to the reference time point among the frame pieces; And (c) providing a second video frame including cropped frame pieces, wherein the first video frame corresponds to a 360 degree image, and a reference point is a point of view viewed by the user on the first video frame. Can be.

In addition, the apparatus for providing a virtual reality video according to another embodiment of the present invention includes a memory in which a program for executing the method for providing a virtual reality video is recorded; And a processor for executing a program, wherein the processor converts the first video frame into a plurality of frame pieces based on the reference view point by executing the program, and a frame corresponding to the reference view point among the frame pieces. Crop the remaining frame pieces of the piece and provide a second video frame including the cropped frame pieces, wherein the first video frame corresponds to a 360 degree image, and the reference point is determined by the user on the first video frame. It may be a point of view.

According to an apparatus and a method for providing a virtual reality video according to an embodiment of the present invention, a display of the entire frames of the transmitted virtual reality video is transmitted by excluding a portion of the virtual reality video frame that is less likely to be displayed on the user terminal. The ratio of parts to be increased can be increased.

In addition, according to the apparatus and method for providing a virtual reality video according to an embodiment of the present invention, when the viewpoint viewed by the user is more than a predetermined distance from the reference viewpoint, the virtual reality video converted based on the new reference viewpoint may be provided. have.

1 is a diagram showing the configuration of a virtual reality video providing system according to an embodiment of the present invention.
2 is a diagram showing the configuration of a server according to an embodiment of the present invention.
3 is an operation flowchart illustrating a virtual reality video providing method performed by a server according to an embodiment of the present invention.
4 is a diagram illustrating a method of dividing an entire frame of a virtual reality video into frame pieces according to an embodiment of the present invention.
5 is a view showing a method of converting the frame pieces to correspond to the cube in accordance with an embodiment of the present invention.
6 is a diagram illustrating a method of performing crop on frame pieces according to an embodiment of the present invention.
7 is a diagram illustrating a frame of a converted virtual reality video according to an embodiment of the present invention.
8 is a diagram illustrating a method of changing a reference time point according to an embodiment of the present invention.

DETAILED DESCRIPTION Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art may easily implement the present invention. As those skilled in the art would realize, the described embodiments may be modified in various different ways, all without departing from the spirit or scope of the present invention. In the drawings, parts irrelevant to the description are omitted in order to clearly describe the present invention, and like reference numerals designate like parts throughout the specification.

Throughout the specification, when a part is "connected" to another part, this includes not only "directly connected" but also "electrically connected" with another element in between. . In addition, when a part is said to "include" a certain component, which means that it may further include other components, except to exclude other components, unless specifically stated otherwise, one or more other features It is to be understood that the present disclosure does not exclude the possibility of the presence or the addition of numbers, steps, operations, components, parts, or combinations thereof.

In the entire specification, the server 100 may mean a virtual reality video providing device.

The following examples are detailed description to aid in understanding the present invention, and do not limit the scope of the present invention. Therefore, the same range of inventions that perform the same functions as the present invention will also fall within the scope of the present invention.

1 is a diagram showing the configuration of a virtual reality video providing system 1 according to an embodiment of the present invention.

Referring to FIG. 1, a virtual reality video providing system 1 according to an embodiment of the present invention includes a server 100 and a user terminal 200. In addition, the server 100 includes a communication network 300 interconnecting the plurality of user terminals 201 and 202.

The server 100 according to an embodiment of the present invention provides a virtual reality video to the user terminal 200, and the user terminal 200 displays the virtual reality video provided from the server 100 to the user. do.

Here, the virtual reality video may be a video in which a part of the original video composed of a 360 degree image is cropped by the server 100.

Here, crop refers to cropping so that only a part of a video or an image remains. For example, in a video or image having a resolution of 1920x1080, when cropping by a resolution of 960x540 from an upper left corner, an area except for an area corresponding to a resolution of 960x540 may be removed.

The user terminal 200 may display a portion of the virtual reality video provided from the server 100 corresponding to the reference viewpoint viewed by the user.

Here, the reference viewpoint may include three-dimensional direction information. For example, the reference time point may include pitch and yaw values.

As an optional embodiment, the reference time point may further include a roll value.

The user terminal 200 may display the virtual reality video through the terminal application. The terminal application may be an application dedicated to playing a virtual reality video.

Here, the virtual reality video reproduction only application may be an application embedded in the user terminal 200 or an application downloaded from an application distribution server and installed in the user terminal 200.

The plurality of user terminals 200 refers to a communication terminal that can use a terminal application in a wired / wireless communication environment. The user terminal 200 may be a portable terminal of the user. In FIG. 1, although the user terminal 200 is illustrated as a smart phone, which is a kind of portable terminal, the spirit of the present invention is not limited thereto, and as described above, the terminal capable of mounting a terminal application is not limited thereto. Can be borrowed.

In more detail, the user terminal 200 may be a handheld computing device (eg, PDA, email client, etc.), any form of cellular phone, a head mounted display (HMD) device, or other type of computing. Or any form of communication platform, but the present invention is not limited thereto.

Meanwhile, the communication network 300 connects the server 100 and the user terminal 200. That is, the communication network 300 refers to a communication network that provides a connection path for transmitting and receiving data after the user terminals 200 access the server 100. The communication network 300 may be, for example, wired networks such as local area networks (LANs), wide area networks (WANs), metropolitan area networks (MANs), integrated service digital networks (ISDNs), wireless LANs, CDMA, Bluetooth, satellite communications, and the like. Although it may encompass a wireless network, the scope of the present invention is not limited thereto.

2 is a diagram showing the configuration of a server 100 according to an embodiment of the present invention.

Referring to FIG. 2, the server 100 according to an embodiment of the present invention includes a communication module 110, a memory 120, a processor 130, and a database 140.

In detail, the communication module 110 provides a communication interface necessary to provide a transmission / reception signal between the server 100 and the user terminal 200 in the form of packet data in cooperation with the communication network 300. In addition, the communication module 110 may receive a data request from the user terminal 200 and transmit a data as a response thereto.

Here, the communication module 110 may be a device including hardware and software necessary for transmitting and receiving a signal such as a control signal or a data signal through a wired or wireless connection with another network device.

In the memory 120, a program for performing a virtual reality video providing method is recorded. In addition, the processor 130 performs a function of temporarily or permanently storing data processed. Here, the memory 120 may include a magnetic storage media or a flash storage media, but the scope of the present invention is not limited thereto.

The processor 130 is a kind of central processing unit that controls the entire process of providing the virtual reality video. Each step performed by the processor 130 will be described later with reference to FIG. 3.

Here, the processor 130 may include all kinds of devices capable of processing data, such as a processor. Here, the 'processor' may refer to a data processing apparatus embedded in hardware having, for example, a circuit physically structured to perform a function represented by code or instructions included in a program. As an example of a data processing device embedded in hardware, a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, and an application-specific integrated device (ASIC) It may include a processing device such as a circuit, a field programmable gate array (FPGA), etc., but the scope of the present invention is not limited thereto.

The database 140 may store an entire frame of the virtual reality video and a video frame cropped for providing to the user terminal 200.

Although not shown in FIG. 2, some of the entire frames and cropped video frames of the virtual reality video may be stored in a database (not shown) that is physically or conceptually separated from the database 140.

3 is an operation flowchart illustrating a virtual reality video providing method performed by a server according to an embodiment of the present invention.

Referring to FIG. 3, first, a virtual reality video providing method performed by a server according to an embodiment of the present invention converts a first video frame into a plurality of frame pieces based on a reference viewpoint (S310).

Here, the virtual reality video may correspond to a 360 degree video, and the first video frame may correspond to a 360 degree image.

In addition, the reference time point may be a point that the user views on the first video frame.

In operation S310, the server may split the first video frame into six frame pieces.

Here, the server may divide the 360-degree image corresponding to the first video frame into six image pieces having an angle of view of 90 degrees, and divide the first video frame into six frame pieces so as to correspond to each of the image pieces. have.

Here, the six frame pieces may include a reference frame piece having a reference point as a center point. In addition, the six frame pieces include a right face, a rear face, a left face, a top face, and a bottom face when the face corresponding to the reference frame piece is the front face. Frame pieces corresponding to the bottom face may be included.

In operation S310, the server may convert the frame pieces so that each of the frame pieces corresponds to each side of the cube.

Here, the server may convert the six frame pieces in which the first video frame is divided into a square shape so as to correspond one-to-one to six surfaces of the cube.

Next, in the virtual reality video providing method performed by the server according to an embodiment of the present invention, cropping is performed on the remaining frame pieces of the frame piece corresponding to the reference time point among the frame pieces (S320).

In operation S320, the server may correspond to the six pieces of the frame one-to-one with the six sides of the cube and expand the surfaces of the cube on a plane to generate a developed view.

Here, when the plane corresponding to the reference frame piece is a reference plane, the server may generate a developed view in which the four surfaces adjacent to the reference plane are in contact with the reference plane on the developed view.

For example, when the six surfaces corresponding to the six frame pieces are front, right, rear, left, top, and bottom, the front becomes the reference plane, and the right, left, top, and bottom adjacent to the front face are in contact with the reference plane in the development view. You can create flat patterns.

According to the form of the development view as described above, when the server performs the crop on the development view, the cropping may be performed on the four surfaces (eg, the right side, the left side, the top side, and the bottom side) that are in contact with the reference plane.

In operation S320, the server may set an extended area in which the reference area extends in a direction with respect to each of the four surfaces in contact with the reference plane, using the area corresponding to the reference plane as a reference area, and perform the crop based on the extended area. have.

Here, the server may set the extension area such that the area ratio of each of the four surfaces included in the extension area is a preset reference area ratio.

For example, the server may set the extended area such that an area ratio of each of the right, left, top, and bottom surfaces included in the extended area is 50%. Details thereof will be described later with reference to FIG. 6.

As an optional embodiment, the server may set the extended area such that the area ratio of each of the four surfaces included in the extended area has a different ratio.

For example, the server may set the extended area such that the area ratio of the right and left surfaces included in the extended area is 100% and the area ratio of the upper and bottom surfaces included in the extended area is 50%.

Accordingly, according to an embodiment of the present invention, in consideration of the case in which the binocular angle of view of the user is wider from side to side, the frame pieces may be cropped to have a wider angle of view from side to side.

Next, the virtual reality video providing method performed by the server according to an embodiment of the present invention provides a second video frame including cropped frame pieces (S330).

For example, when cropping is performed such that the area ratio of the right surface, the left surface, the top surface, and the bottom surface is 50% in step S320, the server is the front surface (100%), the right surface (50%), the left surface in step S330. A second video frame including frame pieces corresponding to (50%), top surface (50%), and bottom surface (50%) may be provided.

In addition, when cropping is performed such that the area ratio of the right and left surfaces is 100% and the area ratio of the top and bottom surfaces included in the extended area is 50% in step S320, the server is connected to the front surface 100 in step S330. A second video frame including frame pieces corresponding to%), right side (100%), left side (100%), top side (50%), and bottom side (50%) may be provided.

As an optional embodiment, in step S330, the server may arrange the cropped frame pieces to form a rectangle.

For example, if the cropped frame pieces are the frame pieces corresponding to the front (100%), the right (50%), the left (50%), the top (50%) and the bottom (50%), the front (100%) Given that the corner length is the reference length (e.g., 1) and the area is the reference area (e.g., 1x1), the frame pieces cropped to be a rectangle three times the reference area since the total area of the frame pieces is three times the reference area. Can be arranged (eg, 2x1.5, 3x1, etc.).

For example, if the cropped frame pieces are frame pieces corresponding to the front (100%), the right (100%), the left (100%), the top (50%) and the bottom (50%), the front (100%) Given that the corner length is the reference length (e.g., 1) and the area is the reference area (e.g., 1x1), the area is four times the reference area, so that the cropped frame pieces are arranged (e.g., four times the reference area). , 2x2, 4x1, etc.).

Meanwhile, the 360 degree video provided to the user terminal in real time includes a plurality of second video frames. In this case, steps S310 to S330 may be repeatedly performed every time a first video frame exists to provide the plurality of second video frames to the user terminal, thereby performing real time streaming of the 360 degree video.

Accordingly, according to an embodiment of the present invention, by transmitting a portion of the virtual reality video frame that is less likely to be displayed on the user terminal, the ratio of the displayed portion of the entire frames of the transmitted virtual reality video can be increased. have.

Although not shown in FIG. 3, the method for providing a virtual reality video performed by the server according to an embodiment of the present invention may include receiving a changed time point (not shown) and whether the changed time point is outside the preset reference angle of view. The method may further include determining whether it is present (not shown), and setting a changed time point as a new reference time point when it is out of the reference angle of view (not shown). Details thereof will be described later with reference to FIG. 8.

4 is a diagram illustrating a method of dividing a frame of a virtual reality video into frame pieces according to an embodiment of the present invention.

Referring to FIG. 4, it can be seen that the entire frame 400 of the virtual reality video is divided into frame pieces 420a to 420f having different layouts according to the reference time point 410.

4A illustrates an example in which the reference point 410 is located at the center of the entire frame 400, and FIG. 4B illustrates a case in which the reference point 410 is located in the upper left of the entire frame 400. Is an example.

The entire frame 400 may correspond to a 360 degree image, and each of the frame pieces 420a to 420f may correspond to an image piece having an angle of view of 90 degrees.

In addition, each of the frame pieces 420a to 420f may correspond to the front surface 420a, the right surface 420b, the rear surface 420c, the left surface 420d, the top surface 420e, and the bottom surface 420f.

5 is a view showing a method of converting the frame pieces to correspond to the cube in accordance with an embodiment of the present invention.

Here, FIG. 5 (a) is an example of the result of dividing the entire frame 400 of the virtual reality video into frame pieces, and FIG. 5 (b) shows frame pieces 423a and 423b which are separated from each other among the frame pieces. ) Is an example of the result 510, Figure 5 (c) is an example of the development view 520 when the frame pieces are converted to correspond to the six sides of the cube.

Here, the developed view 520 may be generated in a form adjacent to the other four surfaces of the reference plane.

For example, with respect to the six faces of the cube, the front view (a) is referred to as the reference plane and the remaining faces are respectively the right face (b), the back face (c), the left face (d), the top face (e), and the bottom face (f). 520 may be formed in a form in which the right surface b, the left surface d, the top surface e, and the bottom surface f are adjacent to the front surface a.

6 is a diagram illustrating a method of performing crop on frame pieces according to an embodiment of the present invention.

Referring to FIG. 6, it can be seen that the frame pieces are cropped according to the extension area 610 set on the developed view 520.

6A illustrates an example in which an extended area 610 is set on the developed view 520, and FIG. 6B illustrates a result 620 of cropping according to the extended area 610. Yes.

The extended area 610 may be set such that the area ratio of the four surfaces b, d, e, and f included in the extended area 610 is a preset reference area ratio.

In the case of FIG. 6A, when the reference area ratio is 50%, it can be seen that the area ratio of the four surfaces b, d, e, and f included in the enlarged area 610 is 50%.

7 is a diagram illustrating a frame 800 of a converted virtual reality video according to an embodiment of the present invention.

Referring to FIG. 7, it can be seen that the frame 800 of the converted virtual reality video corresponds to the image of which the angle of view is limited based on the reference viewpoint 410 viewed by the user 810, not the 360 degree image.

In particular, FIG. 7 is a frame 800 of a virtual reality video converted to have a 50% reference area ratio, and the frame 800 has 180 degrees left and right and up and down with respect to the reference time point 410 viewed by the user 810. An image 820 having an angle of view of 180 degrees may be included.

Although not shown in FIG. 7, when the reference area ratio of the right and left surfaces is converted to 100%, the frame 800 has an angle of view of 270 degrees left and right with respect to the reference time point 410 viewed by the user 810. An image (not shown) may be included.

8 is a diagram illustrating a method of changing a reference time point according to an embodiment of the present invention.

In particular, FIG. 8 illustrates a frame 900 of the virtual reality video converted by the reference viewpoint 410 in a two-dimensional image.

When the viewing angle 910 of the user 810 is 60 degrees left and right, an image 920 having an angle of view of 180 degrees left and right is included in the frame 900 of the converted virtual reality video. Even if it is moved to an extent, an image corresponding to the changed time point may be displayed.

However, since the frame 900 of the converted virtual reality video does not include the entire 360-degree image, when the viewpoint of the user 810 moves more than a predetermined time, the frame of the converted virtual reality video is based on a new reference viewpoint. It is necessary to provide (not shown).

First, the server 100 receives a changed time point (not shown) based on the direction 830 viewed by the user 810. In this case, when the viewing direction 830 or the changed view point (not shown) is out of the reference angle of view area 940, the changed view point (not shown) may be set as a new reference view point.

Accordingly, according to an embodiment of the present invention, when the user's view point moves away from the reference view point by more than a predetermined time point, the converted virtual reality video may be provided based on the new reference view point.

One embodiment of the present invention can also be implemented in the form of a recording medium containing instructions executable by a computer, such as a program module executed by the computer. Computer readable recording media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. In addition, the computer readable recording medium may include a computer storage medium. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

Although the methods and systems of the present invention have been described in connection with specific embodiments, some or all of their components or operations may be implemented using a computer system having a general purpose hardware architecture.

The foregoing description of the present invention is intended for illustration, and it will be understood by those skilled in the art that the present invention may be easily modified in other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are exemplary in all respects and not restrictive. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form.

The scope of the present invention is shown by the following claims rather than the above description, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be construed as being included in the scope of the present invention. do.

1: virtual reality video system
100: server 110: communication module
120: memory 130: processor
140: database 200: user terminal
300: network

Claims (8)

In the virtual reality video providing method performed by the server,
(a) converting the first video frame into a plurality of frame pieces based on the reference viewpoint;
(b) cropping the remaining frame pieces of the frame pieces corresponding to the reference time point among the frame pieces; And
(c) providing a second video frame including the cropped frame pieces to a user terminal;
Including,
In step (a),
Dividing the first video frame into a plurality of frame pieces, and setting each of the frame pieces to correspond to each surface constituting a hexahedral exploded view;
In step (b),
And cropping a portion of the plurality of frame pieces corresponding to the reference time point and a portion of at least one frame piece disposed at an adjacent position with respect to the reference frame piece.
The first video frame corresponds to a 360 degree image,
The reference time point is a point of view viewed by the user on the first video frame.
The 360 degree video provided to the user terminal in real time is composed of a plurality of second video frames, and the steps (a) to (c) are repeatedly performed every time the first video frame exists. 2 is provided by the video frame to the user terminal, the real-time streaming of the 360-degree video, virtual reality video providing method. The method of claim 1,
The frame fragments correspond to the image fragments divided into six pieces so that the 360-degree image has a view angle of 90 degrees, virtual reality video providing method. The method of claim 2,
Step (a) is
And dividing the six frame pieces to include a reference frame piece having the reference point as a center point. The method of claim 3, wherein
Step (b) is
Mapping the six frame pieces one-to-one to the six faces of the cube and performing a crop on a flat pattern spreading the faces of the cube such that the other four faces contact the reference plane corresponding to the reference frame piece. How to deliver a video. The method of claim 4, wherein
Step (b) is
Setting an extended area in which the reference area extends in a direction with respect to each of the four surfaces in contact with the reference plane, using the area corresponding to the reference plane as a reference area, and performing cropping based on the extended area; How to deliver a video. The method of claim 5,
Step (b) is
And setting the extension area such that an area ratio of each of the four surfaces included in the extension area is a preset reference area ratio. The method of claim 1,
(d) receiving a modified time point;
(e) determining whether the changed time point is outside a preset reference angle of view; And
(f) setting the changed viewpoint as a new reference viewpoint when it is out of the reference angle of view area;
More,
The reference angle of view is
And a narrower angle of view than an angle of view corresponding to the second video frame. In the virtual reality video providing device,
A memory in which a program for performing a virtual reality video providing method is recorded; And
A processor for executing the program;
The processor, by executing the program,
The first video frame is converted into a plurality of frame pieces based on a reference viewpoint, and the first video frame is divided into a plurality of frame pieces, so that each of the frame pieces corresponds to each surface constituting a hexahedral exploded view. Set it up,
Cropping the remaining frame pieces of the frame piece corresponding to the reference time point among the frame pieces, wherein the reference frame piece corresponding to the reference time point among the plurality of frame pieces is adjacent to the reference frame piece. Crop a portion of at least one other piece of frame placed in position,
Providing a second video frame including the cropped frame fragments to a user terminal;
The first video frame corresponds to a 360 degree image,
The reference time point is a point of view viewed by the user on the first video frame.
The 360 degree video provided to the user terminal in real time is composed of a plurality of second video frames, and the process of converting the plurality of frame pieces to providing the second video frame to the user terminal includes the first video. The frame is repeatedly performed every time the frame is present to provide the plurality of second video frames to the user terminal, so that the real-time streaming of the 360-degree video, virtual reality video providing device.

Images