KR20170043791A - Image format and apparatus for 360 virtual reality services - Google Patents
Image format and apparatus for 360 virtual reality services Download PDFInfo
- Publication number
- KR20170043791A KR20170043791A KR1020150143301A KR20150143301A KR20170043791A KR 20170043791 A KR20170043791 A KR 20170043791A KR 1020150143301 A KR1020150143301 A KR 1020150143301A KR 20150143301 A KR20150143301 A KR 20150143301A KR 20170043791 A KR20170043791 A KR 20170043791A
- Authority
- KR
- South Korea
- Prior art keywords
- video
- image
- viewpoint
- decoded
- hmd
- Prior art date
Links
Images
Classifications
-
- H04N13/0059—
-
- H04N13/0022—
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
- H04N19/597—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding specially adapted for multi-view video sequence encoding
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/70—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by syntax aspects related to video coding, e.g. related to compression standards
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
Abstract
The present invention provides inter-ocular distance adaptation and motion parallax using a compressed image format and a virtual view synthesis method capable of more efficiently performing 360 VR 2D / 3D video service, And more particularly, to a technique for enabling a user to display a better result.
Description
The present invention relates to an image format and apparatus for 360 virtual reality service, and more particularly, to a compressed image format for 360 VR service and an apparatus for the service.
Recently, with the advancement of the head-mount display (HMD) technology, HMDs using the latest technology have been introduced. In particular, a recent HMD detects the motion of the head in real time when the user wears it and provides the time in the direction regardless of which direction the head is moving. In addition, each of the right and left lenses provides a concave curved panoramic display image, which provides a wide reagent angle so that even when the pupil is moved, the image of the virtual reality can be seen. Recent HMDs provide a sense of enthusiasm in the virtual reality by providing a trekking technique that recognizes the movement of the head, and a binocular image provided to both eyes, providing a stereoscopic image suitable for the user.
As a method of providing multi-view images to such an HMD, it is common to use a method in which two images are formed by attaching images taken by a multi-view camera to each other. Usually, this image is displayed assuming that the user's eye exists in the circle through a sphere projection.
The sphere projection image provided in the 360 VR video service is provided only in the 2D so far, and when the user wears the HMD for a long time, it causes eye fatigue to the user, which makes it difficult to wear for a long time.
A video format and a virtual viewpoint generation method for providing a 360 3D VR video service in order to give a natural immersion or stereoscopic effect by giving i-ocular-distance and motion parallax by generating a virtual viewpoint, and The apparatus is described in the present invention.
According to another aspect of the present invention, there is provided an image format for a 360 virtual reality service, the method comprising the steps of: defining a 360 video image format, the 360 image format including a 360 video image and a 360- A 360 video image format including a 360-degree left video image and a 360-degree right video image, or a 360-degree video image including a 360-degree left video image and a 360-degree video image associated with the left- Video image format.
At this time, 360 video images and related 360 depth images are encoded / decoded for 360 image formats including 360 video images and related 360 depth images, 360 decoded video images and related 360 depth images are displayed on a display such as an HMD And a rendering step.
At this time, 360 video encoding / decoding of the left / right view 360 video format including the left view 360 video image and the right view 360 view video image, and the step of rendering the decoded left / right view 360 video on the display such as the HMD can do.
At this time, the left / right viewpoint 360 video and the left / right viewpoint and the related depths for the 360 image format including the left view 360 video image, the right viewpoint 360 video image, the 360 depth image related to the left viewpoint, Encoding / decoding the image, rendering the decoded left / right viewpoint 360 video, and the left / right viewpoint and the associated depth image to the display such as the HMD.
At this time, the decoded 360 video image and the related 360-depth image are rendered on the display such as the HMD, and the decoded 360 video image and 360 depth video related thereto are used for natural 3D viewing, (Adjusting the binocular disparity -inter-ocular distance) and generating the image, and rendering the decoded 360 video image and the generated virtual viewpoint image on the display such as the HMD.
At this time, the step of rendering the decoded left / right view 360 video on the display such as the HMD, the step of providing the binocular parallax using the decoded left / right view 360 video image or the step of using the decoded left / Calculating a displacement map and adjusting a inter-ocular distance by generating a virtual viewpoint using the calculated displacement map, generating a left / right virtual viewpoint at the corresponding position to provide a motion parallax when the user moves left and right, / Right 360 video or left / right 360 virtual point-in-time video to a display such as an HMD.
At this time, rendering the decoded left / right view 360 video and the left / right view and the related depth image on the display such as the HMD; Using the decoded left / right viewpoint 360 video image to binocular parallax; Or adjusting the inter-ocular distance by generating a left / right virtual viewpoint using the decoded left / right viewpoint 360 video image and the 360 depth image associated with the left / right viewpoint; Generating a left / right virtual viewpoint at a corresponding position to provide a motion parallax when the user moves left / right; And rendering the left / right 360 video generated in the above step on a display such as an HMD.
According to the present invention, 360 VR video service can be more naturally provided to the user wearing the head-mounted display.
1 is a diagram illustrating a 3D 360 video generation process according to an embodiment of the present invention.
The present invention will now be described in detail with reference to the accompanying drawings. Hereinafter, a repeated description, a known function that may obscure the gist of the present invention, and a detailed description of the configuration will be omitted. Embodiments of the present invention are provided to more fully describe the present invention to those skilled in the art. Accordingly, the shapes and sizes of the elements in the drawings and the like can be exaggerated for clarity.
Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the accompanying drawings.
The present invention proposes various image formats provided through the following sphere projection.
To do this, we acquire the video image through 360 3D camera and construct two sphere projected images. The system structure for this is shown in Fig.
1 is a diagram illustrating a 3D 360 video generation process according to an embodiment of the present invention.
In FIG. 1, a 360 3D video generation process is briefly shown. A video is shot with a 360 3D camera configured with multiple left / right cameras arranged in all directions. The camera images thus taken are formed into an equirectangular format through stiching processing. However, the equirectangular projection method used here is one method for sphere projection, and 360 video can be generated by selectively using various projection methods in the spreading process. Finally, equirectangular 360 video images will generate left and right images, respectively. This is a common 360 3D video image format.
However, in this case, since the image provides only the fixed binocular disparity to the user, the viewer who wears the HMD is prevented from enjoying comfortable viewing due to fatigue during the long time viewing.
Therefore, it is possible to generate a virtual viewpoint image using the following image format.
1) 360 video +360 depth video
2) Left 360 video + Right 360 video
3) Left / Right 360 Video + Left / Right Depth Video
1.360 video +360 depth video
360 video refers to an equirectangular projection image format obtained by expanding images taken from a 360 degree camera. The corresponding 360 depth image can be generated and defined as the image format of one 360 3D video. The depth image used here is used to provide three-dimensional information of 360 video, which can be anything that can provide three-dimensional spatial information in addition to the depth image. Therefore, disparity can be provided as it can provide three-dimensional communicative information. The generated 360 3D image format can be served to the user through the following process.
1) 360 video and 360 depth / displacement images are encoded.
2) The compressed image is provided to the user's HMD device in the form of transmission or file.
3) The HMD device produces the virtual image using 360 depth / displacement image and 360 stereoscopic images using the left and right images.
4) At this time, a method of synthesizing a virtual image using 360 depth / displacement image may be implemented by a method of synthesizing only the viewpoints provided through the HMD and a method of synthesizing virtual images of 360 video images at a time.
5) The virtual image at this time can be generated by adjusting the most comfortable state (inter-ocular distance adaption) considering the interval between eyes of the HMD user.
2. Left 360 video + Right 360 video
360 video refers to the left and right image formats of left and right equirectangular projection shapes obtained by expanding images shot from a 360 degree 3D camera. The left / right 360 video images can be provided to the user through the following process.
1) The left / right 360 video images are compressed through an appropriate encoding process.
2) The compressed image is provided to the HMD device or the like in the form of transmission or file.
3) Left and right images can be provided in the left and right eyes of the user wearing the HMD device, respectively, to provide stereoscopic vision.
4) or displacement of the left / right image to generate a disparity map and use it to provide a virtual viewpoint image for inter-ocular distance adjustment.
5) The generated virtual viewpoint image has an advantage that the HMD device can provide a natural motion pallax since the user can provide the viewpoint image when moving left and right.
6) At the same time, based on the left 360 video, the right video can use a virtual image for natural stereoscopic viewing when viewing through a displacement map.
3. Left / right 360 video + left / right depth video
The left / right video shot with a 360-degree 3D camera produces a video format of the left / right 360 video through a spreading process. At the same time, a left / right depth image is provided together. Such an image format can be provided to the user through the following process.
1) Left and right 360 video images and left / right depth images are compressed through appropriate encoding process.
2) The compressed image is provided to the HMD device or the like in the form of transmission or file.
3) In the HMD device, 360 video and left / right depth images are obtained through the decoding process and are provided to the user through the following process.
4) Through the HMD worn by the user, the following services can be provided.
A. 2D video service can be done by showing left image only. However, when the user moves left and right (motion parallax) using the left / right image and the left / right depth image, a virtual view image of the corresponding position can be generated and displayed.
B. 3D video service can be done by showing left / right video.
C. The left image and the user can conveniently generate the virtual view image using the left / right image and the left / right depth image for the right image for 3D viewing. When the user moves left and right, It is possible to generate a natural viewpoint motion parallax.
As described above, the image format and apparatus for the 360 virtual reality service according to the present invention are not limited to the configuration and method of the embodiments described above, but the embodiments can be variously modified All or some of the embodiments may be selectively combined.
Claims (7)
A 360 video image format including a 360-degree video image and a 360-depth image associated therewith, or a 360-video image format including a 360-degree left video image and a 360-degree right video image, And a 360-degree video image format that includes a 360-depth image associated with the right-point and a 360-depth image associated with the right-point.
About 360 video formats including 360 video images and related 360 depth videos
Encoding / decoding 360 video images and 360 depth images associated therewith, and
Rendering the decoded 360 video image and the 360 depth video associated therewith on a display such as an HMD
And a method thereof.
For a 360-image format that includes 360 view video images at the left and 360 view video images at the right viewpoint
A left / right view 360 video encoding / decoding step, and
The decoded left / right viewpoint 360 rendering the video on the display such as the HMD
And a method thereof.
For a 360-image format that includes a 360-degree left video image, a 360-degree right video image, a 360-depth image associated with the left viewpoint, and a 360-
Encoding / decoding the left / right viewpoint 360 video and the left / right viewpoint and related depth images, and
The decoded left / right viewpoint 360 video and the step of rendering on the display such as the HMD using the left / right viewpoint and the related depth image
And a method thereof.
For the step of rendering the decoded 360 video image and the related 360 depth image on the display such as the HMD
A step of adjusting the position of the virtual viewpoint image (adjusting the binocular disparity-inter-ocular distance) according to the distance between the user's eyes for natural 3D viewing using the decoded 360 video image and the 360 depth image related thereto,
Rendering 360 decoded video images and generated virtual viewpoint images on a display such as an HMD
And a method thereof.
Decoded left / right viewpoint 360 For the step of rendering the video on the display such as the HMD
A binocular parallax using the decoded left and right 360 video images, or a displacement map using the decoded left and right 360 video images, and generates a virtual viewpoint using the decoded left and right 360 video images to adjust the inter-ocular distance ,
Generating a left / right virtual viewpoint at the position when the user moves left / right to provide a motion parallax, and
Left / right 360 video or left / right 360 virtual point-in-time rendering video to display such as HMD
And a method thereof.
Rendering the decoded left / right view 360 video and the left / right viewpoint and the associated depth image on a display such as an HMD,
A left / right virtual viewpoint is generated by using the decoded left / right viewpoint 360 video image for the binocular parallax or the decoded left / right viewpoint 360 video image and the 360 depth image related to the left / right viewpoint, adjusting the ocular distance,
Generating a left / right virtual viewpoint at the position when the user moves left / right to provide a motion parallax, and
Rendering the left / right 360 video generated in the above step on a display such as an HMD
And a method thereof.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| KR1020150143301A KR20170043791A (en) | 2015-10-14 | 2015-10-14 | Image format and apparatus for 360 virtual reality services |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| KR1020150143301A KR20170043791A (en) | 2015-10-14 | 2015-10-14 | Image format and apparatus for 360 virtual reality services |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| KR20170043791A true KR20170043791A (en) | 2017-04-24 |
Family
ID=58704427
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| KR1020150143301A KR20170043791A (en) | 2015-10-14 | 2015-10-14 | Image format and apparatus for 360 virtual reality services |
Country Status (1)
| Country | Link |
|---|---|
| KR (1) | KR20170043791A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109388448A (en) * | 2017-08-09 | 2019-02-26 | 京东方科技集团股份有限公司 | Image display method, display system and computer readable storage medium |
| WO2019098728A1 (en) * | 2017-11-16 | 2019-05-23 | 삼성전자 주식회사 | Method and apparatus for processing three-dimensional images |
| CN109862340A (en) * | 2017-11-30 | 2019-06-07 | 宏碁股份有限公司 | Video-unit and its three-dimensional object decoding method |
| KR20190118803A (en) | 2018-04-11 | 2019-10-21 | 주식회사 씨오티커넥티드 | Stereoscopic image generating apparatus |
-
2015
- 2015-10-14 KR KR1020150143301A patent/KR20170043791A/en unknown
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109388448A (en) * | 2017-08-09 | 2019-02-26 | 京东方科技集团股份有限公司 | Image display method, display system and computer readable storage medium |
| CN109388448B (en) * | 2017-08-09 | 2020-08-04 | 京东方科技集团股份有限公司 | Image display method, display system, and computer-readable storage medium |
| WO2019098728A1 (en) * | 2017-11-16 | 2019-05-23 | 삼성전자 주식회사 | Method and apparatus for processing three-dimensional images |
| US11212507B2 (en) | 2017-11-16 | 2021-12-28 | Samsung Electronics Co., Ltd. | Method and apparatus for processing three-dimensional images |
| CN109862340A (en) * | 2017-11-30 | 2019-06-07 | 宏碁股份有限公司 | Video-unit and its three-dimensional object decoding method |
| KR20190118803A (en) | 2018-04-11 | 2019-10-21 | 주식회사 씨오티커넥티드 | Stereoscopic image generating apparatus |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8953023B2 (en) | Stereoscopic depth mapping | |
| EP3287837B1 (en) | Head-mountable display system | |
| KR20160135660A (en) | Method and apparatus for providing 3-dimension image to head mount display | |
| TW201019708A (en) | A method of processing parallax information comprised in a signal | |
| TWI504232B (en) | Apparatus for rendering 3d images | |
| KR20170043791A (en) | Image format and apparatus for 360 virtual reality services | |
| JP2022183177A (en) | Head-mounted display device | |
| US20160070112A1 (en) | Adjustable optical stereoscopic glasses | |
| KR101322187B1 (en) | 3-dimensional moving picture photographing apparatus and camera | |
| JP6687751B2 (en) | Image display system, image display device, control method thereof, and program | |
| JP6207640B2 (en) | 2D image stereoscopic display device | |
| CN110780454B (en) | Three-dimensional image display apparatus and method | |
| KR101172507B1 (en) | Apparatus and Method for Providing 3D Image Adjusted by Viewpoint | |
| CN104952098A (en) | 3D stop-motion animation film shooting and making method | |
| CN108632600B (en) | Method and device for playing stereoscopic image by reflecting user interaction information | |
| KR20110106708A (en) | An apparatus and method for displaying image data in image system | |
| TW201631961A (en) | Method of reproducing images with a three-dimensional appearance | |
| US20200049994A1 (en) | Tilted focal plane for near-eye display system | |
| KR101578890B1 (en) | 3-dimensional moving picture photographing apparatus and camera | |
| JP2014175813A (en) | Stereoscopic video display method and device | |
| US20220232201A1 (en) | Image generation system and method | |
| KR20170059879A (en) | three-dimensional image photographing apparatus | |
| KR20160116145A (en) | HMD using See Through Hologram | |
| KR101433645B1 (en) | 3-dimensional moving picture photographing apparatus and camera | |
| Rhee et al. | Stereoscopic view synthesis by view morphing |