KR102612605B1 - Method and apparatus for generating and reproducing projection multiplexed image based on multiple omnidirectional image - Google Patents

Abstract

A method and device for generating a projection multiplexed image based on multiple omnidirectional images are provided. The projection multiplexed image generation method of the present invention includes the steps of acquiring at least one of a single camera image and an omnidirectional image from a plurality of omnidirectional cameras; When the omnidirectional image is acquired, projecting the acquired omnidirectional image in a predetermined projection format; performing projection multiplexing using at least one of the acquired single camera image and the omnidirectional image projected in the predetermined projection format; and generating one multiplexed omnidirectional image by performing the projection multiplexing.

Description

Method and apparatus for generating and reproducing projection multiplexed images based on multiple omnidirectional images {METHOD AND APPARATUS FOR GENERATING AND REPRODUCING PROJECTION MULTIPLEXED IMAGE BASED ON MULTIPLE OMNIDIRECTIONAL IMAGE}

The present invention relates to a method and device for providing projection multiplexed images based on multiple omnidirectional images. Specifically, the present invention relates to a method and device for projection multiplexing the entire or cut portion of a single camera image or an omnidirectional image acquired by a plurality of omnidirectional cameras into a single omnidirectional image.

The beginning of VR (Virtual Reality) technology was a method of showing only images corresponding to the front view while completely blocking external visual information through a fixed HMD (Head Mounted Display). Since then, VR technology has developed technologically in a way that can maximize the sense of realism and immersion by displaying video from a viewpoint that matches the viewer's movements. In the first step, users were provided with a panoramic viewing space that could freely rotate left and right horizontally, and in the next step, a spherical viewing space was provided so that viewers could rotate 3DoF (Degrees of Freedom) from one fixed location. You can now watch the video with the appropriate perspective while rotating (up and down, left and right, and forward and backward). However, viewers want to watch VR content with an increased sense of immersion and realism, and as a way to achieve this, it is possible to provide viewers with a change of viewpoint based on multiple omnidirectional videos shot with multiple 360-degree cameras located at different viewpoints. Supporting ‘perspective change technology’ is being considered.

FIG. 1A is a diagram illustrating a process for acquiring viewpoint change content using a plurality of omnidirectional cameras according to an embodiment of the present invention.

A plurality of omni-directional cameras used for filming may be located in the same space or different spaces depending on the purpose of the content. For example, referring to FIG. 1A, in the case of sports games such as soccer and basketball, all cameras can be placed in the same space because the viewpoint changes only in a limited area of the stadium. On the other hand, in the case of travel content that explores tourist attractions in various countries, the camera may be positioned in a different space because it must be possible to change the viewpoint to a completely different space.

FIG. 1B is a diagram illustrating a viewing process of viewpoint change content obtained using a plurality of omnidirectional cameras according to an embodiment of the present invention.

Viewers can select a specific camera located at a desired viewpoint through an HMD or 2D display device and watch the corresponding 360-degree video. At this time, in the case of an HMD, a specific camera can be selected by the user staring at an icon displaying the ID of a specific camera located in a specific part of the screen and then blinking or nodding the head. Additionally, in the case of a 2D display, a specific camera can be selected by the user clicking on an icon using a finger or remote control. Meanwhile, a switching delay of a certain amount of time may occur in the process of switching cameras to load an omnidirectional image at a user's desired point of view. Therefore, referring to FIG. 1B, viewers cannot watch images from multiple cameras at the same time, and switching delays may reduce the sense of immersion and realism in the content.

The purpose of the present invention is to provide a method and device for generating a projection multiplexed image based on a plurality of omnidirectional images.

Another object of the present invention is to provide a method and device for projection multiplexing the entire or cut portion of a single camera image or an omnidirectional image acquired by a plurality of omnidirectional cameras into a single omnidirectional image.

According to the present invention, acquiring at least one of a single camera image and an omnidirectional image from a plurality of omnidirectional cameras; When the omnidirectional image is acquired, projecting the acquired omnidirectional image in a predetermined projection format; performing projection multiplexing using at least one of the acquired single camera image and the omnidirectional image projected in the predetermined projection format; and generating a single multiplexed omnidirectional image by performing the projection multiplexing.

The features briefly summarized above with respect to the present disclosure are merely exemplary aspects of the detailed description of the present disclosure described below, and do not limit the scope of the present disclosure.

According to the present invention, a method and device for generating a projection multiplexed image based on a plurality of omnidirectional images can be provided.

In addition, according to the present invention, a method and device can be provided for projection multiplexing the entire or cut portion of a single camera image or an omnidirectional image acquired by a plurality of omnidirectional cameras into one omnidirectional image.

The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by those skilled in the art from the description below. will be.

FIG. 1A is a diagram illustrating a process for acquiring viewpoint change content using a plurality of omnidirectional cameras according to an embodiment of the present invention.
FIG. 1B is a diagram illustrating a viewing process of viewpoint change content obtained using a plurality of omnidirectional cameras according to an embodiment of the present invention.
Figure 2 is a diagram showing a conceptual diagram of a projection multiplexing method according to an embodiment of the present invention.
Figure 3 is a diagram showing a service using projection multiplexed omnidirectional images according to an embodiment of the present invention.
Figure 4 is a diagram showing the operation of a projection multiplexed image generating device according to an embodiment of the present invention.
Figures 5A to 5D are diagrams illustrating a process of projection multiplexing two different omnidirectional images projected in ERP format into one omnidirectional image according to an embodiment of the present invention.
Figure 6 is a diagram showing the operation of a projection multiplexed image generating device according to another embodiment of the present invention.
Figure 7 is a diagram illustrating a process of projection multiplexing and reproducing a tile image corresponding to a viewing area according to an embodiment of the present invention.

Since the present invention can make various changes and have various embodiments, specific embodiments will be illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all changes, equivalents, and substitutes included in the spirit and technical scope of the present invention. Similar reference numbers in the drawings refer to identical or similar functions across various aspects. The shapes and sizes of elements in the drawings may be exaggerated for clearer explanation. For a detailed description of the exemplary embodiments described below, refer to the accompanying drawings, which illustrate specific embodiments by way of example. These embodiments are described in sufficient detail to enable those skilled in the art to practice the embodiments. It should be understood that the various embodiments are different from one another but are not necessarily mutually exclusive. For example, specific shapes, structures and characteristics described herein with respect to one embodiment may be implemented in other embodiments without departing from the spirit and scope of the invention. Additionally, it should be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the embodiment. Accordingly, the detailed description that follows is not to be taken in a limiting sense, and the scope of the exemplary embodiments is limited only by the appended claims, together with all equivalents to what those claims assert if properly described.

In the present invention, terms such as first and second may be used to describe various components, but the components should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another. For example, a first component may be named a second component, and similarly, the second component may also be named a first component without departing from the scope of the present invention. The term and/or includes any of a plurality of related stated items or a combination of a plurality of related stated items.

When a component of the present invention is referred to as being “connected” or “connected” to another component, it may be directly connected or connected to the other component, but other components may exist in between. It must be understood that it may be possible. On the other hand, when it is mentioned that a component is “directly connected” or “directly connected” to another component, it should be understood that there are no other components in between.

The components appearing in the embodiments of the present invention are shown independently to represent different characteristic functions, and do not mean that each component is comprised of separate hardware or a single software component. That is, each component is listed and included as a separate component for convenience of explanation, and at least two of each component can be combined to form one component, or one component can be divided into a plurality of components to perform a function, and each of these components can perform a function. Integrated embodiments and separate embodiments of the constituent parts are also included in the scope of the present invention as long as they do not deviate from the essence of the present invention.

The terms used in the present invention are only used to describe specific embodiments and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In the present invention, terms such as "comprise" or "have" are intended to designate the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, but are not intended to indicate the presence of one or more other features. It should be understood that this does not exclude in advance the possibility of the existence or addition of elements, numbers, steps, operations, components, parts, or combinations thereof. In other words, the description of “including” a specific configuration in the present invention does not exclude configurations other than the configuration, and means that additional configurations may be included in the practice of the present invention or the scope of the technical idea of the present invention.

Some of the components of the present invention may not be essential components that perform essential functions in the present invention, but may simply be optional components to improve performance. The present invention can be implemented by including only essential components for implementing the essence of the present invention excluding components used only to improve performance, and a structure including only essential components excluding optional components used only to improve performance. is also included in the scope of rights of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In describing the embodiments of the present specification, if it is determined that a detailed description of a related known configuration or function may obscure the gist of the present specification, the detailed description will be omitted, and the same reference numerals will be used for the same components in the drawings. Redundant descriptions of the same components are omitted.

Figure 2 is a diagram showing a conceptual diagram of a projection multiplexing method according to an embodiment of the present invention. A single camera image acquired using a plurality of omnidirectional cameras, the entire omnidirectional image, or an image partially cut from the omnidirectional image can be fused into one omnidirectional image through projection multiplexing technology. Meanwhile, viewers can watch images from different omnidirectional cameras at the same time by performing 3DoF rotational movements while wearing the HMD, similar to the way they watch existing VR content, or by dragging with a mouse or finger on the 2D display. . Referring to FIG. 2, an image can be generated by projection multiplexing four omnidirectional images acquired using four different cameras (cameras 1 to 4).

Figure 3 is a diagram showing a service using projection multiplexed omnidirectional images according to an embodiment of the present invention.

Referring to FIG. 3, a process in which an omnidirectional video from a specific camera selected by the viewer is played while watching a projection multiplexed omnidirectional video is explained. In the case of the existing viewing method using viewpoint change content, when the viewpoint change content first starts, the user knows which camera provides the scene from the desired viewpoint because the omnidirectional video corresponding to a specific camera is already playing on the screen. I don't know if it exists. Therefore, in order to find out, you have to go through the cumbersome process of playing each video one by one. However, when the projection-multiplexed video is shown on the initial screen, viewers can immediately see what video each camera is shooting. In addition, viewers can select the video by performing a separate interaction process, such as shaking their head or clicking with the mouse, in front of the video they want to view in more detail, and thus can watch the video in all directions. .

Figure 4 is a diagram showing the operation of a projection multiplexed image generating device according to an embodiment of the present invention.

Specifically, FIG. 4 may show the operation of a system structure that provides the service described above in FIG. 3 based on a plurality of omnidirectional images and projection multiplexed images generated in non-real time.

The image acquisition module 400 can perform shooting with a plurality of omnidirectional cameras and use this to acquire a plurality of single camera images including 360-degree scene information from different viewpoints for each omnidirectional camera.

The image pre-processing module 402 may perform temporal synchronization settings, color correction, stitching, projection processes, etc. on the single camera image acquired by the image acquisition module 400. For example, setting temporal synchronization may mean performing temporal synchronization between single cameras or omnidirectional cameras using an algorithm based on recorded audio signals or movement information of objects in the image. Color correction can mean evenly adjusting color differences that occur due to different physical characteristics between cameras or differences in illuminance depending on the shooting environment. Additionally, color correction may be necessary to create an omnidirectional image without any heterogeneity. Stitching may refer to the process of joining multiple single camera images into one image using a geometric image change technique such as image warping, and in this case, the stitching process may be performed for each omnidirectional camera. The projection process may refer to the process of projecting a stitched image into a specific projection format to reproduce an omnidirectional image in a spatial sense. Types of projection formats include, for example, Equi-rectangular Projection (ERP), Cubemap Projection (CMP), and EAP. There are various formats such as (Equi-Angular Projection) and ISP (Icosaheron Projection).

The projection multiplexing module 404 can generate a single omnidirectional image by multiplexing a single camera image for each omnidirectional camera or an omnidirectional image projected in a specific format. The above process will be described in detail later in FIGS. 5A to 5D.

The encoding module 406 can encode a plurality of omnidirectional images and projection-multiplexed images to generate a bitstream for each image.

The file segment encapsulation module 408 can divide each bitstream into segments according to a predetermined container format. Additionally, the file segment encapsulation module 408 can encapsulate the bitstream divided into the segment format and store it in the transmission server. The predetermined container format may include ISO Base Media File Format (ISOBMFF).

The transmission module 410 may receive the viewer interaction information 418 from the rendering and display module 416 and transmit, based on this, an encapsulated projection multiplexed image or a segment of a specific omnidirectional image selected by the viewer to the client side.

The file segment decapsulation module 412 can decapsulate the transmitted encapsulated segment.

The decoding module 414 can decode a bitstream of a non-encapsulated video.

The rendering and display module 416 can display the decoded image to the viewer through an HMD or 2D monitor.

Figures 5A to 5D are diagrams illustrating a process of projection multiplexing two different omnidirectional images projected in ERP format into one omnidirectional image according to an embodiment of the present invention.

Figure 5a is an example of two omnidirectional images multiplexed in the left and right or up and down directions. Additionally, FIGS. 5B to 5D are examples of extracting and multiplexing only a predetermined region from each omnidirectional image.

A method such as the example in FIG. 5A has the advantage of being very simple to implement, but the multiplexed video may also include a relatively large number of unnecessary areas that viewers do not often view. In the case of camera 1 video in FIG. 5A, most viewers will mainly watch the center portion where an attractive main character appears within the 360-degree viewing space, and will pay relatively less attention to the surrounding portion than the center portion. Therefore, according to an embodiment of the present invention, unnecessary areas can be cut out in various ways, as shown in the examples of FIGS. 5B to 5D.

Referring to Figure 5b, the left and right areas of the main object located in the center can be cut out and the remaining center area can be multiplexed.

Referring to Figure 5c, only the sweet spot area can be multiplexed. The Swiss spot may refer to an area that viewers particularly frequently view among the entire omnidirectional video. The sweet spot is generally set by the content creator, but according to the present invention, it can also be found by statistically analyzing the viewer's content viewing information.

Referring to FIG. 5D, after cutting out highly distorted upper and lower parts of an omnidirectional image, the remaining image can be multiplexed. This takes into account the characteristics of the ERP format, where distortion becomes more severe toward the extreme points.

Meanwhile, in addition to the four methods shown in FIGS. 5A to 5D, various types of projection multiplexed images can be generated by editing and/or fusing each omnidirectional video in various ways according to the intention of the content creator. Although the process described above in FIGS. 5A to 5D only uses the ERP format as an example, according to the present invention, it can also be applied to omnidirectional images projected in various formats such as CMP and EAP format. In addition, generally, the same omnidirectional camera model with the same FoV (field of view), image quality, and focal length is used to acquire viewpoint change content, but down/up sampling and cropping are used. It can also be applied to images taken with different settings or different types of cameras through the same image processing process.

In addition, the process described above in FIGS. 5A to 5D uses an omnidirectional image projected by ERP as an example, but single camera images can also be multiplexed through a process similar to the above process.

According to an embodiment of the present invention, information necessary to provide a service using the projection multiplexed video described above in FIG. 4 can be signaled.

As an example, nb_of_ovids may mean the total number of omnidirectional images required for the service, including projection multiplexed omnidirectional images and omnidirectional images each acquired by a plurality of omnidirectional cameras. Using the above information, a unique ID can be assigned to each omnidirectional image.

As another example, ovid_id may refer to an identifier for distinguishing multiple omnidirectional videos, and using this, when a viewer selects a specific omnidirectional video, it can be loaded as a full 360-degree VR video. Additionally, additional information such as projection format, number of pixels in horizontal/vertical directions, etc. can be given to each omnidirectional image.

As another example, init_ovid_id may mean the ID of the omnidirectional video that is played first when the viewer starts the viewpoint change content.

As another example, init_vp_flag may be information for identifying the part that is played first among the entire 360-degree area when a viewer selects a projection multiplexed image or an image from a specific omnidirectional camera.

As another example, projection_type may mean a value indicating the projection format of each omnidirectional image, and may be used to map a 2D image onto a 3D sphere at the rendering stage.

As another example, ocam_vid_wdth and ocam_vid_hgt may mean the total number of horizontal and vertical pixels of the omnidirectional image, respectively.

As another example, mpx_type may mean the single camera image acquired by each omnidirectional camera or the format in which all or part of the omnidirectional image was projection multiplexed. For example, when multiple images are combined horizontally, the mpx_type value can be set to 0, and when multiple images are combined vertically, the mpx_type value can be set to 1.

As another example, mpx_vid_x_stpos/mpx_vid_y_stpos and mpx_vid_wdth/mpx_vid_hgt may mean the horizontal/vertical start coordinates and horizontal/vertical length of the partial image used in each omnidirectional image during projection multiplexing, and can be used to refer to each omnidirectional image. The location and/or size information of each partial image can be known.

Figure 6 is a diagram showing the operation of a projection multiplexed image generating device according to another embodiment of the present invention.

Specifically, FIG. 6 shows the operation of a system structure for providing a projection multiplexed image with higher quality based on a plurality of omnidirectional images and a tile image generated by dividing each of the plurality of omnidirectional images into specific sizes. You can.

The operations of the image acquisition module 600 and the image pre-processing module 602 may be the same as those of the image acquisition module 400 and the image pre-processing module 402 of FIG. 4 .

The image tiling module 604 can divide each omnidirectional image into several tile images. For example, if the quality of the acquired omnidirectional image is 4Kx2K, the omnidirectional image can be divided into four 2Kx1K tile images.

The encoding module 606 may generate a bitstream by encoding a plurality of omnidirectional images and a tile image of each omnidirectional image. At this time, as the number of tiles increases, the required encoding power may increase.

The file segment encapsulation module 608 can divide each bitstream into several segments, encapsulate them, and store them in the transmission server.

The transmission module 610 may transmit a segment of an encapsulated tile image or omnidirectional video needed by the viewer to the client based on the viewer interaction information 620.

According to one embodiment of the present invention, the viewer interaction information 418 of FIG. 4 is simply used to select a specific omnidirectional image to be viewed among several omnidirectional images, while the viewer interaction information 620 of FIG. 6 ) may also include eye tracking information to retrieve a specific tile image corresponding to the viewing area, in addition to the function of the viewer interaction information 418 of FIG. 4. Accordingly, it can be said that the viewer interaction information 620 of FIG. 6 has a relatively expanded performance range.

The file segment decapsulation module 612 can decapsulate the transmitted encapsulated segment.

The decoding module 614 can decode a bitstream of several non-encapsulated tile images or an omnidirectional image from a specific camera. At this time, the number of decoders required may be equal to the number of tiles corresponding to the viewing area.

When a tile image is input, the projection multiplexing module 616 can perform projection multiplexing on the tile image. Meanwhile, omnidirectional images from a certain camera can be directly transmitted from the decoding module 614 to the rendering and display module 618 without a projection multiplexing process.

The rendering and display module 618 can display the image received from the decoding module 614 or the projection multiplexing module 616 to the viewer through an HMD or 2D monitor.

Figure 7 is a diagram illustrating a process of projection multiplexing and reproducing a tile image corresponding to a viewing area according to an embodiment of the present invention.

Referring to FIG. 7, a combined tile image can be created using viewing area part 1 of omnidirectional image 1 and viewing area part 2 of omnidirectional image 2.

According to the present invention, a projection multiplexing technique can be provided to generate a single omnidirectional image by combining a specific single camera image from each omnidirectional camera or the entire or cut portion of an omnidirectional image projected in a specific format.

In addition, according to the present invention, the utility of the viewpoint change technology that shows the viewer by switching images from various viewpoints based on a plurality of omnidirectional images obtained using a plurality of omnidirectional cameras consisting of two or more single cameras, and Technology can be provided to increase immersion.

In addition, according to the present invention, in order to support simultaneous viewing of images acquired by a plurality of omni-directional cameras, a single camera image from each omni-directional camera and the entire or cut portion of an omni-directional image projected in a specific format are converted into one omni-directional image. Projection multiplexing technology and multiplexed video playback technology can be provided.

In addition, according to the present invention, a method and apparatus can be provided for projecting and multiplexing a single camera image acquired by a plurality of omnidirectional cameras, or the entire or cut portion of an omnidirectional image, into one omnidirectional image, and reproducing the multiplexed image.

In addition, according to the present invention, it is possible to expand the application field of the existing viewpoint change service and further increase the sense of immersion and realism in the content.

In the above-described embodiments, the methods are described based on flowcharts as a series of steps or units, but the present invention is not limited to the order of steps, and some steps may occur in a different order or simultaneously with other steps as described above. You can. Additionally, a person of ordinary skill in the art will recognize that the steps shown in the flowchart are not exclusive and that other steps may be included or one or more steps in the flowchart may be deleted without affecting the scope of the present invention. You will understand.

The above-described embodiments include examples of various aspects. Although it is not possible to describe all possible combinations for representing the various aspects, those skilled in the art will recognize that other combinations are possible. Accordingly, the present invention is intended to include all other substitutions, modifications and changes falling within the scope of the following claims.

Embodiments according to the present invention described above may be implemented in the form of program instructions that can be executed through various computer components and recorded on a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc., singly or in combination. Program instructions recorded on the computer-readable recording medium may be specially designed and configured for the present invention, or may be known and usable by those skilled in the computer software field. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magneto-optical media such as floptical disks. media), and hardware devices specifically configured to store and perform program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include not only machine language code such as that created by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules to perform processing according to the invention and vice versa.

In the above, the present invention has been described with specific details such as specific components and limited embodiments and drawings, but this is only provided to facilitate a more general understanding of the present invention, and the present invention is not limited to the above embodiments. , a person skilled in the art to which the present invention pertains can make various modifications and variations from this description.

Therefore, the spirit of the present invention should not be limited to the above-described embodiments, and the scope of the patent claims described below as well as all modifications equivalent to or equivalent to the scope of the claims fall within the scope of the spirit of the present invention. They will say they do it.

Claims (10)

In the omnidirectional video encoding method,
Obtaining a plurality of omnidirectional images from a plurality of omnidirectional cameras;
Projecting each of the plurality of omnidirectional images in a predetermined projection format;
generating a packed image based on omnidirectional images projected in the predetermined projection format; and
Including the step of encoding the packing image and information about the packing image,
The information includes number information indicating the number of omnidirectional images. delete According to clause 1,
The omnidirectional video encoding method includes identification information for each of the omnidirectional images. According to clause 1,
The omnidirectional video encoding method includes information indicating an omnidirectional video that is played first among the plurality of omnidirectional videos. In the omnidirectional video decoding method,
Decoding a bitstream to obtain a packed image including data for a plurality of omnidirectional images and information about the packed image;
Decoding the packed image; and
Based on the information, including displaying an omnidirectional image corresponding to a specific viewpoint,
The information includes number information indicating the number of the plurality of omnidirectional images. delete According to clause 5,
The information includes identification information for each of the plurality of omnidirectional images. According to clause 5,
The omnidirectional video decoding method includes information indicating the omnidirectional video that is played first among the plurality of omnidirectional videos. A computer-readable recording medium storing a bitstream containing omnidirectional video information encoded according to an omnidirectional video encoding method,
The omnidirectional video encoding method is,
Obtaining a plurality of omnidirectional images from a plurality of omnidirectional cameras;
Projecting each of the plurality of omnidirectional images in a predetermined projection format;
generating a packed image based on omnidirectional images projected in the predetermined projection format; and
Including the step of encoding the packing image and information about the packing image,
The information includes number information indicating the number of omnidirectional images,
A computer-readable recording medium that stores a bitstream. According to clause 9,
The information includes identification information for each of the omnidirectional images. A computer-readable recording medium storing a bitstream.