WO2016195167A1

WO2016195167A1 - Content conversion method, content conversion apparatus, and program for multi-layered hologram

Info

Publication number: WO2016195167A1
Application number: PCT/KR2015/009492
Authority: WO
Inventors: 오병기
Original assignee: 주식회사 쓰리디팩토리
Priority date: 2015-06-01
Filing date: 2015-09-09
Publication date: 2016-12-08
Also published as: KR101754976B1; KR20160141446A

Abstract

An embodiment of the present invention relates to a content conversion method for generating a multi-layered hologram from one 2D image content by a content conversion apparatus for a multi-layered hologram, the method comprising the steps of: extracting a still image from the 2D image content according to a predetermined frame rate; separating, for the extracted still image, a background image and a target image including one or more objects; performing mapping for each of the separated target and background images, using a specific map; and generating a multi-view image for each of the mapped target and background images, using a depth and a three-dimensional value.

Description

Content conversion method, converter and program for stacked hologram

The present invention relates to a method, an apparatus and a program for converting content for stacked holograms.

Unlike the existing 2D (2D) image, the 3D (3D) image technology is a new concept of realistic image media that enhances the quality of visual information unlike the actual 2D image. It is expected to lead the culture.

Such a 3D stereoscopic image may be obtained by directly photographing through a plurality of cameras, or may be obtained through a method of converting a 2D planar image into a 3D stereoscopic image having a stereoscopic feeling.

When a 3D stereoscopic image is generated using a 2D planar image, depth information is applied to an object of the 2D planar image so that the 2D planar image may be converted into a 3D stereoscopic image.

On the other hand, 3D stereoscopic images are displayed in a multi-view autostereoscopic 3D (AS3D; AutoStereoscopic 3D) on a three-dimensional space, and according to an increase in the need to express objects more realistically and consistently, a plurality of 3D images are converted into AS3D images. Technology is being used.

However, a multiview AS3D image integrates and displays a 3D image in a 3D space as a multiview stereoscopic image. Therefore, a problem arises in that the consistency of the three-dimensional effect of the three-dimensional effect is weak at one time point and the three-dimensional effect is relatively strong at another point occurs. This problem causes a ghost phenomenon that causes viewers to experience visual fatigue or dizziness.

As described above, an object of the present invention is to propose a technology for producing a content for displaying a multi-view 3D stereoscopic image in which a stereoscopic effect is improved without ghosting by using 2D image contents.

In order to achieve the above object, an embodiment of the present invention proposes a content production method for providing a hologram with improved three-dimensional effect.

According to an embodiment of the present invention, a content conversion method of a content conversion apparatus for a stacked hologram for generating a stacked hologram from a single 2D video content includes: extracting a still image from the 2D video content according to a predetermined frame rate; Separating the target image and the background image including one or more objects from the extracted still image, performing mapping using the specific map on each of the separated target image and the background image, and the mapped target image And generating a multiview image for each of the background images using depth and stereoscopic values.

Another embodiment of the present invention is an extraction module for extracting a still image according to a predetermined frame rate from the 2D image content, and separating the object image and the background image containing one or more objects with respect to the extracted still image, Stacking type from one 2D image content including a mapping module for mapping each target image and background image using a specific map, and a viewpoint conversion module for generating a multiview image for each mapped target image and background image It is characterized by providing a content conversion apparatus for stacked holograms for generating holograms.

According to an embodiment of the present invention, by converting one 2D video content into a multi-view 3D video content, the target video and the background video can be reproduced on two physically separated displays, thereby creating a three-dimensional image without dizziness or ghosting. Can provide.

1 is a block diagram showing a main configuration of a content conversion apparatus for stacked holograms according to the present invention.

2 is a flowchart illustrating a content conversion method of a content conversion apparatus for a stacked hologram according to an exemplary embodiment of the present specification.

3A and 3B are exemplary views for explaining generation of a target image and a background image according to an embodiment of the present disclosure.

4 is a flowchart illustrating a mapping according to an embodiment of the present disclosure.

5A to 5E are exemplary diagrams for explaining mapping of the target image and the background image according to one embodiment of the present specification.

6A and 6B are exemplary diagrams for describing a method of generating a multiview image, according to an exemplary embodiment.

7 is an exemplary diagram illustrating a format of a target image and a background image content generated according to an embodiment of the present specification.

8 is an exemplary view for explaining a method for synchronizing a target image and a background image into one file according to an embodiment of the present specification.

9 is an exemplary view for explaining a setup method for displaying a target image and a background image according to an embodiment of the present specification.

10 is an exemplary diagram in which a stacked hologram implementing system according to an exemplary embodiment of the present specification is installed.

11 is an exemplary diagram in which a stacked hologram implementing system according to another exemplary embodiment of the present specification is installed.

It is to be noted that the technical terms used herein are merely used to describe particular embodiments, and are not intended to limit the present invention. In addition, the technical terms used in the present specification should be interpreted as meanings generally understood by those skilled in the art unless they are specifically defined in this specification, and are overly inclusive. It should not be interpreted in the sense of or in the sense of being excessively reduced. In addition, when the technical terms used herein are incorrect technical terms that do not accurately represent the spirit of the present invention, it should be replaced with technical terms that can be understood correctly by those skilled in the art. In addition, the general terms used in the present invention should be interpreted as defined in the dictionary or according to the context before and after, and should not be interpreted in an excessively reduced sense.

Also, the singular forms used herein include the plural forms unless the context clearly indicates otherwise. In the present application, terms such as “consisting of” or “comprising” should not be construed as necessarily including all of the various components, or various steps described in the specification, wherein some of the components or some of the steps It should be construed that it may not be included or may further include additional components or steps.

In addition, the suffixes "module" and "unit" for the components used herein are given or mixed in consideration of ease of specification, and do not have meanings or roles that are distinguished from each other.

In addition, terms including ordinal numbers, such as first and second, as used herein may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as the second component, and similarly, the second component may also be referred to as the first component.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings, and the same or similar components will be given the same reference numerals regardless of the reference numerals, and redundant description thereof will be omitted.

In addition, in describing the present invention, when it is determined that the detailed description of the related known technology may obscure the gist of the present invention, the detailed description thereof will be omitted. In addition, it should be noted that the accompanying drawings are only for easily understanding the spirit of the present invention and should not be construed as limiting the spirit of the present invention by the accompanying drawings.

Prior to the description, the terms '3D', 'three-dimensional' and 'stereo' used in the present invention are used interchangeably.

In addition, a display panel is used as a generic term for an image that can be viewed by a person.

1 is a detailed configuration diagram of the main configuration of the content conversion apparatus for a stacked hologram according to the present invention.

As illustrated, the apparatus for converting stacked holograms according to the present invention includes an extraction module 110, a mapping module 120, a viewpoint conversion module 130, and a synchronization module 140.

The above components are expressed separately according to the function of the stacked hologram content conversion apparatus according to an embodiment of the present invention. One component may be divided into a plurality of components or a plurality of components may be integrated into one component. Such embodiments are also within the scope of the present invention.

The extraction module 110 generates a target image and a background image, which are still images, from a video of 2D image content. To this end, the extraction module 110 may extract a still image at a predetermined frame rate from a video of 2D image content. Frame rate refers to the number of frames progressing per second. For example, it consists of 30 frames per second for TV and 24 frames per second for movies. These frames per second can be selected, changed and saved by the user.

The 2D image content acquired by the extraction module 110 may be a single view image. The single-view image refers to an image obtained by photographing an object and a background from the same location through the photographing apparatus.

The extraction module 110 separates the target image including one or more objects from the extracted still image. The target image is separated and the remaining part becomes the background image.

The mapping module 120 converts 2D video content into 3D video. More specifically, the mapping module 120 performs mapping on each of the target image and the background image separated by the extraction module 110. In this case, the specific map may be at least one of a depth map, a displacement map, a texture map, and a combination thereof.

In addition, the mapping module 120 converts a plurality of single-view 2D images into a plurality of 3D images by using various conversion techniques including a 3D image converter or a 3D image conversion program.

The viewpoint conversion module 130 generates a multiview image for each of the target image and the background image converted into the 3D image. The viewpoint transformation module 140 renders the multiview AS3D image in the three-dimensional space through the projection reconstruction in the three-dimensional space.

Accordingly, the viewpoint conversion module 130 displays the plurality of 3D images in a single three-dimensional space and converts the multi-view AS3D images.

The synchronization module 140 synchronizes the multi-view AS3D object image and the multi-view AS3D image background image processed separately from one still image of the original 2D image content.

Hereinafter, a content conversion method of the content conversion apparatus for a stacked hologram will be described with reference to FIGS. 2 to 8.

3 to 6 are exemplary views for explaining a method of extracting a still image according to an embodiment of the present disclosure.

First, referring to FIG. 2, a still image is extracted according to a predetermined frame rate from one 2D image content acquired by the stacked hologram content converter in step S210.

3A is a diagram illustrating a state in which a plurality of still images are extracted. Each still image extracted in this way is 2D content.

The following operations are performed on each of the extracted still images.

Next, in step 220, the stacked hologram content converting apparatus extracts one or more objects from the still image extracted in step 210 as the object image, and separates the object image and the background image by defining the unextracted background as the background image.

One example of the technique used to separate this is rotoscoping techniques.

As a technique of drawing on top of each frame based on the actual photographed image, as shown in FIG. 3B, a contour of a desired object, such as a human, as a target image, is drawn in a CG program, and the image and background of the human are based on the contour. Split the image.

In addition, to extract a specific region from a frame, an optical flow method, a kernel-based mean-shift method using similarity of object distribution, and a contour based detection of an object boundary ) Tracking methods. In addition, it is possible to smoothly track even when occlusion occurs by fully utilizing the depth value.

Since the depth information of each object separated from the 2D plane represents a simple plane shape, a method for representing an actual object is required. Step 230 and 240 are steps for representing such a real object.

In operation 230, mapping of the target image and the background image is performed. This step converts simple planar 2D video content into 3D video content.

Performing mapping will be described below with reference to FIG. 4.

4 is a flowchart illustrating a mapping according to an embodiment of the present invention.

First, in operation 231, it is checked whether an image to be performed is a background image. Since the background image is the remaining image obtained by separating the target image from one image, a blank area 601 as shown in FIG. 5A is formed by separating the target image.

In operation 232, the pixel is filled in the empty area 601 with reference to the pixels around the empty area 601 and the information about the frame before and after the corresponding image.

If the target image is not a background image in step 231, the process proceeds to step 233, which is the next step without going through step 232.

In step 233, when there are a plurality of objects included in the working image, which is either the target image or the background image, the depth of each object is extracted, and in step 234, the object is rearranged by referring to the depth value (depth value).

The depth value of each pixel represents a three-dimensional distance difference between objects in the image and is expressed as a value between 0 and 255. The closer the distance is, the closer to zero. Therefore, an object having a small depth value is placed before an object having a large depth value.

For example, as shown in FIG. 5C, the object T1 having the smallest depth value is disposed at the frontmost surface, the object T2 having the middle depth value is disposed at the middle, and the object having the largest depth value. Place (T3) at the very back (back).

When viewed from the side of the display, as shown in FIG. 5D, an object T1 having the smallest depth value is disposed in front of the display panel D and an object having an intermediate depth value in the display panel D. T2 is disposed, and an object T3 having the largest depth value is disposed behind the display panel D. FIG.

In step 235, the background image and the target image are formed as 3D images by applying a three-dimensional effect to the rearranged object. The object image provided with the three-dimensional effect is illustrated in FIG. 5E.

Without this step, a cardboard effect would appear that would make the object look like a sheet of paper.

Referring to FIG. 3 again, in operation 240, each of the 3D object image and the 3D background image is converted into a multiview image AS3D.

A plurality of viewpoint images are generated for the 3D object image and the 3D background image, respectively. As shown in FIGS. 6A and 6B, in order to generate a multiview image based on an image of a single viewpoint, three-dimensional image information such as disparity map, motion compensation information, and object segmentation information is extracted from the 3D object image and the 3D background image. N virtual view images are generated.

In step 250, the AS3D target image converted into a multiview image and the AS3D background image are synchronized.

A raster file of the target image and a raster file of the multiview image are generated so that each generated AS3D target image and the AS3D background image are reproduced on separate screens. For example, as shown in FIG. 7, assuming that an object image and a background image are each generated as 9 viewpoint images as in the exemplary embodiment of the present specification, a raster file generated according to each viewpoint is 9 tile content. It is assumed that each image has a resolution of 1280 × 720.

In order to synchronize the generated raster files, as illustrated in FIG. 8, each viewpoint image of the target image content and each viewpoint image of the background image content may be bundled into a pair of tiles to generate 9 new tiles. . For example, the one-eye image of the target video content and the one-eye image of the background image content are grouped into a pair of one-eye image. The paired viewpoint images may be processed for synchronization. For example, the target video content and the background video content may include time information for each unit. During playback, synchronization with time information can be achieved.

The image generated by pairing the target video content and the background video content may have a total resolution of 3840 × 4320 or 7680 × 2160.

As shown in Figs. 9A and 9B, display setting can be performed through a display screen of a PC or the like connected to the playback apparatus. The two displays are set to the extended mode in order to reproduce the object image and the background image on the first display panel and the second display panel, respectively.

In the exemplary embodiment of the present specification, since the luminance decreases in proportion to the number of viewpoints and the resolution also decreases with the number of viewpoints, a nine-eye display is adopted to obtain a stable image when playing content for a stereoscopic image array. It is not limited to this. In addition, in the case of adopting a 9-eye display, it is preferable to adopt a high-resolution LCD panel of 4K (3840x2160) or more in that it can prevent the problem of deterioration in brightness and resolution.

As shown in FIG. 9, the first display panel 210 is installed on the rear side of the two display panels separated from each other, and the second display panel is ceiling mounted in front of the first display panel 210. On the floor or on the floor to output the target image. At this time, the target image is plotted by a two-way mirror 230 provided at a 45 degree angle on the second display panel 220 to form a hologram. Therefore, the target image is stacked and displayed in front of the background image output by the first display panel 210 with different depths.

In general, the depth that can be expressed in an autostereoscopic 3d display panel is a maximum pop-out distance and a maximum depth in distance. The maximum pop-out distance and maximum depth in distance are 0.5 times the display panel height H, respectively, and the maximum depth distance that can be expressed in one display is the display panel height.

When the background image and the target image are output by the separated display panel as illustrated in FIG. 9, a depth deeper as much as the physical distance between the first display panel and the second display panel may be expressed, and the physical distance may be adjusted. It is possible.

The first display panel 410 and the second display panel 420 are raised on one plane with each other, and two-way mirrors are provided at the angles of 45 degrees.

The background image is reproduced on the first display panel 410 disposed further behind the viewer, and the object image is reproduced on the second display panel 420 disposed earlier. In this case, the front means a direction closer to the viewer.

The background image output from the first display panel 410 is plotted by the first two-way mirror 430 to generate a first hologram, and the target image output from the second display panel 420 is the second two-way mirror 440. ) To generate a second hologram.

Accordingly, the first hologram and the second hologram are displayed to be stacked with different depths.

That is, only the physical distance between the first display panel 410 and the second display panel 420 may further provide a sense of depth between the first hologram and the second hologram.

At this point, it will be understood that each block of the flowchart illustrations and combinations of flowchart illustrations may be performed by computer program instructions. Since these computer program instructions may be mounted on a processor of a general purpose computer, special purpose computer, or other programmable data processing equipment, those instructions executed through the processor of the computer or other programmable data processing equipment may be described in flow chart block (s). You will create a module that performs the functions. These computer program instructions may be stored in a computer usable or computer readable memory that can be directed to a computer or other programmable data processing equipment to implement functionality in a particular manner, and thus the computer usable or computer readable memory. It is also possible for the instructions stored in to produce an article of manufacture containing an instruction module that performs the functions described in the flowchart block (s). Computer program instructions may also be mounted on a computer or other programmable data processing equipment, such that a series of operating steps may be performed on the computer or other programmable data processing equipment to create a computer-implemented process to create a computer or other programmable data. Instructions for performing the processing equipment may also provide steps for performing the functions described in the flowchart block (s).

In addition, each block may represent a portion of a module, segment, or code that includes one or more executable instructions for executing a specified logical function (s). It should also be noted that in some alternative implementations, the functions noted in the blocks may occur out of order. For example, the two blocks shown in succession may in fact be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending on the corresponding function.

In this case, the term '~ part' used in the present embodiment refers to software or a hardware component such as an FPGA or an ASIC, and '~ part' performs certain roles. However, '~' is not meant to be limited to software or hardware. '~ Portion' may be configured to be in an addressable storage medium or may be configured to play one or more processors. Thus, as an example, '~' means components such as software components, object-oriented software components, class components, and task components, and processes, functions, properties, procedures, and the like. Subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functionality provided within the components and the 'parts' may be combined into a smaller number of components and the 'parts' or further separated into additional components and the 'parts'. In addition, the components and '~' may be implemented to play one or more CPUs in the device or secure multimedia card.

Those skilled in the art will appreciate that the present disclosure may be embodied in other specific forms without changing the technical spirit or essential features of the present disclosure. Therefore, it should be understood that the embodiments described above are exemplary in all respects and not restrictive. The scope of the present specification is indicated by the following claims rather than the detailed description, and all changes or modifications derived from the meaning and scope of the claims and the equivalent concept are included in the scope of the present specification. Should be interpreted.

On the other hand, the present specification and drawings have been described with respect to the preferred embodiments of the present specification, although specific terms are used, it is only used in a general sense to easily explain the technical content of the present specification and to help the understanding of the invention, It is not intended to limit the scope of the specification. It will be apparent to those skilled in the art that other modifications based on the technical spirit of the present disclosure may be implemented in addition to the embodiments disclosed herein.

Conventional multi-view AS3D video has a problem that the coherence of the three-dimensional effect that the three-dimensional effect is weak at one time point, and the three-dimensional effect is relatively strong at another point of view occurs, but the present invention converts one 2D image content into a multi-view 3D image content Accordingly, the object image and the background image can be reproduced on two physically separated displays, thereby providing a three-dimensional image without dizziness or ghosting. This can further stimulate the imaging industry market.

Claims

A content conversion method of a content conversion apparatus for stacked holograms for generating stacked holograms from one 2D video content,

Extracting a still image from the 2D image content at a predetermined frame rate;

Separating the object image and the background image including one or more objects with respect to the extracted still image;

Mapping each of the separated target image and background image using a specific map;

Generating a multiview image by extracting depth and stereoscopic values for each of the mapped target image and background image

Content conversion method for a stacked hologram comprising a.
The method of claim 1,

The mapping of the separated target image and the background image using a specific map may include:

Extracting a depth value of each pixel of at least one object included in the target image and the background image; And

Rearranging the extracted object with larger depth value to the front than the object with smaller depth value

Content conversion method for a stacked hologram comprising a.
The method of claim 2,

Before extracting the depth value of each pixel for one or more objects included in the target image,

Filling pixels by referring to the pixels around the boundary of the empty area in the empty area of the separated background image

Content conversion method for a stacked hologram comprising a.
The method of claim 1,

After generating the multi-view image of the generated target image and the multi-view image of the background image into a single file, a raster file is generated, and a reproduced image is generated by synchronizing the generated raster file for the target image with the raster file for the background image. step

Content conversion method for a stacked hologram further comprising.
The method of claim 1,

And wherein the specific map is at least one of a depth map, a displacement map, a texture map, and a combination thereof.
Combined with hardware,

Extracting a still image according to a predetermined frame rate from the 2D image content;

Separating the object image and the background image including one or more objects with respect to the extracted still image;

Mapping each of the separated target image and background image using a specific map; And

Generating a multi-view image using depth and stereoscopic values for each of the mapped target image and background image

A computer program stored on the medium for executing.
A content conversion device for stacked holograms for generating stacked holograms from one 2D video content,

An extraction module which extracts a still image from the 2D image content at a predetermined frame rate, and separates a target image and a background image including one or more objects from the extracted still image;

A mapping module for mapping each target image and the background image using a specific map; And

A viewpoint conversion module for generating a multiview image for each of the mapped target image and background image

Content conversion device for a stacked hologram comprising a.
The method according to claim 7,

Synchronization module for creating a raster file after combining the multi-view image of the generated target image and the multi-view image of the background image into a single file, and synchronizing the raster file for the target image and the raster file for the background image

Stacked holographic content converter further comprising a.
The method of claim 7, wherein

The extraction module is a content conversion device for a stacked hologram, characterized in that to fill the pixel by referring to the pixel around the boundary of the blank area in the blank area of the background image separated from the separated target image before the mapping.
The method according to claim 7,

The mapping module extracts a depth value of each pixel with respect to one or more objects included in the image, and arranges an object having a larger extracted depth value in front of an object having a smaller depth value. .