KR20140077398A

KR20140077398A - Method and apparatus for scalable multiplexing in three-dimension display

Info

Publication number: KR20140077398A
Application number: KR1020120146161A
Authority: KR
Inventors: 신홍창; 엄기문; 김찬; 정원식; 허남호; 이현
Original assignee: 한국전자통신연구원
Priority date: 2012-12-14
Filing date: 2012-12-14
Publication date: 2014-06-24
Also published as: KR101992767B1

Abstract

An apparatus and a method for scalable multiplexing in a stereoscopic image display are provided. The apparatus creates a virtual viewpoint image using an original image to be displayed on the stereoscopic image display and adjusts the size of the created virtual viewpoint image to perform scalable multiplexing in the stereoscopic image display; detects, from the original image, a region of interest or an object of interest for which scalable multiplexing is to be performed; and adjusts the size and depth of the region of interest or the object of interest. And, the apparatus performs scalable multiplexing by rearranging the region of interest or the object of interest, of which the size and depth are adjusted, in the virtual viewpoint image, of which the size is adjusted.

Description

[0001] The present invention relates to a method and apparatus for hierarchical multiplexing in a stereoscopic image display,

The present invention relates to an apparatus and method for hierarchical multiplexing in a stereoscopic image display, and more particularly, to a method and apparatus for hierarchical multiplexing in a stereoscopic image display, in which a region of interest (ROI) or an object of interest (OOI) To an apparatus and method for hierarchical multiplexing in a stereoscopic image display that adjusts depth.

Currently, a stereoscopic image display (hereinafter, used in combination with a "three-dimensional display") displays an image of two or more viewpoints on a spatially divided display panel and simultaneously reproduces the individual viewpoints. Images reproduced at the same time can be perceived in three dimensions as a viewpoint is separated by a driving method such as a polarizing film, a lenticular sheet, and a parallax barrier, and images of different viewpoints are presented to two eyes of a person.

In the multi-view display for displaying a stereoscopic image, a plurality of views must be simultaneously reproduced. In order to acquire a real multi-view image other than computer graphics (CG), there are many practical problems such as camera arrangement, acquisition method, data storage, and data transmission. Accordingly, a method of providing a multi-view image using a view synthesis technique that generates a virtual view image using a smaller number of reference view images than the number of reproduction views is widely used.

In addition, since a three-dimensional display basically needs to simultaneously reproduce a plurality of viewpoint images, there is a trade-off between the playback space and the resolution per viewpoint. In other words, the left / right image is input as a HD (High Definition) image, and an imaginary view of the same resolution is generated through the intermediate process of image synthesis. As a result, .

Korean Patent Publication No. 10-2012-0063984 ("Multi-view image generation method and apparatus ", published by Electronics and Telecommunications Research Institute, June 18, 2012)

SUMMARY OF THE INVENTION It is an object of the present invention to overcome the problems described above and to provide a method and apparatus for hierarchical multiplexing in a stereoscopic image display capable of more naturally controlling the size and depth of an ROI or an OOI in a stereoscopic image display through hierarchical multiplexing And to provide a method and an apparatus for the same.

The technical objects of the present invention are not limited to the technical matters mentioned above, and other technical subjects not mentioned can be clearly understood by those skilled in the art from the following description.

According to an aspect of the present invention, there is provided a method of hierarchically multiplexing in a stereoscopic image display, the method comprising: generating a virtual viewpoint image using an original image to be displayed on the stereoscopic image display; Adjusting a magnitude and a depth of a region of interest or an object of interest to be hierarchically multiplexed in the original image, and adjusting a magnitude and a depth of the region of interest, And relocating the object.

According to the apparatus and method for hierarchical multiplexing in the stereoscopic image display, hierarchical multiplexing for adjusting the size and depth of a region of interest or an object of interest to be hierarchically multiplexed and rearranging the virtual viewpoint images It is possible to provide a natural depth adjusting effect, thereby improving the resolution per viewpoint of the image displayed on the stereoscopic image display.

1 is a conceptual diagram showing the principle of a general three-dimensional multi-view display.
FIG. 2 is a conceptual diagram of a general stereoscopic-based multi-view image generation.
3 is a conceptual diagram schematically showing a variation control method in a three-dimensional display according to an embodiment of the present invention.
4 is a conceptual diagram schematically showing a depth adjusting method in a three-dimensional display according to an embodiment of the present invention.
5 is a conceptual diagram for explaining hierarchical multiplexing according to an embodiment of the present invention.
6 is a diagram schematically showing a multi-layer multiplexing apparatus according to an embodiment of the present invention.
Fig. 7 is a view schematically showing the multiplexing unit shown in Fig. 6. Fig.
8 is a diagram schematically illustrating a hierarchical multiplexing order according to an embodiment of the present invention.

While the invention is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will herein be described in detail.

It should be understood, however, that the invention is not intended to be limited to the particular embodiments, but includes all modifications, equivalents, and alternatives falling within the spirit and scope of the invention.

The terms first, second, etc. may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component. And / or < / RTI > includes any combination of a plurality of related listed items or any of a plurality of related listed items.

It is to be understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, . On the other hand, when an element is referred to as being "directly connected" or "directly connected" to another element, it should be understood that there are no other elements in between.

The terminology used in this application is used only to describe a specific embodiment and is not intended to limit the invention. The singular expressions include plural expressions unless the context clearly dictates otherwise. In the present application, the terms "comprises" or "having" and the like are used to specify that there is a feature, a number, a step, an operation, an element, a component or a combination thereof described in the specification, But do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the relevant art and are to be interpreted in an ideal or overly formal sense unless explicitly defined in the present application Do not.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In order to facilitate the understanding of the present invention, the same reference numerals are used for the same constituent elements in the drawings and redundant explanations for the same constituent elements are omitted.

1 is a conceptual diagram showing the principle of a general three-dimensional multi-view display. FIG. 2 is a conceptual diagram of a general stereoscopic-based multi-view image generation.

As shown in FIG. 1, a three-dimensional multi-view display displays an image 11 of more than two view points on a spatially divided display panel 10 and simultaneously reproduces the individual viewpoints. The images reproduced at the same time on the display panel 10 are separated by a driving method such as a polarizing film, a lenticular sheet, a parallax barrier, or the like. Then, images of different viewpoints are displayed on the two eyes of the viewer, and they are displayed as multi-viewpoint images 20 that can feel the three-dimensional feeling.

According to this principle, the multi-view image 20 receives the left image 30 and the right image 40 and extracts the three-dimensional image information from the left and right images 30 and 40 as shown in FIG. 2 N virtual viewpoint images 50 are generated, and the generated virtual viewpoint images 50 are displayed on the display panel. In this case, the three-dimensional image information can be extracted by using various information such as a mutation map, motion compensation information, and object division information of the left and right images.

As such, the 3D multi-view display basically requires a plurality of viewpoint images to be played simultaneously, and thus a trade-off occurs between the playback space and the resolution per viewpoint. Accordingly, as shown in FIG. 2, the left and right images are input to the HD image, so that a virtual point of view having the same resolution is generated through an intermediate process of image synthesis. However, .

Hereinafter, an apparatus and method for hierarchical multiplexing in a stereoscopic image display according to an embodiment of the present invention will be described in detail with reference to FIG. 3 to FIG.

3 is a conceptual diagram schematically showing a variation control method in a three-dimensional display according to an embodiment of the present invention.

As shown in Fig. 3, in the three-dimensional display, since there is a disparity between the left image 60a and the right image 60b, the user feels a sense of depth. If the parallax of the object in the video displayed on the left and right images 60a and 60b is larger, the depth of view becomes larger, and the viewer feels as if the object is getting closer. Therefore, when the variation of the region of interest (ROI) or the object of interest (OOI) in the image displayed by the three-dimensional multi-view display is controlled, the depth feeling is controlled.

If it is desired to adjust the depth of the object of interest (OOI) in the video, that is, the depth of the vehicle C11, the position of the vehicle C11 is adjusted in the left / right images 60a and 60b Setting. Then, the viewer feels as if the vehicle C12 of the image displayed by the position adjustment becomes closer.

However, in the real world, when an object approaches a person, it is natural that the size of the object becomes larger. If the depth is controlled by adjusting only the position, the depth of the object becomes geometrically unnatural.

Therefore, the original purpose of the 3D display is to add 3D sensation and real feeling of the scene to the 2D display. Therefore, in order to achieve a more natural depth control, the ROI or the object of interest It should not be solved by adjusting the actual size.

In the following, we will explain more specifically how to control the more natural depth sense.

4 is a conceptual diagram schematically showing a depth adjusting method in a three-dimensional display according to an embodiment of the present invention.

Referring to FIG. 4, when the variation of the vehicle C21, which is the object of interest (OOI) in the image IM1, is adjusted as in FIG. 3, the depth of the object of interest is adjusted so that the vehicle C21 becomes closer I feel. However, when the variation of the vehicle C21 is adjusted by the distance d, the obstacle area R1 is generated because there is no information (background) on the area covered by the first vehicle C21. The occlusion region in the embodiment of the present invention (hereinafter, used in combination with the "shielded region ") is an area that occurs as a region in which information has not appeared due to positional displacement of an object of interest.

To interpolate this occluded region R1, the region of interest or the object of interest is enlarged to such an extent that it can mask both of its shielded regions. In other words, if the shielded area generated by moving the position to give a mutation to the object of interest is enlarged and covered by the object of interest, there is no need to interpolate by post-processing. In addition, the depth control can be made more natural because the depth of the object of interest can be adjusted and the size can be changed.

In other words, if the original size of the vehicle C31 is enlarged like the vehicle C32 before the depth of the image IM2 is adjusted in order to interpolate the shielded area, variations in the object in the leftward or rightward direction or in the up- A free space SR is formed. If the variation is within the limits of the generated clearance space (SR), the variation of the object of interest or the region of interest can be adjusted without interpolating the shielded area. However, although the depth of field is controlled more naturally by enlarging the area of interest or the object of interest, the resolution may be degraded due to enlargement.

The degradation problem that may occur due to enlargement of the area of interest or the object of interest can be solved by hierarchical multiplexing when multi-point multiplexing in a three-dimensional display.

As described above with reference to FIG. 2, in the three-dimensional multi-view display, the output view image becomes smaller than the input original image. This is due to the fact that multiple viewpoints are simultaneously played on a limited resolution display, so that the resolution per viewpoint is reduced and rearranged, which is referred to as multiplexing. When multiplexing, images at all virtual viewpoints are reduced in size at the same rate and multiplexed.

For example, assuming that the input reference image is HD (2K) size and the display outputting it is Ultra High Definition (UHD) (4K) ) To generate nine virtual viewpoint images. When this is reproduced on an ultra-high resolution multi-view display, the resolution of each virtual viewpoint image is reduced to approximately "HD * (4/9)" size. A more detailed description will be given later with reference to FIG.

Accordingly, when the ROI or ROI is detected and the ROI is multiplexed without using an image of a virtual viewpoint having a reduced size when the ROI or the ROI is enlarged, the ROI is multiplexed at a different ratio from the non-detection region, The region of interest or the object of interest. That is, through the scalable multiplexing which multiplexes the non-detection area excluding the ROI or the ROI and the detection area corresponding to the ROI or the ROI at a different ratio, Can be enlarged.

Hereinafter, the process of hierarchical multiplexing according to an embodiment of the present invention will be described more specifically.

5 is a conceptual diagram for explaining hierarchical multiplexing according to an embodiment of the present invention.

5, in order to describe the hierarchical multiplexing according to the embodiment of the present invention, a left-side image INIM1 and a right-side image INIM2 of HD size are input to nine point-in-time images VIM1-VIM9) are reproduced at the same time, and the size of the vehicle C41, which is an object of interest, is increased by about 1.2 times.

First, when the input images INIM1 and INIM2 are received, nine virtual viewpoint images VIM1 through VIM9 are generated through image synthesis, and the sizes are adjusted and multiplexed according to the resolution of the final display for multiplexing. At this time, the vehicle C41, which is an object of interest to adjust the depth sense, is detected from the input images INIM1 and INIM2. Then, the size of the vehicle (C41) is adjusted within the effective enlargement range in which enlargement is possible, and the depth variation is controlled within the effective depth range. The rearranged images RIM1 to RIM9 are generated when the vehicle C41 whose size and variation are adjusted to the images VIM1 to VIM9 at the respective virtual viewpoints is rearranged. The effective enlargement range according to the embodiment of the present invention refers to a scaling factor that can be expanded without degrading the resolution, and the effective enlargement range is expressed by Equation (1).

[ Equation One]

_,

Where S is the enlargement ratio of the region of interest, I is the resolution of the original input image, and O is the resolution of a view image that is multiplexed and output.

The effective depth range is a range in which the depth can be controlled by controlling the variation within the range that the shielding region is not generated by calculating the margin region that can give a variation due to the enlargement of the ROI or the ROI. In addition, the maximum effective depth range may be limited to a range within which the visual fatigue caused by crosstalk between view images presented to the user's eyes does not occur. This may be determined by a number of complex factors such as the size of the display, the distance between the display and the user, the focal length, etc., and the effective range may be limited through direct user input within a range where the user does not feel visual fatigue .

6 is a diagram schematically showing a multi-layer multiplexing apparatus according to an embodiment of the present invention. Fig. 7 is a view schematically showing the multiplexing unit shown in Fig. 6. Fig.

6, a hierarchical multiplexing apparatus 100 according to an embodiment of the present invention includes an image receiving unit 110, a virtual viewpoint generating unit 120, and a multiplexing unit 130.

The image receiving unit 110 receives an original of an image to be displayed on a three-dimensional display (hereinafter referred to as "original image"). The image receiving unit 110 transmits the input original image to the virtual view generation unit 120 for three-dimensional display. In addition, the image receiving unit 110 transmits the original image to the multiplexing unit 130 so that a region of interest or an object of interest for hierarchical multiplexing is detected.

The virtual viewpoint generation unit 120 receives the original image from the image reception unit 110. The virtual viewpoint generation unit 120 generates N virtual viewpoint images from the original image through view synthesis. The virtual viewpoint generation unit 120 transmits the generated virtual viewpoint images to the multiplexer 130 for multiplexing.

The multiplexer 130 receives the virtual viewpoint images from the virtual viewpoint generator 120. The multiplexer 130 adjusts the sizes of the virtual viewpoint images. On the other hand, the multiplexer 130 receives the original image from the image receiver 110. Then, the multiplexer 130 adjusts the size and depth of the ROI or ROI to be hierarchically multiplexed from the original image, rearranges the size of the ROI to the size-adjusted virtual view images, and generates and rearranges the rearranged image.

7, the multiplexing unit 130 includes a virtual view size adjusting unit 131, a depth adjusting unit 132, and an image rearranging unit 133.

The virtual view size adjustment unit 131 adjusts the size of the virtual view images transmitted from the virtual view generation unit 120.

The depth adjusting unit 132 receives the original image from the image receiving unit 110. The depth adjuster 132 determines a region of interest or an object of interest to adjust the depth, and detects a region of interest or an object of interest in the original image. The depth adjusting unit 132 calculates the effective enlargement range of the ROI or the ROI to adjust the depth, and then resizes the ROI according to the enlargement ratio. At this time, the enlargement ratio may be input by the user as a number or may be enlarged stepwise to a predetermined number. The depth adjuster 132 calculates the effective depth control range according to the enlargement of the ROI or the ROI, and adjusts the depth of the ROI or the ROI within the range.

The image rearranging unit 133 receives the adjusted virtual viewpoint images from the virtual viewpoint size adjuster 131. Then, the image rearranging unit 133 receives the information on the region of interest or the object of interest whose size and depth have been adjusted from the depth adjusting unit 132. The image rearranging unit 133 rearranges the image of the ROI or the ROI whose size and depth are adjusted in the resized virtual viewpoint images.

8 is a diagram schematically illustrating a hierarchical multiplexing order according to an embodiment of the present invention.

As shown in FIG. 8, the image receiving unit 110 of the hierarchical multiplexing apparatus 100 according to the embodiment of the present invention receives an original image to be displayed on a three-dimensional display (S100). The image receiving unit 110 transmits the input image for the three-dimensional display to the virtual view generation unit 120, and outputs the original image to the multiplexing unit 120 so that the image of interest or the object of interest to be hierarchically multiplexed can be detected from the input original image. (130).

The virtual viewpoint generation unit 120 generates N virtual viewpoint images from the original image through view synthesis (S110). Then, the virtual viewpoint generation unit 120 transmits the virtual viewpoint images generated for the multiplexing to the multiplexer 130.

The multiplexer 130 adjusts the size of the virtual viewpoint images transmitted from the virtual viewpoint generator 120 (S120). Then, the multiplexer 130 adjusts the size and depth of the ROI or ROI to be hierarchically multiplexed in the original image transmitted from the image receiver 110 (S130) and rearranges the size and depth of the ROI to the size- A rearrangement image is generated (S140).

While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, but, on the contrary, And various modifications may be made by those skilled in the art without departing from the scope of the present invention.

100: Hierarchical multiplexing apparatus 110: Image receiving unit
120: virtual viewpoint generation unit 130:
131: virtual view size resizing unit 132: depth adjusting unit
133: image rearrangement unit

Claims

A hierarchical multiplexing method in a stereoscopic image display,
Generating a virtual viewpoint image using an original image to be displayed on the stereoscopic image display and adjusting a size of the virtual viewpoint image;
Detecting a region of interest or an object of interest to be hierarchically multiplexed in the original image, and adjusting the size and depth; And
Rearranging the region of interest or the object of interest whose size and depth have been adjusted on the image of the virtual time point of which the size has been adjusted
Wherein the method comprises the steps of: