KR101992767B1

KR101992767B1 - Method and apparatus for scalable multiplexing in three-dimension display

Info

Publication number: KR101992767B1
Application number: KR1020120146161A
Authority: KR
Inventors: 신홍창; 엄기문; 김찬; 정원식; 허남호; 이현
Original assignee: 한국전자통신연구원
Priority date: 2012-12-14
Filing date: 2012-12-14
Publication date: 2019-09-30
Also published as: KR20140077398A

Abstract

An apparatus and method for hierarchical multiplexing in a stereoscopic image display are provided. For hierarchical multiplexing in stereoscopic image display, create a virtual viewpoint image using the original image to be displayed on the stereoscopic image display, adjust the size, and detect the region of interest or the object of interest to be hierarchically multiplexed in the original image. Adjust the depth The hierarchical multiplexing is performed by rearranging the ROI or the ROI whose size and depth are adjusted in the virtual view image having the adjusted size.

Description

Method and apparatus for scalable multiplexing in three-dimension display}

The present invention relates to an apparatus and a method for hierarchical multiplexing in a stereoscopic image display, and more particularly, to the size of a region of interest (ROI) or an object of interest (OOI) in a stereoscopic image display. An apparatus and method for hierarchical multiplexing in a stereoscopic image display for controlling depth.

Currently, a stereoscopic image display (hereinafter, used in combination with a "three-dimensional display") displays two or more viewpoints on a spatially divided display panel and simultaneously reproduces individual viewpoints. The images being reproduced at the same time are separated by viewpoints by a driving method such as a polarizing film, a lenticular sheet, and a parallax barrier, so that images of different viewpoints are presented to both eyes of the human being in three dimensions.

In a multi-view display for displaying a stereoscopic image, several viewpoints must be played simultaneously. In order to acquire real-time multi-view images other than computer graphics (CG), there are many practical problems such as camera arrangement, acquisition method, data storage, and data transmission. Therefore, a method of providing a multiview image using a view synthesis technique of generating a virtual view image using a number of reference view images smaller than the number of reproduction viewpoints has been widely used.

In addition, since the 3D display basically needs to reproduce several viewpoint images at the same time, there is a trade-off between the playback space and the resolution per viewpoint. That is, the left / right image is input as a high definition (HD) image, and virtual viewpoints of the same resolution are generated through the intermediate process of image synthesis, but the resolution per view of the image presented on the multi-view display is reduced. There is this.

Korean Patent Publication No. 10-2012-0063984 ("Multi-view Image Generation Method and Apparatus", Korea Electronics and Telecommunications Research Institute, 2012.06.18 published)

Summary of the Invention An object of the present invention for solving the above problems is hierarchical multiplexing in a stereoscopic image display that can more naturally adjust the size and depth of a region of interest (ROI) or an object of interest (OOI) in a stereoscopic image display through hierarchical multiplexing. It is to provide an apparatus and method for.

The technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the following description.

In the hierarchical multiplexing method in a three-dimensional image display according to an embodiment of the present invention for achieving the above object of the present invention, after generating a virtual view image by using the original image to be displayed on the three-dimensional image display size Adjusting the size and depth by detecting a region of interest or an object of interest to be hierarchically multiplexed in the original image, and adjusting the size and depth of the region of interest or the image of the adjusted virtual view And relocating the object.

According to the apparatus and method for hierarchical multiplexing in the above-described stereoscopic image display, by detecting a region of interest or an object of interest to be hierarchically adjusted, adjusting the size and depth, and relocating to virtual viewpoint images, It is possible to provide a natural depth adjustment effect, thereby improving the per-view resolution of the image displayed on the stereoscopic image display.

1 is a conceptual diagram illustrating the principle of a general three-dimensional multi-view display.
2 is a conceptual diagram illustrating a general stereoscopic multiview image generation.
3 is a conceptual diagram schematically illustrating a method of adjusting a shift in a 3D display according to an exemplary embodiment of the present invention.
4 is a conceptual diagram schematically illustrating a depth adjusting method in a 3D display according to an exemplary embodiment of the present invention.
5 is a conceptual diagram illustrating hierarchical multiplexing according to an embodiment of the present invention.
6 is a diagram schematically illustrating a multi-layer multiplexing device according to an embodiment of the present invention.
FIG. 7 is a diagram schematically illustrating a multiplexer illustrated in FIG. 6.
8 is a diagram schematically illustrating a hierarchical multiplexing order according to an embodiment of the present invention.

As the present invention allows for various changes and numerous embodiments, particular embodiments will be illustrated in the drawings and described in detail in the written description.

However, this is not intended to limit the present invention to specific embodiments, it should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention.

Terms such as first and second may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as the second component, and similarly, the second component may also be referred to as the first component. The term and / or includes a combination of a plurality of related items or any item of a plurality of related items.

When a component is referred to as being "connected" or "connected" to another component, it may be directly connected to or connected to that other component, but it may be understood that other components may be present in between. Should be. On the other hand, when a component is said to be "directly connected" or "directly connected" to another component, it should be understood that there is no other component in between.

The terminology used herein is for the purpose of describing particular example embodiments only and is not intended to be limiting of the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, the terms "comprise" or "have" are intended to indicate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, and one or more other features. It is to be understood that the present invention does not exclude the possibility of the presence or the addition of numbers, steps, operations, components, components, or a combination thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in the commonly used dictionaries should be construed as having meanings consistent with the meanings in the context of the related art and shall not be construed in ideal or excessively formal meanings unless expressly defined in this application. Do not.

Hereinafter, with reference to the accompanying drawings, it will be described in detail a preferred embodiment of the present invention. In the following description of the present invention, the same reference numerals are used for the same elements in the drawings and redundant descriptions of the same elements will be omitted.

1 is a conceptual diagram illustrating the principle of a general three-dimensional multi-view display. 2 is a conceptual diagram illustrating a general stereoscopic multiview image generation.

As shown in FIG. 1, the three-dimensional multi-view display displays two or more images 11 on the spatially divided display panel 10 and simultaneously reproduces individual viewpoints. As described above, the images reproduced simultaneously on the display panel 10 are separated by a driving method such as a polarizing film, a lenticular sheet, and a parallax barrier. Then, the viewer's eyes are presented with images of different viewpoints, respectively, and are displayed as multi-view images 20 which can feel a three-dimensional effect.

Based on this principle, the multi-view image 20 receives the left image 30 and the right image 40 and extracts 3D image information from the left and right images 30 and 40 as shown in FIG. 2. N virtual view images 50 are generated, and the generated virtual view images 50 are displayed on the display panel. In this case, the 3D image information may be utilized by extracting a variety of information such as the disparity map, motion compensation information, object segmentation information of the left and right images.

As described above, since a three-dimensional multi-view display basically needs to reproduce several viewpoint images at the same time, a trade-off occurs between the playback space and the resolution per viewpoint. Accordingly, as shown in FIG. 2, the left / right image is input as an HD image, and a virtual view having the same resolution is generated through an intermediate process of image synthesis, but the resolution per view of the image presented on the multi-view display is reduced. There is this.

Hereinafter, an apparatus and method for hierarchical multiplexing in a stereoscopic image display according to an exemplary embodiment of the present invention will be described in detail with reference to FIGS. 3 to 8.

3 is a conceptual diagram schematically illustrating a method of adjusting a shift in a 3D display according to an exemplary embodiment of the present invention.

As shown in FIG. 3, in the three-dimensional display, a sense of depth exists because a disparity exists between the left image 60a and the right image 60b. If the parallax of the object in the image displayed in the left image 60a and the right image 60b becomes larger, the depth is increased and the viewer feels that the object is getting closer. Therefore, when the variation of the region of interest (ROI) or the object of interest (OOI) displayed in the 3D multi-view display is adjusted, the depth is adjusted.

If you want to make the object of interest (OOI) in the image, ie, the depth of the vehicle C11, closer, adjust the position of the vehicle C11 in the left / right images 60a, 60b so that the variation becomes larger. Set it. The viewer then feels as if the vehicle C12 of the image displayed by the position adjustment is getting closer.

However, in the real world, as the object gets closer to a human being, it is natural that the size becomes larger. However, when the position is controlled only by adjusting the position, the depth is geometrically unnatural.

Therefore, the original purpose of the three-dimensional display is to add a three-dimensional and real feeling of the scene than the two-dimensional display, so to adjust the depth more naturally, simply adjust the ROI or the object of interest (OOI) forward or backward. It is not to solve the problem but to adjust the actual size together.

The following describes in more detail a method for adjusting the natural depth.

4 is a conceptual diagram schematically illustrating a depth adjusting method in a 3D display according to an exemplary embodiment of the present invention.

Referring to FIG. 4, as shown in FIG. 3, when the variation of the vehicle C21 that is the object of interest OI in the image IM1 is adjusted, the depth of the object of interest may be adjusted to bring the vehicle C21 closer. I feel it. However, if the variation of the vehicle C21 by the distance d is adjusted, the covered region R1 is generated because there is no information (background) on the region covered by the vehicle C21. The occlusion region (hereinafter, used in combination with the "shielding region") in the embodiment of the present invention is a region generated by newly appearing an area in which there is no information as the object is displaced.

In order to interpolate this obscured region R1, the region of interest or the object of interest is enlarged to a size or more that can obscure all of its shielded regions. In other words, when the object of interest is enlarged to cover the shielding area generated by moving the position to give the object of interest a transition, it is not necessary to interpolate with post-processing. In addition, since the depth of the object of interest can be adjusted at the same time, the depth control becomes more natural.

In other words, if the original size of the vehicle C31 is enlarged like the vehicle C32 before the depth control of the image IM2 is interpolated to interpolate the shielding area, the object is shifted in the left and right or up and down directions so that the shielding area is not generated. Free space SR is formed. If the variation is provided within the limits of the free space SR formed as described above, the variation of the object of interest or the region of interest may be adjusted without interpolating the shielding region. However, although the depth is naturally adjusted by enlarging the region of interest or the object of interest, a resolution deterioration problem may occur.

The resolution degradation problem that may occur due to the expansion of the ROI or the object of interest may be solved by hierarchical multiplexing when performing multi-view multiplexing on a 3D display.

As described above with reference to FIG. 2, in the 3D multiview display, the output view image is smaller than the input original image. The reason for the decrease is that since images of multiple viewpoints are simultaneously reproduced on a display having limited resolution, the resolution per view is reduced and rearranged. This is called multiplexing. In the case of multiplexing, images of all virtual views are reduced in size by the same ratio and multiplexed.

For example, assuming that the input reference image is HD (2K) size and the display outputting the same is Ultra High Definition (UHD) (4K), and nine viewpoints are simultaneously played back, view synthesis is performed. Nine virtual view images are generated through). When reproduced on an ultra high-definition multi-view display, the resolution of each virtual viewpoint image is reduced to approximately "HD * (4/9)" size. More details will be described later with reference to FIG. 5.

Therefore, when a region of interest or an object of interest is enlarged by detecting the region of interest or the object of interest in the original image without using the image of the virtual point of view that is reduced in size, the resolution is multiplexed at a different ratio from that of the non-detection region to reduce resolution. The area of interest or objects of interest can be enlarged. That is, through the multiplexing of the non-detection area except for the ROI or the object of interest and the detection area corresponding to the ROI or the object of interest at different ratios, the corresponding ROI or the object of interest is not reduced. You can zoom in.

Hereinafter, a process of hierarchical multiplexing according to an embodiment of the present invention will be described in more detail.

5 is a conceptual diagram illustrating hierarchical multiplexing according to an embodiment of the present invention.

In FIG. 5, in order to explain hierarchical multiplexing according to an exemplary embodiment of the present invention, an image of nine virtual viewpoints on a multiview display of “4K” as an input image is a left image INIM1 and a right image INIM2 of an HD size. Suppose that the VIM1-VIM9) is simultaneously reproduced and the size of the vehicle C41 that is the object of interest is increased by about 1.2 times.

First, when the input images INIM1 and INIM2 are received, nine virtual viewpoint images VIM1 to VIM9 are generated through image synthesis, and are sized and multiplexed to fit the resolution of the final display for multiplexing. At this time, the vehicle C41, which is the object of interest to adjust the depth, is detected from the input images INIM1 and INIM2. In addition, while controlling the size of the vehicle (C41) within the effective enlargement range that can be enlarged, the depth shift is adjusted within the effective depth range. When the vehicle C41 in which the size and the variation are adjusted is rearranged in the images VIM1-VIM9 of each virtual viewpoint, the relocation images RIM1-RIM9 are generated. The effective enlargement range according to the embodiment of the present invention refers to a scaling factor that can be enlarged without degrading the resolution, and the effective enlargement range is shown in Equation 1.

[ Equation One]

_,

Where S is the magnification of the ROI, I is the resolution of the original input image, and O is the resolution of one viewpoint image which is multiplexed and output.

The effective depth range is a range in which the depth can be adjusted by adjusting the variation within a range in which a shielding region is not generated by calculating a marginal area that may cause variation due to the expansion of the ROI or the object of interest. In addition, the maximum effective depth range may be limited to a range within which the visual fatigue caused by crosstalk between the viewpoint images presented to the eyes of the user is not induced. This may be determined by various complex factors such as the size of the display, the distance between the display and the user, and the focal length, and the effective range may be limited through direct user input within the limit that the user does not experience visual fatigue. .

6 is a diagram schematically illustrating a multi-layer multiplexing device according to an embodiment of the present invention. FIG. 7 is a diagram schematically illustrating a multiplexer illustrated in FIG. 6.

As shown in FIG. 6, the hierarchical multiplexing apparatus 100 according to an exemplary embodiment of the present invention includes an image receiver 110, a virtual view generator 120, and a multiplexer 130.

The image receiving unit 110 receives an original (hereinafter referred to as an "original image") of an image to be displayed on a 3D display. The image receiver 110 transmits the original image input for the 3D display to the virtual view generator 120. In addition, the image receiver 110 transmits the original image to the multiplexer 130 so that the ROI or the object of interest for hierarchical multiplexing can be detected.

The virtual view generator 120 receives the original image from the image receiver 110. The virtual view generator 120 generates N virtual view images as the original image through view synthesis. The virtual view generator 120 transmits the generated virtual view images to the multiplexer 130 to multiplex the generated virtual view images.

The multiplexer 130 receives the virtual view images from the virtual view generator 120. The multiplexer 130 adjusts the sizes of the virtual viewpoint images. On the other hand, the multiplexer 130 receives the original image from the image receiver 110. The multiplexer 130 adjusts the size and depth of the ROI or the object of interest to be hierarchically multiplexed from the original image, and then rearranges the image to the adjusted virtual viewpoint images to generate a hierarchical multiplex.

More specifically, the multiplexer 130 includes a virtual viewpoint size adjuster 131, a depth adjuster 132, and an image repositioner 133, as shown in FIG. 7.

The virtual view size adjusting unit 131 adjusts the sizes of the virtual view images transferred from the virtual view generating unit 120.

The depth adjusting unit 132 receives the original image from the image receiving unit 110. The depth adjusting unit 132 determines a region of interest or an object of interest to adjust the depth, and detects the region of interest or the object of interest in the original image. The depth adjusting unit 132 calculates the effective enlargement range of the ROI or the object of interest to adjust the depth and adjusts the size according to the enlargement ratio. In this case, the enlargement ratio may be input numerically by the user or may be gradually expanded to a predetermined number. The depth controller 132 calculates an effective depth control range according to the enlargement of the corresponding ROI or the object of interest, and adjusts the depth of the ROI or the object of interest within the range.

The image rearranging unit 133 receives the virtual view images whose size is adjusted from the virtual view size adjusting unit 131. The image relocator 133 receives information about the ROI or the object of interest whose size and depth are adjusted from the depth adjuster 132. The image rearranging unit 133 rearranges the image of the ROI or the ROI whose size and depth are adjusted to the virtual view images having the size adjusted.

8 is a diagram schematically illustrating a hierarchical multiplexing order according to an embodiment of the present invention.

As shown in FIG. 8, the image receiving unit 110 of the hierarchical multiplexing apparatus 100 according to an embodiment of the present invention receives the original image to be displayed on the 3D display (S100). The image receiver 110 transmits the image input for the 3D display to the virtual view generator 120 and multiplexes the original image so that the ROI or object of interest to be hierarchically multiplexed can be detected from the inputted image. Forward to 130.

The virtual view generator 120 generates N virtual view images as an original image through view synthesis (S110). The virtual view generator 120 transmits the virtual view images generated for multiplexing to the multiplexer 130.

The multiplexer 130 adjusts the sizes of the virtual viewpoint images transmitted from the virtual viewpoint generator 120 (S120). In addition, the multiplexer 130 adjusts the size and depth of the ROI or the object of interest to be hierarchically multiplexed in the original image transmitted from the image receiver 110 and then rearranges the adjusted virtual viewpoint images. A relocation image is generated (S140).

As mentioned above, although preferred embodiments of the present invention have been described with reference to the accompanying drawings, it should not be construed as limiting the scope of the present invention, and the spirit and scope of the present invention as determined by the claims described below. Various modifications can be made by those skilled in the art without departing from the scope of the invention.

100: hierarchical multiplexing device 110: video receiver
120: virtual view generation unit 130: multiplexer
131: virtual view size control unit 132: depth control unit
133: video repositioning unit

Claims

In the hierarchical multiplexing method in stereoscopic image display,
Generating a virtual view image by combining the original image, and adjusting the size of the virtual view image in consideration of the size and resolution of the stereoscopic image display and the size and resolution of the original image;
Detecting a region of interest or an object of interest from the original image, and adjusting the size and depth of the region of interest or the object of interest in consideration of the size and resolution of the virtual viewpoint image; And
Constructing a multiplexed virtual view image by combining the adjusted virtual view image and the size and depth-adjusted ROI or object of interest;
Method for hierarchical multiplexing in a stereoscopic image display comprising a.

The method of claim 1,
Adjusting the size and depth by detecting the region of interest or the object of interest,
Checking the effective magnification range,
And determining the size of the region of interest or the object of interest in consideration of the effective magnification range.

The method of claim 2,
Confirming the effective enlargement range,
And determining the size ratio of the region of interest or the object of interest calculated based on the resolution of the original image and the resolution of the virtual viewpoint image.

The method of claim 1,
Adjusting the size and depth by detecting the region of interest or the object of interest,
Checking the effective depth range,
And determining the depth of the region of interest or the object of interest in consideration of the effective depth range.

The method of claim 4, wherein
Confirming the effective depth range,
And determining the effective depth range in which the region of interest or the shielded area by the object of interest is not generated in consideration of the size of the stereoscopic image display, the distance between the stereoscopic image display and the user, and a focal length. Method for hierarchical multiplexing in displays.

In the electronic device,
Image receiving unit for receiving the original image,
A virtual viewpoint generating unit for generating a virtual viewpoint image by combining the original image, and adjusting the size of the virtual viewpoint image in consideration of the size and resolution of the stereoscopic image display and the size and resolution of the original image;
Detects the region of interest or the object of interest from the original image, adjusts the size and depth of the region of interest or the object of interest in consideration of the size and resolution of the virtual viewpoint image, and adjusts the size of the image and the size of the adjusted virtual viewpoint And a multiplexer configured to combine a depth-of-interest-interested region of interest or an object of interest to form a multiplexed virtual viewpoint image.

The method of claim 6,
The multiplexer,
Check the effective magnification range,
And determining the size of the region of interest or the object of interest in consideration of the effective magnification range.

The method of claim 7, wherein
The multiplexer,
For the hierarchical multiplexing in the stereoscopic image display, the effective magnification range is determined in consideration of the size ratio of the region of interest or the object of interest calculated based on the resolution of the original image and the resolution of the virtual viewpoint image. Device.

The method of claim 6,
The multiplexer,
Check the effective depth range,
And determining the depth of the ROI or the object of interest in consideration of the effective depth range.

The method of claim 9,
The multiplexer,
And determining the effective depth range in which the region of interest or the shielded area by the object of interest is not generated in consideration of the size of the stereoscopic image display, the distance between the stereoscopic image display and the user, and a focal length. Device for hierarchical multiplexing in displays.