WO2012131752A1

WO2012131752A1 - Depth information updating device, stereoscopic video generation device, and depth information updating method

Info

Publication number: WO2012131752A1
Application number: PCT/JP2011/001795
Authority: WO
Inventors: 山本　純也; 仁尾　寛; 晴子寺井; 大輔加瀬
Original assignee: パナソニック株式会社
Priority date: 2011-03-25
Filing date: 2011-03-25
Publication date: 2012-10-04

Abstract

A depth information updating device (101) is provided with: a depth information acquisition unit (111) which acquires depth information which indicates a plurality of depth values; and a depth information updating unit (112) which updates the depth information by subjecting a depth image configured from the plurality of depth values indicated by the depth information to enhancement processing which is processing that enhances components in a certain band.

Description

Depth information update device, stereoscopic video generation device, and depth information update method

The present invention relates to a depth information update device, a stereoscopic video generation device, and a depth information update method for updating depth information for generating a stereoscopic video.

Conventionally, a video display device using a liquid crystal panel or the like has been used as a device for displaying a two-dimensional video. On the other hand, technological development of a three-dimensional video display device capable of viewing three-dimensional video (stereoscopic video) is progressing by alternately viewing two two-dimensional videos having parallax using active shutter glasses.

Also, in recent years, various techniques for generating stereoscopic video have been developed. For example, Patent Document 1 adjusts the depth value of a depth image for generating a stereoscopic video. And the technique (henceforth prior art A) which produces | generates a stereo image using the depth value after adjustment is disclosed.

JP, 2003-209858, A (paragraph 0060, paragraph 0076)

However, in the prior art A, there is a problem that it is not possible to improve the fine three-dimensional effect in a three-dimensional image.

Specifically, in the prior art A, the depth value is set to a value between the maximum projection amount and the maximum depression amount for generating a stereoscopic image with less discomfort. The setting of the depth value is not particularly considered for the improvement of the fine stereoscopic effect. Therefore, in the prior art A, the fine three-dimensional effect of a plurality of adjacent pixels is not improved.

The present invention has been made to solve the above-described problems, and an object thereof is to provide depth information capable of generating depth information for generating a stereoscopic image with an improved fine stereoscopic expression. It is providing an update apparatus etc.

In order to achieve the above object, a depth information updating apparatus according to an aspect of the present invention is a depth information updating apparatus that performs processing using a depth value corresponding to each pixel constituting a stereoscopic video, A process of emphasizing a component of a specific band with respect to a depth image composed of a plurality of depth information indicating a plurality of depth values to be generated and a depth image including the plurality of depth values indicated by the depth information And a depth information update unit configured to update the depth information by performing the emphasizing process.

That is, the depth information updating apparatus acquires a depth information acquiring unit that acquires depth information indicating a plurality of depth values, and a component of a specific band with respect to a depth image including the plurality of depth values indicated by the depth information. And a depth information updating unit configured to update the depth information by performing an emphasizing process that emphasizes.

Thereby, it is possible to generate depth information in which a change in depth value is emphasized. That is, it is possible to generate depth information for generating a stereoscopic video with an improved expression of fine stereoscopic effect.

In addition, preferably, the depth information update unit determines, in at least a part of the depth image, an absolute value of a difference between two depth values of which an absolute value of a difference between depth values of two adjacent pixels is less than a threshold. The emphasizing process is performed to increase the size.

In addition, preferably, the depth information update unit is an absolute value of a difference between the depth value of the processing target pixel which is the pixel to be processed and the depth value of the pixel adjacent to the processing target pixel in the depth image. A calculation unit which performs processing of calculating a certain difference depth value for all pixels constituting the depth image, and a difference depth value less than the threshold if the calculated difference depth value is less than a threshold And a processing execution unit that performs the enhancement processing on the corresponding processing target pixel.

In addition, preferably, the depth information update unit performs the enhancement process that is a filter process.

In addition, preferably, the depth information updating unit further indicates a plurality of depths indicated by the updated depth information such that absolute values of the plurality of depth values indicated by the updated depth information decrease at substantially the same rate. The updated depth information is updated by changing the value.

In addition, preferably, the depth information update unit updates the updated depth information by multiplying an absolute value of the plurality of depth values indicated by the updated depth information by a coefficient smaller than one.

A stereoscopic video generation device according to an aspect of the present invention includes the depth information update device, and a stereoscopic video generation unit that generates a new stereoscopic video using a plurality of depth values indicated by the updated depth information. .

The depth information update method according to an aspect of the present invention is performed by the depth information update apparatus that performs processing using depth values corresponding to respective pixels that configure a stereoscopic video. The depth information updating method includes a depth information acquisition step of acquiring depth information indicating a plurality of depth values for generating a stereoscopic video, and a depth image including the plurality of depth values indicated by the depth information. And a depth information updating step of updating the depth information by performing an emphasizing process which is a process of emphasizing a component of a specific band.

Note that the present invention may realize all or part of a plurality of components constituting such a depth information update apparatus as a system LSI (Large Scale Integration: large scale integrated circuit).

The present invention may also be implemented as a program that causes a computer to execute the steps included in the depth information update method. Also, the present invention may be realized as a computer readable recording medium storing such a program. Also, the program may be distributed via a transmission medium such as the Internet.

According to the present invention, it is possible to generate depth information for generating a three-dimensional image with an improved fine three-dimensional expression.

FIG. 1 is a diagram showing an example of the configuration of a stereoscopic video viewing system according to the first embodiment. FIG. 2 is a diagram for explaining a stereoscopic video. FIG. 3 is a block diagram showing an example of the configuration of a stereoscopic video reproduction apparatus. FIG. 4 is a diagram showing a depth table as depth information. FIG. 5 is a diagram showing a relationship between a stereoscopic target object in a stereoscopic video and a depth value. FIG. 6 is a view showing an example of a stereoscopic video showing a stereoscopic target object. FIG. 7 is a flowchart of stereoscopic video generation processing. FIG. 8 is a diagram for explaining changes in depth value. FIG. 9 is a flowchart of stereoscopic video generation processing A. FIG. 10 is a diagram showing a relationship between a stereoscopic target object in a stereoscopic video and a depth value. FIG. 11 is a block diagram showing an example of the configuration of the depth information update unit. FIG. 12 is a flowchart of stereoscopic video generation processing B. FIG. 13 is a diagram for explaining a processing target pixel and a pixel close to the processing target pixel.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the following description, the same components are denoted by the same reference numerals. Their names and functions are also the same. Therefore, detailed description about them may be omitted.

First Embodiment
FIG. 1 is a diagram showing an example of the configuration of a stereoscopic video viewing system 1000 according to the first embodiment. In FIG. 1, the X, Y, and Z directions are orthogonal to one another. Each of the X, Y, Z directions shown in the following figures are also orthogonal to one another.

As shown in FIG. 1, a stereoscopic video viewing system 1000 includes a stereoscopic video reproduction device 100, a stereoscopic video display device 200, and active shutter glasses 300.

The stereoscopic video display device 200 is, for example, a plasma display, a liquid crystal display, an organic EL display, or the like. Note that the stereoscopic video display device 200 is not limited to the above display, and may be a display of another type. For example, the stereoscopic video display device 200 may be a volume display type display in which a plurality of liquid crystal panels are arranged in the depth direction.

In the present specification, the video and the stereoscopic video may be either a moving image or a still image.

The stereoscopic video display device 200 includes a display surface 210 for displaying a video. The display surface 210 is assumed to be parallel to the XY plane. As an example, it is assumed that the display surface 210 can display an image composed of a plurality of pixels arranged in m (natural number) rows and n (natural number) columns.

Here, m and n are assumed to be 1080 and 1920, respectively. That is, it is assumed that the display surface 210 can display an image having a size of 1920 × 1080 pixels (hereinafter, also referred to as full HD size). Hereinafter, an image of a size that can be displayed on the display surface 210 is also referred to as a displayable image.

The size of the displayable video is not limited to the full HD size. The size of the displayable image may be, for example, the size of 1280 × 720 pixels.

In the present embodiment, the stereoscopic video display device 200 is, for example, a device that displays a stereoscopic video by a frame sequential method. In this case, the size of the stereoscopic video displayed by the display surface 210 is equal to the size of the displayable video.

Note that the display method of the stereoscopic video of the stereoscopic video display device 200 is not limited to the frame sequential method. The stereoscopic video display method of the stereoscopic video display device 200 may be, for example, a lenticular method. In this case, the size of the stereoscopic video displayed by the display surface 210 is smaller than the size of the displayable video.

The 3D image reproduction apparatus 100 is connected to the 3D image display apparatus 200 by the signal cable 10. The signal cable 10 is a high-definition multimedia interface (HDMI) cable.

The signal cable 10 is not limited to the HDMI cable, and may be, for example, a cable for D terminal, a coaxial cable, or the like. Further, the communication between the stereoscopic video reproduction device 100 and the stereoscopic video display device 200 is not limited to communication using a wire, and may be wireless communication.

The stereoscopic video reproduction device 100 transmits a signal indicating the stereoscopic video 30 as a three-dimensional video to the stereoscopic video display device 200 via the signal cable 10. The stereoscopic video 30 is configured of a video 31 for the left eye and a video 32 for the right eye.

The left-eye video 31 is a video to be shown to the left eye of the viewer (user) (hereinafter also referred to as a first viewpoint). The right-eye image 32 is an image for showing the viewer's right eye (hereinafter, also referred to as a second viewpoint). The left-eye video 31 and the right-eye video 32 are two-dimensional video images having parallax.

The stereoscopic video display device 200 alternately displays the left-eye video 31 and the right-eye video 32 on the display surface 210.

When the left-eye video 31 is displayed on the display surface 210, the active shutter glasses 300 shield the right eye of the viewer. Further, when the right-eye image 32 is displayed on the display surface 210, the active shutter glasses 300 shield the left eye of the viewer.

A viewer who wears the active shutter glasses 300 having such a configuration can view the left-eye video 31 with the left eye, and can view the right-eye video 32 with the right eye. Thereby, the viewer can feel the stereoscopic video 30 stereoscopically. That is, the stereoscopic video 30 is expressed using the first viewpoint video (left-eye video 31) of the first viewpoint and the second viewpoint video (right-eye video 32) of the second viewpoint.

As described above, the display method of the stereoscopic video is not limited to the frame sequential method using the active shutter glasses 300. For example, the display method of the stereoscopic video may be a method using polarized glasses. Further, for example, a method of displaying a stereoscopic image may be a method using a parallax barrier, a lenticular sheet or the like.

Further, the number of viewpoints required for the stereoscopic video display device 200 to display the stereoscopic video 30 is not limited to two, and may be three or more.

Next, stereoscopic video will be described in detail. The stereoscopic image 30 is an image generated using a plurality of depth values. The depth value corresponds to the amount of parallax between the left-eye video and the right-eye video.

FIG. 2 is a diagram for explaining a stereoscopic video. A stereoscopic video is composed of a plurality of pixels. Hereinafter, each pixel forming a stereoscopic video is referred to as a stereoscopic display pixel. A plurality of depth values are respectively associated with a plurality of stereoscopic display pixels constituting a stereoscopic video.

In FIG. 2, the display surface 210 is a parallax zero surface. The parallax zero plane is a plane in which the parallax of the pixels at the same position of the left eye image and the right eye image displayed on the parallax zero plane is zero.

The depth value indicates a value larger than a predetermined parallax zero reference value when a stereoscopic display pixel in the stereoscopic image is disposed (displayed) on the other side of the display surface 210. That is, when the insertion amount of the stereoscopic display pixel from the display surface 210 is a positive value, the depth value indicates a value larger than the parallax zero reference value.

Here, the parallax zero reference value is a value for arranging a stereoscopic display pixel at the position of the parallax zero surface (display surface 210).

The depth value indicates a value less than the parallax zero reference value when stereoscopic display pixels are arranged (displayed) on the front side of the display surface 210 in a stereoscopic image. That is, when the projection amount of the stereoscopic display pixel from the display surface 210 is a positive value, the depth value indicates a value less than the parallax zero reference value.

That is, each depth value is a value indicating the projection amount or back amount of the stereoscopic display pixel corresponding to the depth value from the zero parallax surface (display surface 210).

As an example, assuming that the position of the viewer 40 is 0, the depth value is represented in the range of 0 to 255 so that the value increases as the distance from the position of the viewer 40 increases, and the case where the parallax zero reference value is 123 is An example will be described. The depth value is not limited to the range of 0 to 255, and may be represented, for example, in the range of 0 to 511. Also, if the disparity zero reference value is 0, the depth value may be represented by a positive value and a negative value.

Hereinafter, a pixel for stereoscopic display is also referred to as a stereoscopic display pixel.

First, in the case of arranging a stereoscopic display pixel on the front side of the display surface 210 in a stereoscopic image, that is, a case where the projection amount of the stereoscopic display pixel is a positive value will be described.

In FIG. 2, the position where the depth value is the parallax zero reference value (123) corresponds to the position of the display surface 210 (parallax zero surface) in the Z direction.

As shown in FIG. 2, it is assumed that, on the display surface 210, the stereoscopic display pixels are displayed at, for example, a position P11 in the left-eye video and a position P12 in the right-eye video. In this case, the viewer 40 looks at the stereoscopic display pixels arranged at the position P11 with the left eye 41 by the operation of the active shutter glasses 300. Also, in this case, the viewer 40 looks at the pixel for stereoscopic display disposed at the position P12 with the right eye 42 by the operation of the active shutter glasses 300.

As a result, the viewer 40 looks as if the stereoscopic display pixel is arranged at the position P10 ahead of the display surface 210 by the distance d1 in the stereoscopic video. In this case, the depth value of the stereoscopic display pixel indicates a value less than the parallax zero reference value (123).

The horizontal distance between the position P12 and the position P11 corresponds to the amount of parallax. In addition, the parallax amount also corresponds to the projection amount d1 of the stereoscopic display pixel in the depth value. When the depth value is the parallax zero reference value, the viewer 40 looks as if stereoscopic display pixels are arranged on the display surface 210 (parallax zero surface).

The relation between the parallax amount and the depth value of the stereoscopic display pixel may be performed by a predetermined conversion formula. For example, the amount of parallax is d, the depth value of the stereoscopic display pixel is L, the depth value of the parallax zero plane is L _b , and the distance between the left eye 41 and the right eye 42 of the viewer 40 (interpupillary distance) is e.

It may be At this time, the depth value L is proportional to the difference in distance between the viewer 40 and P10 on the Z axis.

Next, the case where a stereoscopic display pixel is disposed on the other side of the display surface 210 in a stereoscopic image, that is, the case where the insertion amount of the stereoscopic display pixel is a positive value will be described.

As shown in FIG. 2, it is assumed that, on the display surface 210, the stereoscopic display pixels are displayed at, for example, a position P21 in the left-eye video and a position P22 in the right-eye video. In this case, the viewer 40 looks at the stereoscopic display pixels arranged at the position P21 with the left eye 41 by the operation of the active shutter glasses 300. Further, in this case, the viewer 40 looks at the stereoscopic display pixel arranged at the position P22 with the right eye 42 by the operation of the active shutter glasses 300.

As a result, the viewer 40 looks as if the stereoscopic display pixel is arranged at the position P20 on the opposite side of the display surface 210 by the distance d2 in the stereoscopic image. In this case, the depth value of the stereoscopic display pixel indicates a value larger than the parallax zero reference value (123).

Next, the configuration of the stereoscopic video reproduction device 100 will be described.

FIG. 3 is a block diagram showing an example of the configuration of the stereoscopic video reproduction device 100. As shown in FIG.

As shown in FIG. 3, the stereoscopic video reproduction device 100 includes a stereoscopic video generation device 110. The 3D image reproduction apparatus 100 also includes a processing unit and the like (not shown).

The three-dimensional video generation device 110 is a device for generating a three-dimensional video. Although details will be described later, the three-dimensional video generation apparatus 110 acquires depth information of the three-dimensional video to be processed, and updates the depth information. Then, using the updated depth information, a new stereoscopic image is generated. Here, the depth information is information indicating a plurality of depth values respectively corresponding to a plurality of stereoscopic display pixels constituting a stereoscopic video to be processed.

The stereoscopic video generation device 110 receives the stereoscopic video signal SG1. Here, the stereoscopic video signal SG1 is a signal indicating stereoscopic video data and depth information corresponding to the stereoscopic video. The stereoscopic video data is data of a left-eye video and a right-eye video.

Note that the data, information, and the like indicated by the stereoscopic video signal SG1 are not limited to those described above. The stereoscopic video signal SG1 may be, for example, a signal indicating data of a two-dimensional video and depth information for converting the two-dimensional video into a stereoscopic video.

The stereoscopic video signal SG1 is, for example, a signal indicating data read from the recording medium. The recording medium is, for example, a BD (Blu-ray Disc (registered trademark)). In this case, the above-described processing unit included in the stereoscopic video reproduction device 100 performs processing of reading stereoscopic video data and depth information from the recording medium.

Also, the stereoscopic video signal SG1 is not limited to the signals described above. The stereoscopic video signal SG1 may be, for example, a signal acquired from a broadcast wave. In this case, the above-described processing unit included in the stereoscopic video reproduction device 100 has, for example, the functions of a tuner and a demodulation circuit.

Here, depth information will be described. The depth information is expressed, for example, as the following depth table T100.

FIG. 4 is a diagram showing a depth table T100 as depth information. FIG. 4 shows the depth table T100 when the stereoscopic video is one frame of a moving image or a still image.

The depth table T100 indicates a plurality of depth values respectively corresponding to a plurality of stereoscopic display pixels constituting a stereoscopic video. The plurality of depth values indicated by the depth table T100 are values for generating a stereoscopic video. The plurality of depth values indicated by the depth table T100 are arranged in a matrix like the plurality of stereoscopic display pixels constituting the stereoscopic video.

Here, each of the plurality of depth values indicated by the depth table T100 is regarded as a pixel value. In this case, the depth table T100 as depth information indicates an image composed of a plurality of depth values (hereinafter, also referred to as a depth image). The depth image corresponds to a depth map (disparity map). In this case, the value of each pixel constituting the depth image is a depth value.

In FIG. 4, for example, Cmn indicates the depth value of a pixel corresponding to m rows and n columns in a stereoscopic video. Also, for example, C12 indicates a depth value of a pixel corresponding to one row and two columns in a stereoscopic video.

Note that each depth value indicated by the depth table T100 is not limited to the depth value corresponding to one pixel. Each depth value indicated by the depth table T100 may be, for example, a depth value corresponding to four pixels. In this case, the number of depth values indicated by the depth table T100 is one fourth of the number of stereoscopic display pixels constituting the stereoscopic video.

Next, stereoscopic video and depth values will be described.

FIG. 5 is a diagram showing a relationship between a stereoscopic target object in a stereoscopic video and a depth value. Here, a stereoscopic target object is an object to be displayed stereoscopically in a stereoscopic video.

FIG. 5A is a perspective view showing the positional relationship between the stereoscopic target object 50 in the stereoscopic video and the display surface 210. FIG. FIG. 5B is a graph for explaining the depth value of the three-dimensional object 50. In FIG. 5B, the vertical axis indicates the depth value (Z direction). The horizontal axis indicates the X direction (horizontal direction of stereoscopic video).

FIG. 6 is a view showing an example of a stereoscopic video G100 showing the stereoscopic target object 50. As shown in FIG. FIG. 6 is a diagram showing a stereoscopic image G100 parallel to the XY plane. Note that, in FIG. 6, in order to explain the depth value, a depth line L10 not shown in the stereoscopic video G100 is actually shown.

The depth line L10 is a line indicating n depth values respectively corresponding to n stereoscopic display pixels arranged in one line in the stereoscopic video G100.

As illustrated in FIG. 5A, in the stereoscopic image, the stereoscopic target object 50 is disposed, for example, on the front side of the display surface 210. In FIG. 5A, the depth line L10 is shown on the stereoscopic target object 50 and the display surface 210.

FIG. 5B is a graph showing the depth line L10.

In FIG. 5B, the depth line L10 in the region R10 corresponds to the depth line L10 shown in the stereoscopic target object 50. From the depth line L10 in the region R10 of FIG. 5B, it can be seen that the surface of the three-dimensional object 50 has unevenness.

When the depth table T100 is a table corresponding to the stereoscopic video G100, the depth image indicated by the depth table T100 indicates the stereoscopic video G100 expressed in gray scale.

Next, the configuration of the stereoscopic video generation device 110 will be described.

As shown in FIG. 3, the stereoscopic video generation device 110 includes a depth information updating device 101 and a stereoscopic video generation unit 113.

The depth information updating apparatus 101 performs processing using depth values corresponding to respective pixels constituting a stereoscopic video.

The depth information update device 101 includes a depth information acquisition unit 111 and a depth information update unit 112. The depth information acquisition unit 111 and the stereoscopic video generation unit 113 receive the stereoscopic video signal SG1.

The depth information acquisition unit 111 acquires depth information indicated by the received stereoscopic video signal SG1. The depth information is, for example, the depth table T100 of FIG. The depth information acquisition unit 111 transmits the acquired depth information to the depth information update unit 112.

Note that, for example, when the stereoscopic video signal SG1 is a signal indicating a multi-view video, the depth information acquisition unit 111 acquires depth information by performing stereo matching, which is a known technique, for example. When depth information is included in the stereoscopic video signal SG1, a value obtained by passing a specific conversion equation on the depth information may be acquired as depth information.

The depth information update unit 112 updates the received depth information, the details of which will be described later. Then, the depth information update unit 112 transmits the updated depth information to the stereoscopic video generation unit 113.

Hereinafter, the updated depth information transmitted by the depth information update unit 112 to the stereoscopic video generation unit 113 is referred to as updated depth information.

Although details will be described later, the stereoscopic video generation unit 113 generates a new stereoscopic video using stereoscopic video data indicated by the received stereoscopic video signal SG1 and the received updated depth information.

Then, the stereoscopic video generation unit 113 transmits, to the stereoscopic video display device 200, the stereoscopic video signal SG2 indicating the generated new stereoscopic video (video for left eye and video for right eye).

The stereoscopic video display device 200 alternately displays the video for the left eye and the video for the right eye indicated by the received stereoscopic video signal SG2 on the display surface 210 for each frame.

Next, processing for generating a stereoscopic video in the present embodiment (hereinafter, also referred to as stereoscopic video generation processing) will be described.

FIG. 7 is a flowchart of stereoscopic video generation processing.

In step S110, the depth information acquisition unit 111 acquires depth information indicated by the stereoscopic video signal SG1. The depth information is, for example, the depth table T100 of FIG. The depth information acquisition unit 111 transmits the acquired depth information to the depth information update unit 112.

In step S120, depth information update processing is performed. In the depth information update processing, processing in which the depth information update unit 112 emphasizes a component of a specific band for a depth image composed of a plurality of depth values indicated by the depth information (depth table T100) (hereinafter, emphasis processing) The depth information is updated by performing (a). Here, the specific band is a specific frequency band.

The depth information update processing will be described in detail below. Hereinafter, a stereoscopic video to be processed is referred to as a processing stereoscopic video. Here, it is assumed that the processing target stereoscopic video is a still image. Further, the processing target three-dimensional video is described as an example of the three-dimensional video G100. In this case, the depth image processed in the depth information update process corresponds to the stereoscopic video G100.

In step S125, depth enhancement processing is performed. In the depth enhancement processing, the depth information updating unit 112 emphasizes (increases) the change in depth value in a specific band for a depth image configured of a plurality of depth values indicated by the depth table T100 as depth information. ) Perform filter processing. That is, the depth information updating unit 112 performs an emphasizing process, which is a process of emphasizing a component of a specific band, on a depth image composed of a plurality of depth values indicated by the depth information (depth table T100). The enhancement process is, for example, a process of increasing the absolute value of the difference between the two depth values at which the absolute value of the difference between the depth values of two adjacent pixels is less than a threshold in at least part of the depth image. The threshold is a threshold used in step S122 described later. The threshold is, for example, 20.

When the depth image indicated by the depth table T100 is a processing target of the filter processing, for example, emphasizing the change of the depth value of the high frequency can be considered to be equivalent to the sharpening processing in the image processing. That is, emphasizing a change in depth value of a specific band can be performed by, for example, an FIR filter, as with a filter in image processing.

Furthermore, specifically, in depth enhancement processing, the depth information update unit 112 performs filter processing using a two-dimensional FIR (Finite Impulse Response) filter on the depth image indicated by the depth table T100.

Hereinafter, "change in depth value" is also referred to as "change in depth" or "change in depth".

The filter process of step S125 is performed with one pixel among the plurality of pixels forming the depth image as a pixel of interest. Note that each time the process of step S125 is performed, the filter process is performed with the different pixel as the pixel of interest.

Thereby, the depth table T100 as depth information is updated. The depth table T100 is updated each time the process of step S125 is performed. The FIR filter used for the above-mentioned filter processing is, for example, a sharpening filter using an unsharp mask.

The filter used for the filtering process is not limited to the sharpening filter, and may be another filter as long as it is a filter that emphasizes a change in depth value in a specific band.

When the process of step S125 is performed on all the pixels constituting the depth image, the depth information update unit 112 transmits the updated depth table T100 to the stereoscopic video generation unit 113 as updated depth information. . Then, the depth information update process in step S120 ends, and the process proceeds to step S130.

By the depth information updating process of step S120, the amplitude of the depth line L10 in the region R10 shown in FIG. 8A becomes large as shown in FIG. 8B. That is, the depth (three-dimensional effect) of the three-dimensional target object 50 is emphasized by the process of step S125.

In step S130, as described above, the three-dimensional video generation unit 113 uses the three-dimensional video data indicated by the received three-dimensional video signal SG1 and the received updated depth information (the updated depth table T100) to perform a new process. Generate stereoscopic video. The stereoscopic video data indicated by the stereoscopic video signal SG1 is data of a left-eye video and a right-eye video.

Specifically, the stereoscopic video generation unit 113 calculates the parallax amount of the left-eye video and the right-eye video using the plurality of depth values indicated by the updated depth information. Then, using the calculated amount of parallax, the stereoscopic video generation unit 113 updates one or both of the left-eye video and the right-eye video indicated by the stereoscopic video signal SG1. As a result, a new three-dimensional video composed of a new left-eye video and a right-eye video is generated.

Note that the process of generating a stereoscopic video using depth information is, for example, a process according to the MVC (Multiview Video Coding) standard. Therefore, detailed description of the process of calculating the amount of parallax, the process of updating the left-eye video and the right-eye video, and the like is not performed.

When the received stereoscopic video signal SG1 indicates data of a two-dimensional video and depth information for converting the two-dimensional video into a stereoscopic video, the stereoscopic video generation unit 113 determines a plurality of depths indicated by the depth information. The amount of parallax is calculated using the value. The three-dimensional video generation unit 113 generates a new left-eye video and a right-eye video from the two-dimensional video using the calculated parallax amount. As a result, a new three-dimensional video composed of a new left-eye video and a right-eye video is generated.

The three-dimensional video generation processing described above is processing when the processing target three-dimensional video is a still image, but when the processing target three-dimensional video is a moving image, the above-described three-dimensional video is generated for each frame constituting the moving image. The video generation process is repeated.

As described above, according to the present embodiment, the depth information updating unit 112 applies a component (depth of a specific band) to a depth image composed of a plurality of depth values indicated by the depth information (depth table T100). Process to emphasize the change of value).

Thereby, it is possible to generate depth information for sufficiently emphasizing the change in the three-dimensional effect of the portion where the change in depth value is small. That is, it is possible to generate depth information for generating a stereoscopic video with an improved expression of fine stereoscopic effect. That is, it is possible to generate depth information for generating a three-dimensional video with improved quality (three-dimensional effect) of the three-dimensional video.

In addition, a stereoscopic image is generated using such depth information. Therefore, according to the present embodiment, it is possible to generate a three-dimensional video in which the expression of fine three-dimensional effect is improved.

<Modified Example 1 of First Embodiment>
In the first modification of the first embodiment, a process for generating a stereoscopic image in which the sloppy three-dimensional effect is eliminated will be described. Here, the writable stereoscopic effect is, for example, a stereoscopic effect in which each of a plurality of planes representing an object image is disposed at a different depth position. In other words, the simplistic 3D effect is a 3D effect in which the 3D effect of the object image alone is hardly expressed.

The 3D image viewing system in the first modification of the first embodiment is the 3D image viewing system 1000 of FIG. That is, the three-dimensional video reproduction apparatus in the first modification of the first embodiment is the three-dimensional video reproduction apparatus 100 of FIG. Therefore, detailed description of the configuration of stereoscopic video reproduction device 100 will not be repeated. In addition, the stereoscopic video generation device in the first modification of the first embodiment is the stereoscopic video generation device 110 of FIG. 3.

Next, processing for generating a stereoscopic video (hereinafter also referred to as stereoscopic video generation processing A) in the first modification of the present embodiment will be described.

FIG. 9 is a flowchart of stereoscopic video generation processing A. In FIG. 9, the process of the same step number as the step number of FIG. 7 is performed in the same manner as the process described in the first embodiment, and therefore the detailed description will not be repeated. Hereinafter, differences from the first embodiment will be mainly described.

First, the depth information acquisition unit 111 acquires depth information indicated by the stereoscopic video signal SG1 (S110).

Then, the process of step S120A is performed.

In step S120A, depth information update processing A is performed. The depth information update process A differs from the depth information update process of FIG. 7 in that the process of step S125A is performed instead of step S125 and the process of step S126 is further performed. The other processing of depth information update processing A is the same processing as the depth information update processing, and therefore detailed description will not be repeated.

In step S125A, depth enhancement processing A is performed. In the depth enhancement processing A, as in the first embodiment, the depth information update unit 112 determines the depth in a specific band with respect to the depth image including a plurality of depth values indicated by the depth information (depth table T100). Apply a filter process that emphasizes (increases) changes in value. That is, the depth information updating unit 112 performs an emphasizing process, which is a process of emphasizing a component of a specific band, on a depth image composed of a plurality of depth values indicated by the depth information.

The filter process of step S125A is performed with one pixel among the plurality of pixels constituting the depth image indicated by the depth information (depth table T100) as the pixel of interest. Note that each time the process of step S125A is performed, the filter process is performed with the different pixel as the pixel of interest.

When the process of step S125A is performed on all the pixels constituting the depth image, the depth enhancement process A ends, and the process proceeds to step S126.

The depth table T100 as depth information is updated by the depth enhancement process A in step S125A. Hereinafter, the depth information updated by the depth enhancement processing A is referred to as first updated depth information. The first updated depth information is the updated depth table T100. Hereinafter, the updated depth table T100 is also referred to as a first updated depth table.

By the depth enhancement processing A, the amplitude of the depth line L10 in the region R10 shown in FIG. 8A is increased as shown in FIG. 8B. That is, the depth (three-dimensional effect) of the three-dimensional object 50 is emphasized by the process of step S125A.

In step S126, depth compression processing is performed. In the depth compression process, the depth information updating unit 112 sets the plurality of depth values indicated by the updated depth information such that the absolute values of the plurality of depth values indicated by the updated depth information decrease at substantially the same rate. By changing, the updated depth information is updated.

Specifically, the depth information updating unit 112 updates the first updated depth information by multiplying the absolute value of the plurality of depth values indicated by the first updated depth information by a coefficient less than one. A coefficient less than 1 is, for example, a value in the range of 0.4 to 0.7.

By this processing, it is possible to reduce the plurality of depth values indicated by the first updated depth information, centering on the zero parallax surface.

In addition, the process which updates 1st updated depth information is not limited to the said process. For example, the process of updating the first updated depth information may be a process of subtracting a predetermined value from the absolute values of the plurality of depth values indicated by the first updated depth information.

Here, it is assumed that the processing target stereoscopic video is the stereoscopic video G100 of FIG. In this case, as the depth compression process is performed, the amplitude of the depth line L10 shown in FIG. 8B is entirely reduced as shown in FIG. 10A. As a result, as shown in FIG. 10B, the distance between the stereoscopic target object 50 whose stereoscopic effect is enhanced by the processing of step S126 and the display surface 210 (parallax zero surface) is reduced. The greater the distance between the stereoscopic target object 50 and the display surface 210 (parallax zero surface), the harder it is for the viewer to identify the stereoscopic effect of the stereoscopic target object 50.

Therefore, the first updated depth information after update for reducing the distance between the stereoscopic target object 50 in which the stereoscopic effect is emphasized and the display surface 210 (parallax zero surface) is a stereoscopic object in which the writing stereoscopic effect is eliminated. Information for generating a video.

Hereinafter, the first updated depth information after updating is referred to as second updated depth information. The second updated depth information is the first updated depth table after the update.

The depth information update unit 112 transmits the second updated depth information as the updated depth information to the stereoscopic video generation unit 113.

Then, as in the first embodiment, the process of step S130 is performed.

The stereoscopic video generation processing A described above is processing when the processing target stereoscopic video is a still image, but when the processing target stereoscopic video is a moving image, the above-described processing is performed for each frame constituting the moving image. The stereoscopic video generation processing A is repeatedly performed.

As described above, according to the first modification of the present embodiment, the depth information updating unit 112 reduces the absolute values of the plurality of depth values indicated by the first updated depth information at substantially the same rate, The first updated depth information is updated by changing the plurality of depth values indicated by the first updated depth information. The first updated depth information is information generated by the process of step S125A similar to the process of step S125 of the first embodiment.

As a result, it is possible to improve the fine three-dimensional expression and to generate depth information (first updated depth information after update) for eliminating the writing three-dimensional effect. For example, while emphasizing the three-dimensional effect of the three-dimensional target object 50 of the three-dimensional video G100, it is possible to generate depth information for eliminating the draught three-dimensional effect.

Also, a stereoscopic video is generated using such updated first updated depth information. Therefore, according to the modification of the present embodiment, it is possible to improve the expression of the fine three-dimensional effect and to generate a three-dimensional video in which the sloppy three-dimensional effect is eliminated.

In addition, since the absolute values of the plurality of depth values indicated by the first updated depth information decrease at approximately the same rate, the parallax amount when the viewer views the stereoscopic video generated according to the modification of the present embodiment Can reduce eye fatigue caused by being too large.

<Modification 2 of First Embodiment>
In the second modification of the first embodiment, a process for generating a stereoscopic image in which the sloppy stereoscopic effect is eliminated will be described.

The 3D image viewing system in the second modification of the first embodiment is the 3D image viewing system 1000 of FIG. That is, the three-dimensional video reproduction apparatus in the modification 2 of the first embodiment is the three-dimensional video reproduction apparatus 100 of FIG. Therefore, detailed description of the configuration of stereoscopic video reproduction device 100 will not be repeated. In addition, the stereoscopic video generation device in the modification 2 of the first embodiment is the stereoscopic video generation device 110 of FIG. 3.

FIG. 11 is a block diagram showing an example of the configuration of the depth information update unit 112. As shown in FIG.

As shown in FIG. 11, the depth information update unit 112 includes a calculation unit 121 and a processing execution unit 123.

The calculation unit 121 calculates the difference between the depth value of the processing target pixel that is the processing target pixel and the depth value of the pixel near the processing target pixel among the depth images, although the details will be described later. The process of calculating the depth value is performed on all the pixels constituting the depth image.

Although the details will be described later, when the calculated difference depth value is less than a threshold, the process execution unit 123 performs the enhancement process on the processing target pixel corresponding to the difference depth value that is less than the threshold. , Update depth information.

Next, processing for generating a stereoscopic video (hereinafter also referred to as stereoscopic video generation processing B) according to the second modification of the present embodiment will be described.

FIG. 12 is a flowchart of stereoscopic video generation processing B. In FIG. 12, the process of the same step number as the step number of FIG. 7 is performed in the same manner as the process described in the first embodiment, and therefore the detailed description will not be repeated. Hereinafter, differences from the first embodiment will be mainly described.

First, the depth information acquisition unit 111 acquires depth information indicated by the stereoscopic video signal SG1. The depth information acquisition unit 111 transmits the acquired depth information to the depth information update unit 112. (S110).

Then, the process of step S120B is performed.

In step S120B, depth information update processing B is performed. The depth information update process B differs from the depth information update process of FIG. 7 in that the process of step S125B is performed instead of step S125 and the process of steps S121 and S122 is further performed. The other processing of depth information update processing B is the same processing as the depth information update processing, and therefore detailed description will not be repeated.

The depth information update processing B will be described in detail below. Hereinafter, a stereoscopic video to be processed is referred to as a processing stereoscopic video. Here, it is assumed that the processing target stereoscopic video is a still image. Further, the processing target three-dimensional video is described as an example of the three-dimensional video G100. In this case, the depth image processed in the depth information update process B corresponds to the stereoscopic video G100.

In step S121, the calculation unit 121 calculates the absolute value of the difference between the depth values of the two adjacent pixels forming a plurality of pixels forming the depth image indicated by the depth information (depth table T100) indicated by the stereoscopic video signal SG1 ( Hereinafter, the difference depth value is calculated. The depth image corresponds to the processing target stereoscopic video.

Specifically, first, the calculation unit 121 sets one pixel of the plurality of pixels forming the depth image as a processing target pixel.

FIG. 13 is a diagram for explaining a processing target pixel and a pixel close to the processing target pixel.

In FIG. 13, when the processing target pixel P30 is a pixel other than the upper end, the lower end, the left end, and the right end of the processing target stereoscopic video, the pixels P31, P32, P33, and P34 are pixels adjacent to the processing target pixel P30.

The pixel P31 is a pixel that is close to the processing target pixel P30 in the vertical direction. The pixel P32 is a pixel close to the processing target pixel P30 in the left-right direction. The pixel P33 is a pixel close to the processing target pixel P30 in the left-right direction. The pixel P34 is a pixel which is close to the processing target pixel P30 in the vertical direction.

Hereinafter, each of the pixels P31, P32, P33, and P34 is referred to as a processing target proximity pixel.

When the processing target pixel is a pixel in the first row and the first column (upper left corner) of the processing target stereoscopic video, the processing target proximity pixels are only the pixels P33 and P34. Further, when the processing target pixel is a pixel in the first row and the n-th column (upper right end) of the processing target stereoscopic video, the processing target proximity pixels are only the pixels P32 and P34.

When the processing target pixel is a pixel in the m-th row and first column (lower left end) of the processing target stereoscopic video, the processing target neighboring pixels are only the pixels P31 and P33. When the processing target pixel is a pixel in the m-th row and the n-th column (lower right end) of the processing target three-dimensional video, the processing target neighboring pixels are only the pixels P31 and P32.

Note that the processing target proximity pixel may be a pixel that is close to the processing target pixel P30 in the oblique direction, as with pixels P35, P36, P37, and P38 in FIG.

Further, instead of the pixel P31, a pixel separated by two or more pixels such as the pixel P41 may be set as the processing close pixel. At this time, P42 and P33 may be similarly set as the processing target proximity pixels instead of the pixel P32. Further, instead of the pixels P43 and P34, the pixel P44 may be set as the processing target proximity pixel.

Referring again to FIG. 12, in step S121, calculation unit 121 calculates the absolute value of the difference between the depth value of processing target pixel P30 and the depth value of each processing target proximity pixel (hereinafter referred to as difference depth value). calculate.

At this time, when the difference depth value between the pixel P30 and the pixel P41 in FIG. 13 is obtained, the pixel P30 and the peripheral pixels of the pixel P30 are added at a constant ratio instead of the depth value of the pixel P30. It is good. Similarly, instead of the depth value of the pixel P41, the pixel P41 and the peripheral pixels of the pixel p41 may be added at a constant ratio.

For example, instead of the depth value of pixel P30, the depth values of pixels P31, P30, and P32 are added at a ratio of 1: 2: 1, and the depth values of pixels P41A, P41, and P41B instead of the depth value of pixel P41. What added at a ratio of 1: 2: 1 may be used, and the difference between these two values may be used as the difference depth value.

In step S122, the calculation unit 121 determines whether at least one of the plurality of difference depth values calculated by the process of step S121 is equal to or greater than a predetermined threshold.

Here, the threshold is, for example, a value of 5 to 20% of the difference between the maximum value and the minimum value set for the depth value. The threshold is, for example, 20 when the depth value is represented in the range of 0 to 255.

If NO in step S122, the process proceeds to step S125B. That is, when the calculated plurality (all) of the difference depth values is less than the threshold, the process proceeds to step S125B. On the other hand, if YES in step S123, the process of step S121 is performed again on the next pixel.

In step S125B, depth enhancement processing B is performed. In the depth enhancement processing B, the processing execution unit 123 sets the processing target pixel as a target pixel, as in the first embodiment, for a depth image composed of a plurality of depth values indicated by the depth table T100 as depth information. Filter processing to emphasize (increase) the change in depth value in a specific band. That is, the process execution unit 123 performs an emphasizing process, which is a process of emphasizing a component of a specific band, on a depth image composed of a plurality of depth values indicated by the depth information (depth table T100).

Step S125B is executed only in the case of NO at step S122. That is, when the calculated difference depth value is less than the threshold, the processing execution unit 123 performs the enhancement process on the processing target pixel corresponding to the difference depth value that is less than the threshold to obtain depth information. Update.

The above-described processes of steps S121, S122, and S125B are repeated for all the pixels forming the depth image. Note that, each time the process of step S121 is performed, a different pixel is set as the processing target pixel among the plurality of pixels forming the depth image.

Accordingly, the calculation unit 121 calculates a difference depth value which is an absolute value of a difference between the depth value of the processing target pixel which is the processing target pixel and the depth value of the pixel close to the processing target pixel in the depth image. The calculation process is performed on all the pixels constituting the depth image.

Also, as a result, depth information is updated as in the first embodiment.

In addition, depth emphasis processing B is executed only in the case of NO at step S122. Thereby, in the range regarded as the same object (for example, the stereoscopic target object 50) in the depth image indicated by the updated depth information (depth table T100), the change in depth is emphasized, but two objects at different depths The change in depth is not emphasized at the boundaries.

Here, when the change in depth between two objects at different depths is much larger than the change in depth in the same object, the change in depth in the same object is hardly expressed. That is, when the change in depth value at the boundary between the same object and another object is much larger than the change in depth value in the range considered to be the same object, the change in depth in the same object is almost all It is not expressed. In this case, the viewer 40 feels as writing sensible.

However, only when the calculated difference depth value is less than the threshold value by the depth information update process B, the enhancement process is performed on the processing target pixel corresponding to the difference depth value which is less than the threshold value. This makes it possible to emphasize changes in depth within the same object without emphasizing changes in depth between two objects at different depths. In other words, it is possible to eliminate the scribed stereoscopic effect that the change in depth in the same object is hardly expressed.

Then, as in the first embodiment, the process of step S130 is performed.

In the depth information update process B, immediately after the processes of S121, S122, and S125B are repeatedly performed on all the pixels constituting the depth image, the process of step S126 is performed as in the first modification of the first embodiment. Depth compression processing may be performed.

As mentioned above, although the depth information update apparatus or the three-dimensional-video production | generation apparatus in this invention was demonstrated based on embodiment, this invention is not limited to these embodiment. Without departing from the spirit of the present invention, various modifications as may occur to those skilled in the art may be applied to the present embodiment, or a form constructed by combining components in different embodiments is also included in the scope of the present invention. .

For example, the process of updating depth information may be performed on frequency-converted values.

Also, for example, the stereoscopic video generation device 110 may receive a 2D video and generate depth information for generating a stereoscopic video from the 2D video. In this case, the depth information acquisition unit 111 of the three-dimensional video generation device 110, for example, sets the depth value so that the image of the subject in focus in the two-dimensional video is on the near side of the zero parallax surface in the three-dimensional video. calculate. Then, the depth information acquisition unit 111 generates information indicating the calculated depth value as depth information. The depth information acquisition unit 111 transmits the depth information acquired by the generation to the depth information update unit 112.

In addition, two adjacent pixels are not limited to two pixels in contact with each other. The two adjacent pixels may be, for example, pixels arranged to sandwich one or more pixels between the two adjacent pixels.

Further, all or part of the plurality of components constituting the depth information updating apparatus 101 described above may be configured by hardware. Further, all or part of the components constituting the memory management unit may be a module of a program executed by a central processing unit (CPU) or the like.

In addition, all or part of the plurality of components constituting the stereoscopic image generation device 110 or the depth information update device 101 described above may be configured from one system LSI (Large Scale Integration: large scale integrated circuit). . The system LSI is a super multifunctional LSI manufactured by integrating a plurality of components on one chip, and specifically, a microprocessor, a read only memory (ROM), a random access memory (RAM), etc. A computer system configured to include

For example, the stereoscopic video generation device 110 or the depth information update device 101 may be configured from one system LSI (integrated circuit). That is, the stereoscopic video generation device 110 or the depth information update device 101 may be an integrated circuit.

Further, the present invention may be realized as a depth information updating method in which the operation of a characteristic configuration unit included in the stereoscopic video generating device 110 or the depth information updating device 101 is a step. The present invention may also be implemented as a program that causes a computer to execute the steps included in such a depth information update method. Also, the present invention may be realized as a computer readable recording medium storing such a program. Also, the program may be distributed via a transmission medium such as the Internet.

Further, all the numerical values used in the above embodiment are an example of numerical values for specifically explaining the present invention. That is, the present invention is not limited to each numerical value used in the above embodiment.

Further, the configuration of the depth information update device 101 is an example of a configuration for specifically explaining the present invention. That is, the depth information updating apparatus 101 may not include all the components shown in FIG. That is, the depth information updating apparatus 101 according to the present invention only needs to have the minimum configuration that can realize the effects of the present invention. For example, if the process execution unit 123 also performs the process performed by the calculation unit 121, the depth information update unit 112 of the depth information update apparatus 101 may not include the calculation unit 121.

Further, the depth information updating method according to the present invention corresponds to the depth information updating process of FIG. 7, the depth information updating process A of FIG. 9, or the depth information updating process B of FIG. The depth information updating method according to the present invention does not necessarily include all the corresponding steps in FIG. 7, FIG. 9 or FIG. That is, the depth information updating method according to the present invention may include only the minimum steps capable of realizing the effects of the present invention.

Further, the order in which the steps in the depth information update method are performed is an example for specifically explaining the present invention, and may be an order other than the above. Also, some of the steps in the depth information update method and other steps may be performed in parallel independently of each other.

It should be understood that the embodiments disclosed herein are illustrative and non-restrictive in every respect. The scope of the present invention is shown not by the above description but by the scope of claims, and is intended to include all modifications within the scope and meaning equivalent to the scope of claims.

INDUSTRIAL APPLICABILITY The present invention can be used as a depth information updating device capable of generating depth information for generating a stereoscopic video with an improved representation of fine stereoscopic effect.

50 3D Object 100 3D Image Reproduction Device 101 Depth Information Update Device 110 3D Image Generation Device 111 Depth Information Acquisition Unit 112 Depth Information Update Unit 113 3D Image Generation Unit 121 Calculation Unit 123 Processing Execution Unit 200 3D Image Display Device 210 Display Surface 300 Active shutter glasses 1000 3D viewing system

Claims

A depth information updating apparatus that performs processing using a depth value corresponding to each pixel constituting a stereoscopic video, and
A depth information acquisition unit that acquires depth information indicating a plurality of depth values for generating a stereoscopic video;
A depth information updating unit configured to update the depth information by performing an emphasizing process, which is a process of emphasizing a component of a specific band, on a depth image constituted by the plurality of depth values indicated by the depth information; Depth information update device provided.
The emphasizing process of increasing the absolute value of the difference between the two depth values where the absolute value of the difference between the depth values of two adjacent pixels is less than a threshold in at least a part of the depth image The depth information update device according to claim 1.
The depth information update unit is
The process of calculating a difference depth value which is an absolute value of a difference between a depth value of a processing target pixel which is a processing target pixel among the depth images and a depth value of a pixel close to the processing target pixel; A calculation unit performed on all the pixels constituting the image;
The process execution part which performs the said emphasis process with respect to the said process target pixel corresponding to the difference depth value which is less than the said threshold value when the said difference depth value calculated is less than a threshold value is included. Depth information update device.
The depth information updating apparatus according to any one of claims 1 to 3, wherein the depth information updating unit performs the enhancement process which is a filter process.
The depth information updating unit further changes the plurality of depth values indicated by the updated depth information such that absolute values of the plurality of depth values indicated by the updated depth information decrease at substantially the same rate. The depth information updating device according to any one of claims 1 to 4, wherein the depth information after the updating is updated.
The depth information update unit according to claim 5, wherein the depth information update unit updates the updated depth information by multiplying the absolute value of the plurality of depth values indicated by the updated depth information by a coefficient less than one. Update device.
The depth information update device according to any one of claims 1 to 6,
A three-dimensional video generation device comprising: a three-dimensional video generation unit that generates a new three-dimensional video using a plurality of depth values indicated by the updated depth information.
A depth information updating method performed by a depth information updating apparatus that performs processing using a depth value corresponding to each pixel constituting a stereoscopic video, comprising:
A depth information acquisition step of acquiring depth information indicating a plurality of depth values for generating a stereoscopic video;
A depth information updating step of updating the depth information by performing an emphasizing process, which is a process of emphasizing a component of a specific band, on a depth image composed of the plurality of depth values indicated by the depth information; Including depth information update method.