JP4080386B2 - Depth information regeneration method, depth information regeneration device, program, and recording medium - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a depth information regeneration method, a depth information regeneration apparatus, a program, and a recording medium, and more particularly, to a method for regenerating depth information added to an image.
[0002]
[Prior art]
There are two types of methods for adding depth information to an image.
One is a method of actually performing 3D measurement and adding a true depth value, and the other is a method of adding a pseudo depth value by image processing or the like.
As an example of the former, a method of adding a result measured by a stereo method or a light cutting method to an image can be considered. These methods are widely installed in commercial products.
As the latter example, the method described in Patent Document 1 below is known. The method described in Patent Document 1 corresponds to the frame border of the image and the innermost part of the image based on the idea that the closer to the center point, the farther from the shooting position in a photographed image or the like. This is a method of extracting a contour line connecting equidistant points from the bottom boundary line and the center point of the image, and assigning depth information from the photographing position to a region surrounded by the contour line and the frame line.
Further, as a method for assisting perception of a three-dimensional effect, a method described in Patent Document 2 below is known. The method described in Patent Document 2 is a method of removing a frame perception of depth perception by installing a frame positioned at the forefront of a three-dimensional depth display range in displaying a parallax image. is there.
[0003]
As prior art documents related to the invention of the present application, there are the following.
[Patent Document 1]
Japanese Patent No. 3116019 [Patent Document 2]
Japanese Patent No. 2826710 [0004]
[Problems to be solved by the invention]
Conventionally, the added depth information is used as it is for the depth information of the object. However, since these are methods for reproducing the correct depth on the premise of pseudo three-dimensional display on a two-dimensional display, a true three-dimensional display using a three-dimensional display cannot always obtain a strong stereoscopic effect. It was.
For example, when the depth display range of a three-dimensional display is narrow, the depth is compressed. In particular, it is difficult to obtain a clue for depth determination in live-action shooting, and it takes time until a stereoscopic effect is produced.
Further, in the method described in Patent Document 2, a frame having color information different from the image is added around the image, so that the display area of the image becomes narrow, and the color information around the image is modified. There was a problem that would have been.
In addition, since the frame and the depth of the image are separated, there is a problem that in a true three-dimensional display, black spots occur like a book split and give the viewer a sense of discomfort.
[0005]
The present invention has been made to solve the above-described problems of the prior art, and the object of the present invention is to provide a true three-dimensional display without altering the image size and texture color information, so that there is no sense of incongruity. An object of the present invention is to provide a depth information regeneration method and depth information regeneration device capable of emphasizing a three-dimensional effect.
Another object of the present invention is to provide a program for causing a computer to execute the above-described depth information regeneration method.
Another object of the present invention is to provide a recording medium on which the aforementioned program is recorded.
The above and other objects and novel features of the present invention will become apparent from the description of this specification and the accompanying drawings.
[0006]
[Means for Solving the Problems]
Of the inventions disclosed in this application, the outline of typical ones will be briefly described as follows.
In order to solve the above-described problem, in the present invention, pseudo depth information that does not depend on the true depth value is displayed in the frame-like region on the outer edge of the image with respect to the depth information of the image added by the conventional technique. Synthesize.
The depth of the newly added frame area is a clue to stereoscopic perception, and the stereoscopic effect of the image is psychologically emphasized. Note that the original depth value is prioritized for an area to which sufficient depth information is added even at the outer edge of the image.
Further, the depth value of the frame region is given so as to change smoothly so as to be equal to the depth value before modification toward the center of the image, thereby preventing separation from the existing depth value. These processes prevent the observer from feeling uncomfortable.
[0007]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
Note that components having the same function are denoted by the same reference symbols throughout the drawings for describing the embodiment, and the repetitive description thereof will be omitted.
[Embodiment 1]
FIG. 1 is a flowchart showing a processing procedure of the depth information regeneration method according to the first embodiment of the present invention.
The depth information regeneration method shown in FIG. 1 corresponds to the invention described in claim 1 of the present application.
Hereinafter, the depth information regeneration method of the present embodiment will be described with reference to FIG.
Step (S1-1) is a step of reading the depth information Z generated for a certain image.
FIG. 2 is a diagram for explaining the depth information.
2 indicates a texture of an image, and 2-2 indicates depth information of the image. In FIG. 2B, depth information is expressed by dither processing, and the higher the density of stippling, the farther the distance is.
The present invention enhances the stereoscopic effect of the texture 2-1 in the true three-dimensional display by the three-dimensional display by regenerating the depth information 2-2.
Step (S1-2) is a step of determining a threshold value used for discriminating the outer edge portion where the depth value is to be modified and the region that can be modified.
The threshold value is a value representing depth, and can be arbitrarily determined as necessary. In the present embodiment, a depth value representing the innermost part farthest from the observer is set as a threshold value.
[0008]
Step (S1-3) is a step of setting a modifiable region in which the depth value can be modified using a threshold value.
The modifiable region in which the depth value can be modified uses a predetermined threshold value for the depth value from the outer side of the image whose depth value is to be modified and a region within a predetermined range from the outer side. It is set by selecting an area that satisfies the discrimination condition.
Also, once obtained by applying a predetermined predetermined range adjacent to the area obtained by applying the discrimination condition using the same method, and then applying the same or different discrimination condition again. The area may be set as a modifiable area.
In addition, the setting method of a changeable area | region is not limited above, A various setting method may be used as needed.
In the present embodiment, it is assumed that the condition for determining the outer edge part for modifying the depth value is a distance that is greater than or equal to a threshold value, that is, the innermost part. Further, it is assumed that the condition for determining the modifiable area is a distance equal to or greater than a threshold in the 15% area of the image, that is, the innermost part. FIG. 3 is a diagram illustrating an example of a changeable area set in step (S1-3).
3-1 in FIG. 3 represents binarization of the read depth information using a threshold value. The outer edge part and the modifiable area for modifying the depth value are selected from the area filled in with a discriminant.
3-2 in FIG. 3 represents an example after discrimination. A thick line 3-21 indicates an outer edge part for modifying the depth value. A hatched line 3-22 indicates a depth value changeable region.
[0009]
Step (S1-4) is a step of calculating the depth value data of the frame region so that it falls within the modifiable region at the outer side where the depth value is modified.
FIG. 4 is a diagram for explaining a calculation example in step (S1-4).
In FIG. 4A, the outer stipple region 4-11 shows a frame region that is a region where a depth value is newly calculated.
The frame area is created so as to fit within the modifiable area from the outer edge determined to modify the depth value. Therefore, in the frame area, the depth value may be regenerated regardless of the value before modification. In the present embodiment, a frame region having the same width is set both horizontally and vertically.
4-2 of FIG. 4 shows the depth information Z _f of the frame region, the depth value calculated in the framework regions, are represented by the same dither processing and 2-2 in FIG.
The depth value is generated so as to smoothly change so as to become equal to the depth value before modification toward the center of the image.
FIG. 5 is a diagram for explaining the depth value generated in step (S1-4).
In the lower graph of FIG. 5, Z represents a depth value in the foreground (distance closer to the observer) as the value is larger, and represents a depth value in the rear (distance away from the observer) as the value is smaller. A defined depth value.
The depth value before modification of the straight line 12-2 drawn on the depth information 12-1 is represented by a curve 12-3 on the graph. At this time, the frame area on the straight line 12-2 is set to 12-4.
In FIG. 5, 12-51 is a linear function, 12-52 is a quadratic function, and 12-53 is an example of a depth value by a cubic function.
The depth values of 12-51 to 12-53 in FIG. 5 smoothly change so as to become equal to the depth value 12-6 before modification toward the center of the image.
[0010]
Step (S1-5) is a step of synthesizing the depth information Z _f of the generated depth information Z and the frame region for an image.
FIG. 6 is a diagram illustrating an example of the depth information after being combined in step (S1-5).
In FIG. 6A, the depth value is expressed by dither processing.
In the present embodiment, the depth information is synthesized by obtaining the sum of the depth value in the depth information 2-2 before modification and the depth value in the depth information 4-2 of the frame region.
Step (S1-6) is a step of outputting the synthesized depth value.
According to the present embodiment, it is possible to obtain a natural stereoscopic effect without altering the image size or texture color information during true three-dimensional display.
In the present invention, the stereoscopic effect is psychologically enhanced by a frame-like region to which a new depth value is added. With regard to the effect of this stereoscopic effect enhancement, the reproduction of depth information according to the present invention is performed on a three-dimensional display. It has been confirmed that a sufficient effect can be obtained by a subjective evaluation experiment in which images before and after the formation are displayed.
[0011]
[Embodiment 2]
FIG. 7 is a diagram for explaining the process of calculating the depth value of the frame area in the depth information regeneration method according to the second embodiment of the present invention.
The depth information regeneration method shown in FIG. 7 corresponds to the invention according to claim 2 of the present application.
The method described in the first embodiment and the like are used until the determination of the outer edge part and the modification area for modifying the depth information.
Hereinafter, the calculation process of the depth value of the frame area in the present embodiment will be described with reference to FIG.
6-1 of FIG. 7 shows the outer side part which modifies a depth value, and a modifiable area. In FIG. 7A, a thick line 6-11 indicates an outer edge portion for modifying depth information, and a hatched line 6-12 indicates a depth value modifiable region.
The depth value of the frame region is calculated for each side or for each line segment obtained by dividing each side into a plurality of segments so as to be within the modifiable region from the outer side determined to modify the depth value.
In the present embodiment, the depth information regeneration rule is defined as follows.
The width of the frame region for each side is defined as the width until the region intersects the unmodifiable region when each side is moved toward the center of the image.
Reference numerals 6-13 to 6-15 in 6-1 of FIG.
A frame region 6-21 is shown at 6-2 in FIG. Since the depth information is generated so as to change smoothly so as to become equal to the depth value before modification toward the center of the image, if the width of the frame region for each side is different, the change rate of the depth value is also different.
[0012]
In this way, by generating the depth information of the frame area in accordance with the scene, the stereoscopic effect can be emphasized more naturally than when the fixed depth information of the frame area is applied.
For example, in FIG. 7, since a non-modifiable region is in contact with the lower side, it is assumed that some object having a depth value exists continuously outside the image.
In this case, if the depth value of the frame region is added to the lower side, the surrounding texture may be connected to the object, which may cause a sense of discomfort. If the rules of the present embodiment are used, the above case can be handled.
Note that the depth information regeneration rule for each side is not limited to the above-described method. For example, all the modifiable regions may be frame regions, or only the left and right sides may be frame regions.
In addition, the change in depth value is not limited to a linear to cubic function, and is a fourth or higher order function as long as it changes toward the center of the image so as to be equal to the depth value before modification. Also good.
According to the present embodiment, it is possible to obtain a natural three-dimensional emphasis effect that is more effective in accordance with the scene without modifying the image size and texture color information during true three-dimensional display.
[0013]
FIG. 8 is a block diagram showing a schematic configuration of a depth information regeneration device for implementing each of the depth information regeneration methods described above.
The depth information regeneration device shown in FIG. 8 corresponds to the inventions according to claims 4 and 5 of the present application.
In the depth information regeneration apparatus shown in FIG. 8, the depth information input unit 110 executes the above-described step (S1-1), and the depth information Z generated for a certain image is converted into the depth information (Z) storage unit 100. Read from.
The threshold setting unit 130 executes the above-described step (S1-2) and sets a threshold.
The modification area setting unit 120 executes the above-described step (S1-3), and sets a modifiable area in which the depth value can be modified using the threshold set by the threshold setting unit 130.
The depth information calculation unit 140 executes the above-described step (S1-4) or the calculation method according to the above-described second embodiment, and the frame region so as to be within the modifiable region at the outer side portion where the depth value is modified. The depth value data is calculated.
Depth information synthesizing unit 150 performs the above-described step (S1-5), and the depth information Z generated for a certain image, to synthesize the depth information Z _f of the frame region.
The depth information Z _n synthesized by the depth information synthesis unit 150 is stored in the depth information (Z _n ) storage unit 160.
[0014]
[Embodiment 3]
FIG. 9 is a flowchart showing a processing procedure of the depth information regeneration method according to the third embodiment of the present invention.
The depth information regeneration method shown in FIG. 9 corresponds to the invention according to claim 3 of the present application.
Hereinafter, the depth information regeneration method according to the present embodiment will be described with reference to FIG.
The step (S7-1) includes various values such that the frame-shaped region including the outer edge of the image has a depth value that changes from the outer edge toward the center of the image to be equal to a predetermined depth value. This is a step of registering the depth information of the frame region. Here, a plurality of the aforementioned predetermined depth values are set so as to correspond to a predetermined threshold described later.
10, in this embodiment, is a diagram illustrating an example of a frame area data F _n to be registered.
In the present embodiment, it is assumed that the depth information of the four frame regions F _{1 to} F ₄ (8-1 to 8-4) is registered. A region 8-5 represented by pointillism indicates the presence of depth information of the frame region.
Note that step (S7-1) can be omitted if the depth information of the frame region is registered in advance and it is not necessary to register it newly.
Step (S7-2) is a step of reading the depth information Z generated for a certain image.
FIG. 11 is a diagram for explaining the depth information.
In FIG. 11, 9-1 indicates the texture of a certain image, and 9-2 indicates the depth information of the certain image. In FIG. 11, 9-2, the depth information is expressed by dither processing, which indicates that the higher the stipple density is, the farther the distance is.
In step (S7-2), such depth information 9-2 is captured.
[0015]
Step (S7-3) is a step of determining a threshold value used for discriminating an outer edge part where the depth value is to be modified and a modifiable area.
The threshold value is a value representing depth, and can be arbitrarily determined as necessary. In the present embodiment, a depth value representing the innermost part farthest from the observer is set as a threshold value.
Step (S7-4) is a step of setting a modifiable region in which the depth value can be modified using a threshold value.
The modifiable region in which the depth value can be modified uses a predetermined threshold value for the depth value from the outer side of the image whose depth value is to be modified and a region within a predetermined range from the outer side. It is set by selecting an area that satisfies the discrimination condition.
In addition, it was obtained by adding an adjacent predetermined range area to the area obtained by using the above method for applying the discrimination condition, and then applying the same or different discrimination condition again. The area may be set as a modifiable area.
The determination condition using the threshold value is not limited to the above-described method, and may be set as necessary.
In the present embodiment, it is assumed that the condition for determining the outer edge part for modifying the depth value is a case where the outer edge part is a distance equal to or greater than a threshold, that is, the innermost part. Further, it is assumed that the condition for determining the changeable area is a distance equal to or greater than a threshold value, that is, the innermost part in the 15% peripheral area of the image.
[0016]
FIG. 12 is a diagram illustrating an example of the alterable area set in step (S7-4) of the present embodiment.
10-1 in FIG. 12 represents binarized read depth information using a threshold value. The outer edge part and the modifiable area for modifying the depth value are selected from the area filled in with a discriminant.
10-2 in FIG. 12 represents an example after discrimination. The thick line 10-21 of 10-2 of FIG. 12 has shown the outer side part which modifies a depth value. A hatched line 10-22 indicates a region where the depth value can be changed.
Step (S7-5) is a step of selecting from among the depth information F _n of a plurality of frame areas registered in advance, a depth information frame region adapted to depth values modifiable area of the image.
In the present embodiment, as a selection criterion for the depth information F _n of the frame region, an operation that takes the logical product of the modifiable region and the frame region is performed and the one having the largest area is selected, and F ₃ (8-3 ) Is selected.
Note that the selection criterion for the depth information F _n of the frame region is not limited to the above-described method. For example, the determination is performed based on the shape, position, or area of the depth information Z captured in step (S7-2). You may choose.
[0017]
Step (S7-6) is a step for synthesizing depth information _{F n} of depth information Z and the frame region. During the synthesis, the result of the logical product of step (S7-5), to synthesize the depth information F _n of the frame region in an area on the detected Z.
FIG. 13 is a diagram illustrating an example of the depth information after being combined in step (S7-6).
The hatched portion 11-1 in FIG. 13 shows the logical product result. The result of combining the depth information F ₃ frame regions in this region is shown in 11-2.
11-2 of FIG. 13 expresses the depth value by dither processing, similarly to 9-2 of FIG.
In the present embodiment, a region 11 obtained by ANDing the frame region and the modifiable region 10-22 of the depth value in the depth information 9-2 before modification and the depth information F ₃ (8-3) of the frame region. The depth information is synthesized by obtaining the sum of the depth value at -1.
Step (S7-7) is a step of outputting the synthesized depth value.
According to the present embodiment, since the depth value of the frame area can be stored in advance, the calculation load of the frame area is reduced.
Therefore, it is possible to obtain an effect of reducing the calculation load of a computer when emphasizing a three-dimensional feeling without changing the image size and texture color information during true three-dimensional display.
[0018]
FIG. 14 is a block diagram showing a schematic configuration of a depth information regeneration apparatus for implementing the depth information regeneration method according to Embodiment 3 of the present invention.
Note that the depth information regeneration device shown in FIG. 14 corresponds to the invention described in claim 6 of the present application.
In the depth information regeneration apparatus shown in FIG. 14, depth information (F _n ) of various frame regions is stored in a frame depth information (F _n ) storage unit 180.
The depth information input unit 110 executes the above-described step (S7-2) and reads the depth information Z generated for a certain image from the depth information (Z) storage unit 100 or the frame depth information F _n. Are read from the frame depth information (F _n ) storage unit 180.
The threshold setting unit 130 executes the above-described step (S7-3) and sets a threshold.
The modification area setting unit 120 executes the above-described step (S7-4), and sets a modifiable area in which the depth value can be modified using the threshold set by the threshold setting unit 130.
The depth information selection unit 170 executes the above-described step (S7-5), and the depth of the frame region that matches the depth value changeable region of the image from the depth information F _n of the plurality of frame regions registered in advance. Select information.
Depth information synthesizing unit 150 performs the above-described step (S7-6), to synthesize the depth information _{F n} of depth information Z and the frame region.
The depth information (Z _n ) synthesized by the depth information synthesis unit 150 is stored in the depth information (Z _n ) storage unit 160.
In the above description, the depth information regeneration method according to each embodiment of the present invention can be executed by a computer. In this case, the depth information regeneration according to each embodiment of the present invention is performed. The method is performed by causing a computer to execute a program stored in a hard disk or the like in the computer. This program is supplied by various recording media such as a CD-ROM and a DVD-ROM, or downloaded via a network.
Although the invention made by the present inventor has been specifically described based on the above-described embodiment, the present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the scope of the invention. Of course.
[0019]
【The invention's effect】
The effects obtained by the representative ones of the inventions disclosed in the present application will be briefly described as follows.
(1) According to the present invention, it is possible to obtain a natural stereoscopic effect without changing the image size and texture color information during true three-dimensional display.
(2) According to the present invention, it is possible to obtain a natural three-dimensional enhancement effect that is more effective for a scene without modifying the image size and texture color information during true three-dimensional display. Become.
(3) According to the present invention, when emphasizing a three-dimensional feeling without changing the image size and texture color information during true three-dimensional display, the effect of reducing the computational load of the computer is achieved. Can be obtained.
[Brief description of the drawings]
FIG. 1 is a flowchart illustrating a processing procedure of a depth information regeneration method according to a first embodiment of the present invention.
FIG. 2 is a diagram for explaining depth information;
FIG. 3 is a diagram showing an example of a modifiable region set in the depth information regeneration method according to the first embodiment of the present invention.
FIG. 4 is a diagram for explaining a calculation example in the depth information regeneration method according to Embodiment 1 of the present invention;
FIG. 5 is a diagram for explaining a depth value generated in the depth information regeneration method according to Embodiment 1 of the present invention;
FIG. 6 is a diagram showing an example of combined depth information in the depth information regeneration method according to the first embodiment of the present invention.
FIG. 7 is a diagram for explaining a calculation process of a depth value of a frame region in the depth information regeneration method according to the second embodiment of the present invention.
FIG. 8 is a block diagram showing a schematic configuration of a depth information regeneration apparatus for implementing the depth information regeneration method according to Embodiments 1 and 2 of the present invention.
FIG. 9 is a flowchart showing a processing procedure of a depth information regeneration method according to the third embodiment of the present invention.
[10] In depth information regenerating method according to the third embodiment of the present invention, is a diagram illustrating an example of a frame area data F _n to be registered.
FIG. 11 is a diagram for explaining depth information;
FIG. 12 is a diagram showing an example of a modifiable region set in the depth information regeneration method according to the third embodiment of the present invention.
FIG. 13 is a diagram illustrating an example of combined depth information in the depth information regeneration method according to the third embodiment of the present invention.
FIG. 14 is a block diagram showing a schematic configuration of a depth information regeneration device for implementing the depth information regeneration method according to Embodiment 3 of the present invention;
[Explanation of symbols]
DESCRIPTION OF SYMBOLS 100 ... Depth information (Z) storage part, 110 ... Depth information input part, 120 ... Modified area setting part, 130 ... Threshold setting part, 140 ... Depth information calculating part, 150 ... Depth information synthetic | combination part, 160 ... Depth information (Z _n ) Storage unit, 170 ... Depth information selection unit, 180 ... Frame depth information ( _Fn ) storage unit.

Claims (8)

A method of regenerating depth information for regenerating depth information generated for an arbitrary image,
Depth value can be modified from an outer edge part of the image and an area within a predetermined range predetermined from the outer edge part, where the depth value of the image satisfies a determination condition using a predetermined threshold value Process 1 to set as a safe area,
In a frame-like region including the outer edge in the region where the depth value can be modified, a depth value that changes so as to be equal to the depth value before modification from the outer edge toward the center of the image. Process 2 for calculating and generating depth information of the frame region;
A depth information re-generation method comprising a step 3 of combining the depth information of the image and the depth information of the generated frame region. In the step 2, the shape of the frame-like region is determined for each outer edge of the image or for each line segment obtained by dividing the outer periphery of the image into a plurality of depths, and a depth value is determined in each determined region. The depth information regenerating method according to claim 1, further comprising a calculation step. A method of regenerating depth information for regenerating depth information generated for an arbitrary image,
Depth value can be modified from an outer edge part of the image and an area within a predetermined range predetermined from the outer edge part, where the depth value of the image satisfies a determination condition using a predetermined threshold value Process 1 to set as a safe area,
A plurality of types are registered in advance, and each is equal to a predetermined depth value corresponding to the predetermined threshold value from the outer side toward the center of the image in a frame-like region including the outer side of the image. A step 2 of selecting depth information of the frame region that matches a region in which the depth value can be modified from depth information of a plurality of frame regions having varying depth values;
A method of regenerating depth information, comprising a step 3 of combining depth information of the image and depth information of the selected frame region. A depth information regeneration device for regenerating depth information generated for an arbitrary image,
Depth value can be modified from an outer edge part of the image and an area within a predetermined range predetermined from the outer edge part, where the depth value of the image satisfies a determination condition using a predetermined threshold value Means 1 for setting as a safe area;
In a frame-like region including the outer edge in the region where the depth value can be modified, a depth value that changes so as to be equal to the depth value before modification from the outer edge toward the center of the image. Means 2 for calculating and generating depth information of the frame area;
An apparatus for regenerating depth information, comprising: means 3 for synthesizing the depth information of the image and the depth information of the generated frame region. The means 2 determines the shape of the frame-shaped region for each outer edge of the image or for each line segment obtained by dividing the outer edge of the image into a plurality, and sets the depth value in each determined region. The depth information regenerating apparatus according to claim 4, further comprising a calculating unit. A depth information regeneration device for regenerating depth information generated for an arbitrary image,
Depth value can be modified from an outer edge part of the image and an area within a predetermined range predetermined from the outer edge part, where the depth value of the image satisfies a determination condition using a predetermined threshold value Means 1 for setting as a safe area;
A plurality of types are registered in advance, and each is equal to a predetermined depth value corresponding to the predetermined threshold value from the outer side toward the center of the image in a frame-like region including the outer side of the image. Means 2 for selecting depth information of the frame region that matches a region in which the depth value can be modified from depth information of a plurality of frame regions having varying depth values;
An apparatus for regenerating depth information, comprising: means 3 for combining the depth information of the image and the depth information of the selected frame region. A program for causing a computer to execute the depth information regeneration method according to any one of claims 1 to 3. A recording medium on which the program according to claim 7 is recorded.

Images