WO2013077508A1  Device and method for depth map generation and device and method using same for 3d image conversion  Google Patents
Device and method for depth map generation and device and method using same for 3d image conversion Download PDFInfo
 Publication number
 WO2013077508A1 WO2013077508A1 PCT/KR2012/003347 KR2012003347W WO2013077508A1 WO 2013077508 A1 WO2013077508 A1 WO 2013077508A1 KR 2012003347 W KR2012003347 W KR 2012003347W WO 2013077508 A1 WO2013077508 A1 WO 2013077508A1
 Authority
 WO
 WIPO (PCT)
 Prior art keywords
 depth map
 image
 value
 input image
 initial
 Prior art date
Links
 238000006243 chemical reactions Methods 0.000 title claims abstract description 33
 230000000875 corresponding Effects 0.000 claims description 26
 239000000284 extracts Substances 0.000 claims description 9
 238000000605 extraction Methods 0.000 abstract 1
 238000010586 diagrams Methods 0.000 description 9
 239000000203 mixtures Substances 0.000 description 9
 241000239290 Araneae Species 0.000 description 6
 238000004590 computer program Methods 0.000 description 6
 281000009961 Parallax, Inc. (company) companies 0.000 description 4
 238000000034 methods Methods 0.000 description 4
 238000004458 analytical methods Methods 0.000 description 2
 230000000694 effects Effects 0.000 description 2
 238000000926 separation method Methods 0.000 description 2
 230000000007 visual effect Effects 0.000 description 2
 280000156839 Program Products companies 0.000 description 1
 239000003086 colorants Substances 0.000 description 1
 230000001276 controlling effects Effects 0.000 description 1
 238000007796 conventional methods Methods 0.000 description 1
 238000010191 image analysis Methods 0.000 description 1
 238000003384 imaging method Methods 0.000 description 1
 230000002452 interceptive Effects 0.000 description 1
 238000004519 manufacturing process Methods 0.000 description 1
 230000004048 modification Effects 0.000 description 1
 238000006011 modification reactions Methods 0.000 description 1
 239000004065 semiconductors Substances 0.000 description 1
 230000001131 transforming Effects 0.000 description 1
Images
Classifications

 G—PHYSICS
 G06—COMPUTING; CALCULATING; COUNTING
 G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
 G06T7/00—Image analysis
 G06T7/50—Depth or shape recovery

 H—ELECTRICITY
 H04—ELECTRIC COMMUNICATION TECHNIQUE
 H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
 H04N13/00—Stereoscopic video systems; Multiview video systems; Details thereof
 H04N13/10—Processing, recording or transmission of stereoscopic or multiview image signals
 H04N13/106—Processing image signals
 H04N13/128—Adjusting depth or disparity

 H—ELECTRICITY
 H04—ELECTRIC COMMUNICATION TECHNIQUE
 H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
 H04N13/00—Stereoscopic video systems; Multiview video systems; Details thereof
 H04N2013/0074—Stereoscopic image analysis
 H04N2013/0081—Depth or disparity estimation from stereoscopic image signals
Abstract
Description
The present invention relates to an apparatus and method for generating a depth map, and to an apparatus and method for converting a stereoscopic image using the same, more particularly, extracting at least one characteristic information of an input image, and based on the characteristic information, After generating an initial depth map, performing a Four Fourier Transform (FFT) on the input image and converting it into a frequency image, obtaining a correlation value using the representative value of the frequency image and the average value of the initial depth map, A depth map generating apparatus and method for determining a final depth map based on the correlation value, and a stereoscopic image converting apparatus and method using the same.
Recently, as interest in 3D images is amplified, research on 3D images is being actively conducted.
In general, it is known that humans feel the most threedimensional effect by the parallax between both eyes. Thus, 3D imaging can be implemented using these characteristics of humans. For example, by distinguishing a particular subject into a left eye image seen through the viewer's left eye and a right eye image seen through the viewer's right eye, the viewer simultaneously displays the left eye image and the right eye image so that the viewer views the 3D image as a 3D image. I can make it visible. As a result, the 3D image may be implemented by manufacturing a binocular image divided into a left eye image and a right eye image and displaying the same.
In order to convert a monocular 2D image without depth information into a 3D image, it is necessary to add depth information to the 2D image to render.
In general, stereoscopic conversion is divided into manual and automatic methods. The manual method literally creates a depth map while watching the image according to the subjective judgment of a person about all the images. This process is based on the subjective judgment of the person who can predict the depth map while watching the video. Therefore, a person directly produces a depth map for each image, and the error of the depth map is very small. However, a lot of time and effort is required because a person directly intervenes in each video to create a depth map of the video.
Automatic stereoscopic transformation means analyzing the characteristics of an image to extract an appropriate depth map and using it to generate left and right stereoscopic images. In this process, since the image itself does not have information about the depth map, the depth map is generated by using general image characteristics such as edge characteristics, color, brightness characteristics, and vanishing point characteristics of the image. However, these features often do not match the stereoscopic characteristics of the image itself, so the effect of stereoscopic is not large.
In addition, a single image includes various types of image contents. It is virtually impossible to extract a depth map through image processing of each image, and the depth map obtained through image processing also includes many errors.
The error of the depth map obtained through such image processing can be divided into two types.
One is an error or inversion of the depth map in the partial region of the image and a combination thereof, and another is an error or inversion of the depth map in the entire image. Of course, the error of the depth map is not easy to distinguish by technical methods such as image processing.
Therefore, there is a need for a technique capable of automatically detecting an error of a depth map from an objective point of view only with an image.
SUMMARY OF THE INVENTION The present invention has been made to solve the abovedescribed problems, and an object of the present invention is to provide a depth map generating apparatus capable of correcting an error of a depth map for stereoscopic representation generated during an automatic stereoscopic conversion process of an image; A method and an apparatus and method for converting a stereoscopic image using the same are provided.
Another object of the present invention is to provide an apparatus and method for generating a depth map that can objectively detect and correct an error of a depth map of an image generated through image processing during automatic stereoscopic image conversion, and an apparatus and method for stereoscopic image conversion using the same. have.
Another object of the present invention is to provide a depth map generating apparatus and method for correcting errors in a depth map and minimizing errors in image conversion by converting a 2D image into a 3D image using the corrected depth map. And a stereoscopic image conversion apparatus and method using the same.
According to an aspect of the present invention, a feature information extractor for extracting at least one feature information of an input image, a depth map initializer for generating an initial depth map for the input image based on the feature information, the input image An FFT transform unit converts a frequency image by performing a fast fourier transform (FFT), and obtains a correlation value by using a representative value of the frequency image and an average value of the initial depth map, and finally obtains a correlation value based on the correlation value. There is provided a depth map generator including a depth map determiner for determining a depth map.
The feature information extracting unit extracts feature information including at least one of edge information, color information, luminance information, motion information, and histogram information.
The depth map initialization unit divides a plurality of pixels constituting the input image into at least one block, and then sets an initial depth value for the at least one block to set an initial depth map ( Create a depth map.
The depth map determiner obtains a representative value by summing pixel values corresponding to a high frequency region in the frequency image, and obtains an average value of depth values corresponding to a block region corresponding to the region obtained from the initial depth map. A correlation value is obtained using the representative value and the average value.
The depth map determiner calculates a correlation value (CRV) using the following equation.
[Equation]
CRV = Σ (FFT (n) * Depth (n))
Here, the FFT (n) is a representative value representing the block sharpness of the frequency image, the Depth (n) is the average value of the initial depth map corresponding to the block area matching the block area of the FFT (n), n Is the index of each block.
The depth map determiner may determine the initial depth map as the final depth map when the correlation value is greater than or equal to a predetermined threshold value, and invert the depth values of the initial depth map when the correlation value is not greater than the threshold value. Determine the map as the final depth map.
According to another aspect of the present invention, an image analysis unit for extracting at least one feature information by analyzing a twodimensional input image, generates an initial depth map for the input image based on the feature information, After converting to a frequency image by performing an FFT, a correlation value is obtained by using a representative value of the frequency image and an average value of the initial depth map, and a depth map setting for determining a final depth map based on the correlation value. A stereoscopic image converting apparatus including a stereoscopic image generator for converting the input image into a threedimensional stereoscopic image using the final depth map is provided.
The depth map setting unit may include a depth map initializer configured to generate an initial depth map of the input image based on the characteristic information, an FFT converter configured to perform an FFT on the input image, and convert the image into a frequency image; And a depth map determiner that obtains a correlation value by using a representative value and an average value of the initial depth map, and determines a final depth map based on the correlation value.
According to another aspect of the present invention, in the method for generating a depth map by the depth map generating apparatus, extracting at least one characteristic information for the input image, the initial depth for the input image based on the characteristic information Generating a map, performing an FFT on the input image, converting the image into a frequency image, obtaining a correlation value using a representative value of the frequency image and an average value of the initial depth map, and calculating the correlation value. A depth map generation method is provided that includes determining a final depth map as a basis.
The obtaining of the correlation value by using the representative value of the frequency image and the average value of the initial depth map may include obtaining a representative value by adding pixel values corresponding to a high frequency region in the frequency image, and representing the representative value in the initial depth map. Obtaining an average value of depth values corresponding to a block area coinciding with the obtained area, and obtaining a correlation value using the obtained representative value and the average value.
The determining of the final depth map based on the correlation value may include determining the initial depth map as the final depth map when the correlation value is greater than or equal to a predetermined threshold value, and determining the final depth map as the final depth map. And inverting the depth values to determine the inverted depth map as the final depth map.
According to another aspect of the invention, the step of extracting at least one characteristic information for the input image, generating an initial depth map for the input image based on the characteristic information, performing an FFT on the input image Converting the frequency image into a frequency image, obtaining a correlation value using an average value of the representative value of the frequency image and the initial depth map, and determining a final depth map based on the correlation value. A recording medium is provided which is produced by a program and can be read by an electronic device.
According to another aspect of the present invention, in a method for automatically converting a stereoscopic image by a stereoscopic image converting apparatus, analyzing the input twodimensional input image to extract at least one characteristic information, based on the characteristic information Generating an initial depth map with respect to the input image, determining a final depth map by checking the validity of the initial depth map, and converting the input image into a threedimensional stereoscopic image using the final depth map. There is provided a stereoscopic image conversion method comprising the step.
The determining of the final depth map may include generating an initial depth map of the input image based on the characteristic information, performing an FFT on the input image, and converting the input image into a frequency image. Obtaining a representative value by summing pixel values corresponding to an area, obtaining an average value of depth values corresponding to a block area corresponding to the area where the representative value is obtained from the initial depth map, and representing the representative value of the frequency image and the initial value Obtaining a correlation value using an average value of the depth map; when the correlation value is equal to or greater than a predetermined threshold, determining the initial depth map as a final depth map; and when the correlation value is not greater than or equal to the threshold, a depth value of the initial depth map. Inverting them to determine the inverted depth map as the final depth map.
According to another aspect of the invention, the step of extracting at least one characteristic information by analyzing the input twodimensional input image, generating an initial depth map for the input image based on the characteristic information, the initial depth Determining a final depth map by checking whether a map is valid, and converting the input image into a threedimensional stereoscopic image using the final depth map. A readable recording medium is provided.
Therefore, according to the present invention, it is possible to correct an error of a depth map for the stereoscopic representation that occurs during the automatic stereoscopic conversion of the image.
In addition, an error in a depth map of an image generated through image processing during automatic stereoscopic image conversion may be objectively detected and corrected.
In addition, the error of the depth map may be corrected, and the error of the image conversion may be minimized by converting the 2D image into the 3D image using the corrected depth map.
1 is a block diagram showing the configuration of a stereoscopic image conversion apparatus according to the present invention.
Figure 2 is a block diagram schematically showing the configuration of the depth map generating apparatus according to the present invention.
3 is an exemplary view for explaining the difference between the initial depth map and the final depth map when the input image is converted into a threedimensional stereoscopic image according to the present invention.
4 is an exemplary diagram for explaining an image of an FFT high frequency component according to the present invention;
5 is an exemplary view of a block representing the 8x8 block in the frequency domain through the FFT according to the present invention.
6 is a diagram illustrating a method for converting a twodimensional input image into a threedimensional stereoscopic image by the stereoscopic image conversion apparatus according to the present invention.
7 is a flowchart illustrating a method of generating a depth map by a depth map generating apparatus according to the present invention;
Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the description with reference to the accompanying drawings, the same or corresponding components will be given the same reference numerals and redundant description thereof will be omitted.
Hereinafter, a depth map representing a stereoscopic image is generated and corrected for an image object satisfying the following two prerequisites.
Precondition 1 is "usually a bright color or a strong brightness is often near." For example, the more distant the mountain in the picture you see, the more the color looks achromatic compared to the nearby mountain. Or, the farther and brighter the object, the less saturated the color becomes. Such a feature is not only a visual characteristic of a person but also a feature of a camera photographing it.
Precondition 2 is "usually, the distance between the clear parts of the image is often close." For example, when the same object is near and far, the sharpness of the object is different, which occurs because of the limitations of the resolution of the camera being visually or photographed.
The present invention intends to transform the stereoscopic and correct it by utilizing the two characteristics as described above. Precondition 1 is a condition for converting threedimensional, and precondition 2 is a means for correcting this. That is, the depth map is extracted from the color and brightness information of the image, and the error of the depth map is detected and corrected through objective verification of whether the depth map is partially or totally valid.
1 is a block diagram showing the configuration of a stereoscopic image conversion apparatus according to the present invention.
Referring to FIG. 1, the 3D image converting apparatus 100 includes an image analyzer 110, a depth map setting unit 120, and a 3D image generating unit 130.
The image analyzer 110 extracts at least one characteristic information by analyzing a twodimensional input image. The characteristic information includes edge information, color information, luminance information, motion information, histogram information, and the like.
The depth map setting unit 120 generates an initial depth map of the input image based on the characteristic information extracted by the image analyzer 110, and performs a fast fourier transform (FFT) on the input image. After converting to a frequency image, a correlation value is obtained using the representative value of the frequency image and the average value of the initial depth map, and a final depth map is determined based on the correlation value.
A detailed description of the depth map setting unit 120 will be described with reference to FIG. 2.
The stereoscopic image generator 130 converts the twodimensional input image into a threedimensional stereoscopic image using the final depth map determined by the depth map setting unit 120. For example, the stereoscopic image generation unit 130 may generate parallax information using the final depth map, and generate a 3D stereoscopic image using the parallax information. The 3D stereoscopic image generated in this way looks more stereoscopic as the depth values for each pixel in each frame vary.
Herein, the stereoscopic image generator 130 converts the 2D image into a 3D stereoscopic image using parallax information. However, the stereoscopic image generator 130 converts the input image using the final depth map. The method of converting to a stereoscopic image follows various conventional methods.
The stereoscopic image converting apparatus 100 configured as described above may convert a 2D input image into a 3D image by setting a depth value for the input image based on the characteristic information of the input image.
2 is a block diagram schematically illustrating a configuration of a depth map generating apparatus according to the present invention, and FIG. 3 illustrates a difference between an initial depth map and a final depth map when converting an input image into a 3D stereoscopic image. 4 is an exemplary diagram for explaining an image of an FFT high frequency component according to the present invention, and FIG. 5 is an exemplary diagram of a block in which a 8x8 block according to the present invention is expressed in a frequency domain through an FFT.
In FIG. 1, the depth map setting unit is described, but in FIG. 2, the depth map generator 200 will be described.
Referring to FIG. 2, the depth map generator 200 may include a feature information extractor 210, a depth map initializer 220, a fast fourier transform (FFT) transformer 230, and a depth map determiner 240. Include.
The characteristic information extractor 210 extracts at least one characteristic information with respect to the input image. The input image may be a monocular image.
The feature information extracted by the feature information extractor 210 may be edge information, color information, luminance information, motion information, or histogram information.
The feature information extractor 210 extracts feature information in an image through various analysis methods in units of pixels or blocks in order to collect information that is a basis for generating a depth map.
The depth map initializer 220 generates an initial depth map of the input image based on the feature information extracted by the feature information extractor 210.
That is, the depth map initialization unit 220 divides a plurality of pixels constituting the input image into at least one block, and then initializes an initial depth value of the at least one block. Set to create an initial depth map.
The depth map initializer 220 generates a depth map for each frame of the 2D image based on the extracted characteristic information. That is, the depth map initializer 220 extracts depth values for each pixel of each frame from a depth map of the 2D image. Here, the depth map is a data structure storing depth values of each pixel per frame for the 2D image.
An example of converting an input image into a 3D stereoscopic image using the initial depth map generated by the depth map initialization unit 220 will be described with reference to FIG. 3.
Based on the observer, the spider is placed closest to the observer, the next flower is located, and the background is farthest from the observer. For example, the spider, flower, and background can be separated in three stages according to the position. Do. Although it can be expressed in more detail, it will be divided into three steps as an example for explanation.
This is a subjective human visual characteristic. This analysis is possible, but if it is to be analyzed automatically through image processing, the spider that is considered to be the most front using the precondition 1 is achromatic and dark and is the farthest. Will be an expression.
FIG. 3A illustrates a stereoscopic image converted into a depth map according to Precondition 1. FIG.
In addition, if the depth map generated by the depth map initialization unit 220 is changed like a film image inverted like a print film of a photograph, a normal depth map may be formed.
That is, if the depth map is reversed like a print film of a photograph, the initial depth map of FIG. 3A can be converted into a depth map as shown in FIG. 3B, and FIG. 3B can be a threedimensional expression with less error than that of FIG.
In the image of FIG. 3B, the spider is bright because the spider is in front, and then the flower area, and the background is dark because there is little change. As a result, when the depth map of FIG. 3B is selected, the stereoscopic representation is possible while improving the error.
Therefore, the depth map generating apparatus 200 should determine whether to select and convert an image to the depth map of FIG. 3A or to convert to a depth map of FIG. 3B.
The depth map generating apparatus 200 performs a fast fourier transform (FFT) on the input image to determine whether to select and convert to the depth map of FIG. 3A or to convert to the depth map of FIG. 3B. The final depth map is generated using the frequency image.
Therefore, the depth map generating apparatus 200 includes an FFT converter 230 and a depth map determiner 240.
The FFT converter 230 performs an FFT on the input image to convert the input image into a frequency image in a frequency domain. That is, the FFT converter 230 converts a spatial input image into a frequency image.
The FFT converter 230 performs an FFT with reference to FIG. 4 for an image obtained by converting an input image into a frequency image. Referring to FIG. 4, the brighter portion has a clearer or stronger outline. In other words, if the high frequency area in the frequency domain, that is, the area expressing sharpness or strong exterior, is expressed in bright colors, the outline of the spider is bright in the image, and then the flower area and the background are dark because there is little change. do.
The image representation as shown in FIG. 4 is a characteristic of the FFT. FIG. 5 (a) represents an arbitrary region of 8x8 pixels in the input image, and FIG. 5 (b) shows (a) as the frequency region through the FFT.
In FIG. 5B, region A refers to a low frequency band and region B refers to a high frequency band. That is, the B region is a high frequency component. If the image has a clear or strong outline, the value of the B region is increased, and the simpler the value of the upper portion of the A region is, the smaller the value of the B region is.
As such, the sum of pixel values belonging to the high frequency region in the frequency image, that is, the sum of the B region representing the sharpness in the 8x8 block belonging to the partial region of the image and the large deviation between each pixel of the image is expressed as 8x8. It is defined as a representative value corresponding to pixel. In this way, when the size value of each block unit is expressed as an image, the image is expressed using the FFT high frequency component as shown in FIG. 4.
The depth map determiner 240 obtains a correlation value by using a representative value of the frequency image generated by the FFT converter 230 and an average value of the initial depth map, and based on the correlation value, a final depth map. Determine.
That is, the depth map determiner 240 obtains a representative value FFT (n) representing a block sharpness in arbitrary 8 × 8 block units in the frequency image formed by the FFT converter 230. Here, the representative value is obtained by summing pixel values corresponding to a high frequency region in the frequency image. Then, the depth map determiner 240 obtains an average value Depth (n) of the 8x8 block of the initial depth map that matches the block area of the FFT (n), and then correlates the representative value with the average value. Obtain the value (CoRelation Value).
The depth map determiner 240 obtains a correlation value (CRV (CoRelation Value)) using Equation 1.
[Equation 1]
CRV = Σ (FFT (n) * Depth (n))
Here, the FFT (n) is a representative value representing the block sharpness of the frequency image, Depth (n) is the average value of the initial depth map corresponding to the block area that matches the block area of the FFT (n), n is It means the index of each block. The FFT (n) refers to a sum of pixel values corresponding to a high frequency region in an FFTconverted frequency image, and an average value of the initial depth map corresponds to a block region corresponding to a region obtained by obtaining the representative value in the initial depth map. The mean value of the depth values.
When the correlation value is obtained through Equation 1, the depth map determiner 240 determines the initial depth map as the final depth map when the correlation value is greater than or equal to a predetermined threshold value, and when the correlation value is not greater than or equal to the threshold value. The depth values of the initial depth map are inverted to determine the inverted depth map as the final depth map.
As a result, the depth map determiner 240 is a depth map for automatically converting an image to a stereoscopic image by using a frequency image converted into an image of the frequency domain as shown in FIG. 4. It is concluded that
In addition, the depth map determiner 240 is a precondition if the condition that the color of the precondition 1 is stronger and brighter than the condition determined to be in front and the condition that the sharp part is in front match, that is, the correlation value is greater than or equal to a predetermined threshold. It is determined that the depth map of FIG. 3A generated as 1 is valid.
If the correlation value is not greater than or equal to a predetermined threshold value, the depth map determiner 240 determines that the depth map reflecting the inversion phenomenon is a more suitable depth map as shown in FIG. 3B.
The depth map generation apparatus 200 configured as described above extracts an initial depth map using characteristic information such as color and brightness information of an image, and performs an objective verification on whether the initial depth map is partially or entirely valid. Detects the error and generates the final depth map.
6 is a diagram illustrating a method of converting a twodimensional input image into a threedimensional stereoscopic image by the stereoscopic image conversion apparatus according to the present invention.
Referring to FIG. 6, the stereoscopic image conversion apparatus analyzes a twodimensional input image to extract at least one characteristic information (S602). Here, the characteristic information includes edge information, color information, luminance information, motion information, histogram information, and the like.
After performing the step S602, the 3D image conversion apparatus generates an initial depth map of the input image based on the characteristic information (S604).
Then, the stereoscopic image conversion apparatus determines the final depth map through objective verification of whether the initial depth map is partially or entirely valid (S606).
That is, the stereoscopic image converting apparatus performs an FFT on the input image and converts the image into a frequency image. Then, the stereoscopic image conversion apparatus obtains a representative value by summing pixel values belonging to a high frequency region in the frequency image, and obtains an average value of depth values corresponding to a block region that matches the region where the representative value is obtained in the initial depth map. Then, the 3D image conversion apparatus obtains a correlation value using Equation 1 using the obtained representative value and the average value.
Thereafter, the stereoscopic image conversion apparatus determines a final depth map based on the correlation value.
After performing S606, the stereoscopic image conversion apparatus converts the input image into a 3D stereoscopic image using the determined final depth map (S608).
7 is a flowchart illustrating a method of generating a depth map by a depth map generating apparatus according to the present invention.
Referring to FIG. 7, the depth map generating apparatus extracts at least one characteristic information of an input image, and generates an initial depth map of the input image based on the extracted characteristic information (S702). That is, the depth map generator extracts characteristic information such as edge information, color information, luminance information, motion information, histogram information, and the like. Then, the depth map generating apparatus divides a plurality of pixels constituting the input image into at least one block, and then sets an initial depth value for the at least one block to initialize the depth map. Create a depth map.
After performing the operation S702, the depth map generating apparatus performs an FFT on the input image and converts the frequency image into a frequency image (S704).
After performing S704, the depth map generator determines whether the converted frequency image is an image that satisfies a precondition (S706). In this case, the precondition is a precondition 2, and the precondition 2 is " usually, a clear portion in the image is often close in distance. &Quot;
If the determination result of the step S706 is an image satisfying the precondition, the depth map generator obtains a correlation value using the representative value of the frequency image and the average value of the initial depth map (S708). A detailed description of the method for obtaining the correlation value will be given with reference to FIG. 2.
After performing S708, the depth map generator determines whether the obtained correlation value is greater than or equal to a predetermined threshold value (S710).
When the determination result of S710 is greater than or equal to a threshold, the depth map generator determines the initial depth map as a final depth map (S712).
If the correlation value is not greater than or equal to a threshold as a result of the determination in S710, the depth map generator inverts depth values of the initial depth map (S714), and converts a depth map including the inverted depth values into a final depth map. Determine (S716).
In the present invention, a computer program (also known as a program, software, software application, script, or code) included in a depth map generating device or a stereoscopic image conversion device and executing a method according to the present invention is a compiled or interpreted language or a priori. Or written in any form of programming language, including procedural languages, and may be deployed in any form, including standalone programs or modules, components, subroutines, or other units suitable for use in a computer environment. A computer program does not necessarily correspond to a file in a file system. A program may be in a single file provided to the requested program, in multiple interactive files (eg, a file that stores one or more modules, subprograms, or parts of code), or part of a file that holds other programs or data. (Eg, one or more scripts stored in a markup language document). The computer program may be deployed to run on a single computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communication network.
Computerreadable media suitable for storing computer program instructions and data include, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, such as magnetic disks such as internal hard disks or external disks, magnetooptical disks, and CDROMs. And all forms of nonvolatile memory, media and memory devices, including DVDROM discs. The processor and memory can be supplemented by or integrated with special purpose logic circuitry.
Implementations of the functional operations and subject matter described herein may be implemented in digital electronic circuitry, in computer software, firmware, or hardware including the structures and structural equivalents disclosed herein, or in a combination of one or more of these. Can be. Implementations of the subject matter described in this specification are one or more modules relating to computer program instructions encoded on a program storage medium of tangible type for controlling or by the operation of one or more computer program products, ie data processing apparatuses. It can be implemented as.
Implementations of the subject matter described herein may include, for example, a backend component such as a data server, or include a middleware component such as, for example, an application server, or a web browser or graphical user, for example, where a user may interact with the implementation of the subject matter described herein. It can be implemented in a computing system that includes a front end component such as a client computer having an interface or any combination of one or more of such back end, middleware or front end components. The components of the system may be interconnected by any form or medium of digital data communication such as, for example, a communication network.
Although the specification includes numerous specific implementation details, these should not be construed as limiting to any invention or the scope of the claims, but rather as a description of features that may be specific to a particular embodiment of a particular invention. It must be understood. Certain features that are described in this specification in the context of separate embodiments may be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable subcombination. Furthermore, while the features may operate in a particular combination and may be initially depicted as so claimed, one or more features from the claimed combination may in some cases be excluded from the combination, the claimed combination being a subcombination Or a combination of subcombinations.
Likewise, although the operations are depicted in the drawings in a specific order, it should not be understood that such operations must be performed in the specific order or sequential order shown in order to obtain desirable results or that all illustrated operations must be performed. In certain cases, multitasking and parallel processing may be advantageous. Moreover, the separation of the various system components of the abovedescribed embodiments should not be understood as requiring such separation in all embodiments, and the described program components and systems will generally be integrated together into a single software product or packaged into multiple software products. It should be understood that it can.
As such, those skilled in the art will appreciate that the present invention can be implemented in other specific forms without changing the technical spirit or essential features thereof. Therefore, the abovedescribed embodiments are to be understood as illustrative in all respects and not as restrictive. The scope of the present invention is shown by the following claims rather than the above description, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present invention. do.
The present invention can objectively detect and correct errors in the overall depth map of an image through image processing during automatic stereoscopic image conversion, and convert the 2D image into a 3D image by using the corrected depth map. The present invention can be applied to a depth map generating apparatus and method and a stereoscopic image converting apparatus and method using the same.
Claims (15)
 A feature information extractor configured to extract at least one feature information of the input image;A depth map initialization unit generating an initial depth map of the input image based on the characteristic information;An FFT transform unit performing a fast fourier transform (FFT) on the input image to convert the image into a frequency image; AndA depth map determination unit obtaining a correlation value by using a representative value of the frequency image and an average value of the initial depth map, and determining a final depth map based on the correlation value;Depth map generation device comprising a.
 The method of claim 1,The feature information extractor extracts feature information including at least one of edge information, color information, luminance information, motion information, and histogram information. Map generator.
 The method of claim 1,The depth map initializer divides a plurality of pixels constituting the input image into at least one block, and then sets an initial depth value for the at least one block. Depth map generation device characterized in that for generating a depth map).
 The method of claim 1,The depth map determiner obtains a representative value by summing pixel values corresponding to a high frequency region in the frequency image, and obtains an average value of depth values corresponding to a block region corresponding to the region obtained from the initial depth map. And a correlation value is calculated using the representative value and the average value.
 The method of claim 4The depth map determination unit obtains a correlation value (CRV (CoRelation Value)) using the following equation.[Equation]CRV = Σ (FFT (n) * Depth (n))Here, the FFT (n) is a representative value representing the block sharpness of the frequency image, the Depth (n) is the average value of the initial depth map corresponding to the block area that matches the block area of the FFT (n), n is each The index of a block.
 The method of claim 1,The depth map determiner determines the initial depth map as a final depth map when the correlation value is greater than or equal to a predetermined threshold value, and inverts the depth values of the initial depth map when the correlation value is not greater than the threshold value to determine the inverted depth map. Depth map generation device characterized in that for determining the final depth map.
 An image analyzer extracting at least one characteristic information by analyzing a twodimensional input image;After generating an initial depth map of the input image based on the characteristic information, performing an FFT on the input image, converting the image into a frequency image, and using a representative value of the frequency image and an average value of the initial depth map. A depth map setting unit obtaining a correlation value and determining a final depth map based on the correlation value; AndA stereoscopic image generator for converting the input image into a threedimensional stereoscopic image using the final depth map;Stereoscopic image conversion device comprising a.
 The method of claim 7, whereinThe depth map setting unit,A depth map initialization unit generating an initial depth map of the input image based on the characteristic information;An FFT converter which performs an FFT on the input image and converts the frequency image into a frequency image; AndAnd a depth map determiner which obtains a correlation value by using the representative value of the frequency image and the average value of the initial depth map, and determines a final depth map based on the correlation value.
 In the depth map generating apparatus generates a depth map,Extracting at least one characteristic information with respect to the input image;Generating an initial depth map of the input image based on the characteristic information;Performing an FFT on the input image and converting the same to a frequency image;Obtaining a correlation value using an average value of the representative value of the frequency image and the initial depth map; AndDetermining a final depth map based on the correlation value;Depth map generation method comprising a.
 The method of claim 9,Obtaining a correlation value by using the representative value of the frequency image and the average value of the initial depth map,Obtaining a representative value by summing pixel values corresponding to a high frequency region in the frequency image;Obtaining an average value of depth values corresponding to a block area corresponding to the area from which the representative value is obtained in the initial depth map; AndAnd obtaining a correlation value using the obtained representative value and the average value.
 The method of claim 9,Determining a final depth map based on the correlation value,If the correlation value is greater than or equal to a predetermined threshold value, the initial depth map is determined as the final depth map. If the correlation value is not greater than the threshold value, the depth values of the initial depth map are inverted to determine the inverted depth map as the final depth map. Depth map generation method characterized in that.
 Extracting at least one characteristic information with respect to the input image;Generating an initial depth map of the input image based on the characteristic information;Performing an FFT on the input image and converting the same to a frequency image;Obtaining a correlation value using an average value of the representative value of the frequency image and the initial depth map;And determining a final depth map based on the correlation value. The recording medium may be recorded by a program and read by an electronic device.
 In the stereoscopic image conversion apparatus automatically converts a stereoscopic image,Extracting at least one characteristic information by analyzing the input twodimensional input image;Generating an initial depth map of the input image based on the characteristic information, and determining a final depth map by checking whether the initial depth map is valid; AndConverting the input image into a threedimensional stereoscopic image using the final depth map;Stereoscopic image conversion method comprising a.
 The method of claim 13,Determining the final depth map,Generating an initial depth map of the input image based on the characteristic information;Performing an FFT on the input image and converting the same to a frequency image;Obtaining a representative value by summing pixel values corresponding to a high frequency region in the frequency image;Obtaining an average value of depth values corresponding to a block area corresponding to the area from which the representative value is obtained in the initial depth map;Obtaining a correlation value using an average value of the representative value of the frequency image and the initial depth map; AndIf the correlation value is greater than or equal to a predetermined threshold value, the initial depth map is determined as the final depth map. If the correlation value is not greater than the threshold value, the depth values of the initial depth map are inverted to determine the inverted depth map as the final depth map. Stereoscopic image conversion method comprising the step of performing.
 Extracting at least one characteristic information by analyzing the input twodimensional input image;Generating an initial depth map of the input image based on the characteristic information, and determining a final depth map by checking whether the initial depth map is valid; AndAnd converting the input image into a threedimensional stereoscopic image using the final depth map, wherein the stereoscopic image conversion method is recorded by a program and can be read by an electronic device.
Priority Applications (2)
Application Number  Priority Date  Filing Date  Title 

KR1020110123374  20111124  
KR1020110123374A KR101660808B1 (en)  20111124  20111124  Apparatus and Method for generating Depth Map, stereoscopic image conversion apparatus and method usig that 
Publications (1)
Publication Number  Publication Date 

WO2013077508A1 true WO2013077508A1 (en)  20130530 
Family
ID=48469924
Family Applications (1)
Application Number  Title  Priority Date  Filing Date 

PCT/KR2012/003347 WO2013077508A1 (en)  20111124  20120430  Device and method for depth map generation and device and method using same for 3d image conversion 
Country Status (2)
Country  Link 

KR (1)  KR101660808B1 (en) 
WO (1)  WO2013077508A1 (en) 
Cited By (1)
Publication number  Priority date  Publication date  Assignee  Title 

KR20160095515A (en) *  20150203  20160811  연세대학교 산학협력단  3 dimensional optical measurement of defect distribution 
Families Citing this family (1)
Publication number  Priority date  Publication date  Assignee  Title 

CN108076338A (en)  20161114  20180525  北京三星通信技术研究有限公司  Image vision processing method, device and equipment 
Citations (3)
Publication number  Priority date  Publication date  Assignee  Title 

KR20100019804A (en) *  20080811  20100219  한국전자통신연구원  Stereo vision system and control method thereof 
KR20100062360A (en) *  20081202  20100610  삼성전자주식회사  Device and method for depth estimation 
KR20100064196A (en) *  20081204  20100614  삼성전자주식회사  Method and appratus for estimating depth, and method and apparatus for converting 2d video to 3d video 
Family Cites Families (2)
Publication number  Priority date  Publication date  Assignee  Title 

KR100873638B1 (en) *  20070116  20081212  삼성전자주식회사  Image processing method and apparatus 
KR20100008677A (en)  20080716  20100126  광주과학기술원  Device and method for estimating death map, method for making intermediate view and encoding multiview using the same 

2011
 20111124 KR KR1020110123374A patent/KR101660808B1/en active IP Right Grant

2012
 20120430 WO PCT/KR2012/003347 patent/WO2013077508A1/en active Application Filing
Patent Citations (3)
Publication number  Priority date  Publication date  Assignee  Title 

KR20100019804A (en) *  20080811  20100219  한국전자통신연구원  Stereo vision system and control method thereof 
KR20100062360A (en) *  20081202  20100610  삼성전자주식회사  Device and method for depth estimation 
KR20100064196A (en) *  20081204  20100614  삼성전자주식회사  Method and appratus for estimating depth, and method and apparatus for converting 2d video to 3d video 
Cited By (2)
Publication number  Priority date  Publication date  Assignee  Title 

KR20160095515A (en) *  20150203  20160811  연세대학교 산학협력단  3 dimensional optical measurement of defect distribution 
KR101700109B1 (en) *  20150203  20170213  연세대학교 산학협력단  3 dimensional optical measurement of defect distribution 
Also Published As
Publication number  Publication date 

KR20130057586A (en)  20130603 
KR101660808B1 (en)  20160929 
Similar Documents
Publication  Publication Date  Title 

US10540806B2 (en)  Systems and methods for depthassisted perspective distortion correction  
US10200678B2 (en)  System and method for combining text with threedimensional content  
US20200160102A1 (en)  Keypoint unwarping for machine vision applications  
CN104980646B (en)  Stop detection method and electronic device  
AU2018247216B2 (en)  Systems and methods for liveness analysis  
US9384588B2 (en)  Video playing method and system based on augmented reality technology and mobile terminal  
JP5952001B2 (en)  Camera motion estimation method and apparatus using depth information, augmented reality system  
CN102985933B (en)  Distinguish real and flat surfaces  
US8406619B2 (en)  Stereo camera with automatic control of interocular distance  
US8456518B2 (en)  Stereoscopic camera with automatic obstruction removal  
TWI524734B (en)  Method and device for generating a depth map  
Zhu et al.  Depth image based view synthesis: New insights and perspectives on hole generation and filling  
KR101310213B1 (en)  Method and apparatus for improving quality of depth image  
CA2668941C (en)  System and method for model fitting and registration of objects for 2dto3d conversion  
US9189857B2 (en)  Apparatus and method for reconstructing three dimensional faces based on multiple cameras  
US8385595B2 (en)  Motion detection method, apparatus and system  
WO2016009811A1 (en)  Method for calibrating one or more cameras  
CN107408205B (en)  Discriminating between foreground and background using infrared imaging  
CN101742123B (en)  Image processing apparatus and method  
CN101674491B (en)  Apparatus, method, and computer program for analyzing image data  
WO2010101362A2 (en)  Metadata generating method and apparatus and image processing method and apparatus using metadata  
CN101673395B (en)  Image mosaic method and image mosaic device  
US8441521B2 (en)  Method and apparatus for determining view of stereoscopic image for stereo synchronization  
EP2153669B1 (en)  Method, apparatus and system for processing depthrelated information  
EP3095090B1 (en)  System and method for processing input images before generating a high dynamic range image 
Legal Events
Date  Code  Title  Description 

121  Ep: the epo has been informed by wipo that ep was designated in this application 
Ref document number: 12852318 Country of ref document: EP Kind code of ref document: A1 

NENP  Nonentry into the national phase in: 
Ref country code: DE 

122  Ep: pct application nonentry in european phase 
Ref document number: 12852318 Country of ref document: EP Kind code of ref document: A1 