KR20140118091A - System for producing image or video and method for generating disparity map - Google Patents

Abstract

Disclosed are a system for producing an image and a method for generating a displacement map. More specifically, the system for producing an image comprises: a displacement setting unit to set a minimum displacement value and a maximum displacement value in a certain frame within an image sequence based on color images acquired from multiple color cameras which are moved to photograph in certain directions; a function generating unit to generate a correlation function which represents a relationship between an actual depth value in the certain frame that is calculated based on the set minimum displacement value and maximum displacement value and a depth value in the certain frame that is acquired from one or more depth cameras which are moved to photograph in certain directions; and a displacement calculating unit to calculate the minimum displacement value and the maximum displacement value in an arbitrary frame within the image sequence based on the correlation function, wherein the image is produced using a disparity map which is depth information of each frame within the image sequence generated based on calculation results in the displacement calculating unit.

Description

TECHNICAL FIELD [0001] The present invention relates to a video production system and a displacement map generation method,

The present invention relates to an image production system and a displacement map generation method, and more particularly, to an image production system and an image generation method for producing an image using a displacement map, which is depth information of each frame in an image sequence.

Recently, 3D stereoscopic image production technology has been widely used and many video images have been produced, and related fields have been actively studied.

In particular, techniques for producing more realistic 3D stereoscopic images or multi-view / ambience stereoscopic images in various environments are under research and development.

Korean Patent Laid-Open Publication No. 2013-0005702 (entitled " 3-D Zoom Image Generation Method and Apparatus of Stereo Camera ") discloses a method and apparatus for generating a 3-dimensional zoom image using a zoom image obtained by a stereo camera, Dimensional zoom image without warping by warping and restoring the three-dimensional zoom image. Such a technique can produce a zoom image such as a zoom-in or zoom-out effect by applying an image processing process to an image acquired from a camera in a fixed state.

SUMMARY OF THE INVENTION The present invention has been made in order to solve the above problems of the prior art and it is an object of the present invention to provide a method and apparatus for generating a displacement map which is depth information of each frame in an image sequence, A production system, and a displacement map generation method.

In addition, some embodiments of the present invention provide a video production system and a displacement map generation method capable of efficiently generating a displacement map with a small amount of calculation and high accuracy with respect to an image sequence having a zoom-in or zoom-out effect There is a purpose.

It is to be understood, however, that the technical scope of the present invention is not limited to the above-described technical problems, and other technical problems may exist.

According to an aspect of the present invention, there is provided an image production system, comprising: a color image generation unit that generates a color image based on a color image obtained from a plurality of color cameras photographed in a predetermined direction, A displacement setting unit for setting a minimum displacement value and a maximum displacement value in the frame; A difference between an actual depth value in the predetermined frame calculated on the basis of the set minimum displacement value and the maximum displacement value and a depth value in the predetermined frame obtained from one or more depth cameras photographed while moving in the predetermined direction A function generating unit for generating a relational function representing a relation; And a displacement calculation unit for calculating a minimum displacement value and a maximum displacement value in any frame in the image sequence based on the relational function, wherein the displacement calculation unit calculates the minimum displacement value and the maximum displacement value of each frame in the image sequence generated based on the calculation result of the displacement calculation unit The image is created using a disparity map which is depth information.

According to another aspect of the present invention, there is provided an image production system including: a photographing device including a plurality of color cameras and at least one depth camera installed so as to be photographable while being moved in a predetermined direction; A disparity map, which is depth information of each frame in the video sequence, is generated based on the color image and the depth image obtained by photographing while moving the photographing apparatus in the predetermined direction, and based on the generated displacement map, The image authoring apparatus includes a moving image generating unit for generating a moving image based on the color image and a displacement setting unit for setting a minimum displacement value and a maximum displacement value in a predetermined frame in the image sequence based on the color image, part; Generating a function that represents a relationship between an actual depth value in the predetermined frame calculated based on the set minimum displacement value and the maximum displacement value and a depth value in the predetermined frame obtained from the depth image, part; And a displacement calculator for calculating a minimum displacement value and a maximum displacement value in any frame in the image sequence based on the relational function.

A method of generating a disparity map, which is depth information of each frame in a video sequence according to an exemplary embodiment of the present invention, is a method of generating a disparity map based on a color image obtained from a plurality of color cameras Setting a minimum displacement value and a maximum displacement value in a predetermined frame in the image sequence; A difference between an actual depth value in the predetermined frame calculated on the basis of the set minimum displacement value and the maximum displacement value and a depth value in the predetermined frame obtained from one or more depth cameras photographed while moving in the predetermined direction Generating a relation function indicating a relationship; And calculating a minimum displacement value and a maximum displacement value in any frame in the image sequence based on the relational function.

According to the above-described object of the present invention, it is possible to quickly and accurately calculate a high-quality displacement map even in an environment in which the camera moves, thereby enabling a more dynamic and multi-view stereoscopic video or multi- There are advantages.

Further, according to the above-mentioned object of the present invention, it is possible to cope more effectively with the existing method even when the displacement in a predetermined frame in the video sequence is significantly changed by some objects appearing suddenly, It is advantageous to produce rest / multi-point video.

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a diagram generally showing an image production system according to an embodiment of the present invention;
FIG. 2 is a view showing an example of a color image obtained by photographing with the color camera shown in FIG. 1;
FIG. 3 is a view showing an example of a depth image obtained by photographing with a depth camera shown in FIG. 1;
FIGS. 4A and 4B are views showing an example of a state in which the photographing apparatus shown in FIG. 1 is moved in a predetermined direction;
FIG. 5 is a block diagram showing each configuration included in an image production system according to another embodiment of the present invention; FIG.
6 is a view showing an example of a color image for explaining the operation of the displacement setting unit shown in Fig. 5,
FIG. 7 is a graph showing an example of a relation function generated by the function generating unit shown in FIG. 5,
FIG. 8 is a flowchart for explaining a displacement map generating method according to an embodiment of the present invention; FIG.
9 to 11 are diagrams showing the difference in displacement value when the technique proposed by the present invention and the conventional technique are applied.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, which will be readily apparent to those skilled in the art. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. In order to clearly illustrate the present invention, parts not related to the description are omitted, and similar parts are denoted by like reference characters throughout the specification.

Throughout the specification, when a part is referred to as being "connected" to another part, it includes not only "directly connected" but also "electrically connected" with another part in between . Also, when an element is referred to as "comprising ", it means that it can include other elements as well, without departing from the other elements unless specifically stated otherwise.

Throughout this specification, when a member is " on " another member, it includes not only when the member is in contact with the other member, but also when there is another member between the two members.

Throughout this specification, when an element is referred to as "including " an element, it is understood that the element may include other elements as well, without departing from the other elements unless specifically stated otherwise. The word " step (or step) "or" step "used to the extent that it is used throughout the specification does not mean" step for.

Hereinafter, an image production system proposed by the present invention will be described with reference to FIG.

FIG. 1 is a view showing an entire image production system according to an embodiment of the present invention.

The image production system according to an embodiment of the present invention includes a photographing apparatus 100 and an image authoring apparatus 200. The photographing apparatus 100 and the image producing apparatus 200 are connected to each other by wire / do.

The photographing apparatus 100 includes a plurality of color cameras 110 and one or more depth cameras 120 provided so as to be photographable while moving in a predetermined direction. The plurality of color cameras 110 may be installed substantially side by side, and the depth camera 120 may be installed near the color camera 110 as shown in FIG. 1, and the installation position is not particularly limited. However, when the number of the color cameras 110 is increased beyond a predetermined number, it is impossible to cover the area captured by the plurality of color cameras 110 by using the one depth camera 120, ) Must be installed. In addition, the color camera 110 and the depth camera 120 preferably use the same kind, but are not limited thereto. For example, the color camera 110 may use a Basler pylon model capable of shooting at 30 fps with a resolution of up to 1920 x 1080 (HD), and the depth camera 120 may use a resolution of 176 x 144 (QCIF) Possible Mesa imaging SR4000 models are available.

FIG. 2 is a view showing an example of a color image obtained by photographing with the color camera shown in FIG. 1. FIG.

Since the specific region is photographed by the two color cameras 110 installed at different positions, the first color image and the second color image have a slight parallax.

In general, the depth information of a specific image can be expressed as a depth map represented by 8 bits from 0 to 255 or a displacement map expressing a parallax with respect to an adjacent point of the corresponding pixel in pixel units. The depth and disparity They have an inverse relationship with each other. That is, although the displacement of an object close to the actual distance from the color camera 110 is large, the depth value is small. On the contrary, the displacement value of the object or the background which is far from the actual distance from the color camera 110 is relatively small, but the depth value is large.

As shown in FIG. 2, when there are two color images having a time difference from each other, a displacement value for a specific pixel in the first color image can be obtained by searching where the corresponding pixel is located in the second color image at an adjacent time point. The depth can be calculated. The first color image and the second color image may correspond to the left image and the right image, respectively. Such a displacement search method is called stereo matching and is used as the most general depth information generating method.

Specifically, a block of a second color image in blocks (a ₁ to a ₅ ) located on the same scan line as a block a ₀ including a specific pixel in the first color image is best A corresponding point searching process for determining whether or not the matching is performed is performed as shown in FIG. At this time, it is assumed that both binocular or multi-point color images used for stereo matching are rectified.

More specifically, there are various methods for stereo matching, and are largely divided into a local method and a global method. The local method has the advantages of simple implementation and fast processing speed, but it has disadvantages that it does not find the correct matching position according to the characteristics of the image. In addition, good matching results can not be obtained in the occlusion region obscured by the non-patterned region or the object. The global method has the advantage that more precise depth information can be obtained because the whole image is optimized considering the values of the current pixel and the surrounding pixels, but the processing and calculation process are complicated.

Hereinafter, a general procedure for performing stereo matching will be described. First, two color images for the right and left viewpoints are selected, and the maximum parallax (the displacement value of the object with the closest actual distance to the color camera 110) and the minimum parallax (the actual distance between the color camera 110 and the color camera 110) Distant object or background displacement value). This process is usually done manually.

According to one example, the maximum / minimum displacement values thus obtained can be input to the stereo matching algorithm, and the range from the minimum displacement value to the maximum displacement value can be determined as the search range of the displacement candidate group. Thereafter, the most suitable displacement value is determined for each pixel in the color image based on the determined search range. The candidate having the smallest energy among the displacement candidates may be determined as the displacement value of the corresponding pixel. If this method is applied under a fixed camera environment, since the background is fixed even when the foreground moves back and forth, the minimum / maximum displacement value can be easily set. For example, the minimum displacement value can be a fixed value, and the maximum displacement value does not substantially change significantly. However, if this method is applied under the camera environment moving in the forward and backward direction, the minimum / maximum displacement value of each frame continuously changes, so that the minimum / maximum displacement value can not be set at a reasonable time.

According to another example, 0, which is the smallest value that the 8-bit displacement map can have, is set as the minimum displacement value, and the largest value 255 can be set as the maximum displacement value. That is, 0 and 255 can be input to the stereo matching algorithm, and the range of 0 to 255 can be determined as the search range of the displacement candidate group. In this case, it takes a long time to calculate, and it is difficult and inefficient to obtain an accurate displacement value in a moving camera environment.

That is, in the case of the image sequence generated in the camera environment moving in the forward or backward direction or the other direction, since the minimum / maximum displacement value changes for each frame, it is difficult to consistently maintain the displacement in the time axis direction. Can occur. In addition, in the case of the conventional displacement map generation algorithm, it is difficult to designate the minimum / maximum displacement value for each frame, thereby causing a problem of lowering the image quality and raising the calculation complexity.

In order to solve such a problem, the present invention utilizes a depth image obtained from a depth camera or a time-of-flight (TOF) camera. FIG. 3 is a view showing an example of a depth image obtained by photographing with the depth camera shown in FIG. 1. FIG.

The depth camera 120 can acquire a depth image containing depth information of a scene using an optical signal. The depth camera 120 may be installed in the vicinity of the color camera 110 and may take a scene (fixed object, moving object, background, etc.) while moving in the same direction as the color camera 110. The depth camera 120 and the above-described color camera 110 can photograph a scene at substantially the same point in time. Also, the depth camera 120 can acquire a depth image including depth information about the nearest place in the scene and depth information about the farthest place while moving in the back and forth direction. The depth image thus obtained is used to quickly and accurately generate a disparity map, which is depth information of each frame in the video sequence, in an environment in which the photographing apparatus 100 moves while moving in a predetermined direction.

1, the photographing apparatus 100 may further include a structure 130 for fixing the plurality of color cameras 110 and the one or more depth cameras 120. [ At this time, the structure 130 is movable in a predetermined direction along the rail 140 installed on the ground, and is positioned on the rail 140 and can move according to the user's intention. For example, the predetermined direction may include a front-back direction or a left-right direction.

4A and 4B are views showing an example of a state in which the photographing apparatus shown in FIG. 1 is moved in a predetermined direction. 4A and 4B, in capturing a certain scene A by using the color camera 110 and the depth camera 120, the user moves the structure 130 in a predetermined direction B, . As an example, the structure 130 may be freely movable on a rail 140 installed in the anteroposterior direction.

The image authoring apparatus 200 generates a displacement map, which is depth information of each frame in the image sequence, based on the color image and the depth image obtained by photographing while moving the photographing apparatus 100 in a predetermined direction, And writes an image including a plurality of frames on the basis of the displacement map.

In particular, the video authoring apparatus 200 includes a displacement setting unit, a function generating unit, and a displacement calculating unit.

The displacement setting unit sets a minimum displacement value and a maximum displacement value in a predetermined frame in the image sequence based on the color image.

The function generating unit generates a relational function representing a relationship between an actual depth value in a predetermined frame calculated based on the set minimum displacement value and the maximum displacement value and a depth value in a predetermined frame obtained from the depth image.

The displacement calculation unit calculates the minimum displacement value and the maximum displacement value in any frame in the video sequence based on the relation function.

A detailed description of each of these configurations will be described later.

Using the image production system described above, a high-quality displacement map can be calculated quickly and accurately even in an environment where the camera is moving, and a binocular / multi-point video having a zoom-in or zoom-out effect can be produced.

5 is a block diagram showing each configuration included in an image production system according to another embodiment of the present invention. The image production system according to another embodiment of the present invention includes a displacement setting unit 210, a function generation unit 220, a displacement calculation unit 230, and a correction unit 240.

These configurations may be implemented in the video authoring apparatus 200 described above, and some of the configurations may be implemented outside the video authoring apparatus 200. Hereinafter, for the sake of convenience of explanation, it is assumed that these configurations are implemented in the video authoring apparatus 200. However, since this is only an example, it will be appreciated that these configurations may operate in the following manner with different combinations or some modified forms.

The displacement setting unit 210 sets a minimum displacement value and a maximum displacement value in a predetermined frame in the image sequence based on the color image obtained from the plurality of color cameras 110 while moving in a predetermined direction. At this time, the color camera 110 is fixed to the movable structure 130, and the structure 130 may be moved in the front-rear direction along the rail 140 installed on the ground.

6 is a diagram showing an example of a color image for explaining the operation of the displacement setting unit shown in FIG.

Specifically, the displacement setting unit 210 determines a point (x _{L , min} ) and a nearest point (x _{L , max} ) that are the farthest from the color camera 110 in the first image in the predetermined frame (X _{R , min} ) and the nearest point (x _{R , max} ) that are the farthest from the color camera in the second image in the predetermined frame. Here, x _{L , min} , x _{R , min} , x _{L , max} , x _{R , and max} are the values of the respective horizontal axes, the first image is the left view image, have. In addition, in the image shown in Fig. 6, the farthest distance from the color camera 110 is the background portion C, and the closest point is the foot portion of the doll D.

In addition, the displacement setting unit 210 sets the displacement (x _{R , min} ) between the horizontal axis coordinate value (x _{L , min} ) of the furthest point in the first image and the horizontal axis coordinate value The minimum displacement value d _min in a predetermined frame can be set as in Equation (1).

Similarly, the displacement setting unit 210 sets the displacement of the first image by the difference between the horizontal axis coordinate value (x _{L , max} ) of the nearest point in the first image and the horizontal axis coordinate value (x _{R , max} ) The maximum displacement value d _max in a predetermined frame can be set as in Equation 2. [

The function generation unit 220 generates a function for generating a depth value of a predetermined frame based on the minimum displacement value and the maximum displacement value set by the displacement setting unit 210 and one or more depth cameras 120 in a predetermined frame. At this time, the depth camera 120 is fixed to the structure 130 movable together with the color camera 110, and the structure 130 may be moved in the front-rear direction along the rail 140 installed on the ground.

Specifically, the function generating unit 220 may include a calculator 221 and a matching unit 222, which will be described with reference to FIG. FIG. 7 is a graph illustrating an example of a relation function generated by the function generator shown in FIG. 5. FIG.

The calculator 221 utilizes Equation (3) which is established in two color images having a time difference from each other.

That is, the calculator 221 calculates the minimum (minimum) and maximum (maximum) values of the color camera 110 based on the information about the color camera 110 and the set minimum displacement value d _min and the maximum displacement value d _max , The actual depth value Z _near and the maximum actual depth value Z _far can be calculated. Here, the information (hardware / software) about the color camera 110 is information on the focal length (f) of the color camera 110 and the distance between the two cameras on which the first image and the second image are acquired And a baseline (B).

The matching unit 222 compares the maximum depth value D _max and the minimum depth value D _min in a predetermined frame obtained from the depth camera 120 with a minimum actual depth value Z _near calculated by the calculator 221. [ And the maximum actual depth value (Z _far ), respectively. Here, the largest value of the pixel value in the depth image may be the maximum depth value (D _max ), and the minimum value of the pixel value may be the minimum depth value (D _min ). In addition, since the pixel value D of the depth camera 120 is a value expressed by linearly quantizing the actual depth value Z, the matching unit 222 can generate a relation function as shown in FIG. As shown in FIG. 7, the relationship function M representing the relationship between the actual depth value Z and the depth value D corresponds to a result value (D _max , Z _near ) and (D _min , Z _far ), and can be a linear function with a negative slope.

A displacement calculating unit 230 is at least the displacement value (d _{min, n)} and the maximum displacement value of the function generator a relationship function (M), any frame (n) within the video sequence based on the generated from the (220) (d _{max , n} ). 7, the displacement calculation unit 230 can calculate the actual depth value Z _k for an arbitrary depth value D _k obtained from the depth camera 120, over at least the displacement value in the n-th frame (d _{min, n)} and the maximum displacement value to the (d _{max, n)} can be calculated as equation 6 and 7.

The correction unit 240 adds an offset to the minimum displacement value d _{min , n} and the maximum displacement value d _{max , n} in an arbitrary frame calculated by the displacement calculation unit 230 Can be corrected. That is, before applying to the stereo matching, the correcting unit 240 corrects the offset k for preventing the matching failure and preventing the error from occurring, together with the integerization of the minimum displacement value d _{min , n} and the maximum displacement value d _{max , n} The final minimum displacement value d ' _{min , n} and the final maximum displacement value d' _{max , n} can be obtained. The final minimum displacement value d ' _{min , n} may be a result obtained by adding a negative offset to the minimum displacement value d _{min , n as} shown in Equation 8, and the final maximum displacement value d' _{max , n} ) may be a result of adding a positive offset after the rounding process for the maximum displacement value d _{max , n} .

At this time, the step of performing the descending process on the minimum displacement value d _{min , n} may mean the maximum integer value smaller than the number, and the step of performing the up-processing on the maximum displacement value d _{max ,} It can mean the minimum integer value. This is the same principle as adding the offset, taking a smaller value for the final minimum displacement value (d ' _{min , n} ), and taking the larger value for the final maximum displacement value (d' _{max , n} ) Can be improved.

The displacement map of each frame in the image sequence can be generated quickly and accurately through the final minimum displacement value d ' _{min , n} and the final maximum displacement value d' _{max , n} , ) Phenomenon can be reduced. In addition, the phenomenon of finding an erroneous displacement value is remarkably reduced, and a displacement map can be efficiently generated even in a camera environment in which a moving image is taken.

As described above, the image production system proposed in the present invention uses a disparity map, which is depth information of each frame in a video sequence generated based on the calculation result of the displacement calculation unit 230 or the correction result of the correction unit 240 So that high-quality multi-view / binocular images can be produced.

A method of generating a disparity map, which is depth information of each frame in a video sequence according to an embodiment of the present invention, will be described with reference to FIG. 8 is a flowchart illustrating a method of generating a displacement map according to an embodiment of the present invention. For this purpose, the above-described image production system may be utilized, but is not limited thereto. However, for convenience of explanation, a method of generating a displacement map using the image production system will be described.

First, a plurality of color cameras 110 and depth cameras 120 are moved and photographed in a predetermined direction (S810). These cameras may be fixed to the structure 130, and the structure 130 may move in the front-rear direction along the rails 140 installed on the ground.

Subsequently, based on the color image obtained from the plurality of color cameras 110 photographed while moving in a predetermined direction, the image production system sets the minimum displacement value and the maximum displacement value in a predetermined frame in the image sequence (S820 ).

This setting step S820 may include a first determining step and a second determining step. Specifically, the first determination step is a step of determining a point (x _{L , min} ) and a nearest point (x _{L , max} ) that are the farthest from the color camera 110 in the first image in the predetermined frame, , And the second determining step determines a point (x _{R , min} ) and a nearest point (x _{R , max} ) that are the farthest from the color camera 110 in the second image in the given frame Process. At this time, the difference between the horizontal axis coordinate value (x _{L , min} ) of the farthest point determined in the first determination step and the horizontal axis coordinate value (x _{R , min} ) The minimum displacement value d _min can be set as shown in Equation (1). The difference between the horizontal axis coordinate value (x _{L , max} ) of the closest point determined in the first determination step and the horizontal axis coordinate value (x _{R , max} ) of the closest point determined in the second determination step is is the maximum displacement value (d _max) may be set like the following expression (2).

Subsequently, the image production system compares the actual depth value in the predetermined frame calculated on the basis of the set minimum displacement value and the maximum displacement value with the actual depth value in the predetermined frame obtained from one or more depth cameras 120 A relation function indicating the relationship between depth values in the frame is generated as shown in FIG. 7 (S830).

The generating step S830 may include a step of calculating and corresponding to the step S830. Specifically, the calculating step calculates a predetermined displacement d _min based on the information (hardware / software) of the color camera 110 and the minimum displacement value d _min and the maximum displacement value d _max set in the setting step S820 And calculating the minimum actual depth value Z _near and the maximum actual depth value Z _far in the frame. In addition, the step of mapping corresponds to calculating a maximum depth value D _max and a minimum depth value D _min in a predetermined frame obtained from the depth camera 120 to a calculated minimum actual depth value Z _near and a maximum actual depth value (Z _far ), respectively. In this matching step, a linear relationship function with a negative slope can be generated based on the corresponding value.

(D _{min , n} ) and a maximum displacement value (d _{max , n} ) in an arbitrary frame in an image sequence based on the relational function generated in the creating step S 830 are expressed by Equation And 7 (S840).

In addition, the image production system adds a negative offset to the minimum displacement value (d _{min , n} ) in an arbitrary frame, rounds down the maximum displacement value (d _{max , n} ) After the processing, a positive offset may be added and corrected (S850).

9 to 11 are views showing the difference in displacement value when the technique proposed in the present invention and the conventional technique are applied. In the case of FIG. 9, the distance between the camera and the object to be imaged is the longest, and in FIG. 11, the distance between the camera and the object to be imaged is closest. That is, it can be seen that the photographing was performed while gradually advancing for a total of 200 frames.

In the prior art, the search range of the displacement candidate group is collectively determined, and it can be confirmed that the search range is 40 to 88 in all the frames. On the other hand, in the case of the technique proposed in the present invention, the search range of the displacement candidate group is determined for each frame, and it can be confirmed that the minimum displacement value and the maximum displacement value are different from each other in the search range in each frame.

It can be confirmed that the value of the background portion is in an incorrect state under the environment in which the camera is moved while shooting in the related art. On the other hand, according to the present invention, a considerable accuracy is exhibited even in an environment in which a camera is photographed while moving, and flickering on a time axis can be efficiently reduced. This allows for stereo matching based on more accurate values.

As described above, by utilizing the displacement map generating method proposed in the present invention, it is possible to quickly and accurately generate the displacement map of each frame in the video sequence, thereby reducing the flickering of the background portion . In addition, the phenomenon of finding an erroneous displacement value is remarkably reduced, and a displacement map can be efficiently generated even in a camera environment in which a moving image is taken.

Meanwhile, each component shown in FIG. 5 may be configured as a 'module'. The term 'module' refers to a hardware component such as software or a Field Programmable Gate Array (FPGA) or an Application Specific Integrated Circuit (ASIC), and the module performs certain roles. However, a module is not limited to software or hardware. A module may be configured to reside on an addressable storage medium and may be configured to execute one or more processors. The functionality provided by the components and modules may be combined into a smaller number of components and modules or further separated into additional components and modules.

While the apparatus and method of the present invention has been described in connection with specific embodiments, some or all of those elements or operations may be implemented using a computer system having a general purpose hardware architecture.

In addition, an embodiment of the present invention may also be embodied in the form of a recording medium including instructions executable by a computer, such as program modules, being executed by a computer. Computer readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. In addition, the computer-readable medium may include both computer storage media and communication media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Communication media typically includes any information delivery media, including computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave, or other transport mechanism.

It will be understood by those skilled in the art that the foregoing description of the present invention is for illustrative purposes only and that those of ordinary skill in the art can readily understand that various changes and modifications may be made without departing from the spirit or essential characteristics of the present invention. will be. It is therefore to be understood that the above-described embodiments are illustrative in all aspects and not restrictive. For example, each component described as a single entity may be distributed and implemented, and components described as being distributed may also be implemented in a combined form.

The scope of the present invention is defined by the appended claims rather than the detailed description and all changes or modifications derived from the meaning and scope of the claims and their equivalents are to be construed as being included within the scope of the present invention do.

100: displacement map generating device 110: color camera
120: depth camera 130: structure
200: Video production device

Claims (11)

In a video production system,
A displacement setting unit for setting a minimum displacement value and a maximum displacement value in a predetermined frame in a video sequence based on a color image acquired from a plurality of color cameras photographed while moving in a predetermined direction;
A difference between an actual depth value in the predetermined frame calculated on the basis of the set minimum displacement value and the maximum displacement value and a depth value in the predetermined frame obtained from one or more depth cameras photographed while moving in the predetermined direction A function generating unit for generating a relational function representing a relation; And
And a displacement calculator calculating a minimum displacement value and a maximum displacement value in any frame in the image sequence based on the relational function,
And generates an image using a disparity map which is depth information of each frame in the video sequence generated based on the calculation result of the displacement calculation unit
Image production system.
The method according to claim 1,
Wherein the color camera and the depth camera are fixed to a movable structure, the structure moving back and forth along a rail installed on the ground.
2. The apparatus of claim 1, wherein the function generator
A calculator for calculating a minimum actual depth value and a maximum actual depth value in the predetermined frame based on the information about the color camera and the set minimum displacement value and maximum displacement value; And
And a matching unit which associates the maximum depth value and the minimum depth value in the predetermined frame obtained from the depth camera with the calculated minimum actual depth value and the maximum actual depth value, respectively,
Wherein the relational function is generated so as to have a linear and negative slope based on the resultant value corresponding to by the matching unit.
The method according to claim 1,
And a correction unit for adding and correcting an offset with respect to a minimum displacement value and a maximum displacement value in the arbitrary frame calculated by the displacement calculation unit.
In a video production system,
A photographing apparatus including a plurality of color cameras and at least one depth camera provided so as to be photographable while being moved in a predetermined direction;
A disparity map, which is depth information of each frame in the video sequence, is generated based on the color image and the depth image obtained by photographing while moving the photographing apparatus in the predetermined direction, and based on the generated displacement map, And an image authoring device for authoring an image including a plurality of frames,
The video authoring device
A displacement setting unit that sets a minimum displacement value and a maximum displacement value in a predetermined frame in the image sequence based on the color image;
Generating a function that represents a relationship between an actual depth value in the predetermined frame calculated based on the set minimum displacement value and the maximum displacement value and a depth value in the predetermined frame obtained from the depth image, part; And
And a displacement calculator for calculating a minimum displacement value and a maximum displacement value in any frame in the image sequence based on the relational function
Image production system.
6. The apparatus of claim 5, wherein the photographing apparatus further comprises a structure for fixing the color camera and the depth camera,
Wherein the structure is movable back and forth along a rail provided on the ground.
A method of generating a disparity map, which is depth information of each frame in a video sequence,
Setting a minimum displacement value and a maximum displacement value in a predetermined frame in the image sequence based on a color image obtained from a plurality of color cameras photographed while moving in a predetermined direction;
A difference between an actual depth value in the predetermined frame calculated on the basis of the set minimum displacement value and the maximum displacement value and a depth value in the predetermined frame obtained from one or more depth cameras photographed while moving in the predetermined direction Generating a relation function indicating a relationship; And
And calculating a minimum displacement value and a maximum displacement value in any frame in the video sequence based on the relational function
Displacement map generation method.
8. The method of claim 7, wherein the setting comprises:
A first determining step of determining, in a first image in the predetermined frame, a point which is the farthest from an actual distance from the color camera and a closest point; And
And a second determining step of determining a point which is the farthest from the actual camera and the closest point in the second image in the predetermined frame,
Setting a minimum displacement value in the predetermined frame through a difference between the horizontal axis coordinate value of the furthest point determined in the first determination step and the horizontal axis coordinate value of the furthest point determined in the second determination step,
And a maximum displacement value in the predetermined frame is set through a difference between a horizontal axis coordinate value of the closest point determined in the first determination step and a horizontal axis coordinate value of the closest point determined in the second determination step, Way.
8. The method of claim 7, wherein the generating comprises:
Calculating a minimum actual depth value and a maximum actual depth value in the predetermined frame based on the information about the color camera and the set minimum displacement value and maximum displacement value; And
And associating the maximum depth value and the minimum depth value in the predetermined frame obtained from the depth camera with the calculated minimum actual depth value and the maximum actual depth value, respectively,
And generating a linear relationship function having a negative slope based on the corresponding value in the matching step.
8. The method of claim 7,
Further comprising the step of adding a negative offset after rounding off the minimum displacement value in the arbitrary frame and adding a positive offset after rounding up the maximum displacement value in the arbitrary frame , A displacement map generation method.
A computer-readable recording medium recording a program for performing each step of the method according to any one of claims 7 to 10 on a computer.

Images