KR101181962B1 - Apparatus for transforming 3d image and the method therefor - Google Patents

Abstract

The present invention relates to a 3D stereoscopic image converting apparatus and a conversion method for producing a stereoscopic image having parallax for each scene and object unit using an original image.

In the present invention, the 3D stereoscopic image converting apparatus comprises: a 2D image converting apparatus to a 3D image having a stereoscopic feeling, comprising: an image collecting apparatus for collecting images; A server equipped with a 3D stereoscopic image authoring tool for producing a stereoscopic image having parallax for each scene and object unit from the image collected by the image collecting device; And an image display device for outputting an image processed by the server.

With this configuration, the present invention can work with 2D video scene by unit with semi-automatic 3D conversion software, and the most important depth map generation of 3D video creation can be easily given depth through user input in 3D space. By providing a UI that can set the volume and surface / positive disparity of the surface, it is possible to produce realistic depth maps.

Project management also saves intermediate artifacts in the workflow, making it easy to make manual corrections when needed.

2D, 3D transformation, object segmentation, object tracking, depth map, object volume, authoring tool

Description

3D stereoscopic image conversion device and conversion method {APPARATUS FOR TRANSFORMING 3D IMAGE AND THE METHOD THEREFOR}

The present invention relates to an authoring tool for converting a 2D image into a 3D image, and more particularly, to a 3D stereoscopic image converting apparatus and a conversion method for producing a stereoscopic image having parallax for each scene and object unit using an original image.

In general, 3D stereoscopic imaging technology will lead the next generation of digital visual culture as a new concept of realistic visual media that enhances the quality of visual information by several dimensions, unlike real 2D plane images. Is viewed. In addition, as the next generation of real-world information and communication service, advanced technology with fierce social advancement and competition for demand and technology development is expected to develop 2D TV and all existing industrial products such as information and communication, broadcasting, medical, film, game, animation, etc. It is expected to establish itself as a core technology applied.

On the other hand, 3D TV is to realize a realistic image by adding a depth image to the existing 2D TV, the absolute lack of stereoscopic content is expected in the era of 3D TV realistic broadcasting at home since 2012.

However, it is difficult to produce a lot of content in a short time because the production of 3D content for 3D TV realistic broadcasting requires a lot of cost, time, and related shooting and editing equipment. Not only is it difficult to provide high-quality video that is commercially available by converting to 3D, but it also takes a lot of time and money to convert such as manual labor.

Accordingly, there is a need for a 2D image conversion authoring tool capable of freely editing and storing images and efficiently securing high quality stereoscopic contents.

In order to solve the above problems, the present invention develops an authoring tool that can take an existing 2D image (still and moving image) as an input and convert the extracted object to give a depth for this purpose. The present invention provides a 3D stereoscopic image converting apparatus and a converting method using techniques such as tracking, extracting depth of an object, generating depth maps, and tracking generated depth maps.

A 3D stereoscopic image converting apparatus according to the present invention comprises: a 2D image converting apparatus to a 3D image having a stereoscopic feeling, comprising: an image collecting apparatus for collecting images; A server equipped with a 3D stereoscopic image authoring tool for producing a stereoscopic image having parallax for each scene and object unit from the image collected by the image collecting device; And an image display device for outputting an image processed by the server.

The 3D stereoscopic image authoring tool may work on the 2D image in a scene unit.

The 3D stereoscopic image authoring tool may generate the depth map of the stereoscopic image on an object basis.

The 3D stereoscopic authoring tool is characterized in that it is possible to set the volume processing and positive parallax of the object surface.

A 3D stereoscopic image conversion method according to the present invention includes a method for converting a 2D image into a 3D image having a stereoscopic feeling, comprising: a first step of collecting a 2D or 3D image in an image collecting device; A second step of converting the image collected by the image acquisition apparatus into a stereoscopic image having parallax in a server equipped with a 3D stereoscopic image authoring tool; And a third step of outputting the processed image to the image display apparatus after processing in the server.

The second step may include opening a new work file or project; Selecting a key frame from an image collected by the image collecting device; And extracting the object and the depth information automatically using a conventional automatic conversion method according to the selected key frame and converting the same into a stereoscopic image.

The second step includes: A step of opening a new work file or project; Selecting a key frame from an image collected by the image collecting device; Dividing and merging regions of each object in the selected key frame; Tracking the key frame; E step of collecting depth information of the key frame; And an F step of generating a parallax image using the depth information.

The step C may include adjusting a region segmentation result using a region segmentation menu of a MeanShift Tool after selecting the key frame; Editing an invalid object region in each object region of the region division result; Merging a wrong object region in the region division result edited using a drawing tool; And detecting the contour point of the object region and detecting the automatic contour using the Lasso tool.

The step D may include adding an object of the key frame to an object list; Automatically setting an initial object contour to the contour point of the object area; Tracking an object contour based on the key frame; And editing an object contour point generated after the object contour tracking.

The step E may include generating a label image for the image; Collecting depth information using the generated label image; Storing the collected depth information in a depth table; And tracking the depth information input to the depth table to generate a depth map of the entire image.

The step F may include selecting a stereoscopic image output method using the depth map.

3D stereoscopic image converting apparatus and conversion method according to the present invention is a semi-automatic stereoscopic conversion software can work 2D image by scene unit, and the most important depth map generation of 3D image generation in object unit easily through user input in 3D space By providing a UI that can set the volume and positive / negative disparity of the object surface, it is possible to create a realistic depth map.

Project management also saves intermediate artifacts in the workflow, making it easy to make manual corrections when needed.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings and embodiments.

1 is a view showing a 3D stereoscopic image converting apparatus according to an embodiment of the present invention.

Referring to FIG. 1, a 3D stereoscopic image converting apparatus according to an embodiment of the present invention includes a 2D image converting apparatus into a 3D image having a stereoscopic sense, and includes an image collecting apparatus 100 and an image collecting apparatus 100. Server 102 equipped with a 3D stereoscopic image authoring tool 106 for producing a stereoscopic image with parallax, and an image display device 104 for outputting an image processed by the server 102. It is configured by.

The image collecting apparatus 100 is provided to collect an image to be collected by using a device such as at least one camera. In this case, the image collecting apparatus 100 stores the collected image as an image file in an image file format such as avi, wmv, asf, mpeg, and transmits the collected image to the server 102.

The server 102 is provided to receive and process image files collected by the image collecting apparatus 100. Here, the server 102 converts the received image file into a 3D stereoscopic image using the 3D stereoscopic image authoring tool 106. In this case, the 3D stereoscopic image authoring tool 106 mounted on the server 102 may work on the image collected by the image capturing apparatus 100 in a scene unit, and when generating a depth map of the stereoscopic image of the image, Can be generated as In addition, the 3D stereoscopic authoring tool may set volume processing and positive parallax on an object surface.

As shown in FIG. 2, the main frame of the 3D stereoscopic authoring tool 106 includes an object segmentation related view 110, a depth setting related view 144, a left image generating view 152, and an image sequence related view ( 200).

First, the Object Extraction View 110 includes a Region Segmentation 120, a Merge 122, a Draw 124, an Eraser 126, a Brush Size Set button 128, and a Region checkbox 130. It includes a drawing tool and a lasso tool.

As shown in FIG. 4, the region segment 120 may include the spatial radius of the window (Hs), the range radius of the window (Hr), and the munumum density a region of the selection parameter setting dialog 132. may have in the resulting segmented image).

Merge 122 selects an object by merging the wrong area among the divided areas with the mouse drag in the region division result.

Draw 124 edits an unnatural area upon merging with Merge 122.

Eraser 126 merges with Merge 122 and deletes portions that are not natural even though edited with Draw 124.

The brush size setting button 128 may selectively change the size of the brush when using the Draw 124 and the Eraser 126.

The Region checkbox 130 is used to identify the selected region.

The drawing tool that finds the outline of an object is divided into a clustering region (134) and a draw region (136).

The clustering region 134 merges the contour region of the object by dragging the mouse while dividing the image in advance in the MeanShift Tool.

The draw region 136 draws the region of interest as much as the brush region by dragging the mouse in the state of dividing the image in advance in the MeanShift Tool.

Lasso of the Lasso Tool selects the contour point while moving the object outline with the mouse, but can automatically detect the outline.

The object list 138 registers the object to be contoured to the object list using the insert button 140 or deletes the object using the delete button 142.

The depth setting related view 144 for controlling the depth includes a Set Depth Information button 146, a Depth Edithing button 148, and a Get DepthMap button 150.

The Set Depth Information button 146 opens a dialog screen for setting the depth of the selected frame when clicked. At this time, the depth setting dialog screen, as shown in Figure 5, the Brush Tool check box 232 to give a volume to the object by setting the radius (Radious), pressure (Pressure), density (DEnsity) using a brush and A negative checkbox (234) to reverse the volume, a Gradiention Tool (236) to tilt the selected object or region by entering Min and Max values, a Layer Depth (238) to adjust the depth of the object, , The Zero Parallax scroll bar 240 for adjusting Zero Parallax values, the Selection button 242 for selecting objects of interest, the Path Selection Tool 244 for selecting an arbitrary area within an object, and the depth of all objects. Initialize button 246 for setting to 0, Vol Init button 248 for setting the base depth value of the selected object, Averaging button 250 for setting the depth value of the selected object to the average depth value, and edited whole Get Depth to check the depth value It comprises a button 252 and the Undo and Redo buttons 254 and 256 serving as Undo and Redo.

Depth Edithing button 148 clicks to modify the depth map to perform an operation on the depth setting dialog screen.

The Get DepthMap button 150 may obtain a depth map when clicked.

Left view control 152 that can set depth map and positive / negative parallax includes Depth Level and Zero Parallax.

Depth Level plays a role of bringing down the level set to 256 levels as the depth map has 256 values with 8 bits. At this time, the calculation formula of Depth Level is New Depth Value = 256 / DelthLevel.

Zero Parallax is for setting up positive / negative parallax. When the value of Zero Parallax is 128, the value below 128 gives positive parallax and the value above 128 gives negative parallax. This has the effect of turning or popping out of or out of the monitor.

An image sequence related view 200 for tracking an image is divided into contour tracking and depth tracking, as shown in FIG. 6.

Contour Tracking consists of an Inintialization button, Init F Tracking button, Init B Tracking button, F Tracking button, B Tracking button and Contour Copy.

The initialization button 202 automatically sets the initial object outline using the contour point set in the Lasso Tool when clicked.

Init F Tracking button 204 tracks omnidirectional object contours based on key frames.

Init B Tracking button 206 tracks backward object contours based on key frames.

The F Tracking and B Tracking buttons 208 and 210 may retrack after setting the object outline with the Init A Tracking button 204 and the Init B Tracking button 206. In this case, the F Tracking and B Tracking buttons 208 and 210 may track after modifying the outline of the initially tracked object in an arbitrary frame instead of a key frame.

The Contour Copy button 212 is used when copying is required rather than tracking because there is no movement of an object in each frame.

The user interval check box 214 checks and inputs the number of frames to be tracked in the blank to perform tracking and copying based on the current frame.

The BM Tracking checkbox 216 is used when a block matching method is to be applied. The default setting is the Lucas-Kannade Tracking method.

Extra points that remove unnecessary parts such as background parts inside the selected object area are configured to include Select, Tracking, and Reset buttons.

The select button 218 sets the point by clicking with the mouse after selecting the select button and makes a closed area by right clicking when obtaining the final outline.

The tracking button 220 sets a tracking section to track the closed area.

The Reset button 222 resets the setting after the edit of the Extra Points.

Depth Tracking for tracking the depth value is Est. It consists of a Depth button, Block Depth button, and a BackGround checkbox.

Generate Label Image 224 generates a label image for setting the depth map.

Est. Depth button 226 generates a depth map using the average depth value.

The block depth button 228 generates a depth map by predicting a depth value based on the block.

The BackGround checkbox 230 is used to apply hole filling to the automatically generated background image. It is especially useful in scenes with no background movement.

The image display device 104 is provided to output an image processed by the server 102. In this case, the image display device 104 includes an electronic paper, a liquid crystal display panel, a field emission display panel, a plasma display panel, and a light emitting display panel. (Light Emitting Display Panel).

7 is a diagram illustrating a conversion process of a 3D stereoscopic image conversion apparatus according to an embodiment of the present invention.

Referring to FIG. 7, the 3D stereoscopic image converting method according to an embodiment of the present invention is a method for converting a 2D image into a 3D image having a three-dimensional effect, the first step of collecting a 2D or 3D image by an image collecting device ( S1), a second step of converting the image collected by the image acquisition apparatus into a stereoscopic image having parallax in a server equipped with a 3D stereoscopic image authoring tool (S2), and the image processed by the image display apparatus after processing in the server; It is configured to include a third step (S3) to display.

First, the stereoscopic image conversion method of the 3D stereoscopic image authoring tool mounted on the server, as shown in Figure 8, the step A of opening a new work file or project (S10), the key from the image collected by the image acquisition device Step B of selecting a frame (S12), step C of dividing and merging the area of each object in the selected key frame (S14), step D of tracking a key frame (S16), and E step (S18) of collecting depth information and F step (S20) of generating a parallax image using the depth information is configured.

Here, in step S14 of dividing and merging the regions of each object in the selected key frame, as shown in FIG. Step (S30), editing the wrong object area in each object area of the area division result (S32), merging the wrong object area in the area division result edited using a drawing tool (S34); And detecting the contour point of the object region and detecting the automatic contour using the Lasso tool (S36).

First, after selecting a key frame and adjusting the object region segmentation result by using the Region Segmentation button of the MeanShift Tool in the window window (S30), as shown in FIG. 10, the region segment of the main frame is selected to display the parameter setting dialog. The spatial radius of the widow (Hs), the range radius of the window (Hr), and the munumum density a region may have in the resulting segmented image (M) scroll bar are adjusted, as shown in FIG. Adjust the results.

Subsequently, in step S32 of editing an invalid object region in each object region of the region segmentation result, object division is selected by merging the divided regions by clicking the Merge button and dragging the mouse as illustrated in FIG. 12. And, if the merge is not natural, edit it with Draw.

Subsequently, as illustrated in FIG. 13, the wheel angle of the object is edited using a clustering region and a draw region, which are drawing tools. In addition, using the Lasso Tool, as shown in FIG. 14, object outline point selection and outline detection are automatically detected while moving the object outline with mouse drag and click.

In step S16 of tracking the key frame, as illustrated in FIG. 15, the step S40 of adding an object of the key frame to the object list and automatically setting the initial object contour to the contour point of the object area are performed. Step S42, tracking the object contour based on the key frame (S44), and editing the object contour point generated after the object contour tracking (S46).

First, in step S40 of adding an object of a key frame to an object list, in order to set an outline of an object, an object must be added to an object list of a main frame.

Subsequently, in step S42, the initial object outline is automatically set to the outline point of the object region. As shown in FIG. 17, the initial object outline is set to the outline point set by the Lasso Tool by clicking the Initialization button of the image sequence related window. Is set automatically.

Subsequently, in step S44 of tracking the object outline based on the key frame, the object outline is tracked forward or backward based on the key frame.

Subsequently, in step S46 of editing the object contour point generated after the object contour tracking, as shown in FIG. 20, the object is moved by moving the mouse to an object contour point that is incorrectly predicted in the selected frame. Straighten and edit the outline of the.

Next, as shown in FIG. 21, unnecessary parts such as a background part are removed in the selected object area using extra points.

Subsequently, as illustrated in FIG. 22, the contour point may be added or deleted using the Insert Point or Delete Point menu by right-clicking on the object.

E step (S18) of collecting the depth information of the key frame, as shown in Figure 23, generating a label image for the image (S50), and using the generated label image depth information Collecting (S52), storing the collected depth information in the depth table (S54), and tracking the depth information input to the depth table to generate a depth map of the entire image (S56). It is configured by.

Generating a label image for the image (S50), as shown in Figure 24, generates a label image using the Generate Label Image button.

In the collecting of depth information by using the generated label image (S52), when the set depth information is clicked on the main frame, the depth values may be adjusted in 3D space on the divided objects in the depth setting dialog screen.

25 to 33 are diagrams illustrating that the depth setting is changed through selection of an object selection on a depth setting dialog screen and a brush tool, gradient tool, layer depth, zero parallax, path selection tool, initialize, and VOL int.

The step S54 of storing the collected depth information in the depth table is to set the depth change of the object in the scene, as shown in FIG. 34, and the last frame as shown in FIG. 35. 36, the depth value in the Peak frame is set as shown in FIG. 36, and when the Calculate button is clicked, as shown in FIG. 37, the total depth values are set to be automatically calculated. do.

The step S56 of tracking the depth information input to the depth table to generate a depth map of the entire image may include estimating Depth Tracking. Depth tracking, Block Depth tracking, and BackGround are used to create depth maps.

When the depth map is predicted for all frames in the scene, as shown in FIG. 39, the depth level and zeropararx of the left view control are set, and as shown in FIG. 40, the stereoscopic image of the Render as in the file input / output menu. Select the output method. In this case, the output method may be a stereoscopic video and a Tga sequence.

41 is a view showing a file format in a project folder created at the time of working according to an embodiment of the present invention.

Referring to FIG. 41, an object mask, a depth map, an image shifted by a depth level, a hole filled image, point information, etc., which are intermediate products during operation, are automatically stored in a project folder. At this time, the file name format in the project folder is object mask file name format is object [object number] _f [frame number] .bmp, and the object point information file name format is point [object number] _f [frame number] .txt and depth map. The file name format consists of DepthImage [frame number] .bmp, the depth information name format DepthInfo_ [frame number] .txt and the layer image file name format consists of LayerImage [frame number] .bmp.

The 3D stereoscopic image converting device and the conversion method configured as described above are semi-automatic stereoscopic conversion software that can work 2D images on a scene basis, and the most important depth map generation of 3D images is easily performed by user input in 3D space. You can create a realistic depth map by providing a UI to set the volume and positive / negative difference of the object surface.

Project management also saves intermediate artifacts in the workflow, making it easy to make manual corrections when needed.

It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit or scope of the invention. Will be clear to those who have knowledge of.

1 is a view showing a 3D stereoscopic image converting apparatus according to an embodiment of the present invention.

2 is a view showing a 3D stereoscopic image authoring tool according to an embodiment of the present invention.

3 is a view showing a 3D stereoscopic image authoring tool according to an embodiment of the present invention.

4 is a diagram illustrating a region segment dialog of a 3D stereoscopic authoring tool according to an embodiment of the present invention.

5 is a diagram illustrating a depth setting dialog of a 3D stereoscopic image authoring tool according to an embodiment of the present invention.

6 is a view showing an image sequence related view for tracking an image according to an embodiment of the present invention.

7 is a flowchart illustrating a conversion process of a 3D stereoscopic image conversion apparatus according to an embodiment of the present invention.

8 is a flowchart illustrating a method of converting a stereoscopic image of a 3D stereoscopic image authoring tool according to an embodiment of the present invention.

9 is a flowchart illustrating a process of dividing and merging regions of each object in a key frame according to an embodiment of the present invention.

10 is a view showing a view for adjusting the object region segmentation result in the 3D stereoscopic authoring tool according to an embodiment of the present invention.

11 is a view showing the object region segmentation result in the 3D stereoscopic authoring tool according to an embodiment of the present invention.

12 is a view showing a result of merging object regions in a 3D stereoscopic authoring tool according to an embodiment of the present invention.

FIG. 13 is a diagram illustrating editing an outline of an object in a 3D stereoscopic authoring tool according to an embodiment of the present invention. FIG.

14 is a diagram for selecting and detecting contour points of an object in a 3D stereoscopic authoring tool according to an embodiment of the present invention.

15 is a flowchart illustrating a process of tracking a frame in a 3D stereoscopic authoring tool according to an embodiment of the present invention.

16 is a diagram illustrating an object list adding window according to an embodiment of the present invention.

17 is a diagram illustrating automatically setting an initial object outline according to an embodiment of the present invention.

18 is a diagram illustrating back and forth tracking based on a key frame according to an embodiment of the present invention.

19 illustrates a wrongly predicted contour point according to an embodiment of the present invention.

20 illustrates editing an incorrectly predicted contour point according to an embodiment of the present invention.

FIG. 21 is a diagram illustrating removing unnecessary portions inside an object area according to an exemplary embodiment of the present invention. FIG.

22 is a diagram illustrating adding and deleting contour points according to an embodiment of the present invention.

23 is a flowchart illustrating a process of collecting depth information according to an embodiment of the present invention.

24 illustrates generating a label image according to an embodiment of the present invention.

25 to 33 are views illustrating changes of images for each button of the depth setting dialog screen according to an embodiment of the present invention.

34 to 37 are diagrams illustrating that a depth map is generated by inputting a value into a depth table according to an embodiment of the present invention.

38 is a diagram of generating a depth map by generating a depth map of the entire image according to an embodiment of the present invention.

39 is a diagram illustrating a left image generation process according to an embodiment of the present invention.

40 is a diagram illustrating a method of outputting a stereoscopic image according to an embodiment of the present invention.

41 is a view showing the configuration of a project file according to an embodiment of the present invention.

<Explanation of symbols for the main parts of the drawings>

100: image acquisition apparatus 102: server

104: image display device 106: 3D stereoscopic image authoring tool

Claims (11)

delete delete delete delete In the method for converting a 2D video into a 3D video with a three-dimensional effect, A first step of collecting a 2D or 3D image in the image collecting apparatus; a second step of converting the image collected by the image collecting apparatus into a stereoscopic image having parallax in a server equipped with a 3D stereoscopic image authoring tool; Including a third step of outputting the processed image to the image display apparatus after processing in, The second step, Opening a new work file or project; Selecting a key frame from an image collected by the image collecting device; And extracting the object and the depth information automatically using the existing automatic conversion method according to the selected key frame and converting the same into a stereoscopic image. delete 6. The method of claim 5, The second step, A step of opening a new work file or project; Selecting a key frame from an image collected by the image collecting device; Dividing and merging regions of each object in the selected key frame; Tracking the key frame; E step of collecting depth information of the key frame; And And a step F of generating a parallax image by using the depth information. The method of claim 7, wherein The step C, Adjusting the segmentation result by using a region segmentation menu of a MeanShift Tool after selecting the key frame; Editing an invalid object region in each object region of the region division result; Merging a wrong object region in the region division result edited using a drawing tool; 3. The method of claim 1, further comprising detecting contour points of the object region and detecting automatic contours using an Lasso tool. 9. The method of claim 8, The step D, Adding an object of the key frame to an object list; Automatically setting an initial object contour to the contour point of the object area; Tracking an object contour based on the key frame; And editing an object contour point generated after the object contour tracking. The method of claim 9, The step E, Generating a label image for the image; Collecting depth information using the generated label image; Storing the collected depth information in a depth table; And generating a depth map of the entire image by tracking with the depth information input to the depth table. 11. The method of claim 10, The F step, 3D stereoscopic image conversion method comprising the step of selecting a stereoscopic image output method using the depth map.

Images