TW201412094A - Device for generating a depth map - Google Patents

Abstract

Device (100) for generating a depth map (142) based on a two-dimensional [2D] image (144) for enabling three-dimensional [3D] rendering of the 2D image, comprising: a display output (160) for displaying a marker (320) in the 2D image, the marker corresponding to a component of a geometric template (300), the geometric template comprising depth information for defining a global depth profile (360) of a reference scene; a user input (120) for enabling a user to position the marker in the 2D image so as to enable the user to match the geometric template to an actual scene shown in the 2D image by positioning the component with respect to the 2D image; and a depth map generator (140) for establishing the global depth profile as the depth map (142) based on a position (122) of the marker in the 2D image.

Description

Device for generating a depth map

The present invention relates to a device for generating a depth map based on a two-dimensional [2D] image to enable three-dimensional [3D] visualization of the 2D image.

The invention further relates to a method for generating a depth map based on a 2D image to enable 3D rendering of the 2D image, and the invention relates to a computer program comprising instructions for causing a processor system to execute the method product.

More and more display devices, such as televisions, digital photo frames, tablets, and smart phones, include 3D displays to provide user depth perception when viewing content on such a device. For this purpose, such 3D displays can (by themselves or with the glasses worn by the user) provide different images in the user's eyes to provide the user with a depth perception based on the stereoscopic sense.

3D displays usually require content with in-depth information. The depth information can be implicitly provided in the 3D content. For example, in the case of what is called stereoscopic content, the depth information is provided by the difference between the left image and the right image of the stereoscopic content. The depth information can also be explicitly provided in the 3D content. For example, in 3D content called image + depth format encoding, the depth information is provided by a depth map, which may include depth values, aberration values, and/or parallax displacement values, all of which indicate objects within the image. The distance from the camera.

A large amount of content currently available (eg, movies, television shows, images, etc.) is 2D content. This content needs to be converted to 3D to enable rendering of its 3D display on a 3D display Now. Converting to 3D can include generating a depth map of a 2D image (eg, each 2D image for a 2D video). This depth map can be automatically generated. For example, a device can estimate the distance of an object within the 2D image from the camera, based on which a depth map of the 2D image is generated. The user can also manually generate the depth map. For example, a digital photo frame may provide the consumer with the possibility to draw a depth map using one of the touch sensitive surfaces of the digital photo frame. As another example, executing a software tool on a workstation can provide the possibility for a professional user to add depth to a 2D image by drawing a depth map using a digital pen. The depth map can also be generated semi-automatically, wherein the device generates the depth map based on user input. Here, the user provides depth information to the device, such as a relative depth step between the objects. Based on the depth information, the device then generates the depth map.

Half of the automatically generated depth map can have a higher quality than a fully automatic depth map. However, the user may be inconvenienced by the need to provide depth information to the device.

January-February 2011, IEEE Vol. 31, No. 1, pp. 36-48, Computer Graphics and Applications, B. Ward et al., entitled "Depth Director: A System for Adding Depth to Movies" The product description is used to transform an existing 2D data film into an interactive system of stereoscopic movies. The system is said to provide automated depth designation for scenes with one of the mobile cameras using a motion structure. It is said that the user can arbitrarily emphasize or not emphasize the part of the scene. This is done through a stroke-based interface along with a more complex depth template to refine the computer vision results. A depth template consists of a triangular mesh that can be transposed, scaled, and rotated in 3D to fit a selected area. Examples are a flat surface, a spherical surface, a cylinder, a box, a car, and a face template. These planar templates are said to be useful for making a ground plane.

One of the shortcomings of the above interactive system is that it is difficult for a novice user to use the above interactive system to convert an existing 2D data film into a stereo movie.

One of the objects of the present invention is to provide an apparatus and/or method for enabling a novice user to more easily generate a depth map of a 2D image.

A first aspect of the present invention provides an apparatus for generating a depth map based on a two-dimensional [2D] image to enable three-dimensional [3D] visualization of the 2D image, comprising: a display output for display One of the 2D images, the marker corresponding to one of the geometric templates, the geometric template including depth information for defining a global depth distribution of a reference scene; a user input for making a The user can place the marker in the 2D image to enable the user to match the geometric template to one of the actual scenes displayed in the 2D image by placing the component with respect to the 2D image; and a depth map a generator for establishing the global depth profile as the depth map based on a location of a marker in the 2D image.

Another aspect of the present invention provides a method for generating a depth map based on a two-dimensional [2D] image to enable three-dimensional [3D] visualization of the 2D image, comprising: displaying one of the 2D images, The marker corresponds to a component of a geometric template comprising depth information for defining a global depth distribution of a reference scene; enabling a user to place the marker in the 2D image for use The geometric template can be matched to one of the actual scenes displayed in the 2D image by placing the component with respect to the 2D image; and the global depth distribution is established as the depth based on a position of the marker in the 2D image. Figure.

Another aspect of the present invention provides a computer program product comprising a processor system for performing the method.

Embodiments of the invention are defined in the accompanying technical solutions.

The above method generates a depth map based on a 2D image. A depth map generally includes A depth value corresponding to a portion of the 2D image and an estimated distance from the camera. Here, the term camera refers to the use of one of a real or a virtual camera when capturing or generating the 2D image. Therefore, for each part of the 2D image, a corresponding portion of the depth map exists, which reflects its distance from the camera. The depth map can be used for 3D rendering of the 2D image, for example, using techniques commonly referred to as view rendering. Here, one or more additional 2D images are generated by replacing portions of the 2D image based on the depth map. The 2D image is used for the generation of the depth map as follows.

The device includes a display output that enables a marker to be displayed to the user. The marker is displayed within the 2D image. For example, the marker can be overlaid on the 2D image. The user is then presented with the 2D image and one of the markers displayed. The display output can also enable display of the 2D image. Alternatively, the 2D image can be displayed by a different device or component of the device. The display output can be connected to a display or can include a display.

The marker corresponds to one of the geometric templates, that is, a mathematically defined shape composed of one or more components. The geometry of the geometric template is defined by the relative 2D position of the one or more components. The location of the geometric template is defined by the absolute 2D position of the one or more components. The geometry template includes depth information. For example, one or more depth values can be assigned to one or more components of the geometric template. The geometric template, along with the depth information, defines a global depth distribution of one of the reference scenes. Here, the depth information provides the depth of the global depth distribution and the shape and/or position of the geometric template provides the shape and/or position of the global depth distribution. The scene is a reference scene in which it constitutes an instance of a certain type of scene, such as a landscape scene or an indoor scene. The term scene refers to a combination of a space and an object (eg, a person) between them. Examples of spaces are, for example, a room, a narrow alley, a flat landscape, and the like. The global depth distribution indicates how depth is distributed globally within the scene. Here, the term global refers to a general overview of the depth distribution giving a depth of the scene, ie, a preliminary estimate, omitting many or all of the details Section. In general, the global depth distribution corresponds to the depth distribution of the space rather than the object therein, as this generally provides a general overview of the depth of the scene.

The device includes a user input that enables the user to place the marker in the 2D image, for example, by receiving a suitable command from a user input tool operated by the user. The marker is positioned 2D, ie a 2D position is generated. By establishing the 2D position of the marker, the user establishes a position of the component of the geometric template with respect to the 2D image. Therefore, the positioning of the component is also 2D, that is, a 2D position is generated. The position of the component of the geometric template affects the position and/or shape of the geometric template as a whole. The user can thus influence the position and/or shape of the geometric template with respect to the 2D image by appropriately placing the marker in the 2D image. The geometric template is known to define the global depth profile, and the user can thus adjust the global depth profile by adjusting the position and/or shape of the geometric template. Therefore, the user adjusts the global depth distribution by placing the marker appropriately in the 2D image. The user can adjust the marker such that the geometric template (and thus the global depth profile defined by the geometric template) matches one of the actual scenes displayed in the 2D image. The user can thus compensate for the difference between the reference scene and the actual scene by placing the marker appropriately. For example, the user can compensate for differences in composition, such as one of the horizons in two scenes, one of the vanishing points, and the like.

The device includes a depth map generator. The depth map generator generates a depth map of the 2D image based on a position of a component of the geometric template in the 2D image. The above location is provided by the location of the marker. The depth map generator generates the depth map by establishing a global depth profile defined by the geometric template. The global depth profile may have been adjusted by the user by placing the marker appropriately in the 2D image. The depth map generator takes these adjustments into consideration by taking the position of the marker into consideration, that is, establishing a global depth distribution for adjusting the actual scene.

The present invention is based in part on the recognition that a novice user uses even an interaction It is also difficult to create a depth map of a 2D image. One reason for this problem is that known interactive systems identify all elements of a scene as individual objects, and the user must specify a depth for the individual objects. Disadvantageously, the user may be reluctant to assign a depth to all elements of the scene. The inventors realized that it is especially important to correctly establish the global depth of a scene. One reason for this is as follows. A user can primarily focus on one of the objects within the scene. In the depth perception of the object, its absolute depth plays an important role. However, the depth of an object is often judged by the space in which the object is placed. Therefore, the relative depth of the object also plays an important role. Therefore, even when trying to convincingly copy one of the depths of one of the objects in a scene, it is desirable to correctly establish the depth of the surrounding space. In addition, the global depth plays an important role in establishing a global impression of one of the depths of a scene.

The present invention enables the user to conveniently establish a global depth distribution of the 2D image by providing a global depth distribution of a reference scene and enabling the user to depth the global depth by placing a marker appropriately The distribution matches one of the actual scenes shown in the 2D image. When the marker is placed, the device then automatically generates a depth map by establishing the possibly adjusted global depth profile as the depth map. Advantageously, the user does not need to specify a depth for individual elements of the space in which the object is placed (eg, a floor, wall, and ceiling). More specifically, the device enables the user to generate a depth map of the entire space in a single step. Advantageously, the user does not need to place the marker in 3D to establish a depth of the 2D image. More precisely, the depth information is provided separately.

Note that in the above, the term "map" refers to data that is configured with columns and rows. In addition, the adjective "depth" should be understood to mean the depth of the portion of the 2D image to the camera. Thus, the depth map can be formed by depth values, but also by, for example, aberration values or disparity displacement values. In essence, the depth map can thus constitute an aberration map or a parallax displacement map. Here, the term "aberration" refers to the use of one of the user's left eye or one One of the differences in the position of an object when the right eye perceives. The term "parallax displacement" refers to the displacement of one of the objects between two views to provide the above aberration to the user. Aberration and parallax displacement are usually inversely related to distance or depth. All of the above types of diagrams and/or conversion devices and methods are known.

Optionally, the geometric template defines a depth gradient, and the depth map generator is configured to establish a depth gradient in the depth map based on the position of the marker. A depth gradient is well suited as a global depth distribution because the depths in many scenes are globally distributed in a similar gradient. For example, the depth in many landscape scenes can be expressed as a gradient toward one of the horizons, and the sky has a depth corresponding to one of the depths at one of the ends of the gradient.

Optionally, the position of the marker forms a starting point or an ending point of the depth gradient. Here, one of the geometric templates defines a start point or an end point of the depth gradient, and the marker corresponds to the above component. The user can thus conveniently adjust one of the depth gradients in position and/or shape by placing the marker properly in the 2D image.

Optionally, the location of the marker constitutes a vanishing point in the global depth profile. Here, one of the geometric templates defines a vanishing point, and the marker corresponds to the above components. A vanishing point is well suited as one of the components of the global depth distribution because the depths in many scenes are aggregated toward one of the vanishing points in the scene. Therefore, the global depth distribution is well suited for many scenarios by including a vanishing point in the global depth distribution, or by forming a vanishing point of the global depth distribution. The user can conveniently adjust the position of the vanishing point by placing the marker properly in the 2D image.

Optionally, the marker is one of a plurality of markers, the plurality of markers corresponding to a plurality of components that together define the global depth distribution, and the user input is configured to enable the user to place the individual Each of the plurality of markers is configured to adjust the global depth distribution to adapt the actual scene to the 2D image. The user can therefore Multiple markers are placed locally to adjust multiple components of the global depth distribution. By individually adjusting the above components, the user can better match the global depth distribution to the actual scene displayed in the 2D image.

Optionally, the display output is configured to display one or more connections between the plurality of markers, and the depth map generator is configured to be based on one of the one or more connections in the 2D image Position to establish one or more edges of the global depth distribution. One or more connections between the markers can thus be presented to the user, for example, as a line. The one or more connections correspond to edges of the global depth profile, and thus a depth edge or depth transition is generated in the depth map. By presenting one or more of the connections, the user can better match the global depth distribution to the actual scene displayed in the 2D image. One reason for this is that the user can also (in addition to or as an alternative to placing the markers in the 2D image relative to the feature points) the markers are also placed to match the one or more connections to The characteristic structure in the 2D image. These characteristic structures can be edges or transitions in the 2D image. Advantageously, the user can more accurately match the global depth distribution to the actual scene displayed in the 2D image.

Optionally, the plurality of markers and the one or more connections correspond to one of the global depth profiles. A horizon is usually a characteristic structure in a scene. In addition, a horizon is well suited as one of the components of the global depth distribution when one of the horizons or a transition between the depth gradient of the horizon and the homogenous depth of the sky. The user can thus match the global depth distribution to the actual scene displayed in the 2D image by matching the markers to the one or more connections to the horizontal line. Advantageously, the user can more easily match the global depth distribution to the actual scene displayed in the 2D image.

Optionally, the plurality of markers and the one or more connections form at least one of: a line segment, a polyline, or a polygon.

Optionally, the user input is configured to enable the user to add a marker to The plurality of markers and/or the markers are removed from the plurality of markers. The user can thus change the number of markers. Advantageously, the user can adapt the number of markers to the actual scene in front of the eye, ie if the geometry template is better matched to the actual scene displayed in the 2D image, then a marker is added, or if not needed One or more markers remove the marker by matching the geometric template to the actual scene displayed in the 2D image.

Optionally, the user input is configured to enable the user to select one of the foreground objects in the 2D image, the foreground object constituting an object in the 2D image having one of the foreground depths insufficiently represented by the global depth distribution; And the depth map generator is configured to i) establish a foreground depth of the foreground object, and ii) include a depth representation of the foreground object in the depth map based on the foreground depth.

The user input enables the user to select an object in the 2D image, for example, by pointing to the object or clicking on the object. The object constitutes a foreground object. This means that the object constitutes a separate object that the user can distinguish from the background. The background is typically provided by the space in which the object in the scene is placed, wherein the foreground object covers or blocks a portion of the space. The foreground system is insufficiently represented by the global depth distribution because the global depth distribution contains depth values at corresponding portions of the depth map that cannot or inaccurately reflect such foreground objects, and if the foreground objects If it does not exist, it can reflect this environmental space. The depth map generator establishes a depth of the foreground object and uses the depth to include a depth representation of one of the foreground objects in the corresponding portion of the depth map.

Advantageously, the user (in addition to conveniently establishing the global depth of a scene) may also specify a depth to one of the foreground objects of the scene. Advantageously, the user can conveniently and quickly obtain one of the 2D images suitable for the depth map by first establishing the global depth profile as the depth map, thereby obtaining a depth map of the space, and then selecting the scene. One or more foreground objects are represented in the depth map.

Optionally, the depth map generator is configured to establish the foreground depth based on a depth at a boundary of a global depth distribution and a depth representation of the foreground object. A foreground system is often connected to its environmental space by, for example, standing upright on a floor or hanging from the ceiling. This global depth distribution may indicate the depth of the space. Therefore, the depth of the foreground object is at least largely determined from the depth of the global depth distribution and the boundary of a portion of the depth map corresponding to the foreground object, since the depth of the space is equal to the foreground object depth. Advantageously, the foreground depth can be automatically determined.

Optionally. The user input is configured to enable the user to establish a foreground object that is erected or hung in the actual scene displayed in the 2D image, and the depth map generator is configured to establish the boundary based on the upright or suspension One point. A suspension system is usually attached to the space around it at the top of it. Similarly, an upright system is typically attached to the space at its bottom. For example, when the foreground system is one person, the person's foot forms the connection of the person to the ground. By indicating that the foreground object is the actual scene displayed erected or hung in the 2D image, the device thus has information on where to establish the point along the edge of the foreground object.

Optionally, the user input is configured to enable the user to provide the foreground depth to the depth map generator. The user can manually determine the depth of the foreground object by providing the foreground depth to the depth map generator (eg, by typing a value into a text box or moving a depth slider to a position).

100‧‧‧ device

110‧‧‧ display

120‧‧‧User input

122‧‧‧ position

130‧‧‧User interface device

132‧‧‧Location information

140‧‧‧Depth map generator

142‧‧Deep map

144‧‧‧Two-dimensional [2D] imagery

160‧‧‧ Display output

162‧‧‧ Display information

164‧‧‧ cursor

190‧‧‧ Prospect objects

192‧‧‧ foreground objects

194‧‧‧ Prospect objects

196‧‧‧depth representation

197‧‧‧ border

198‧‧‧Deep representation

199‧‧‧ border

200‧‧‧ method

210‧‧‧ display

220‧‧‧Enable

230‧‧‧ Established

250‧‧‧Computer-readable media

260‧‧‧Computer Program Products

300‧‧‧Geometric template

320‧‧‧ marker

322‧‧‧Marker

324‧‧‧Marker

326‧‧‧Marker

340‧‧‧Connect

360‧‧‧Global depth distribution

These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described herein. In the drawings, Figure 1 shows a device for generating a depth map; Figure 2 shows one method for generating the depth map; Figure 3 shows a computer program product for performing the method; Figure 4a shows a two-dimensional [2D] image; 4b shows a geometric template including depth information; FIG. 4c shows a global depth distribution of a reference scene, the global depth distribution is defined by the geometric template; FIG. 4d shows that a plurality of markers are displayed in the 2D image, The plurality of markers correspond to the components of the geometric template; FIG. 4e shows that the user places the plurality of markers in the 2D image to match the geometric template to one of the actual scenes displayed in the 2D image; FIG. 4f Demonstrating a global depth distribution established by the user as a depth map based on the placement of the plurality of markers; Figure 5a shows another geometric template including depth information; Figure 5b shows the user placing a plurality of markers on the The 2D image is matched to the actual scene displayed in the 2D image; FIG. 5c shows a global depth distribution established by the user as a depth map based on the placement of the plurality of markers; FIG. 5d Demonstrating that the user selects one of the foreground objects in the 2D image; FIG. 5e shows that one of the foreground objects is represented by a depth representation based on the foreground object standing upright in the actual scene Figure 5f shows that one of the foreground objects has a depth representation that is included in the depth map based on the foreground object suspended in the actual scene; Figure 6a shows a geometric template including depth information, the geometric template definition including One of the vanishing points is a global depth distribution; and Figure 6b shows the user placing a marker in the 2D image to match the vanishing point to one of the actual scenes displayed in the 2D image.

It should be noted that items having the same component symbols in different drawings have the same structural features and the same functions, or are the same signals. If the function and/or structure of this item has been elucidated, there is no need to repeat the explanation in the embodiment.

1 shows an apparatus 100 for generating a depth map 142 based on a two-dimensional [2D] image 144. The device 100 includes a display output 160 for displaying one of the markers 320 in the 2D image 144. For this purpose, display output 160 is shown to provide display data 162 to a display 110 that is coupled to display output 160 of device 100. Display material 162 can include 2D image 144 itself. Device 100 can include display 110. The device 100 further includes a user input 120 for enabling a user to place the marker 320 in the 2D image 144. For this purpose, the display user input 120 receives location data 132 from a user interface device 130, such as a computer mouse, keyboard, touch screen, and the like. The apparatus 100 further includes a depth map generator 140 for generating a depth map 142 based on one of the markers 122 in the 2D image 144. For this purpose, the presentation depth map generator 140 receives the location 122 of the marker 320 from the user input 120.

One of the operations of the device 100 can be briefly explained as follows. The marker 320 corresponds to one of the components of a geometric template. The geometric template includes depth information for defining a global depth distribution of one of the reference scenes. The user places a marker 320 to match the geometric template to one of the actual scenes shown in the 2D image 144. This is enabled because the marker 320 corresponds to one of the geometric templates, and thus by placing the marker 320, the user also places the component with respect to the 2D image 144. Due to this placement, a position 122 of the marker 320 is obtained. The depth map generator 140 establishes the global depth distribution as the depth map 142 based on the position 122 of the marker 320 in the 2D image 144. As a result, a depth map 142 of the 2D image 144 is obtained, which enables three-dimensional [3D] visualization of the 2D image 144. For this purpose, depth map 142 can be sent along with 2D image 144 to a 3D display that performs 3D rendering using techniques such as view rendering, which are known per se from the art of 3D displays. Display 110 can be a 3D display to provide a preview of the 3D representation. However, display 110 can also be a 2D display.

It is noted that the geometric template can be selected by the user from a plurality of geometric templates. The user's selection can be determined by which of the plurality of geometric templates best matches the actual scene displayed in the 2D image 144. For this purpose, device 100 can indicate which reference scene a geometric template corresponds to. Thus, in the case where the 2D image 144 exhibits a landscape scene, the user can select a geometric template that also corresponds to a landscape scene.

2 shows a method 200 for generating a depth map based on a two-dimensional [2D] image to enable three-dimensional [3D] visualization of the 2D image. The method 200 includes (in one step named "DISPLAYING MARKER IN 2D IMAGE") displaying 210 one of the 2D images, the marker corresponding to one of the geometric templates, the geometric template including Refer to the depth information of the global depth distribution of one of the reference scenes. The method 200 further includes (in a step named "ENABLING USER TO POSITION MARKER") enabling 220 a user to place the marker in the 2D image to enable the user to view the component with respect to the 2D The image is placed to match the geometric template to one of the actual scenes shown in the 2D image. The method 200 further includes (in one step named "ESTABLISHING DEPTH MAP BASED ON POSITION OF MARKER") establishing 230 the global depth profile as the depth map based on one of the markers in the 2D image. Method 200 can correspond to one of the operations of device 100. However, method 200 can also be performed separately from device 100.

3 shows a computer readable medium 250 including a computer program product 260 for causing a processor system to perform the method in accordance with the present invention. For this purpose, computer program product 260 includes instructions for the processor system that, when executed, cause the processor system to perform the method. Computer program product 260 can include a series of elements on a computer readable medium 250 that are labeled as a series of machine readable entities and/or have different electrical properties (eg, magnetic properties), or optical properties or values.

The operation of device 100 can be further described with reference to Figures 4a and subsequent figures. Figure 4a shows a two-dimensional [2D] image 144. In this particular example, the 2D image 144 depicts a scene comprising a sailboat 190 sailing through the sea, wherein the sea extends to a horizon and the horizon The line forms a boundary between the sea and the sky. In the above scenario, the sailboat 190 constitutes an object 190 within the environmental space of the sea and the sky. The sailboat 190 further constitutes a foreground object 190 because it covers or blocks portions of the space where the sea and portions of the sky are positioned and thus covered by the foreground object 190.

Figure 4b shows one geometric template 300 including depth information. The change in depth information is schematically represented by one of the density of the lines, which is schematically represented in Figure 4b, where a high density represents a large depth and a low density represents a small depth or proximity. It will be appreciated that the geometric template 300 can be represented by data that allows for the generation of a schematic representation as shown in Figure 4b, but which takes a different form, such as a list of targets and associated depth information. 4b further shows a plurality of markers 320, 322, ie, a first marker 320 and a second marker 322 that define a line within the geometric template 300.

Figure 4c shows a global depth distribution 360 as defined by the geometric template 300 of Figure 4b. The intensity within the global depth distribution 360 represents the degree of depth, i.e., the dark area corresponds to a large depth or large distance, and the bright area corresponds to a small depth or proximity. The global depth distribution 360 belongs to a reference scene. The reference scene may be identified from the global depth distribution 360 because the reference scene includes a ground plane that is tilted toward a horizon, and a sky plane extends upward from the horizon. Figure 4c also shows a plurality of markers 320, 322 and connections 340 between the markers, the connections 340 being shown in a dashed form. The visible connection 340 corresponds to the horizon in the reference scene because it forms one of the global depth distributions 360 between the depth gradient of the ground plane that gradually increases toward the horizon and the flat depth plane of the sky extending over the horizon. .

The geometric template 300 as shown in Figure 4b defines a global depth profile 360 as shown in Figure 4c, wherein one of the plurality of markers 320, 322 and the connection 340 indicate where the horizon is located at the global depth distribution 360. The geometric template 300 can further define the shape of the depth gradient associated with the ground plane, the degree of depth within the global depth distribution 360, and the like.

FIG. 4d shows a plurality of markers 320, 322 displayed in the 2D image 144. Here In the example, a plurality of markers 320, 322 are overlaid on top of the 2D image 144. In addition, the connection 340 between the plurality of markers 320, 322 is displayed in the 2D image 144. Figure 4d shows that the plurality of markers 320, 322 and connections 340 initially correspond to the locations shown in Figures 4b and 4c, i.e., their position is such that the 2D image 144 exhibits the reference scene.

Figure 4d further shows the display of one of the cursors 164. The cursor 164 can be provided by the display output 160, i.e., can be part of the display material 162. The user can move the cursor 164 on the screen by appropriately operating the user interface device 130. To display the cursor 164 at a suitable location on the screen, the display output 160 can receive the location data 132 from the user input 120. The user can use the cursor 164 to place one or more of the plurality of markers in the 2D image 144, i.e., to reposition one or more of the plurality of markers with respect to their initial position. For example, the user can reposition the first marker 320 by dragging and dropping one of the first marker 320 to the new location of the 2D image 144. Using the drag and drop, the user can place a plurality of markers 320, 322 such that the geometric template 300 is matched to the actual scene displayed in the 2D image. For this purpose, the user can understand that a plurality of markers 320, 322 are components of a geometric template that represent one of the global depth distributions 360. The user may also not understand the concept of the geometric template, but the user can understand that the plurality of markers 320, 322 represent a horizon. Thus, the user can place a plurality of markers 320, 322 to match the horizon displayed in the 2D image 144, ie the horizon of the actual scene.

It is noted that the user may only need to place one or more of the plurality of markers 320, 322, wherein the connection 340 is automatically drawn by the display output 160. Therefore, there is no need to manually place the connection 340. Moreover, instead of using upstream markers 164 and/or drag and drop operations, an alternative method of placing a plurality of markers 320, 322 can be used. For example, device 100 can be configured to directly create a new marker at a location indicated by the user. Therefore, there is no need to reposition a marker. Thus, the user can instead establish the initial position of the marker directly in the 2D image 144. For example, display 110 can include a touch sensitive meter And the user can establish the initial position of the marker by touching an appropriate position on the touch-sensitive surface.

Figure 4e shows the result of one of the positioning of the plurality of markers 320, 322. Here, the positions of the first marker 320 and the second marker 322 are such that the connection 340 between the markers matches the horizon of the actual scene displayed in the 2D image 144. 4f shows a global depth distribution 360 that is a depth map 142 of one of the 2D images 144 based on the positioning of the plurality of markers 320, 322 by the user. Comparing with FIG. 4c, the boundary between the depth gradient of the ground plane extending on the horizon and the flat depth plane of the sky is now placed at different vertical positions, ie, at a lower position within the global depth distribution, and having a The angle, wherein the position and angle correspond to the position and angle of the horizontal line shown in the 2D image 144.

It will be appreciated that the depth map 142 can be used for 3D visualization of the 2D image 144. Thus, the user will perceive the sea tilting towards the horizon, and the sky extends upward from the horizon. However, since the depth representation of the foreground object (i.e., the sailboat 190) is not explicitly included in the depth map 142, the user will perceive the sailboat 190 as having a depth of the environmental space based only on the stereoscopic perception. Therefore, the hull of the sailboat 190 will be perceived as tilting towards the horizon, while the sail of the sailboat 190 will be perceived as extending upward from the horizon. Essentially, after 3D visualization, the sailboat 190 is established as a depth as part of the space between the sea and the sky.

Figures 5a through 5f illustrate one of the optional features of the present invention that allows one of the foreground objects to be represented in depth within the depth map 142.

Figure 5a shows another example of a geometric template 300 that includes depth information. Again, one of the depth information is schematically represented by one of the linear densities, i.e., a high density indicates a large depth and a low density indicates a small density or proximity. Figure 5a further shows a plurality of markers 320-326 that define a rectangle within the geometric template 300. From Figure 5a, it can be seen that the geometric template 300 can define, for example, a global depth distribution of a room, an interior of a box, and the like.

Figure 5b shows a 2D image 144 depicting one of the scenes of one of the people standing upright in a room, One of the doors is in a wall at one of the far sides of the room and a light is suspended from the ceiling. In this scenario, the person and the light constitute an object within the space provided by the room. The person and the light further constitute a foreground object, as each of these covers or blocks portions of the space, i.e., portions of the room form a background with respect to the person and the light. Figure 5b further shows the user placing a plurality of markers 320-326 in the 2D image 144 to match the geometric template 300 of Figure 5a to the actual scene shown in the 2D image 144. This can be done using, for example, dragging and dropping.

Figure 5c shows the result of the user placing each of the plurality of markers 320-326 appropriately in the 2D image 144, and the device then establishes a global depth based on the location of the plurality of markers 320-326 by the user. The distribution 300 is taken as a depth map 142 of the 2D image 144. It will be appreciated that the floor, ceiling, side walls and far walls of the room can be identified in depth map 142.

To account for the presence of one or more foreground objects in the space, the user input 120 of the device 100 can be configured to enable the user to select one of the foreground objects 192, 194 in the 2D image 144 that constitutes the 2D An image having one of the foreground depths is not adequately represented by the global depth distribution 360, and the depth map generator 140 can be configured to i) establish a foreground depth of the foreground object, and ii) based on the foreground depth, including One of the foreground objects in the depth map 142 has a depth representation 196, 198.

The operation of the above apparatus 100 can be further explained with reference to Figures 5d-5f. Figure 5d shows the user selecting a foreground object in the 2D image, the foreground system person 192. The user can select the person 192 by, for example, moving the cursor 164 onto the person 192 in the 2D image 144 and then clicking on the person 192 by appropriately operating the user interface device 130. The click can indicate to the device 100 that the foreground system is located at the location of the cursor 164. In response, device 100 can establish an outline of one of the foreground objects in 2D image 144 by separating the foreground object. The depth map generator 140 may thus separate the person 192 by placing a seed for one of the seed-based separation algorithms, for example, at one of the 2D images 144 corresponding to the position of the cursor 164. However, an alternative separation tool can also be used (or the general outline of the foreground object can be created) alternative method). For example, the user can manually draw the outline of the foreground object using the cursor 164. Thus, as a result, the depth map generator 140 obtains a position and a contour of a person 192 in the 2D image 144.

5e shows a result of a depth map generator 140 including a depth representation 196 of a person 192 in the depth map 142 having substantially the position and contour as the person 192 in the 2D image 144, but including the depth value in place of the brightness and / or chroma value. The depth map generator 140 can obtain the foreground depth from the user. For this purpose, user input 120 can be configured to enable the user to provide foreground depth to depth map generator 140. For example, the user can provide a foreground depth using a slider, dial or similar user interface element that can be displayed on display 110. Alternatively or additionally, the user input 120 can be configured to enable the user to indicate a point in the space surrounding the foreground object that has the same or similar depth as one of the foreground objects, and the depth map generator 140 can be configured The foreground depth is established based on the spatial depth at the point.

Alternatively or additionally, the depth map generator 140 may be configured to establish the foreground depth based on a depth at one of the boundaries 197 of the depth representation 360 and the depth representation of the foreground object 196. The depth map generator 140 can automatically determine which point on the boundary corresponds to a point at which the foreground object is in contact with the space. Depending on the preset, the depth map generator 140 can assume that this point is at a lower portion of the boundary of the foreground object, for example, based on a hypothesis that most of the objects are upright within a scene. The depth map generator 140 can also be configured to estimate whether the foreground object is upright, floating, or suspended within the actual scene displayed in the 2D image 144, and based on the above estimate, appropriately establish a point on the boundary of the foreground object . Alternatively or additionally, the user input 120 can be configured to enable the user to manually indicate the point at which the foreground object is in contact with its environmental space, for example by clicking on the point in the 2D map 144.

It is noted that the term suspension is understood to include both suspensions from a ceiling and suspensions from a wall (eg, wall-mounted). Therefore, when it is determined that the foreground object is suspended, The depth map generator 140 may additionally determine that the foreground system is suspended from a ceiling or suspended from a wall.

Alternatively or additionally, the user input 120 can be configured to enable the user to determine that the foreground system is upright or hung in the actual scene displayed in the 2D image 144, and the depth map generator 140 can be configured to A point is established on the boundary based on the upright or suspension. For example, in the case of FIG. 5e, the user can indicate that the person 192 is upright in the room, and thus, the depth map generator 140 can establish a point on the lower boundary 197 of the depth representation 196 of the person 192, and subsequently The depth of the global depth distribution 360 at that location is used as the foreground depth. In essence, depth map generator 140 may propagate the depth at a point on the lower boundary 197 of the foreground object upward to fill the depth representation of the foreground object with the foreground depth. It is noted that since the global depth distribution 360 has been established as the depth map 142 in the previous step, the foreground depth may also be established from the corresponding point in the depth map 142.

FIG. 5f shows an example of a user additionally selecting a light 194 in the 2D image 144. The user can further indicate that the foreground object (i.e., light 194) is suspended in the room. Subsequently, depth map generator 140 may establish a point on boundary 199 above depth representation 198 of lamp 194 and propagate the depth at that point down on upper boundary 199 in the manner previously described. Depth map generator 140 may also automatically determine that the foreground system is suspended or erect, for example, based on the top or bottom of the foreground object being located in an upper or lower half of one of the 2D images 144. The depth map generator 140 may also consider the depth of the space surrounding the foreground object to determine whether the foreground system is suspended or erect. For example, the depth map generator 140 may determine that the person 192 constitutes an environmental space erected thereto based on the bottom of the person 192 shown in FIG. 5d being located on a lower depth gradient that may constitute one of the floors within the global depth distribution. One of the objects.

The device 100 thus enables the user to repeatedly create a depth map 142 of a 2D image 144 by first establishing a global depth profile as the initial depth map 144, and then selecting one or more of the depth maps 144 to be included. A foreground object. As a result, one of the depth maps 142 is obtained to be repeatedly optimized.

Figure 6a shows another example of a geometric template 300 that includes depth information. Furthermore, one of the changes in the depth information is represented roughly by a change in the linear density, that is, a high density indicates a large depth and a low density indicates a small depth or close. Figure 6a further shows a marker 320 that constitutes a vanishing point in the global depth distribution 360 as defined by the geometric template 300. Figure 6b shows an example of a 2D image 144 comprising one of the actual scenes, wherein the global depth of the actual scene can be well established by the global depth distribution of the geometric template 300 of Figure 6a. Accordingly, the user can place the marker 320 in the 2D image 144 to match the vanishing point of the global distribution to one of the actual vanishing points displayed within the 2D image 144.

In general, user input 120 can be configured such that the user can add markers to a plurality of markers and/or remove markers from a plurality of markers. For example, the user can add a marker by clicking on a connection 340 between a first marker 320 and a second marker 322, wherein a third marker is added at the location and the connection 340 is separated For the two connections, the first connection connects the first marker 320 with the third marker, and the second connection connects the third marker with the second marker 322. However, alternative methods of adding and/or removing markers can also be used.

In general, a plurality of markers and the one or more connections may form a line segment, i.e., it includes two markers and one of the connections between the two markers. The plurality of markers and the one or more connections may also form a fold line, ie, include a plurality of connecting line segments. The plurality of markers and the one or more connections may also form a polygon, i.e., it includes a plurality of connecting segments that form a closed shape. A user interface can be provided that allows a user to select among, for example, the aforementioned line segments, polylines, or polygons to enable the user to select between associated geometric templates. Alternatively, the geometric templates can be represented by names, and when selected, a line segment, polyline, or polygon is automatically provided. For example, when the user selects “Horizon”, a line segment can be provided. When the user selects “vanishing point”, a single marker can be provided, and when the user selects “indoor”, a rectangular polygon can be provided. .

It should be noted that the above-described embodiments are illustrative and not limiting, and those skilled in the art will be able to devise various alternative embodiments. In particular, it is noted that the invention is not limited to such geometric templates or global depth distributions that have been depicted in such figures. Of course, those skilled in the art will be able to design many advantageous alternatives.

In the scope of the patent application, any reference signs placed between parentheses shall not be construed as limiting the technical solution. The use of the verb "comprise" and its conjugations does not exclude the presence of elements or steps in the elements of the invention. The article "a" or "an" or "an" or "an" The invention can be implemented by means of a hardware comprising a plurality of different components, and a suitably stylized computer. In the device technical solution enumerating several components, several of these components can be embodied by one and the same item. Although certain measures are proposed in mutually different subsidiary technical solutions, it does not mean that one of these measures cannot be used in an advantageous manner.

100‧‧‧ device

110‧‧‧ display

120‧‧‧User input

122‧‧‧ position

130‧‧‧User interface device

132‧‧‧Location information

140‧‧‧Depth map generator

142‧‧Deep map

144‧‧‧Two-dimensional [2D] imagery

160‧‧‧ Display output

162‧‧‧ Display information

164‧‧‧ cursor

320‧‧‧ marker

Claims (15)

A device (100) for generating a depth map (142) based on a two-dimensional [2D] image (144) to enable three-dimensional [3D] visualization of the 2D image, comprising: a display output (160) for Displaying one of the 2D images (320), the marker corresponding to one of a geometric template (300), the geometric template including depth information for defining a global depth distribution (360) of a reference scene; a user input (120) for enabling a user to place the marker in the 2D image to enable the user to match the geometric template to the component by placing the component with respect to the 2D image One of the actual scenes is displayed in the 2D image; and a depth map generator (140) is configured to establish the global depth distribution as the depth map based on the position (122) of the marker in the 2D image (142) . The apparatus (100) of claim 1, wherein the geometric template (300) defines a depth gradient, and wherein the depth map generator (140) is configured to be based on the location (122) of the marker (320) The depth gradient in the depth map (142) is established. The device (100) of claim 2, wherein the location (122) of the marker (320) constitutes a start point or an end point of the depth gradient. The device (100) of any one of claims 1 to 3, wherein the location (122) of the marker (320) constitutes a vanishing point in the global depth distribution (360). The apparatus (100) of any one of claims 1 to 3, wherein the marker (320) is one of a plurality of markers (320-326), wherein the plurality of markers correspond to define the global domain together a plurality of components of depth distribution (360), and wherein the user input (120) is configured to enable the user to individually place each of the plurality of markers to adjust the global depth profile to adapt the The actual scene in the 2D image (144). The apparatus (100) of claim 5, wherein the display output (160) is configured to display one or more connections (340) between the plurality of markers (320-326), and wherein the depth map is generated The device (140) is configured to establish one or more edges of the global depth profile (360) based on a location of the one or more connections in the 2D image (144). The apparatus (100) of claim 5, wherein the plurality of markers (320-326) and the one or more connections (340) correspond to one of the global depth distributions (360). The device (100) of claim 5, wherein the plurality of markers (320-326) and the one or more connections (340) form at least one of: a line segment, a polyline, or a polygon. The device (100) of claim 5, wherein the user input (120) is configured to enable the user to add a marker to the plurality of markers (320-326) and/or from the plurality of markers (320-326) Remove the marker. The device (100) of any one of claims 1 to 3, wherein: the user input (120) is configured to enable the user to select one of the foreground objects (190-194) of the 2D image (144) The foreground object constitutes an object having a foreground depth in the 2D image, the foreground depth is not adequately represented by the global depth distribution (360); and the depth map generator (140) is configured to i) establish The foreground depth of the foreground object, and ii) based on the foreground depth, including a depth representation of the foreground object in the depth map (142) (196, 198). The apparatus (100) of claim 10, wherein the depth map generator (140) is configured to be based on the global depth distribution (360) and the depth representation (196, 198) of the foreground object (190-194) ( One of the depths at 197, 199) establishes the depth of the foreground. The device (100) of claim 11, wherein the user input (120) is configured to enable the user to determine that the foreground object (190-194) is erected or hung in the 2D image (144) Displayed in the actual scene, and wherein the depth map generator (140) is configured to establish a point on the boundary (197, 199) based on the upright or suspension. The device (100) of claim 10, wherein the user input (120) is configured to enable the user to provide the foreground depth to the depth map generator (140). A method (200) for generating a depth map based on a two-dimensional [2D] image to enable three-dimensional [3D] visualization of the 2D image, comprising: displaying (210) one of the 2D images, the mark Corresponding to one of the geometric templates, the geometric template includes depth information for defining a global depth distribution of a reference scene; enabling (220) a user placing the marker in the 2D image to enable The user can match the geometric template to an actual scene displayed in the 2D image by placing the component relative to the 2D image; and establishing based on a position of the marker in the 2D image (230) This global depth distribution is used as the depth map. A computer program product (260) comprising instructions for causing a processor system to perform the method of claim 14.