WO2009118156A2

WO2009118156A2 - M,method for generating a 3d-image of a scene from a 2d-image of the scene

Info

Publication number: WO2009118156A2
Application number: PCT/EP2009/002156
Authority: WO
Inventors: Markus Klippstein; Alexander Schmidt
Original assignee: Visumotion Gmbh
Priority date: 2008-03-27
Filing date: 2009-03-25
Publication date: 2009-10-01
Also published as: DE102008016553A1; WO2009118156A3

Abstract

The invention relates to a method for generating a three-dimensional image A3D of a scene S1 from a two-dimensional image A2D of the same scene S1, said scene S1 having objects at varying distances from the recording position. For the purposes of the invention, objects are bodies, figures characters, items or things which generally occur in a scene. According to invention the following method steps are provided: f) Creation or use of a data set of images of scenes Si (i=2...n), each image being provide with depth information, g) comparison of the image A2D with the images of the scenes Si (i=2...n) with regard to objects with identical or similar features, h) selection of at least one scene Sk (22D, i) transmission of the depth information for each object contained in the scene Sk to the object with comparable features in scene A2D, j) generation of a three-dimensional image A3D by relating the objects contained therein to the transmitted depth information.

Description

title

A method for generating a 3D image of a scene from a 2D image of the scene

FIELD OF THE INVENTION The invention relates to a method for generating a three-dimensional image of a scene from a two-dimensional image of the same scene, wherein the scene contains entities at different distances from the position from which the image of the scene was taken. For the purposes of the invention described below, entities are to be understood as bodies, figures, characters, objects or things which generally occur in scenes, and also details which are formed thereon.

State of the art

It is known to generate images using computers. For example, the term 2D computer graphics is used in connection with the generation of images whose contents can be displayed in one plane.

In 3D computer graphics, on the other hand, the image content is processed in three dimensions. The entirety of the entities to be displayed in a 3D computer graphic is called a scene, and the generation of an image or movie based on such scenes is known as rendering. 3D computer graphics can be represented as true 3D stereoscopic images. Various rendering techniques are applicable, such as raytracing, a ray-based obfuscation algorithm that determines the visibility of objects from a particular point in space, and radiosity, another method of calculating light propagation within one Scene.

3D computer graphics are created using commercial 3D CAD display programs such as Pro / ENGINEER or AutoVue. Especially with the latter is a Rendering in several 3D modes to display three-dimensionally recognizable technical structures in the form of a shaded model, a wire representation or in outline with hidden line removal.

With the increasingly available storage capacity associated with the advancement of computer technology, it is more recently sought to produce three-dimensional mappings of scenes from two-dimensional mappings. Thus, for example, WO 98/10584 A2 describes a "display system" for the sequential representation of two image planes, in which a first image source, which provides the foreground, and a second image source, which reproduces the background, with the aid of an optical beam splitter The entire scene is superimposed in such a way that the viewer perceives the background at a greater distance than the foreground and thus obtains a spatial impression For the optical adaptation of the aspect ratio between foreground image and background image, monofocal Fresnel cylinder lenses or cylindrical mirrors are used.

Under the name "Automatic Photo Pop Up", Siggraph 2005, an annual trade show for specialists in 2D and 3D graphics, has presented a process in which a single photograph is used to reconstruct a 3D view of the imaged objects in space becomes. It is estimated where in this photograph vertically aligned objects are and which parts of the picture belong to the ground or to the sky in order to determine from this knowledge the position of the horizon.

This and the aforementioned method has the disadvantage that the quality of the three-dimensional representations obtainable therewith does not meet higher requirements and, moreover, the manual proportion of the time required for producing the three-dimensional representations is relatively high.

DESCRIPTION OF THE INVENTION Based on this, the object of the invention is to develop a method for generating a three-dimensional image of a scene from a two-dimensional image of the same scene in which a better image quality is achieved with a higher proportion of automatable process steps.

According to the invention, in a method for generating a three-dimensional image A _{30 of} a scene S1 from a two-dimensional image A _{2D of} the scene S1, the following method steps are provided: a) creating or using a data set of images of scenes Si (i = 2... N), contain the entities at different distances from the shooting position and at b) comparing the image A _{20 of} the scene S1 with the images of the scenes Si (i = 2 ... n) with respect to the presence of entities having the same or similar Characteristics, c) Selection of at least one scene Sk (2 <k <n), in which entities occur that have the same or similar features of the entities that are also included in the image A _2D of scene S1, d) transmit the Depth information from the respective structures contained in the scene Sk on the structures with the same or similar features in the image A _{20 of} the scene S1, e) generating a three-dimensional image A _{3D of} the scene S1 by linking the entities contained therein with the transmitted depth information in the form a record.

Thus, the three-dimensional image A _{30 is} initially stored in a data record and can now be displayed visually for one or more observers with conventional display methods. For example, the display can be made on an electronically controllable LC display suitable for 3D displays, by means of a projector on a reflective screen (with the aid of 3D glasses for the observer) or on a holographic screen. The LC display is suitable for 3D displays, for example, when it is equipped with a parallax barrier screen or a lenticular screen, so that the light emitted from the display comes separately to both eyes of an observer.

The depth information in the sense of the present invention correspond in a figurative sense of the visual experience of humans; the method according to the invention uses this visual experience of humans. As such, another step of the method of the present invention is to obtain the set of images of scenes Si (i = 2... N) including the depth information from a plurality of three-dimensional or quasi-three-dimensional images generated in a conventional manner. such as images taken with 3D cameras or rendered 3D images.

To measure sizes and distances in images and thus to obtain the depth information, for example, a known under the name "VisualSize" software can be used, which requires only two different images of the same scene, which have been taken from different angles. The two images are computationally compared on the basis of identical image contents, and the result of this comparison is used to create a virtual scale, which is used to deduce sizes and distances or the depth information. The depth information is stored as codings, preferably in the form of gray tones, brightness values or color values, patterns, hatchings or similar characterizations, in the data records of the image stock of scenes Si (i = 2... N). For example, they can be assigned to the image in the form of a vector graphic.

The storage of the data set of the images, both of the stock of images of scenes Si (i = 2... N) and of the image A _2D , which is to be converted into a 3D representation, preferably takes place in electronic form, as a result of which the comparison Figure A ₂₀ with the individual illustrations of the stock can be made automatically using the electronic image data.

In the context of the invention is also the compressed storage of image data in electronic form as well as the storage in any other form that allows automation of the method according to the invention, such as storage based on optical data.

In the comparison, the similarity of the features of entities included in the map A _{20 is} evaluated with the similarity of the features of entities contained in the stored images, the similarity of the contours, the size, the color or Grayscale and / or structures is compared. Also, the comparison can only be based on image sections.

For this comparison, e.g. a method known as "scene completion" is used which was originally developed to supplement a scene obscured and thus invisible in a recorded scene by searching for scenes with similar image content in a plurality of different images, the missing entity as a frame taken from a comparable representation and is used in the image to be supplemented.

The correspondence of a found in the imaging stock illustration of a scene with the Sk shown in Figure ₂₀ A scene S1 is in this case determined on the basis of similar shapes, colors or gray steps and / or structures of the identified structure. With a maximum match found, the depth information from the entities contained in the respective image from the data set is transferred to the entities contained in the map A _2D by, for example, mapping A ₂₀ a so-called depth map, for example in the form of a vector graphic or a gray value image. In this way, a structure contained in the scene S1 is assigned the depth information which is assigned to a comparable entity from the images of scenes Si (i = 2... N).

Has e.g. a record of a map and an associated depth map in the form of a vector graphics memory 2 MB memory would be sufficient for the storage of a data supply of 10,000 images including the associated depth map, a storage capacity of 20 GB.

From the data set of the three-dimensional image A _{30 of} the scene S1, which consists essentially of the data set from the image A _2D and a depth map assigned to this image, several perspective views are now generated by means of known methods. For this purpose, for example, the teaching of WO 97/23097 "PARALLACTIC DEPHT-DEPENDENT PIXEL SHIFTS" is used in which pixel depths are converted into pixel shifts and from these shifts perspective views are generated, which include, for example, objects in image planes of different depths. The application of other einschlagiger method is of course possible and is within the scope of the present invention.

It is within the scope of the invention if in the data sets of the three-dimensional image A ₃₀ of scene S1 or scenes Si additionally undercut information, transparency information or other additional image or scene information of one-, two- or three-dimensional nature are stored for Further processing of the three-dimensional image A ₃₀ may be advantageous.

The data set of the three-dimensional image A ₃₀ of scene S1 or of scenes Si can be further refined by means of conventional depth estimation methods in the sense of a specification of the depth information. For example, the evaluation of shadows, perspective, occlusion, shape from motion or shape from shadow are used.

The method according to the invention is advantageously applicable to the spatially perceptible representation of moving or stationary objects within a scene. The application for the spatially perceptible representation of terrain sections that change relative to a viewer and that contain one or more structures at different distances from the viewer is within the scope of the invention.

The invention will be explained in more detail with reference to an embodiment. According to the invention, a multiplicity of images of scenes Si (i = 2... N) are stored in a preferably electronic memory in a first method step, which can take place in several substeps. Images of scenes of any kind are stored, representing entities such as bodies, figures, characters, objects or things that occur naturally or are artificially created. Since the number of naturally occurring or artificially created structures is finite in principle, they are likely to occur repeatedly in the large number of stored images.

As a rule, an image of a scene Si contains entities of different types, which also have different distances from the position in space within the scene Si from which the scene Si was recorded or - in the case of 3D computer graphics - apparently (virtual) recorded ,

The scenes Si to be stored are recorded, for example, with one camera or several cameras from at least two viewing angles, or they are generated as SD computer graphics with a commercially available software program for generating three-dimensional views, whereby an apparent (virtual) recording position in space results.

In each of these figures, depth information can be determined with reference to the depicted entities. Depth information in the sense of the present invention is preferably an equivalent for a distance indication between the pickup position and the respective structure. In addition, however, also depth information come into consideration, which correspond to a distance indication between a selectable fixed point and the respective structure. For the purposes of this invention, structures are also to be understood as constitutions which are constituents of bodies, figures, characters, objects or things, such as, for example, a canopy on a building, a branch on a tree, a handle on a door, and as details from entities to the scene.

The determination of the depth information from the images taken from several angles or the 3D computer graphics is carried out by means known from the prior art for image evaluation, eg photogrammetry or using conventional stereo matching algorithms or depth detection methods. The depth information obtained in this way is stored as coding associated with the corresponding structures or components of structures shown in the image of the relevant scene Si. The coding can be carried out, for example, in the form of shades of gray, brightness or color values, patterns, hatching or similar characterizations. The entirety of the depth information associated with the entities represented in an image is assigned to this mapping, for example, as a depth map in the form of a vector graphic. As soon as such a supply of stored images, which has been applied after the first method step, is present, the next method steps can be carried out. It is part of the inventive idea that this supply of stored images is always extended or supplemented, which has the advantageous consequence that the subsequent process steps with progressive supplementation of the stock can be performed more and more precise.

In the next method step, the two-dimensional image A _{2D of} a scene S1, which is also electronically stored, is compared with each one of the stored images of scenes Si, in view of the presence of entities having the same or similar features. For example, the features of the structures both in the stored images and in the two-dimensional image A _{2D are} characterized by their outlines, size, color or gray values and / or structures. By way of example, the aforementioned "scene completion" method can be used to carry out the comparison: If the two-dimensional image A _{2D contains,} for example, the representation of one or more buildings and one or more persons, the electronically stored image contents are combined with one another with the aid of this method The comparisons can only be based on the same or similar representations and the comparison can be made automatically using appropriate software and / or hardware.

As a result of the comparison, ideally, there will be correspondences in outline, size, color, and / or texture of the entities depicted in the two-dimensional map A ₂₀ with entities in one or more of the images of scenes Si within the memory found. Then, the depth information is taken from the memory and subsequently assigned to the images in the two-dimensional image A ₂₀ . If matching features for all structures shown in the two-dimensional image A ₂₀ can be found within a single stored image, this again corresponds to a-very possible-ideal case.

In general, however, and in particular when there is a relatively small number of images within the memory for the time being, entities with only similar features and only in several different ones of the stored images will be found. In these cases, the depth information is taken from the storage space and transferred to the structures in the two-dimensional image A _2D , which are the most similar to the structures shown there in terms of outlines, size, color or gray scale and / or structure. The totality of the depth information transmitted to the entities within the two-dimensional map A ₂ Q is, for example, again assigned in the form of a vector graphic to this mapping. As necessary, adjustments of the depth information are made to the size of the respective structure.

In a next method step, the data set of a three-dimensional image A _{30 of} the scene S1 is generated from the combination of the image contents within the two-dimensional image A _2D with the associated depth information according to conventional methods. This data set is now also stored in electronically stored form and can be transmitted via wired or wireless data networks to facilities with which a three-dimensional perceptible representation of the scene S1 for a viewer can be displayed.

The data set of a three-dimensional image A _{30 of} the scene S1 can be converted into a plurality of views for the purpose of the display, which can be perceived three-dimensionally by a viewer, for example according to the already mentioned method according to WO 97/23097.

Part of a display device suitable for the display can be, for example, an LC display that is equipped with a parallax barrier screen or a lenticular screen, so that the light emanating from the LC display is separated in part separately from the one and the other eye the observer is directed, which in turn perceives each viewer's eye different views.

Claims

claims

A method for generating a three-dimensional image A _{30 of} a scene S1 from a two-dimensional image A _{2D of} the scene S1, wherein the scene S1 contains entities at different distances from the position from which the image A _{2D was} taken, characterized by the following method steps a) applying or using a data set of images of scenes Si (i = 2... n) containing entities at different distances from the photographing position and in which each entity is assigned depth information as a measure of its distance from the photographing position; b) comparing the image A _{20 of} the scene S1 with the images of the scenes Si (i = 2 ... n) with respect to the presence of entities having the same or similar features, c) selecting at least one scene Sk (2 <k <n ), in which formations occur which have the same or similar features of the formations which are also contained in Figure A ₂₀ of scene S1, d) transmission the depth information from the respective parts contained in the scene Sk to the structures with the same or similar features in the figure A _2D of

Scene S1, e) generating a three-dimensional image A _{30 of} the scene S1 by linking the image contents contained therein with the transmitted depth information, preferably in the form of a data record.

2. The method of claim 1, wherein the supply of images of scenes Si (i = 2 ... n) including the depth information is obtained from three-dimensional images of these scenes.

3. The method of claim 1 or 2, wherein the stock of images and the image A ₂₀ are stored in the form of electronic image data, and the comparison of the image A ₂₀ with the individual images of the stock is made automatically based on the electronic image data.

4. Method according to one of the preceding claims, in which the equality or similarity of the features of entities is determined by comparing their contours, size, color or gray values and / or structure.

5. The method of claim 1 or 2, wherein the depth information within the stock of images of scenes Si (i = 2 ... n) as encodings, preferably in the form of shades of gray, brightness or color values, patterns, hatching or the like Characterizations in which records are stored.

6. The method of claim 5, wherein the images contained in the scene S1, the depth information are assigned by the coding, which is associated with a structure of the images of scenes Si (i = 2 ... n), a structure with the same or similar features in scene S1.

7. The method according to any one of the preceding claims, wherein the scene S1

- on an electronically controlled flat display,

- Is displayed spatially perceptible on a holographic screen by means of a projector on a as a reflective screen with the aid of 3D glasses for the viewer, or.

8. The method of claim 7, wherein the LC display is equipped with a parallax barrier screen, so that the light emanating from the LC display is directed separately to the two eyes of the viewer.

9. Application of the method according to claims 1 to 8 for the spatially perceptible representation of moving or stationary objects within a scene.

10. Application of the method according to claims 1 to 8 for the spatially perceptible representation of terrain sections that change relative to a viewer and contain one or more entities at different distances from the viewer.