DE102008016553A1 - A method for generating a 3D image of a scene from a 2D image of the scene - Google Patents

Abstract

The invention relates to a method for generating a three-dimensional image A3D of a scene S1 from a two-dimensional image A2D of the same scene S1, the scene S1 containing images at different distances from the recording position. For the purposes of the invention, structures are bodies, figures, signs, objects or things that generally occur in scenes. According to the invention, the following method steps are provided: f) creating or using a data set of images of scenes Si (i = 2... N) in which each entity is assigned depth information, g) comparing the image A2D with the images of the scenes Si (FIG. i = 2 ... n) with regard to structures having the same or similar features, h) selecting at least one scene Sk (2 2D, i) transferring the depth information from the respective structures contained in the scene Sk to the structures having comparable features in the figure A2D, j) generating a three-dimensional image A3D by linking the entities contained therein with the transmitted depth information.

Description

The The invention relates to a method for generating a three-dimensional Illustration of a scene from a two-dimensional image of the same Scene, where the scene entities at different intervals from the position that contains the picture of the scene has been recorded. Under structures in the sense of the following described body, figures, characters, To understand objects or things that are common in scenes and also details that have been developed.

It It is known to generate images using computers. So will For example, the term 2D computer graphics related to the creation of images whose contents are in one Level.

In 3D computer graphics, on the other hand, make the image content three-dimensional edited. The entirety of the images to be displayed in a 3D computer graphic Entity is referred to as a scene, and the creation of an image or film based on such scenes is under the name Rendering known. 3D computer graphics can be considered true stereoscopic 3D images are displayed. There are different rendering methods applicable, such as raytracing, on the broadcast ray based algorithm for occlusion calculation and thus determining the visibility of objects from a particular point in the room, and the radiosity, also a method of calculation the spread of light within a scene.

to Generation of 3D computer graphics serve commercial 3D CAD viewing programs such as for example, Pro / ENGINEER or AutoVue. Especially with the latter Rendering is done in several 3D modes to technical entities optionally in the form of a shaded model, a wire representation or in outline with hidden line removal three-dimensional perceptible.

With the increasingly available storage capacity associated with the advancement of computer technology, it is more recently sought to produce three-dimensional mappings of scenes from two-dimensional mappings. For example, in WO 98/10584 A2 describe a "display system" for the rearrangement of two image planes in which a first image source, which provides the foreground, and a second image source, which represents the background, are superimposed by means of an optical beam splitter into an overall scene so that the viewer can see the background perceives at a greater distance than the foreground and in this way conditionally receives a spatial impression. For optical adaptation of the aspect ratio between the foreground image and the background image, monofocal Fresnel cylinder lenses or cylindrical mirrors are used.

Under the term "Automatic Photo Pop Up" is on the Siggraph 2005, an annual fair for specialists of 2D and 3D graphics, a procedure been presented in which a single photograph of a 3D view of the imaged objects in the room is reconstructed. It is estimated where in this photograph vertically aligned objects are and which image parts to the bottom surface or belonging to heaven, out of this realization the situation of the horizon.

this as well as the aforementioned method has the disadvantage that the quality of the achievable three-dimensional Representations does not meet higher requirements and also the manual share of the time required for the preparation of the three-dimensional representations required is relatively high.

From that The invention is based on the object, a method for generating a three-dimensional image of a scene from a develop two-dimensional image of the same scene, in which with a higher proportion of automatable process steps a better picture quality is achieved.

• a) Anlegen oder Verwenden eines Datensatzes von Abbildungen von Szenen Si (i = 2...n), die Gebilde mit unterschiedlichen Abständen von der Aufnahmeposition enthalten und bei denen jedem Gebilde eine Tiefeninformation als Maß für dessen Abstand von der Aufnahmeposition zugeordnet ist,
• b) Vergleichen der Abbildung A_2D der Szene S1 mit den Abbildungen der Szenen Si (i = 2...n) hinsichtlich des Vorhandenseins von Gebilden mit gleichen oder ähnlichen Merkmalen,
• c) Auswahl von mindestens einer Szene Sk (2 ≤ k ≤ n), in der Gebilde vorkommen, welche die gleichen oder ähnliche Merkmale der Gebilde haben, die auch in der Abbildung A_2D der Szene S1 enthalten sind,
• d) Übertragen der Tiefeninformation von den jeweiligen in der Szene Sk enthaltenen Gebilden auf die Gebilde mit gleichen oder ähnlichen Merkmalen in der Abbildung A_2D der Szene S1,
• e) Generieren einer dreidimensionalen Abbildung A_3D der Szene S1 durch Verknüpfung der darin enthaltenen Gebilde mit den übertragenen Tiefeninformationen in Form eines Datensatzes.

According to the invention, in a method for generating a three-dimensional image A _{3D of} a scene S1 from a two-dimensional image A _{2D of} the scene S1, the following method steps are provided:

• a) applying or using a set of images of scenes Si (i = 2 ... n) containing entities at different distances from the capture position and where each entity is assigned depth information as a measure of its distance from the capture position;
• b) comparing the image A _{2D of} the scene S1 with the images of the scenes Si (i = 2... n) with regard to the presence of entities having the same or similar features,
• c) selecting at least one scene Sk (2 ≤ k ≤ n) in which entities occur that have the same or similar features of the entities that are also included in the image A _{2D of} the scene S1,
• d) transferring the depth information from the respective structures contained in the scene Sk to the structures having the same or similar features in the image A _{2D of} the scene S1,
• e) Generate a three-dimensional image A _{3D of} the scene S1 by linking the entities contained therein with the transmitted depth information in the form of a data set.

Thus, the three-dimensional image A _{3D is} initially stored in a data record and can now be displayed visually with conventional display methods for one or more observers. For example, the display can be made on an electronically controllable LC display suitable for 3D displays, by means of a projector on a reflective screen (with the aid of 3D glasses for the viewer) or on a holographic screen. The LC display is suitable for 3D displays, for example, when it is equipped with a parallax barrier screen or a lenticular screen, so that the light emitted from the display comes separately to both eyes of an observer.

The Depth information according to the present invention correspond in the figurative sense of the visual experience of man; the inventive method uses this visual experience of the human. In this respect, there is a further step of the invention Method is to use the record of images of scenes Si (i = 2 ... n) including the depth information from one Variety of three-dimensional or quasi-three-dimensional pictures to win, which has been produced in a conventional manner such as images taken with 3D cameras or rendering generated 3D images.

To the Measuring sizes and distances in pictures and thus for obtaining the depth information can, for example one known as VisualSize Software that uses only two different images and the same scene needed, which differ from each other Viewing angles have been recorded. The two pictures are based identical image content computationally compared, and The result of this comparison becomes a virtual scale created with the on sizes and distances or on the depth information is closed.

The Depth information is called codings, preferably in the form of Shades of gray, brightness or color values, patterns, hatchings or similar characterizations, in the datasets of the Image stock of scenes Si (i = 2 ... n) stored. You can for example in the form of a vector graphic of the relevant image be assigned.

The storage of the data set of the images, both of the stock of images of scenes Si (i = 2... N) and of the image A _2D , which is to be converted into a 3D representation, preferably takes place in electronic form, as a result of which the comparison Figure A _2D with the individual illustrations of the stock can be made automatically using the electronic image data.

in the The scope of the invention is also the compressed storage the image data in electronic form as well as the storage in any other form that automates the invention Allow process, such as storage based on optical data.

In the comparison, the similarity of the features of entities included in the map A _{2D is} evaluated with the similarity of the features of entities contained in the stored images, the similarity of the outlines, the size, the color or Grayscale and / or structures is compared. Also, the comparison can only be based on image sections.

To this comparison can z. B. one known as "Scene Completion" Procedures are used, originally developed became one hidden in a recorded scene and therefore not to complement visible structure by adding in a variety from different pictures to scenes with similar picture content searched, the missing structure as image section from a comparable representation and inserted into the picture to be supplemented.

The match of an image of a scene Sk found in the image stock with the scene S1 shown in FIG. _2D is thereby determined on the basis of comparable outlines, colors or gray levels and / or structures of the recognized entities. With a maximum match found, the depth information from the entities contained in the relevant map from the data set is transferred to the entities contained in the map A _2D by, for example, mapping A _2D a so-called depth map, for example in the form of a vector graphic or a gray value image.

In order to For example, the depth information is given to a structure contained in the scene S1 assigned to a comparable entity from the illustrations of scenes Si (i = 2 ... n) are assigned.

Has z. B. a record from a map and an associated depth map in the form of a vector graphic 2 MB of memory space, so would for storing a data stock of 10,000 pictures including the respectively associated depth map one Storage capacity of 20 GB is sufficient.

From the data set of the three-dimensional image A _{3D of} the scene S1, which consists essentially of the data set from the image A _2D and a depth map assigned to this image, several perspective views are now generated by means of known methods. For this example, the doctrine of WO 97/23097 "PARALLACTIC DEPHT-DEPENDENT PIXEL SHIFTS", where pixel depths are converted into pixel shifts and perspective shifts are generated from these shifts, including, for example, objects in image planes of different depths. The use of other relevant methods is of course possible and is within the scope of the present invention.

It is within the scope of the invention if in the data sets of the three-dimensional image A _{3D of} the scene S1 or scenes Si also undercutting information, transparency information or other additional image or scene information one-, two- or three-dimensional nature are stored, for another Processing the three-dimensional image A _3D can be beneficial.

The data set of the three-dimensional image A _{3D of} the scene S1 or of the scenes Si can be further refined by means of conventional depth estimation methods in the sense of a specification of the depth information. For example, the evaluation of shadows, perspective, occlusion, shape from motion or shape from shadow are used.

The inventive method is advantageous to spatially perceivable representation moving or stationary Forms within a scene applicable. Also the application to spatially perceivable representation of terrain sections, that change relative to a viewer and the one or more structures at different distances Included by the viewer is within the scope of the invention.

The Invention will be described below with reference to an embodiment be explained in more detail.

According to the invention in a first process step, in several substeps can run a variety of pictures of scenes Si (i = 2 ... n) stored in a preferred electronic memory. saved become images of scenes Si of any kind, the entities, such as Body, figures, characters, objects or things represent that occur naturally or artificially are created. Since the number naturally occurring or artificially created structures in principle finally They are in the variety of stored pictures high probability also repeatedly.

In The rule contains an image of a scene Si entities different kind, the scene within the Si also different Have distances from the position in the room from which the Scene Si recorded or - in the case of 3D computer graphics - seemingly (virtual) has been recorded.

The For example, scenes Si to be stored are with a camera or multiple cameras from at least two angles, or they are called 3D computer graphics with a commercially available Software program generates three-dimensional views, where there is an apparent (virtual) recording position in space results.

In each of these illustrations can be based on the Represented entities - determine depth information. Depth information in the sense of the present invention is preferred an equivalent for a distance indication between the shooting position and the structure concerned. Come next but also depth information into consideration, that of a distance indication between a selectable fixed point and the relevant one Correspond to entities. Under structures in the sense of this invention are to understand also training, the components of bodies, Figures, characters, objects or things are, such as a canopy on a building, a branch on a tree, a Handle on a door, and as details of formations belong to the scene.

The Determination of depth information from several angles taken pictures or 3D computer graphics with known from the prior art means for image analysis, z. As the photogrammetry or using conventional Stereo matching algorithms or depth detection methods. The thus obtained depth information is encoded as the corresponding, in the image of the scene Si in question or components of structures assigned stored. The coding can be used, for example, in the form of shades of gray, brightness or color values, patterns, hatchings or similar characterizations be made. The set of depth information that are assigned to the structures shown in a figure, is for example as a depth map in the form of a vector graphic of this Map assigned.

As soon as such a supply of stored images, which has been applied after the first method step, is present, the next method steps can be carried out. It belongs to the idea of the invention that this supply of stored images is always extended or supplemented, which has the advantageous consequence that the subsequent process steps with progressive supplementation of the stock can be performed more and more precise.

In the next method step, the two-dimensional image A _{2D of} a scene S1, which is also electronically stored, is compared with each one of the stored images of scenes Si, in view of the presence of entities having the same or similar features. For example, the features of the structures both in the stored images and in the two-dimensional image A _{2D are} characterized by their outlines, size, color or gray values and / or structures. To execute the comparison, the aforementioned "scene completion" method can be used by way of example. If the two-dimensional image A _{2D contains,} for example, the representation of one or more buildings and one or more persons, the electronically stored image contents are compared with one another using this method and the same or similar representations are searched for in the image stored as a reserve. Also, the comparison can only be based on image sections. The comparison runs automatically using appropriate software and / or hardware.

As a result of the comparison, in the ideal case, matches in terms of the contours, the size, the color or gray values and / or the structure of the structures shown in the two-dimensional image A _2D with structures in one or more of the images of scenes Si within the memory found. Then, the depth information is taken from the storage and subsequently assigned to the formations in the two-dimensional image A _2D . If matching features can be found for all structures shown in the two-dimensional image A _2D with structures within a single one of the stored images, this in turn corresponds to a - perfectly possible - ideal case.

In general, however, and in particular when there is a relatively small number of images within the memory for the time being, entities with only similar features and only in several different ones of the stored images will be found. In these cases, the depth information is taken from the storage space and transferred to the structures in the two-dimensional image A _2D , which are the most similar to the structures shown there in terms of outlines, size, color or gray scale and / or structure.

The totality of the depth information transmitted to the structures within the two-dimensional image A _2D is, for example, again assigned in the form of a vector graphic to this image. As necessary, adjustments of the depth information are made to the size of the respective structure.

In a next method step, the data set of a three-dimensional image A _{3D of} the scene S1 is generated from the combination of the image contents within the two-dimensional image A _2D with the associated depth information according to conventional methods. This data set is now also stored in electronically stored form and can be transmitted via wired or wireless data networks to facilities with which a three-dimensional perceptible representation of the scene S1 for a viewer can be displayed.

The record of a three-dimensional image A _{3D of} the scene S1 can, for example, according to the three-dimensional perceptible representation for a viewer, for example, according to the aforementioned method according to WO 97/23097 be converted into multiple views.

component a display device suitable for the display For example, it can be an LC display with a parallax barrier screen or a lenticular screen, so that the separately from the LC display light partially to the one and the other eye of the observer is directed, which in turn every viewer's eye perceives different views.

QUOTES INCLUDE IN THE DESCRIPTION

This list The documents listed by the applicant have been automated generated and is solely for better information recorded by the reader. The list is not part of the German Patent or utility model application. The DPMA takes over no liability for any errors or omissions.

Cited patent literature

• WO 98/10584 A2 [0005]
• WO 97/23097 [0021, 0037]

Claims (10)

Method for generating a three-dimensional image A _{3D of} a scene S1 from a two-dimensional image A _{2D of} the scene S1, wherein the scene S1 contains entities at different distances from the position from which the image A _{2D was} taken, characterized by the following method steps: a ) Creating or using a set of images of scenes Si (i = 2 ... n) containing entities at different distances from the capture position and where each entity is assigned depth information as a measure of its distance from the capture position; b) Comparing the image A _{2D of} the scene S1 with the images of the scenes Si (i = 2... N) with regard to the presence of entities having the same or similar features, c) selecting at least one scene Sk (2≤k≤n), occur in the formations which have the same or similar features of the formations which are also contained in the Figure A _{2D of} the scene S1, d) Ov the depth information of the respective structures contained in the scene Sk on the structures with the same or similar features in the Figure A _2D scene S1, e) generating a three-dimensional image A _{3D of} the scene S1 by linking the image contents contained therein with the transmitted depth information , preferably in the form of a data record. The method of claim 1, wherein the supply of Pictures of scenes Si (i = 2 ... n) including the Depth information from three-dimensional images of these scenes is won. Method according to claim 1 or 2, in which - the stock of images and the image A _2D are stored in the form of electronic image data, and - the comparison of the image A _2D with the individual images of the stock is made automatically on the basis of the electronic image data. Method according to one of the preceding claims, in which the equality or similarity of the characteristics of Forms by comparing their outlines, size, Color or gray scale and / or structure is determined. Method according to Claim 1 or 2, in which the depth information within the stock of pictures of scenes Si (i = 2 ... n) as codings, preferably in the form of shades of gray, brightness or color values, patterns, hatchings or similar characterizations, stored in the records. The method of claim 5, wherein the in the scene S1 containing entities are assigned the depth information, by coding a picture from the pictures of scenes Si (i = 2 ... n), a structure with the same or similar Characteristics in the scene S1 is assigned. Method according to one of the preceding claims, where the scene S1 - on an electronically controllable Flat display, - by means of a projector on one as a reflective screen with the help of 3D glasses for the viewer, or - on a holographic screen is spatially perceptible. Method according to Claim 7, in which the LC display equipped with a parallax barrier screen, so that the from the LC display outgoing light separated to the two eyes of the Viewer is directed. Application of the method according to the claims 1 to 8 for the spatially perceptible representation of moving or immobile objects within a scene. Application of the method according to the claims 1 to 8 for spatially perceptible representation of terrain sections, that change relative to a viewer and the one or more structures at different distances included by the viewer.