DE102019201600A1 - Method for generating a virtual, three-dimensional model of a real object, system for generating a virtual, three-dimensional model, computer program product and data carrier - Google Patents

Abstract

The invention relates to a method for generating a virtual, three-dimensional model (10) of a real object (14) located in an environment (12), with the steps (S1-S4):
- Provision of camera data (16) in which at least part of the object (14) and / or part of the surroundings (12) is or are recorded (S1) by means of a camera device (18);
- Generating a virtual pre-model (20) on the basis of the camera data (16) (S2);
- Provision of at least one scan data set (22) in which at least part of the object (14) and / or part of the surroundings (12) is or will be detected by means of a scanner device (24) (S3); and
- Generating the virtual, three-dimensional model (10) of the object (14) at least by arranging the at least one scan data set (22) on the previous model (20) (S4).
The invention also relates to a system (33), a computer program product and a data carrier.

Description

The invention relates to a method for generating a virtual, three-dimensional model of a real object located in an environment. The invention also relates to a system for generating a virtual, three-dimensional model of a real object. The invention also relates to a computer program product by means of which a method for generating a model can be carried out. Finally, the invention relates to a computer-readable data carrier which in particular contains the computer program product.

Nowadays, due to digitization, there is a wide field of application in which it is useful to have a, in particular three-dimensional, digital copy of real objects and thus a virtual, three-dimensional model of the real object. For example, such a virtual model can be useful for inspection tasks, for navigation, in general for planning, for reverse engineering, in the area of asset management and beyond in other areas. The type of objects can be virtually any and range from particularly small to particularly large objects. Depending on the type of object from which the model is to be created, different technologies, both hardware and software, can be used.

There is already a large number of different 3D scanners that can already capture real objects, so that a virtual model can be generated from each of them. There is software for the scanners that enables manual or semi-automatic assembly of the scans. Some scanner manufacturers offer automatic digitization solutions, but these are limited to limited space for the object. It is thus already possible to obtain an automated solution which, however, is limited to controlled situations in an environment of the object and conceptually tends to fail in general application scenarios. There are a number of different types of 3D scanners, such as laser scanners, structured light sensors, which record the scans using different methods. The scans or individual scan data records are generally recorded as a three-dimensional point cloud of the object and / or a three-dimensional point cloud of the surroundings of the object.

The problem often arises that, in particular as a function of an object size of the object, several scans are necessary, in particular from different positions. In order to obtain a 3D reconstruction, that is to say the model, from this plurality, often in particular unorganized, scans, in particular through what is known as automatic stitching of the 3D scans, there are already a number of solutions which attempt to combine the various scans, and so on to get that model. There are different solutions for both industrial and academic applications.

A fully automated industrial solution requires excessive use of photogrammetry markers. For example, one manufacturer has an integrated system for scanning 3D objects with a limited size, in which markers that are attached to the object, the photogrammetry markers, are required and a robot arm is used to move a camera.

A disadvantage of this system is that it is not suitable for scanning large environments such as factories or other large objects. It is also very expensive.

On the software side, there are solutions in which a merging of 3D point clouds is made possible, whereby this works in particular for a small number of scans, but there must be a particularly large overlap of the scans and thus of the point clouds. Furthermore, this software is sensitive to dynamic scenes or environments in which, for example, an object is moving.

There is the possibility of merging point clouds, the merging taking place using the markers placed in the scene or environment or on the object. A large number of commercially available scanners, such as laser scanners or structured light scanners, are already supplied with software that can capture or generate the point clouds.

The solutions mentioned are all based on the paper "Fully automatic registration of multiple 3D datasets" published by Huber et al in Image and Vision Computing Issue ( 7th ), Vol. (21), pp 637 to 650 , from 2012. In the method presented by Huber et al, disordered 3D scans are aligned in a uniform overall 3D model. The method calculates key points of each individual scan and calculates an associated descriptor, this comparing the key points between overlapping scans. The method calculates which scans overlap. The model boundaries are then clipped on the basis of a visibility and space violation check. In the last step, a global optimization takes place, which is usually based on an iterative Iterative Closest Point (ICP) error measurement based.

The object of the present invention is to provide a method, a system, a computer program product and a data carrier, each of which gives the possibility of offering a completely automated, generic, easy-to-use method that is independent of the type of 3D scanner , in particular its resolution or field of view, to combine the scan data obtained by the scanner into a uniform digital or virtual, three-dimensional model of a real object in a particularly efficient manner.

According to the invention, this object is achieved by the subjects of the independent claims. Advantageous refinements and developments of the invention are specified in the dependent claims as well as in the description and in the drawing.

• In einem ersten Schritt des Verfahrens erfolgt ein Bereitstellen von Kameradaten, bei welchen zumindest ein Teil des Objekts und/oder ein Teil der Umgebung mittels einer Kameraeinrichtung, welche insbesondere beispielsweise als RGB-Kameraeinrichtung und somit sensitiv für den sichtbaren Bereich des elektromagnetischen Spektrums ausgebildet ist, erfasst wird beziehungsweise werden. Die Kameradaten umfassen somit insbesondere Bilder in Form von insbesondere Filmaufnahmen und/oder mehreren, einzelne Aufnahmen, auf welchen insbesondere zumindest ein Teil des Objekts, dessen Modell erstellt werden soll, abgebildet ist.

A first aspect of the invention relates to a method for generating a virtual, three-dimensional model of a real object located in an environment. In order to be able to generate the model particularly advantageously and thus in particular particularly precisely by means of the method according to the invention, the method comprises several steps:

• In a first step of the method, camera data is provided, in which at least part of the object and / or part of the environment is made by means of a camera device, which is designed in particular, for example, as an RGB camera device and thus sensitive to the visible range of the electromagnetic spectrum, is or will be recorded. The camera data thus include, in particular, images in the form of, in particular, film recordings and / or several, individual recordings, on which in particular at least part of the object whose model is to be created is mapped.

In a second step of the method, a virtual pre-model is generated using the camera data. In other words, information about the object contained in the camera data is used to form the virtual model using the information, for example by a method such as photogrammetry, in which the information can be obtained from the camera data, for example by measuring. In this case, the pre-model maps the object at least essentially three-dimensionally, at least in partial areas that have been captured by the camera data or are contained in the camera data.

In a third step of the method, at least one scan data set is provided, that is to say exactly one data set or, advantageously, several scan data sets are provided in which at least part of the object and / or part of the environment is provided by means of a scanner device, which in particular is a 3D Scanner can be designed, is or are detected. The captured part of the object and / or the captured part of the environment can be the same as or different from the part of the object captured by the camera device and / or the part of the environment captured by the camera device. The scanner device detects, for example, in particular three-dimensional point clouds of the part of the object and / or its surroundings, which can be stored or formed for each scan carried out, in particular as one of the scan data records.

In a fourth step of the method, the virtual, three-dimensional model of the object is generated at least by arranging the at least one scan data set on the previous model. This means that a respective orientation of the respective scan data set, which in particular captures point cloud information of at least a part of the object, is aligned on the pre-model generated using the camera data, whereby the spatial contained in the scan data sets in the event that more than one scan data set was captured Information about the object, in particular in the form of point clouds, are arranged relative to one another. The point clouds or scan data records can in particular be designed in such a way that, for example, there need not be any overlapping areas between the scan data records. With the method according to the invention, an arrangement relative to one another is possible in spite of the non-overlapping, in particular because of the preliminary model. After the arrangement or during the arrangement, a further refinement of the arranged scan data sets are carried out to each other. For example, generating the virtual, three-dimensional model of the object can essentially also be a two-stage step, with the arrangement of the respective scan data set or, in particular, of the point clouds contained therein or spatial information about the object analogous to point clouds taking place in the first stage . The scan data sets become one another in particular brought to a common coordinate system. In the second stage, the scan data records that have already been arranged are refined or smoothed relative to one another, so that the model generated describes the real object particularly well.

The provision of the camera data and the provision of the at least one scan data record can be carried out in the method according to the invention, for example, by the camera device or the scanner device, in particular in such a way that the recorded data, i.e. the camera data or the scan data or the respective scan data record, for example via an interface can be made available in which the respective data can be called up. In the method according to the invention, for example, the camera data, in particular live, can be made available and thus made available. Alternatively, the camera data or the at least one scan data record can, for example, already be stored on a storage device of an electronic computing device and thus made available by it, so that the respective data or the data record can be recorded independently of the method. The method can therefore in particular be implemented by computer.

The method provides a particularly advantageous, in particular automated, possibility of creating what is known as stitching, that is, the joining together of several scan data sets, which in particular each comprise a scan of particularly high quality, so that a virtual model of particularly high quality is generated can. The type of scanner device, and thus the method by which the scanner device detects the object and / or the surroundings, can be selected virtually freely. The method according to the invention is not restricted to one type of scanner device. However, the quality of the at least one scan data set has an influence on the quality of the virtual, three-dimensional model.

In other words, in the method, the scanner device captures, in particular mutually independent, scan data records which therefore do not have to be related and / or ordered to one another. Furthermore, the method uses a camera device, which can in particular be, for example, an ordinary RGB video camera, which allows filming in the surroundings and thus filming of the object. The object from which the virtual, three-dimensional model is to be generated is filmed. A camera from a portable device, such as a mobile phone or a tablet, for example, could be used as the camera device.

In the method according to the invention, information, in particular in the form of the camera data and the at least one scan data set, which originate from two different sources, namely one from the camera device and the other from the scanner device, is advantageously combined. The three-dimensional model can thus be generated or created in a particularly advantageous manner by the method. For this purpose, the camera data, in particular in the form of a video or video film of the object and / or the environment, in particular if the environment is also to be recorded, are used to generate the pre-model, for example by means of a structure-from-motion method , SfM). The SfM method is essentially a form of photogrammetric distance mapping technology, which can be used to generate three-dimensional structures from two-dimensional image sequences and thus the video can be used to create the model of the object.

It is advantageous that the object, which serves as a template for the model, is moved relative to the camera while the video is being generated by the camera device. The movement can be translatory and / or rotary. In the simplest case, the camera is moved, in particular on a known path. The SfM process can also be supplemented or alternatively carried out by the so-called SLAM process (Simultaneous Localization And Mapping, in German simultaneous position determination with map creation). With the SLAM method, the environment and thus the object are mapped and, at the same time, a relative movement between the camera device and the object is determined. Options for carrying out the SLAM process and / or the SfM process are already known and can be obtained, for example, as software. The virtual pre-model obtained therefrom is now used in a further step of the method to bring the entirety of the 3D scans, that is to say the scan data sets, which each include at least one scan, in particular in the form of a point cloud, to the same coordinate system. In this case, two scan data sets are not joined to one another in pairs, as was customary in the prior art, but according to the invention the scan data sets are arranged and thus the common coordinate system is generated on the pre-model. The arrangement of the scan data records and thus the generation of the virtual, three-dimensional model can advantageously be carried out using statistical methods such as probabilistic voting and / or, for example, using a point-pair feature method, such as the point using features. In this context, preliminary work on which the invention is based has shown that reliable voting systems (probabilistic voting) can be developed for an adaptation or arrangement of the scan data records to a CAD model. Thus, the possibility would also be given of combining or replacing the virtual pre-model generated using the camera data with an already, in particular at least roughly present, CAD model of the object as an alternative pre-model, in order to carry out the arrangement of the scan data records.

The pre-model generated on the basis of the camera data can be coarse, that is to say not particularly exact, since in the method according to the invention it is sufficient that the scan data sets are initially only roughly aligned or arranged on the pre-model. If such an alignment or the arrangement of a scan data record fails, this is visible to a user of the method, for example, if, for example, the preliminary model and / or the model are displayed by a display device, such as a screen. For example, a user can use other parameters in the propabilistic voting process and / or a scan data set which cannot be inserted into the model can be ignored, for example, when the model is refined. In the process step of generating the model, as already mentioned, there is the possibility of carrying out a further stage in which the, in particular roughly or simply, aligned scan data sets, which are each individually aligned with the pre-model, using an ICP method, for example be aligned. The pre-model can be completely removed and thus only the scan data sets, which are usually more precisely recorded in space, are used for further alignment, in particular with one another. The use of the alignment method, which in particular represents an optimization method, can be selected as desired for the global alignment of the scan data records with one another.

The method according to the invention has the advantage, on the one hand, that the method can be carried out particularly easily by a user, in particular because it can be automated, whereby the method is particularly user-friendly. Another advantage of the method is that the type of scanner device used to generate the scan data sets can be selected as desired and, moreover, even scan data sets from different scanner devices can be combined. An additional advantage is that these scan data sets, in particular the point clouds contained in them, do not have to overlap in order to be able to be arranged next to one another, whereby the method according to the invention also makes it possible in a particularly advantageous manner to generate the scan data sets independently of one another. The method according to the invention makes it possible in a particularly advantageous manner to combine camera data with, in particular, high-quality 3D scans in the form of the scan data records in such a way that a compilation or generation of the virtual, three-dimensional model of the real object that is to be recorded is automatically performed , can be created. Individual intermediate steps in the process are interchangeable and can be selected according to the best available methods or the best performance. Thus, the method makes it possible in a particularly advantageous manner to be able to carry out an entire scanning process, that is to say the acquisition and generation of the virtual, three-dimensional object, in a completely automated manner and, moreover, to enable complete and intuitive control of the method. The method also provides the possibility of separating the object from the surroundings in a particularly advantageous manner, so that the model has a particularly small number of artifacts which can occur, for example, when the surroundings are also recorded in otherwise customary methods.

In an advantageous embodiment of the invention, the camera data include a film, in particular a video film, and / or several recordings of the object, or in particular at least part of the object and / or at least part of the surroundings. A position of the camera device is varied relative to the object during the filming and / or between the recordings. In other words, the object and / or the part of the environment is or are recorded from several angles. A film comprises several, individual recordings, which have a fixed time interval between them. Alternatively, the recordings can be recorded without a fixed temporal reference to one another. If the camera data is a number of individual recordings, in particular without a fixed time sequence, the camera device is preferably moved between the recordings relative to the object, or the object can be moved relative to the camera. If a film is captured as the camera data, the camera device can be moved relative to the object, for example by means of a camera arm and / or the like, relative to the object, whereby the object can thus be captured from several angles. This has the advantage that the object is represented in the camera data captured from several perspectives, whereby information is advantageously obtained from the camera data from the camera data when the virtual pre-model is generated in order to be able to create the previous model particularly well, i.e. particularly precisely, for example.

In a further advantageous embodiment of the invention, the camera data include a position profile, in particular a movement profile, of the camera device. That is, the camera device has, for example, a position detection device which comprises a respective position or a respective movement of the camera device. In particular, a position of the camera device in space or relative to the surroundings can be recorded. This has the advantage that when generating the virtual pre-model based on the movement recorded for the recordings or film, the camera data can be used particularly advantageously to create the virtual pre-model, since there is a relationship between the respective recordings, in particular the film, and the particular position of the camera device can be produced particularly precisely during the recording, whereby methods such as photogrammetry can be used particularly advantageously.

In a further advantageous embodiment of the invention, a Lambertian reflection property is assigned to a surface of the real object recorded in the camera data. As a rule, the real object comprises at least one area, in particular a surface, by means of which it is in particular delimited from the surroundings. This surface can consist of different materials and accordingly have different reflective properties. Depending on the nature of the surface, a variation of the angle of view of the object can be direction-dependent, so the color and / or brightness of the surface of the object can be shown in recordings of the object from different positions, in particular depending on the light conditions in the vicinity of the object Camera data vary. This can lead to undesirable effects when generating or reconstructing the pre-model from the camera data, since the surface, for example due to different reflection properties in a first recording and in a second recording made from a different perspective, when generating the pre-model, for example by software, cannot be correctly recognized as identical. The Lambertian reflection property assigns a diffuse reflection property to each surface, so that the surface has the same brightness and / or color regardless of the viewing angle. If the reflection property is now assigned to the surfaces recorded by the camera data, the virtual pre-model can thereby be generated in a particularly advantageous manner.

In a further advantageous embodiment of the invention, the pre-model is generated from the camera data at least partially by photogrammetry and / or a structure-from-motion method, the structure-from-motion method (SfM method) and / or by means of simultaneous position determination with map creation, the SLAM process and / or a multi-view stereo process (a so-called multi-view stereo process, the MVS process). The MVS method, analogous to photogrammetry, is a method in which an object recorded in the recordings, in particular two-dimensionally, can be converted into a three-dimensional model from several recordings contained in the camera data. In contrast to conventional photogrammetry, for example, the MVS method can be used in such a way that no separate markers need to be attached to the object, which are used, for example, for photogrammetry to provide an orientation for joining the recordings to create the pre-model. The list made here of the possibilities for generating the pre-model is not to be regarded as complete, but several possibilities for generating the pre-model are shown. In this case, for example, which of the methods is used can be selected as a function of the positions of the camera device relative to the object. Furthermore, the selection for one of the methods or a combination of the methods can be made on the basis of, for example, the size of the object to be detected. This results in the advantage that the method according to the invention can be designed to be particularly flexible, so that the greatest possible number of different types of the real object can be converted into a virtual, three-dimensional model

In a further advantageous embodiment of the invention, the at least one scan data set is arranged by means of statistical methods, in particular the aforementioned propabilistic voting and / or a point-pair feature method, the so-called point per features method. By using at least one of the methods, the scan data set can be arranged and thus aligned in a particularly advantageous manner, in particular on the coordinate system of the pre-model. In other words, the individual scan data records can be transformed to a common coordinate system, as a result of which the virtual, three-dimensional model can be generated particularly advantageously.

In a further advantageous embodiment of the invention, the three-dimensional model is generated by means of an iterative next point method. The iterative next point procedure is referred to in English as the Iterative Closest Point method, or ICP method for short. The ICP process is essentially an algorithm that makes it possible to adapt point clouds, i.e. scan data sets, to one another. For the application of the ICP method, the point clouds should already be approximately aligned with one another, which is done in the fourth step of the method according to the invention when generating the virtual model, in particular using the above-mentioned statistical methods and / or the point-pair feature method. In other words, the ICP method is suitable for refining the model after the scan data records have been aligned on a common coordinate system, in particular, for example, the coordinate system of the pre-model. The ICP method thus makes it possible, in a particularly advantageous manner, to further refine or adapt the model so that, for example, flat surfaces that the real object has are actually present as flat surfaces in the virtual model. A preferred form of the ICP process is the so-called Levenberg-Marquardt-ICP process, or LM-ICP for short.

A second aspect of the invention relates to a system for generating a virtual, three-dimensional model of a real object located in an environment. In order to generate the model particularly advantageously by the system, the system comprises a camera device, by means of which at least part of the object and / or part of the surroundings can be detected and can be generated on the basis of the detection or on the basis of the captured camera data. The system further comprises a scanner device by means of which at least part of the object and / or part of the surroundings can be detected and at least one scan data set can be generated on the basis of the detection or on the basis of the detected. The system further comprises an electronic computing device, by means of which a virtual pre-model can be generated using the camera data, by means of which the virtual, three-dimensional model of the object can be generated at least on the basis of an arrangement of the scan data set on the pre-model.

In an advantageous embodiment of the invention, the camera device is designed to capture light in at least one area of the electromagnetic spectrum, in particular in the RGB area (RGB: red, green, blue), that is to say in the area visible to the human eye. In addition, the camera device can also be designed in such a way that it captures light, for example, in a multispectral range and can thus be used as a multispectral camera. Additionally or alternatively, the camera device could, for example, capture light in the infrared range and / or in the UV range. This makes it possible in a particularly advantageous manner to provide the camera data.

In a further advantageous embodiment of the invention, the scanner device is designed as a 3D scanner, by means of which point cloud information in particular can be recorded for the scan data set. In other words, the scanner device is designed at least essentially as a 3D scanner which is suitable for capturing the spatial, that is to say three-dimensional, structure of the object.

In a further advantageous embodiment of the invention, the scanner device comprises a laser scanner and / or a coordinate measuring device and / or a time of flight camera and / or a triangulation unit by means of which a triangulation can be carried out and / or a conoscopic system in which Interferences from light waves are used to detect the spatial structure of an object and / or a strip light topometry device, in which patterns of certain geometric shapes are drawn or projected onto the object by means of light. Depending on the shape of the object, a distortion of the pattern occurs, through which the spatial quality and thus the three-dimensional structure of the object can be determined. Furthermore, in a further advantageous embodiment, the scanner device additionally or alternatively comprises a modulated light 3D scanner (modulated light 3D scanner) and / or a tomography device, by means of which a tomography of the object can be carried out. This selection is also not conclusive and is intended to show that the system or the method according to the invention can be operated particularly advantageously with a scanner device or a 3D scanner which is designed in a wide variety of ways to detect the object. In particular, the scanner device is selected on the basis of the size of the object to be detected or the surroundings, in particular, for example, such as whether the surroundings are an indoor or an outdoor area. In addition, a combination of the scanner devices can be possible. The at least one scan data set can thus be generated in a particularly advantageous manner.

A third aspect of the invention relates to a computer program product according to the invention, which implements a method according to the invention on an electronic computing device. The computer program product can in particular be in the form in which it can be loaded directly into a memory or a memory area of the electronic computing device. The computer program product has Program means, in particular in the form of commands, which, when executed on the electronic computing device, cause the latter to carry out the method according to the invention.

A fourth aspect of the invention relates to a computer-readable data carrier. The data carrier according to the invention comprises control information stored on it, which at least include the computer program product according to the invention and / or are designed such that a method according to the invention is carried out when the data carrier is used in an electronic computing device.

The properties and developments of the method according to the invention specified so far and below, as well as the corresponding advantages, are to be applied mutatis mutandis to the system according to the invention, the computer program product according to the invention and / or the computer-readable data carrier and vice versa. For this reason, a respective explicit formulation of the respective individually named features is dispensed with both for the method according to the invention and for the system according to the invention, the computer program product and the data carrier.

Further advantages, features and details of the invention emerge from the following description of a preferred exemplary embodiment and with reference to the drawings.

• 1 ein schematisches Ablaufdiagramm eines Verfahrens zum Erzeugen eines virtuellen, dreidimensionalen Modells eines realen Objekts; und
• 2 eine schematische Ansicht eines Systems zum Erzeugen des virtuellen, dreidimensionalen Modells des realen Objekts.

It shows:

• 1 a schematic flow diagram of a method for generating a virtual, three-dimensional model of a real object; and
• 2 a schematic view of a system for generating the virtual, three-dimensional model of the real object.

• In einem ersten Schritt S1 des Verfahrens erfolgt ein Bereitstellen von Kameradaten 16, bei welchen zumindest ein Teil des Objekts 14 und/oder ein Teil der Umgebung 12 mittels einer Kameraeinrichtung 18 erfasst wird beziehungsweise werden. In einem zweiten Schritt S2 des Verfahrens erfolgt ein Erzeugen eines virtuellen Vormodells 20 anhand der Kameradaten 16. In einem dritten Schritt S3 des Verfahrens erfolgt ein Bereitstellen wenigstens eines Scandatensatzes 22, bei welchem zumindest ein Teil des Objekts 14 und/oder ein Teil der Umgebung 12 mittels einer Scannereinrichtung 24 erfasst wird beziehungsweise werden. Schließlich erfolgt in einem vierten Schritt S4 des Verfahrens ein Erzeugen des virtuellen, dreidimensionalen Modells 10 des Objekts 14 zumindest durch Anordnen des wenigstens einen Scandatensatzes 22 an das Vormodell 20.

1 shows a schematic flow diagram of a method for generating a virtual, three-dimensional model 10 one in an environment 12th real object located 14th . So that the model 10 can be designed particularly advantageously and thus an image of the real object that is as accurate as possible 14th the process comprises several steps:

• In a first step S1 camera data is provided during the process 16 , in which at least part of the object 14th and / or part of the environment 12th by means of a camera device 18th is or will be recorded. In a second step S2 During the process, a virtual pre-model is generated 20th based on the camera data 16 . In a third step S3 During the process, at least one scan data set is provided 22nd , in which at least part of the object 14th and / or part of the environment 12th by means of a scanner device 24 is or will be recorded. Finally, in a fourth step S4 of the method is generating the virtual, three-dimensional model 10 of the object 14th at least by arranging the at least one scan data set 22nd to the previous model 20th .

The method creates the possibility of the at least one scan data set in a particularly advantageous manner 22nd to create the model 10 particularly advantageous to orient spatially. Several scan data sets are advantageously used in the method 22nd are used, which advantageously in particular do not need to have an overlap area, since during the reconstruction or generation of the virtual model 10 an adaptation or arrangement to the previous model 20th can take place, whereby a so-called stitching, a stitching together, the scan data is particularly efficient. The method can in particular be automated by an electronic computing device 26th and thus in particular can be carried out in a computer-implementable manner.

So that the method can be carried out particularly advantageously, the camera data 16 a film and / or several recordings of the object 14th and / or at least part of the environment 12th detected, wherein a position of the camera device 18th during filming and / or between shots relative to the object 14th and thus especially to the environment 12th varies what by the arrow 28 is shown. This makes it possible to use the object 14th and / or the part of the environment 12th to be captured from several perspectives, thereby generating the virtual pre-model 20th is particularly advantageous. The camera data include so that the method can be carried out particularly advantageously 16 a position profile, in particular, for example, also a movement profile, of the camera device 18th , that is to say the movement or the position in which the camera is in relative to the object, particularly during recording or filming 14th has taken, are stored or retrievable, whereby the reconstruction or the generation of the virtual pre-model 20th based on the camera data 16 can be carried out particularly advantageously.

The creation of the virtual pre-model 20th can be carried out by at least one method, for example by means of a photogrammetry method in which, for example, markers when capturing the camera data on the object 14th are attached. In particular through the markers, there is thus a relation between individual recordings, which the camera data 16 include, in particular with regard to a spatial arrangement, possible, so that a construction of the pre-model 20th is possible. Especially when the camera data 16 Film recordings or video recordings and thus a tracking shot of the camera device 18th around the object 14th include, for example, the structure from motion method (in English Structure-from-Motion, SfM method) can also be used in a particularly advantageous manner. In this case, in a particularly advantageous manner, due to the change in the viewing angle due to the movement of the camera device 18th relative to the object 14th Conclusions about the three-dimensional structure of the object 14th are closed, which is therefore particularly advantageous in the previous model 20th knock down.

In addition or as an alternative, the SLAM process can also be used (SLAM Simultaneous Localization And Mapping, German: simultaneous position determination and map creation). The SLAM method is used in particular in robotics or when a robot finds its way around in an unknown environment, but it can also be used in a particularly advantageous manner to generate the pre-model 20th be used. An additional or alternative method is the multi-view stereo method (multi-view stereo, MVS method), which is essentially a further development of a photogrammetry method, which, however, in contrast to classic photogrammetry, for example, without extra on the object 14th to be attached marker is used to determine the position of the recordings relative to each other. In the MVS method, for example, certain key images are extracted from the recordings or from the video, which can be used to spatially assign the individual recordings to one another. The position profile of the camera device is advantageously 18th by an inertial sensor 30th detected. For example, the inertia sensor 30th be designed in such a way that during the recording or filming of the camera data 16 by the camera device 18th Position data are available virtually in real time, so that the previous model is more or less in real time thanks to a suitable process such as the SLAM process or the SfM process 20th can be generated.

The more precisely a camera trajectory is known and in particular the internal geometries of the camera device in addition or in addition 18th , such as focal length or the like, for example the MVS method can be carried out particularly well. The MVS method can be carried out using the selected key images in such a way that, compared to other photogrammetric methods, a minimized photogrammetric discrepancy between overlapping individual images or individual recordings in the camera data 16 can occur. Furthermore, a calculated 3D shape of the pre-model is created using the MVS method 20th assuming known camera poses and parameters, i.e. the position data or the other internal camera parameters, constant lighting and known material properties, the real object 14th reproduce particularly well.

In this case, the lighting is advantageously assumed to be constant and, in addition, it is assumed that the surface of the object 14th is a Lambertian surface, which means that its intensity and / or color changes with the change of the viewing angle between shots in the camera data 16 does not change. That is, the captured surface of the real object 14th is assigned a Lambertian reflective property.

The preliminary model obtained in this way, particularly advantageous through the MVS process 20th is thus already a three-dimensional model that is based on the scan data sets 22nd can now be refined in such a way that a particularly exact actual model 10 of the object 14th can be generated. For this purpose, the at least one scan data record is now arranged 22nd which advantageously has more than one scan data set 22nd especially the point clouds of the object contained therein 14th , on the previous model 20th . This essentially means that the scan data sets are learned 22nd a coordinate transformation so that, for example, it is based on the coordinate system of the pre-model 20th are oriented. The arrangement can advantageously take place using statistical methods, such as, for example, so-called probabilistic voting and / or advantageously using a point-pair feature method, which can be carried out as a so-called point per features.

This is the step S4 of the method is advantageously at least partially fulfilled, since the step S4 can advantageously be designed in two stages. So can in a second stage of the step S4 of the method advantageously by means of an iterative next-point method, that is to say the ICP method, which can be carried out in particular by means of the Levenberg-Marquardt algorithm, and an adaptation of the scan data sets 22nd to each other. That is, the previous model 20th will be used for further refinement and optimization of the model 10 no longer required, since after arranging and thus providing the common coordinate system for the entirety of the scan data sets 22nd , these can now be arranged relative to one another by means of the ICP method mentioned, whereby the model 10 becomes particularly precise.

2 shows a system in a schematic view 33 , by means of which the presented method can be carried out. To do this, the system includes 33 , which is used to create a virtual, three-dimensional model 10 one in an environment 12th real object located 14th serves, at least the camera device 18th , by means of which at least part of the object 14th and / or part of the environment 12th is or are detectable and based on the detection of camera data 16 are producible. The system also includes 33 the scanner device 24 , by means of which at least part of the object 14th and / or part of the environment 12th is or are detectable and based on the detection at least one scan data set 22nd can be generated. The system also includes 33 the electronic computing device 26th through which based on the camera data 16 a virtual model 20th can be generated and by which at least on the basis of an arrangement of the scan data set 22nd on the previous model 20th the virtual, three-dimensional model 10 of the object 14th is produced.

Here is the camera device 18th advantageously designed to capture at least one region of the electromagnetic spectrum, in particular the visible region, that is to say the so-called RGB region. This corresponds to the camera data 16 For example, normal photo or camera recordings in the visible range of light. Additionally or alternatively, the camera device 18th to, for example, the method in particular of ambient light conditions 12th to make independent, for example also be designed as a hyperspectral camera and / or an infrared camera or include such.

The scanner device is advantageous 24 designed as a 3D scanner through which for the at least one scan data set 22nd in particular point cloud information of the object 14th are detectable.

Here is the scanner device 24 for example designed in such a way that they in particular a laser scanner and / or a coordinate measuring apparatus and / or a time of flight camera, a so-called time-of-flight camera, and / or a triangulation unit and / or a conoscopic system and / or a strip light topometry device and / or a modulated light Includes 3D scanner and / or a tomography device. That is, the system 33 can have at least one arbitrarily designed 3D scanner or the method can be carried out with an arbitrarily designed 3D scanner, so that the method or the system 33 is particularly flexible to use. The choice of the type of 3D scanner or the scanner device is made 24 advantageously depending, for example, on the size of the object 14th or the nature of the environment 12th .

The method presented is advantageously designed in such a way that it can be implemented in a computer, for example in particular on the electronic computing device 26th , is. This includes the provision of the camera data 16 in step S1 or the provision of the at least one scan data set 22nd in step S3 in particular such that in a computer-implemented method camera data 16 and / or the at least one scan data set 22nd in a memory area of the electronic computing device, for example 26th are held and thus made available.

By the method or system presented 33 a possibility is created in each case by which, in a particularly advantageous manner, with virtually any, in particular also non-overlapping, scan data records 22nd , in particular automated, a particularly particularly precise, virtual, three-dimensional model 10 of a real object 14th can be generated in a particularly simple manner. It is particularly advantageous that in addition to the scanner device 24 the camera device in the process 18th is used so that, for example, in particular by RGB film recordings in the camera data 16 an automated stitching of the 3D scans, i.e. the scan data sets 22nd , is feasible. As shown, this involves the intermediate steps, for example generating the pre-model 20th in step S2 or the generation of the virtual model S4 , virtually arbitrarily, for example to the size of the object 14th adaptable and a correspondingly advantageous, in particular high-performance, method can be used for this in each case. For example, for the step S2 the MVS method can be applied and for the step S4 For example, a combination of statistical methods can be used for arranging and then optimizing in the second stage of the step S4 can be used by, for example, the ICP method. In addition, it is particularly advantageously possible for a user of the method to monitor the method and, if necessary, correct it, for example by varying the parameters for one of the methods mentioned, such as the MVS parameters, for example in addition to the automatic, completely independent implementation of the method to enable fine adjustment, for example.

The method described herein can also be in the form of a computer program or a computer program product, which the method within, for example, the electronic computing device 26th implemented. Likewise, there can be a computer-readable data carrier with computer-readable control information stored thereon, which comprises at least one described computer program product and is designed in such a way that, when the data carrier is used in, for example, the electronic computing device 26th perform the procedure described or the system described 33 can control.

List of reference symbols

10

model

12th

Surroundings

14th

object

16

Camera data

18th

Camera setup

20th

Previous model

22nd

Scan data set

24

Scanner setup

26th

electronic computing device

28

arrow

30th

Inertial sensor

33

system

Claims (13)

Method for generating a virtual, three-dimensional model (10) of a real object (14) located in an environment (12), with the steps (S1-S4): - Provision of camera data (16) in which at least part of the object (14) and / or part of the surroundings (12) is or are recorded (S1) by means of a camera device (18); - Generating a virtual pre-model (20) on the basis of the camera data (16) (S2); - Provision of at least one scan data set (22) in which at least part of the object (14) and / or part of the surroundings (12) is or will be detected by means of a scanner device (24) (S3); and - Generating the virtual, three-dimensional model (10) of the object (14) at least by arranging the at least one scan data set (22) on the previous model (20) (S4). Procedure according to Claim 1 , the camera data (16) comprising a film and / or several recordings of the object (14) and / or at least a part of the surroundings (12), a position of the camera device (18) during the filming and / or between the recordings being relative to the object (14) is varied. Procedure according to Claim 1 or 2 wherein the camera data (16) include a position profile of the camera device (18). Method according to one of the preceding claims, wherein a surface of the real object (14) recorded in the camera data (16) is assigned a Lambertian reflection property. Method according to one of the preceding claims, wherein the generation of the pre-model (20) from the camera data (16) at least partially by photogrammetry and / or structure-from-movement and / or simultaneous position determination with map creation method and / or multi- View stereo procedure is done. Method according to one of the preceding claims, wherein the arrangement of the at least one scan data set (22) takes place by means of statistical methods and / or a point-pair feature method. Method according to one of the preceding claims, wherein the generation of the three-dimensional model (10) takes place by means of an iterative next point method. System (33) for generating a virtual, three-dimensional model (10) of a real object (14) located in an environment (12) with: - A camera device (18) by means of which at least part of the object (14) and / or part of the surroundings (12) can be detected and camera data (16) can be generated on the basis of the detection; - A scanner device (24) by means of which at least part of the object (14) and / or part of the surroundings (12) can be detected and at least one scan data set (22) can be generated on the basis of the detection; - An electronic computing device (26) by which a virtual pre-model (20) can be generated based on the camera data (16) and by which the virtual, three-dimensional model (10) is at least based on an arrangement of the scan data set (22) on the pre-model (20) of the object (14) can be generated. System according to Claim 8 wherein the camera device (18) is designed to capture light in at least one region of the electromagnetic spectrum, in particular RGB. System according to Claim 8 or 9 wherein the scanner device (24) is designed as a 3-D scanner, by means of which point cloud information in particular can be recorded for the scan data set (22). System according to Claim 10 , wherein the scanner device (24) comprises a laser scanner and / or a coordinate measuring device and / or a time-of-flight camera and / or a triangulation unit and / or a conoscopic system and / or a strip light topometry device and / or a modulated light 3D scanner and / or a Includes tomography facility. Computer program product, which can be loaded into a memory of an electronic computing device (26), with program means which, when executed on an electronic computing device (26), initiate a method according to one of the Claims 1 to 7th execute. Computer-readable data carrier with control information stored on it, which at least one computer program product according to Claim 9 include and / or are designed in such a way that when using the data carrier in an electronic computing device (26) a method according to one of the Claims 1 to 7th is carried out.

Images