DE102015011427B4 - Image acquisition system and image evaluation system - Google Patents

Abstract

Image recording system and image evaluation system (10) for determining geometric points in space and / or object points, with a light field camera (12) and with an evaluation unit (20), which is adapted from the data of the light field camera 3D coordinates to selected object points and / or space points determine, characterized in that in a subsequent evaluation via a phase reconstruction, a wavefront is determined and from the wavefront for the selected space points and / or object points new 3D coordinates are determined with improved resolution.

Description

BACKGROUND OF THE INVENTION

FIELD OF THE INVENTION

The present invention relates to an image acquisition system and image evaluation system for determining geometric points in space and / or object points, with a light field camera and with an evaluation unit, which is adapted to determine from the data of the light field camera 3D coordinates to selected object points and / or space points

The invention further relates to a method for operating an image acquisition system and image evaluation system for determining geometric spatial points and / or object points, comprising the steps of aligning the image acquisition system with the light field camera to the measurement area to be acquired, taking an image with the light field camera, determining 3D coordinates selected spatial points and / or object points from the image data of the light field camera.

STATE OF THE ART

In the prior art, different image acquisition systems are known, which can capture 3D coordinates for selected spatial points and / or object points.

The recording of image stacks or layer images, which are each recorded at different distances from the measurement object, are a common method. Depending on the desired depth resolution, the distance between the individual images is selected and typically the same distance is used for the entire image stack.

Furthermore, camera-based stereoscopic measuring methods are used. In the case of a known arrangement of the at least two cameras relative to one another, 3D coordinates for selected spatial points and / or object points are then determined in the downstream image analysis via triangulation relationships. Such systems are used for example for the measurement of technical objects in industrial metrology or in medical technology.

The triangulation principle is also applied to plenoptic cameras. The terms plenoptic camera and light field camera are sometimes used interchangeably. In contrast to "normal" cameras, a plenoptic camera or even a light field camera not only records the light intensity of the light field in 2D incident on the individual pixels of an image sensor, but also the direction of the light rays from which the detected light field exists. Consequently, one speaks in plenoptic cameras or light field cameras partly from a 4D detection of the light field. This is typically achieved by additionally arranging an array of microlenses between the objective and the image acquisition sensor. Each lens of this microlens array images the light field detected by the lens in sections onto a partial region of the image sensor. The plenoptic sensor of the plenoptic camera, which consists of a microlens array and image acquisition sensor, is in fact divided into a plurality of laterally separated, separate microcameras via the lenses of the microlens array. The images taken by neighboring microcameras then at least partially overlap. The images of the neighboring microcameras are shifted depending on the direction of incidence of the light beams against each other and may also appear distorted. Therefore, it is possible to determine not only the 2D position of spatial points and / or object points but also their distance from the plenoptic camera. For this purpose, triangulation algorithms are used in the evaluation in order to reconstruct the original object location from the images of at least two separate microcameras. Thus, 3D coordinates can be specified for selected spatial points and / or object points. Plenoptic cameras are in the patents US 2015/0 138 402 A1 . WO 2015/100 105 A1 and EP 2 244 484 A1 described.

C. Perwaß et al., In "Single-lens 3D camera with extended depth-of-field", Proc. SPIE 8291, Human Vision and Electronic Imaging XVII, 829108 (17 February 2012); doi: 10.1117 / 12.909882 describe that by placing a microlens array in front of an image sensor from a normal camera, a 3D camera can be formed with a lens, can be changed in focus and angle when taking a picture in retrospect. While the concept of such plenoptic cameras has been known since 1908, the practical application of plenoptic cameras has become possible only because of the high computing power and cost-effective hardware available today, as well as the advances in the manufacture of microlens arrays. The authors present a detailed analysis of plenoptic cameras as well as the introduction of a new type of plenoptic camera with an extended depth of focus and a maximum effective resolution of up to a quarter of the sensor resolution.

Another approach is used in triangulation sensors. Widely used are laser triangulation sensors and sensors with fringe projection. With laser projection sensors, the measurement object becomes fan-shaped with one in a plane illuminated expanded laser beam. The image recording system with the camera is inclined by the triangulation angle with respect to the fan and arranged symmetrically to the fan. The optical system of the image recording system is designed as Scheimpflugsystem. In the case of sensors for strip projects, a striped pattern is projected onto the measurement object and the pattern resulting on the object surface is recorded with at least one camera. Typical structures of such sensors and applications are in the scriptures WO 2012/075 978 A1 . EP 2 282 164 A1 and DE 10 2013 011 848 A1 described.

Furthermore, focus methods are known in which selected areas of the measuring volume are brought into the focus of the image-receiving optics. To capture larger measurement volumes then image stacks are taken from images with different focal positions.

With chromatic sensors, different levels of the measurement volume are sharply imaged onto the sensor for different colors of the light. For separation of the color information, a confocal diaphragm is usually used, from which the emerging illumination light is imaged onto the measurement object with an optical system which has a known chromatic aberration in the direction of the optical axis. For the wavelength at which the focus of the illumination lies on the object surface, the light is more efficiently reflected back into the illumination beam path and imaged back onto the confocal diaphragm. Thus, the light color for the light reflected at the surface with the least loss is imaged again through the aperture. Behind the confocal aperture, the returning light is spectrally evaluated. The light wavelength with the highest intensity represents the object surface. The depth information of the measurement object appears corresponding to the color splitting of the light in the optics in a color coding in the image. Commonly used are different embodiments of chromatic sensors with punctiform, linear or planar measuring sensors.

The construction of the above-described plenoptic camera with a microlens array is similar to the construction of a Hartmann-Shack wavefront sensor. Hartmann-Shack sensors are known and used to determine the wavefront of an electromagnetic wave. The wavefront contains the complete information about the measured light field, ie the laterally resolved amplitude distribution and the associated laterally resolved phase distribution at each point of the wavefront. With the Hartmann-Shack wavefront sensor, a plurality of images are respectively recorded at axially different positions in the light field to be measured from individual and clearly separated luminous points in an image stack or slice image. To reconstruct the wavefront from these slice images or image stacks, various algorithms for phase reconstruction, also known as phase retrieval, are known. One was published by RG Lane in the conference proceedings of the IEEE International Conference on Image Processing in 1997 in the article entitled "Phase retrieval as a means of wavefront sensing". The patent DE 103 27 019 A1 describes the use of phase reconstruction for extended light distributions of unknown objects. For the reconstructed wavefront, key figures for the quality of the imaging optics used are subsequently determined.

DISCLOSURE OF THE INVENTION

OBJECT OF THE INVENTION

It is therefore an object of the present invention to provide an image acquisition system and image evaluation system and a method of the type mentioned, which allow efficient way to a higher measurement accuracy.

TECHNICAL SOLUTION

According to one aspect of the present invention, this object is achieved by an image acquisition system and image evaluation system of the type mentioned above by determining a wavefront in a subsequent evaluation via a phase reconstruction and new 3D coordinates with improved resolution from the wavefront for the selected spatial points and / or object points be determined.

According to a further aspect of the invention, this object is achieved by a method of the type mentioned above, with the further steps: reconstruction of the phase distribution of the light field recorded with the light field camera, evaluation of the reconstructed optical light field for determining 3D coordinates for the selected space points and / or object points.

The new image acquisition system and image evaluation system uses a plenoptic camera to record the measurement images. The plenoptic camera or light field camera consists of a combination of plenoptic sensor unit and a lens, which may optionally be designed to be changeable.

The 3D coordinates for selected spatial points and / or object points determined with the aid of the light field camera serve as starting values for the reconstruction of the phase of the light field recorded with the light field camera. From the reconstructed light field again 3D coordinates for the selected space points and / or object points are determined and these represent the geometric position of the selected space points and / or measurement points, which are determined using the new image recording system and image evaluation.

The plenoptic sensor unit consists of at least one image acquisition sensor and a microlens array. The arrangement of image acquisition sensor and microlens array to each other is known and can be done for example via a support structure, which can also be designed to minimize thermal effects. Preferably, the planes of the image sensor and the microlens array are arranged parallel to one another and at a defined optical distance. This defined distance can optionally be made variably adjustable so that it can be changed between successive image recordings. For this purpose, for example, the optical path length between the microlens array and the image acquisition sensor via the refractive index of the medium in this area and / or a change in the mechanical distance of the microlens array and the image acquisition sensor can be changed. Alternatively or additionally, the refractive power of the lenses in the microlens array can be made variable. The lateral extent of a microlens in the microlens array can be favorably chosen as an integer multiple of the edge length of a pixel of the image sensor. It is particularly preferred if the extent of a microlens corresponds to an odd-numbered multiple of the edge length of a pixel. It is particularly advantageous if the array of the plurality of microlenses and the pixel pattern of the image sensor are aligned with one another in such a way that the pitch between adjacent microlenses coincides with the pixel pitch of the image sensor. As a result, adjacent portions of the imaging sensor that receive the image from different microlenses are optimally separated from each other and minimizes possible interference between adjacent independent microcameras. In addition, the center of each subarea of the image sensor is thus centered on the optical axis of the respective microlens. With an odd number of pixels, the central pixel of each subarea of the image sensor is thus centered on the optical axis of the associated microlens. The arrangement of the microlenses in the microlens array can preferably take place in a rectangular, square or hexagonal arrangement in order to obtain the highest possible fill factor of the lens surface on the entire surface of the microlens array.

In a further refinement, the image acquisition system and the image evaluation system are designed to take into account the different imaging properties of the microlenses of the microlens array during the evaluation.

This embodiment can take into account the actual property of the microlenses in the microlens array. All microlenses can be the same or the arrangement of the microlenses can have a spatially known distribution with lenses of different refractive power. Furthermore, the microlenses may have different chromatic aberrations, which should be considered in the evaluation. Furthermore, it is possible that the refractive power of the microlenses for a recording can be variably adjusted. In the image acquisition system and image evaluation system, the information about the optical effect of each individual lens of the microlens array can be stored.

In a further embodiment, the image acquisition system and image evaluation system has a diaphragm grille which is embodied with webs or an array of pinhole diaphragms which covers the transitions from a microlens to the microlenses adjacent to the array.

This refinement advantageously makes it possible to reduce stray light effects or the crosstalk from the image area of a microcamera to a microlens in an adjacent image area of another microcamera with a further microlens. As a result, this refinement advantageously contributes to improving the signal-to-noise ratio in the imaging and thus to increase the measurement accuracy when determining the 3D coordinates with the light field camera.

In a further embodiment, the light field camera has a telecentric lens.

A telecentric lens has the special property that the so-called entrance pupil and / or the exit pupil lies at infinity. If both the entrance pupil and the exit pupil lie at infinity, then there is a two-sided telecentric objective. If only the entrance pupil lies at infinity, the objective is telecentric on the object side. If only the exit pupil lies at infinity, then the objective is telecentric on the image side. In the case of the telecentric imaging, the symmetry axes of the light cone used in the imaging are parallel to the optical axis of the optics for each object point and / or pixel. As a result, the image scale of the image within the telecentric operating range of the optics is largely independent of the object distance and remains approximately constant. This property of a telecentric optics can - contrary to previous expectations - also be used in combination with a plenoptic camera. This is confirmed by the results of investigations by the applicant, the even demonstrate the particular suitability of telecentric optics for metrological tasks with plenoptic cameras on different measurement objects and also with different wavelength ranges of the measurement wavelength in the visible and near infrared range. It is particularly advantageous if there is a specific set of calibration data for the plenoptic camera which the image acquisition system and image evaluation system contain, which is separate and independent of the calibration data of the objective used. For technical application, it is particularly advantageous if the lens can be coupled with the plenoptic camera stable and with known relative arrangement, so that a complete calibration record for the imaging system used from the calibration data set for the plenoptic sensor unit and the calibration data set for the lens used in the image acquisition system and image evaluation can be determined.

The measurement of object points refers to the detection and determination of 3D coordinates of points in a range of measurement objects accessible for optical measurements. The object points and their immediate surroundings are directly detectable with the plenoptic camera. For this purpose, the object point and its immediate surroundings must be sufficiently cooperative for the optical image, so that sufficient measurement signal, for example of structures, color changes and / or surface scattering of the measurement object due to roughness and / or textures, can be detected with the camera. For the imaging of object points, a homogeneous illumination of the object area to be detected is sufficient. For example, it is possible to use the natural illumination, the ambient light or the light of a lighting unit, which is additionally provided, for example, by the image recording system and image evaluation system in a further embodiment. A high-contrast image of the measurement object is the basis for the reliable function of the evaluation algorithm for the plenetic images for determining the first 3D data set and for calculating a picture stack.

The measurement of spatial points refers to the detection and determination of 3D coordinates of points in an area of a measurement object accessible for optical measurements. The spatial points and their immediate surroundings are not directly detectable with the plenoptic camera with an image sufficiently rich in contrast for the subsequent evaluation. Space points and their immediate surroundings are not a priori sufficiently cooperative for optical imaging. The new image acquisition system and image evaluation system alternatively and additionally has a lighting unit with a pattern projector, which is designed to project a defined light pattern with a multiplicity of spaced-apart light figures into the detection area of the light field camera. The light patterns can be generated by local variation of the illuminance and / or illumination wavelength. On the measurement object in the detection area of the light field camera, light patterns thus arise due to the illumination, which can be used for the subsequent evaluation. It is also possible to capture spatial points and object points in an image acquisition with the illumination with pattern projector.

In a further refinement, the image acquisition system and image evaluation system can also be designed to record a sequence of images with the plenoptic camera, in which different light patterns are respectively projected into the detection range of the plenoptic camera. The image acquisition system and image evaluation system can also be designed to associate the projected light pattern with the respective measurement image.

In a further refinement, the image acquisition system and image evaluation system can be configured to control the pattern projector in such a way that the projected light patterns with the adjacent light figures are selected such that the measurement object is detected with a predetermined number of spatial points in the detection range of the plenoptic camera. Particularly preferably, the image acquisition system and image evaluation system for determining the light patterns for the control of the pattern projector can use the results of an earlier image acquisition and image evaluation of an image of the plenoptic camera.

In this embodiment, the control with adapted light patterns is particularly advantageous, especially with unknown measurement objects or known measurement objects in an unknown arrangement in the detection range of the plenoptic camera. In addition, an adaptation of the light pattern offers the advantage of minimizing the number of images required for a specific measurement accuracy. This minimizes the required measurement time for a complete measurement and increases the productivity of the image acquisition system and image evaluation system.

The conventional method or functional principle for evaluating the images in the plenoptic camera can be described as follows for a rough overview: As already described above, each microlens of the plenoptic sensor unit forms a microcamera with an associated section of the image acquisition sensor. This micro-camera forms at least a section of the light field detected by the lens a section of the image sensor from. For each pixel, a direction from which the light beam has come for this pixel can then be determined by means of a back projection from the point of incidence of the light on the image acquisition sensor through the center of the microlens array used microlens used.

For adjacent microlenses, the image of the respective microcamera on the imaging sensor is shifted in position and / or size relative to that of the first-viewed camera and possibly distorted. Consequently, the back projection to the corresponding pixel for each of the adjacent microcameras results in a different tilted beam. All backprojected beams to images from adjacent microcameras on the imaging sensor containing the corresponding pixel intersect at an intersection point in an ideal optical system with no aberrations. This intersection determines the distance to a reference plane of the optical system or the plane of the image sensor and thus the third coordinate of the pixel. The lateral 2D coordinates of this pixel result from the image of that microcamera, where the corresponding pixel on the image sensor is located on the optical axis of the associated microlens. If this condition is not met exactly, the position from the images of neighboring microcameras can be interpolated accordingly. In order to determine the locations of corresponding pixels in the adjacent micro-camera images on the imaging sensor, known correlation methods and correlation algorithms are typically used in digital image processing.

In a further refinement, the image acquisition system and image evaluation system can be designed to provide an index of the accuracy of the determined depth information.

In this embodiment, the image acquisition system and image evaluation system to increase the measurement accuracy is further developed to take into account the technical limitations of real optical systems. It may happen that the back-projected rays of a pixel do not intersect or not all in one point. Each of these two back-projected rays either intersect at one point or are skewed to each other. In the case of wind-shafts, the location of the shortest distance can be taken as a substitute for the intersection, for example by determining the center of the shortest connection between the two skewed beams. Consequently, an area is formed in which the intersections or substitute points for intersections lie with the backprojected beams. The size of this area and in particular its lateral and longitudinal extent can serve as a quality criterion or characteristic number for the accuracy for the determined distance to the reference plane and thus for the depth information.

In a further refinement, the image acquisition system and image evaluation system can be designed to perform edge, shape, object and / or structure recognition in the images of the cameras on the image acquisition sensor for more precise determination of the 3D pixels on the image acquisition sensor. Such algorithms are known and described in detail in industrial image processing or are readily available in open source software such as OpenGL (www.opengl.org) as executable computer code. As a result, the individual pixels of selected object points and / or spatial points can be determined more precisely. Typically, the gain in accuracy through the averaging process of one-step structure determination is in the range of 5 to 20 and is often reported as a factor of 10 in the literature. If more complex structures such as circles or other planar structures are determined, the accuracy gain can also reach a factor of 100 and greater.

In the context of this patent application, it is of particular importance that the increase in accuracy in the determination of object points and / or spatial points before the backprojection for determining the 3D coordinate can increase the accuracy of the determination of this 3D coordinate by a corresponding factor.

In a further refinement, the image acquisition system and image evaluation system can be designed to more precisely determine specific 3D coordinates of the pixels from selected image points and / or spatial points from the images of the microcameras on the image acquisition sensor of the plenoptic camera via edge, shape and / or texture recognition. In a preferred variant of the embodiment, an image stack having a multiplicity of image planes along the optical axis of the imaging optics of the plenoptic camera can be determined from the image data of the plenoptic camera.

In a further embodiment, the image acquisition system and image evaluation system can be designed to calculate the associated light field via a phase reconstruction to the 3D data of the plenoptic camera. Particularly preferably, the characteristics for the accuracy of the determined depth information can be taken into account in the phase reconstruction.

For this purpose, the image acquisition system and image evaluation system advantageously uses a iterative algorithm. An advantageous method for the phase reconstruction is in DE10327019 and in the cited Lane / Irwan publication, both of which are incorporated herein by reference. Advantageously, the information from the first evaluation in the form of 3D data and / or image stack serve to define starting conditions. Such starting values are typically required for the iteration calculation and also in the result when the calculated phase surface is converted in a so-called phase unwrapping into a mostly continuous phase surface without phase jumps of 2 pi. Overall, the starting values can be used to improve the convergence of the iterative algorithm and to advantageously reduce the number of iteration steps. With the phase reconstruction, the light field is completely determined, as it provides both the spatial information about the light distribution and the temporal information for the propagation of the light field. In this sense, the light field represents the optical and / or geometric properties of the measurement object. For this complete description of the light field, the term wavefront is also used.

In a further refinement, the image acquisition system and image evaluation system can be designed to provide a calibration data record for compensating geometrical distortions in the computationally determined light field and to carry out a correction.

When determining the light field, systematic geometric distortions can occur which can be determined with measurement objects of known geometry and arrangement and can be described using a correction data record. In subsequent measurements geometric distortions can then be corrected with a computational correction.

In a further embodiment, the image acquisition system and image evaluation system can be designed to evaluate the light field with respect to geometric properties of objects.

The light field contains the available information necessary to calculate the light propagation. Thus, the light field for the selected space points and / or object points can be evaluated and thus new 3D coordinates of the selected space points and / or object points can be determined, which have an improved accuracy compared to the 3D coordinates from the first evaluation. In the calculated light field, it is also possible, for example, to determine the best focus area that corresponds to the surface of the measurement object. Furthermore, for example, the surface normal to areas of the surface around the selected object points and / or spatial points can be determined. Thus, instead of or in addition to the height information from the 3D coordinate, direction information about the direction of the surface normal can also be determined. Alternatively, images from the measurement object can also be calculated from the light field, which would have produced a camera with a predetermined relative arrangement to the measurement object and given imaging parameters. For example, new image stacks with predetermined imaging parameters and distances of adjacent focal planes can be determined from the light field. The images thus obtained can be thoroughly analyzed using the known methods of image processing. Alternatively, the results for the 3D points may be provided as point clouds.

In a further embodiment, the image acquisition system and image evaluation system can be designed to be able to carry out the evaluation of 3D coordinates for selected object points and / or spatial points if the measurement object is embedded in a medium with refractive index n.

In this embodiment, the measurement object is embedded in an optical medium whose refractive index is greater than 1 and in particular greater than 1.1. In this case, the image acquisition system and image evaluation system is designed to take into account the refractive index of the medium during image acquisition and / or image evaluation. Due to the adapted evaluation, precise measurements of the 3D coordinates, e.g. for selected points in space and / or object points are embedded, which are embedded in gaseous, liquid or solid media. Examples of such media are water, oils, alcohols or other liquids, glass, crystals, plastics, resins or other solid particles which are at least partially transparent in the region of the measurement wavelength. The light field camera can be separated from the medium, for example, by a protective housing with viewing window for the lens, in which the measurement object is embedded. Alternatively, the plenoptic camera can look at the medium from the outside and take into account the interface between air and the medium in the subsequent evaluation. It is possible and advantageous if the plenoptic camera determines the interface between air and the medium via selected points in space on the surface and takes into account the so determined interface in the evaluation. The refractive index of the medium may be previously known or determined from the measurement of corresponding object points in the images of adjacent microcameras.

In a further embodiment, the image recording system and image evaluation system can be designed to be connected to a carrier system to take measurements on DUTs.

In this embodiment, the image acquisition system and image evaluation system and in particular the light field camera or the protective housing via an interface for coupling to a support system, which may be a tripod, a robot or generally a displacement unit to the light field camera at locations in space at a predetermined position with predetermined Orientation of the optical axis of the light field camera to position an image capture.

In a further refinement, the image acquisition system and image evaluation system can be designed to store the results of the evaluations in a data record and / or to transmit or transfer them to another system for data processing and / or output to an output device and / or on a display system display.

It is understood that the features mentioned above and those yet to be explained can be used not only in the respective specified combination but also in other combinations or alone, without departing from the scope of the present invention.

list of figures

• 1 eine schematische Darstellung für ein Bildaufnahmesystem und Bildauswertesystem,
• 2 eine weitere schematische Darstellung für ein Bildaufnahmesystem und Bildauswertesystem mit einer telezentrischen Optik,
• 3 ein schematische Darstellung für Beispiele möglicher Anordnungen von Mikrolinsen im Mikrolinsenarray, (a) quadratisch, (b) hexagonal, (c) quadratisch mit 2 Linsentypen, (d) rechteckig mit unterschiedlich gekrümmten Linsenflächen in Richtung der horizontalen (1) und vertikalen (2) Achse in der Zeichenebene, (e) quadratisch mit einer veränderlichen Brechkraft, die beispielsweise über ein elektrisches Feld eingestellt werden kann, das über Elektroden angelegt wird,
• 4 eine Prinzipdarstellung für einen Bildstapel,
• 5 eine vereinfachte Darstellung für ein Bildauswertesystem für eine Auswertung (a) und für eine erweiterte Auswertung (b) des Bildes der plenoptischen Kamera,
• 6 eine vereinfachte Darstellung für ein Bildauswertesystem für eine Auswertung (a) und für eine erweiterte Auswertung (b) zur Phasenrekonstruktion und Bestimmung von 3D Koordinaten
• 7 eine schematische Darstellung für ein Bildaufnahmesystem und Bildauswertesystem mit einer telezentrischen Optik, mit einem Schutzgehäuse mit Sichtfenster um die plenoptische Kamera und in
• 8 eine Prinzipdarstellung zum Ablauf des Verfahrens für ein Bildaufnahmesystem und Bildauswertesystem.

The accompanying drawings show embodiments of the invention. They will be explained in more detail in the following description. Show it:

• 1 a schematic representation of an image recording system and image evaluation system,
• 2 a further schematic representation of an image acquisition system and image evaluation system with a telecentric optics,
• 3 a schematic representation of examples of possible arrangements of microlenses in the microlens array, (a) square, (b) hexagonal, (c) square with 2 lens types, (d) rectangular with differently curved lens surfaces in the direction of horizontal (1) and vertical (2) Axis in the plane of the drawing, (e) square with a variable refractive power, which can be adjusted, for example, by an electric field applied via electrodes,
• 4 a schematic diagram for a picture stack,
• 5 a simplified representation for an image evaluation system for an evaluation (a) and for an extended evaluation (b) of the image of the plenoptic camera,
• 6 a simplified representation for an image evaluation system for an evaluation (a) and for an extended evaluation (b) for the phase reconstruction and determination of 3D coordinates
• 7 a schematic representation of an image recording system and image evaluation system with a telecentric optics, with a protective housing with viewing window to the plenoptic camera and in
• 8th a schematic diagram of the procedure for an image acquisition system and image evaluation system.

EMBODIMENTS

Further advantages, characteristics and features of the following invention will become apparent in the following description of the embodiments. However, the invention is not limited to these embodiments.

In 1 is the new image acquisition system and image evaluation system in its entirety with the reference numeral 10 designated. The image acquisition system and image evaluation system 10 contains a plenoptic camera 12 , a lighting unit 14 , a sample projector 16 , an image capture control 18 , an evaluation unit 20 , for example, a first evaluation 20a and a second evaluation unit 20b contains a data output interface 24 , a communication interface 26 and a display unit 28 , The plenoptic camera 12 consists of a plenoptic sensor module 40 that from the image sensor 42 with a variety of pixels 43 , the microlens array 44 with a variety of microlenses 45 and a support body 48 (not shown here), and a lens 50 , A microlens 45 forms together with a number of pixels 43 a section of the image sensor 42 a micro camera 46 , The lighting unit 14 illuminates the coverage area 30 the plenoptic camera 12 with the light cone 32 at least partially. The pattern projector 16 creates a variety of light figures 34a . 34b . 34c in the coverage area 30 the plenoptic camera 12 be projected. The arrows in the illustration are examples of possible connections between the system components. These connections serve, for example, the power supply, the data transmission or the data exchange, for the transmission of control signals such as trigger signals, latch signals or time signals. These connections can be made, for example, by wire to the electrical and / or optical connection or wirelessly optically or electromagnetically by radio signals.

The plenoptic camera 12 and the pattern projector 16 together form a triangulation sensor in the preferred embodiments, with the aid of which 3D Spatial coordinates of spatial points and / or object points within the detection range of the light field camera can be detected with very high accuracy. In doing so, this triangulation sensor advantageously uses the property inherent in the plenoptic camera in order to be able to determine 3D coordinates to selected spatial points and / or object points, including the light figures 34a . 34b . 34c Based on the previously determined 3D coordinates of selected points in space and / or object points clearly distinguish and identify each other. About the triangulation of light figures 34 and the known triangulation angle between the plenoptic camera 12 and the pattern projector 16 For example, the 3D coordinates can be determined with a higher accuracy than the plenoptic camera alone. A further increase in the determination of the 3D coordinates can be achieved in the preferred embodiment, if in the sub-images of the micro-cameras on the image sensor for selected object points and / or space points together with the light figures 34 of the pattern projector 16 a geometry, structure and / or object recognition is performed.

2 shows the new image acquisition system and image evaluation system in its entirety in the event that the lens 50 the plenoptic camera 12 designed as a telecentric lens. The same reference numerals indicate - as well as in the following figures - the corresponding system components described. In the plenoptic sensor module is the support body 48 shown, the image sensor 42 , the microlens array 44 and possibly also the lens 50 connects with each other.

In 3 are several embodiments for the microlens array 44 shown. 3 (a) shows a microlens array 44 with a square arrangement of the microlenses 45 in a row and column structure.
3 (b) shows a microlens array in which the microlenses 45 with a circular cross-sectional area in a hexagonal structure for the closest spatial arrangement in the plane of the microlens array 44 are arranged.
3 (c) shows an extension of the 3 (a) , where in the spaces between the lenses a second type of microlenses 52 is arranged, which have other optical imaging properties. 3 (d) shows an array of elliptical cross-section microlenses in a rectangular array in which the lens surface in the example shown has a different radius of curvature in the horizontal 54 and the vertical direction 55 have the drawing level. The different radii of curvature lead to different powers of the microlenses in the two excellent directions.

In 3 (e) are additionally exemplary electrodes 56 and 57 shown, for example, on the opposite surfaces of the microlens array 44 are applied as transparent electrodes and, in the case of an electro-optical lens material, to a specific change in the refractive power of the microlenses via an electrical control 58 can be used. The electrical control 58 for the microlens array can be advantageous with the image pickup control 18 be connected. Even if in the representations of the 3 only refractive lenses are shown, lenses with diffractive effect, holographic lenses, graded index lenses, so-called GRIN lenses, or also controllable lenses with variable refractive power in the microlens array can be used in a corresponding manner. It is also possible to construct combinations of refractive, diffractive, holographic, controllable lenses and GRIN lenses, and to use these combinations as lens structures in the microlens array.

In 4 is schematically a picture stack 60 shown. It has a multiplicity N of mutually parallel image planes 62 on that with 62_1 to 62_N Marked are. The variety of image levels 62 is displaced along an axis orthogonal to the image planes, which is parallel to the propagation direction of the light field or the optical axis 64 of the camera system is located. The distance Δ of adjacent image planes is denoted by Δ _{i, i + 1} , where i = 1 to i = N-1. In principle, the distances between adjacent planes may be different. For practical reasons, however, usually equidistant distances are chosen.

The 5 and 6 show a simplified representation for an image evaluation system ( 20 ) of the novel image acquisition system and image evaluation system ( 10 ). Since the image evaluation for determining the 3D coordinates and the reconstruction of the phase of the light field is preferably iterative, in the image evaluation system in FIG 1 two parts shown by the reference numerals 20a and 20b are provided. It is of course also possible to evaluate the image of the plenoptic camera in the same part of the evaluation system 20 execute and evaluate in one step. The separate representation with the evaluation systems 20a and 20b is chosen for illustration purposes and can also take account of the fact that different computing systems that can also be physically separated are used for the different evaluations for 3D coordinates and phase reconstruction can. In addition, a division of the evaluation unit can lead to an accelerated determination of more precise 3D coordinates and thus enable a higher image sequence in the evaluation.

5 schematically shows the evaluation for the image of the light field camera 12 , 5a shows that from the image of the light field camera for selected object points and / or space points 3D Coordinates are determined. Furthermore, optionally from the image of the light field camera also a picture stack 60 be calculated. 5b shows the upstream geometry detection and / or pattern recognition and / or structure recognition, which in the 5b denoted with feature extraction.

6 schematically shows the phase reconstruction of the light field and determining the 3D coordinates to selected object points and / or space points, as in 6a shown. In the 5 certain results for 3D coordinates or a picture stack serve as an input quantity or starting values. 6b shows an extended process when several sets of coordinate values or image stack should be evaluated together. The representation is chosen schematically for n data sets.

In the embodiment of the 7 is the new imaging system and image evaluation system shown in its entirety in the event that the lens 50 the plenoptic camera 12 is designed as a telecentric lens and the plenoptic camera 42 with a protective housing 70 surrounded, which is a viewing window 72 for the lens 50 and the lighting unit 14 and the pattern projector 16 having. Between the unit of plenoptic camera 12 , Objective 50 , Lighting unit 14 and sample projector 16 in the protective housing 70 and the image pickup control 18 is over the double arrow 74 a protected connection shown schematically.

The image pickup sensor 42 has a logarithmic characteristic in some of the embodiments. This means that of the pixels 43 At least in sections, the pixel values generated depend on the intensity of the incident light in logarithmic or quasi-logarithmic characteristics. In this case, a quasi-logarithmic characteristic can be approximated, for example, by at least two linear characteristics of different sensitivity, between which switching takes place as a function of the intensity of the incident light.

8th schematically shows a method 80 for operating an image acquisition system and image evaluation system for determining geometric points in space and / or object points. The process involves the following steps: 82 Aligning the image recording system with the light field camera on the measuring range to be detected, 84 Adjusting the lighting, 86 Taking a picture with the lightfield camera, 88 Determining 3D coordinates to the selected spatial points and / or object points from the image data of the light field camera, 90 Determination of a picture stack from the data of the light field camera, 92 Reconstruction of the phase distribution of the light field recorded with the light field camera and 94 Evaluation of the reconstructed optical light field for determining 3D coordinates for the selected spatial points and / or object points. The final step of the process shows the 96 Data provision or data output for the 3D coordinates and / or the light field and / or a picture stack.

Although the present invention has been described in detail with reference to the embodiments, it will be understood by those skilled in the art that the invention is not limited to these embodiments, but rather modifications in the manner that individual features omitted or other combinations of features can be made as long as the scope of protection of the appended claims is not abandoned. The present disclosure includes all combinations of the features presented.

LIST OF REFERENCE NUMBERS

10

Image acquisition system and image evaluation system

12

Plenoptic camera, light field camera

14

lighting unit

16

pattern projector

18

Image capture control

20

evaluation

24

Output data interface

26

Communication Interface

28

display unit

30

Detection range of the plenoptic camera

32

Light cone of the illumination unit 14

34

Light figures of the pattern projector 16

40

Plenoptic sensor module

42

Image sensor

43

Pixels in the image sensor

44

Microlens array

45

Lens in the microlens array

46

micro camera

48

supporting body

50

lens

52

microlenses

54

Radius of curvature of the microlens 52 in the horizontal direction

55

Radius of curvature of the microlens 52 in the vertical direction

56

Electrode 1

57

Electrode 2

58

control

60

stacks

62

image plane

64

optical axis

70

housing

72

window

74

protected connection

80

Method for image acquisition system and image evaluation system

82

Align the imaging system

84

Adjust the lighting

86

Take a picture with the light field camera

88

Determining first set of 3D coordinates from data light field camera

90

Determining image stacks from data light field camera

92

Phase reconstruction to the light field recorded by light field camera

94

Evaluation light field with phase values for determining 3D coordinates

96

Deploying or outputting results

Claims (19)

Image recording system and image evaluation system (10) for determining geometric points in space and / or object points, with a light field camera (12) and with an evaluation unit (20), which is adapted from the data of the light field camera 3D coordinates to selected object points and / or space points determine, characterized in that in a subsequent evaluation via a phase reconstruction, a wavefront is determined and from the wavefront for the selected space points and / or object points new 3D coordinates are determined with improved resolution. Image acquisition system and image evaluation system according to Claim 1 characterized in that the image recording system has a lighting unit (14) which is designed to illuminate the detection area of the light field camera at least in sections. Image acquisition system and image evaluation system according to Claim 1 or 2 characterized in that the imaging system comprises a pattern projector (16) adapted to generate a defined light pattern having a plurality of spaced-apart light patterns (34) in the detection area of the light field camera. Image recording system and image evaluation system according to one of Claims 1 or 3 , characterized in that the image evaluation system (20) is adapted to identify a defined light pattern (34) from a plurality of spaced-apart light figures based on the 3D coordinates. Image recording system and image evaluation system according to one of Claims 1 to 4 , characterized in that the light field camera (12) comprises a 2D image sensor (42) having a plurality of pixels (43) and a microlens array (44) having a plurality of microlenses (45) disposed on the object side in front of the 2D image sensor and comprising the Multitude of pixels (43) subdivided into pixel arrays. Image recording system and image evaluation system according to one of Claims 1 to 5 , characterized in that the light field camera (12) has a telecentric optic (50). Image recording system and image evaluation system according to one of Claims 1 to 6 , characterized in that in the pixel arrays features on a structure recognition and / or geometry detection and / or edge detection are determined and based on these features 3D coordinates to selected spatial points and / or object points are determined. Image recording system and image evaluation system according to one of Claims 1 to 7 , characterized in that from the image of the light field camera, an image stack with a plurality of image planes at different distances to a reference plane is determined. Image recording system and image evaluation system according to one of Claims 1 to 8th , characterized in that in the pixel arrays a structure recognition and / or geometry recognition and / or edge detection for determining 3D coordinates for object points and / or space points is carried out, and then an image stack (60) with a Variety of image planes (62) is determined at different distances to a reference plane. Image recording system and image evaluation system according to one of Claims 1 to 9 , characterized in that the distance of adjacent image planes in the image stack is in the range of 0.3 times to 1 times the value of the depth of field of the imaging optics used. Image recording system and image evaluation system according to one of Claims 1 to 10 , characterized in that to the 3D coordinates for the selected object points and / or space points, a ratio for the quality of the depth information is determined. Image recording system and image evaluation system according to one of Claims 1 to 11 , characterized in that it comprises protective housing (70) and a viewing window (72) for the light field camera (12). Image recording system and image evaluation system according to one of Claims 1 to 12 , characterized in that it can take into account in the determination of 3D coordinates to selected object points and / or spatial points a medium with a refractive index greater than 1.1, in which the measurement object is at least partially embedded. Image recording system and image evaluation system according to one of Claims 1 to 13 , characterized in that the image evaluation system (20) is designed to jointly evaluate a plurality of image stacks (62) which have been recorded with different light figures (34) and / or different viewing directions of the light field camera on the measurement object. Image recording system and image evaluation system according to one of Claims 1 to 14 , characterized in that the evaluation system can make a correction of geometric distortions with predetermined correction values. Image recording system and image evaluation system according to one of Claims 1 to 15 , characterized in that the evaluation system can perform a correction of the optical properties of the microlenses (45) of the microlens array (44) and / or the objective (50) in the evaluation. Image recording system and image evaluation system according to one of Claims 1 to 16 , characterized in that the image sensing sensor belongs to the class of high dynamic range (HDR) sensors and / or sensors with linear and logarithmic (LinLog) sensitivity distribution. Method for operating an image acquisition system and image evaluation system for determining geometric points in space and / or object points, comprising the steps of aligning the image acquisition system with the light field camera to the measurement area to be acquired recording an image with the light field camera determining 3D coordinates to the selected spatial points and / or object points The image data of the light field camera characterized by the further steps: Reconstruction of the phase distribution of the light field recorded with the light field camera Evaluating the reconstructed optical light field to determine 3D coordinates for the selected space points and / or object points. Method for operating an image acquisition system and image evaluation system according to Claim 18 characterized by the further steps of: determining an image stack to the selected object points and / or spatial points reconstruction of the phase distribution of the optical field associated with the image stack.

Images