DE102019008472B4 - Multi-lens camera system and method for hyperspectral imaging - Google Patents

Abstract

Multi-lens camera system (1) for hyperspectral recording of images, comprising a flat image sensor (3), a location-sensitive spectral filter element (4) and an imaging system, the spectral filter element (4) - comprising a linearly variable filter with filter lines and/or - a mosaic -Filter comprises, wherein the mosaic of the mosaic filter is arranged in such a way that large wavelength steps are on the inside, while smaller intervals are on the outside and wherein the imaging system comprises a flat lens matrix (2) with a multiplicity of individual lenses (2a) which are arranged in this way are that they generate a plurality of grid-like arranged first images (AS) of a motif (M) in a first area on the image sensor (3) at a first recording time, and additionally comprises a further lens (2b) which is in the plane of the lens matrix (2) or offset from the plane of the lens matrix (2) and is designed to produce a second image (AO) of the motif (M) in a second area of the image sensor (3) and/or at a second recording time the image sensor (3), the further lens (2b) having a focal line and/or a different focal length compared to the plurality of individual lenses, and being positioned such that its second image (AO) differs from the first images ( AS) differs at least in one surface dimension of the image sensor (3) with regard to the size and/or with regard to the image information of the motif (M) present in front of the filter element (4).

Description

The invention relates to a multi-lens camera system and a method for hyperspectral recording of images and in particular also for processing these images.

In many areas of business and science, cameras are used which, in addition to spatial resolution, also have a spectral resolution that often goes beyond the visible spectrum. For example, when surveying the earth's surface from the air, cameras are often used that not only have a normal RGB color resolution, but also deliver a high-resolution spectrum, possibly extending into the UV or infrared range. Using these measurements, for example, it is possible to identify individual planting areas in agricultural areas. This can be used, for example, to determine the state of growth or the health of plants or the distribution of various chemical elements such as chlorophyll or lignin.

For these measurements, a spectrally high-resolution imaging technique known as "hyperspectral imaging" has proven itself in recent decades. This hyperspectral imaging allows, for example, different chemical elements to be recognized and differentiated on the basis of the spatially resolved spectrum recorded.

The original hyperspectral imaging system setup uses a so-called "pushbroom" scanning setup, where on a two-dimensional image sensor one dimension is used for spatial determination and the other dimension for spectral determination. New approaches in hyperspectral imaging and the development of higher-resolution sensors and computing hardware have enabled so-called snapshot full-frame hyperspectral systems.

A very compact system for hyperspectral imaging is described, for example, in the publication "Ultra-compact micro-optical system for multispectral imaging" by M. Hubold et al. (Proc. SPIE 10545, MOEMS and Miniaturized Systems XVII, 105450V; 22 February 2018).

U.S. 2014/267849 A1 describes a spectral camera for generating a spectral output. It has an objective lens for forming an image, an optical duplicator, an array of filters and a sensor array arranged to capture the filtered duplicate images simultaneously on different parts of the sensor array. A field stop also defines an outline of the replica images projected onto the sensor array. The filters are integrated on top of the sensor array, which has a planar structure with no vertical physical barriers to prevent crosstalk between each of the adjacent optical channels. The field stop allows adjacent copies of the image to be seamlessly matched for such barriers. Thanks to the integrated filters, there are no parasitic cavities that cause crosstalk between adjacent image copies. This eliminates barriers between neighboring projected image copies and the sensor area can thus be better utilized.

WO 2014/143 338 A2 discloses an imager array that can be provided as part of an imaging system. It includes multiple infrared imaging modules, each infrared imaging module containing multiple infrared sensors associated with an optical element. The infrared imaging modules are essentially aligned in one plane and configured to capture images of the same scene. The infrared imaging modules may contain filters or lens coatings to selectively capture desired areas of infrared radiation.

However, the disadvantage of the prior art is still that a high two-dimensional image resolution always comes at the expense of a high spectral resolution and vice versa.

The object of the present invention was to overcome the disadvantages of the prior art and to provide a multi-lens camera system that achieves both a high spectral resolution and a high spatial resolution of a motif.

This object is achieved by a device and a method according to the claims.

The multi-lens camera system according to the invention for hyperspectral recording of images comprises a planar image sensor, a location-sensitive spectral filter element and an imaging system.

The imaging system comprises a flat lens matrix with a large number of individual lenses which are arranged in such a way that at a first recording time they produce a large number of first images of a motif arranged in a grid in a first region on the image sensor. Furthermore, the imaging system also includes a further lens, which is arranged in the plane of the lens matrix or offset from the plane of the lens matrix (preferably in front of the lens matrix) and is designed to to generate a second image of the subject on the image sensor in a second area of the image sensor and/or at a second recording time. The lenses are, for example, spherical lenses, cylindrical lenses, holographic lenses or Fresnel lenses or lens systems (eg lenses) made up of several such lenses. For better understanding, only “lenses” are used below, without restricting the generality.

The additional lens has a focal line and/or a different focal length compared to the plurality of individual lenses and is positioned in such a way that its second image differs from the first images at least in one surface dimension of the image sensor with regard to size and/or with regard to the Filter element present image information of the subject differs.

Areal image sensors are basically known to the person skilled in the art. These are particularly preferably pixel detectors which allow image points (“pixels”) to be recorded electronically. Preferred pixel detectors are CCD (charge-coupled device) sensors or CMOS (complementary metal-oxide-semiconductor) sensors. ). Sensors based on silicon are particularly preferred, but also InGaAs sensors and sensors based on lead oxide or graphene, in particular for wavelength ranges outside the visible range.

A spectral filter element that is designed in such a way that it transmits different spectral components of incident light at different positions on the surface of the filter element and does not transmit other spectral components is referred to here as a "location-sensitive spectral filter element", which is also referred to as a "location-dependent spectral filter element". could become. It is used to filter the images generated by the imaging system on the image sensor according to different (small) spectral ranges.

There are several options for arranging the filter element. For example, the filter element can be positioned directly in front of the lens matrix or between the lens matrix and the image sensor. It is also preferred that components of the imaging system are designed as filter elements, in particular the lens matrix. For example, the substrate of the lens matrix can be configured as a filter element.

A lens matrix within the meaning of the invention comprises a multiplicity of lenses which are arranged in a grid in relation to one another, ie in a regular arrangement, in particular on a carrier. The lenses are preferably arranged in regular rows and columns or offset from one another. A rectangular or square or a hexagonal arrangement is particularly preferred. The lenses can be, for example, spherical lenses or cylindrical lenses, but aspheric lenses are also preferred in some applications. A preferred lens array is described in more detail below.

According to the invention, the lens matrix is arranged in relation to the image sensor in such a way that it generates a large number of images of a motif arranged in a grid on the image sensor. Since there are two types of different images within the scope of the invention, the images of the individual lenses are referred to as “first images”. In a special subspecies of conventional spectral cameras (see also 1 ) only these first mappings (and no second mappings that differ from them) are available.

An object or a scenery is referred to as a "motif", i.e. basically any area of the room that is recorded by the multi-lens camera system. The motif can be a (geographical) floor area, for example, but also individual objects, e.g. products in a production line.

As a rule, in the case of hyperspectral recordings, the first images always show the same area of the subject (or the entire subject). These first images are recorded by the image sensor with different (light) wavelengths or in different wavelength ranges through the filter element. It is noted here that with regard to the required difference in the image information of the motif, this spectral difference generated by the filter is not meant, since the image information of the motif present in front of the filter element is meant, i.e. the type of representation of the images as they are would take place without a filter element.

The multi-lens camera system includes a further lens in addition to the lens matrix. It is explicitly pointed out here that the expression "one" (as is clearly stated again at the end of the description) should be read as "at least one", and thus "one" can also mean "two or more". This further lens can therefore be a single further lens, but also two or more lenses of the same or a different shape.

The additional lens is arranged in the plane of the lens matrix, in particular on the lens matrix, or offset from the plane of the lens matrix not. The further lens is preferably arranged in front of the lens matrix, ie in such a way that the lens matrix lies between the further lens and the image sensor. If there are two or more additional lenses, the indication of the positioning is to be understood in such a way that all additional lenses can lie in the plane of the lens matrix, in particular on the lens matrix (or on another common plane) or else at different distances can be arranged to the lens matrix. There can certainly be two or more lens matrices, for example a lens matrix with individual lenses and a lens matrix with additional lenses. The further lens can therefore be arranged on a further lens matrix or form a further lens matrix. With a single additional lens this would be a 1×1 matrix, with several additional lenses an A×B matrix.

The second images differ from the first images and are generated by the second lens (or multiple second images are generated by a number of additional lenses) in a second area of the image sensor and/or at a second recording time. If several further lenses are present, then as a rule there are also several second images, since each further lens preferably produces a second image. With regard to the second recording time, this case is equivalent to the feature that the further lens is adjustable or belongs to variable optics. In this preferred case, by adjusting the lens or by changing the optics, the further image is changed in terms of its size or with regard to other parameters relating to the image information and there is automatically a different image at a different recording time, even if the motif does not change.

The further lens has a focal line (in this case it is a cylindrical lens). Alternatively or additionally, it has a different focal length compared to the multiplicity of individual lenses. In addition, the further lens (or all further lenses are) positioned in such a way that its second image differs from the first image at least in one surface dimension of the image sensor with regard to size and/or with regard to the image information of the motif present in front of the filter element. As already mentioned above, what is meant are images as they would exist on the image sensor without a filter element.

This means that the second image differs from the first images. This difference is preferably relative to both surface dimensions of the image sensor (i.e. both its x and y coordinates), but in principle only has to be present in one surface dimension (i.e. either in the x or y direction of the sensitive surface of the image sensor). The difference relates to the size of the images, which means that the second image occupies a smaller or larger area on the image sensor than the first images (it is preferred that the second image is larger than the first image and thus generates a higher spatial resolution). Alternatively or additionally, however, the difference can also relate to the image information of the motif that is present in front of the filter element. In other words, the image information that has not yet been filtered by the filter element. This means that apart from the effect of the filter element, the image information of the second image has different image information than the first images (e.g. the image of a cylindrical lens as the second image and the images of spherical lenses as the first images). The influence of the filter element is therefore not significant (image information is viewed here as if the filter element were not there).

The image information particularly preferably relates to the image resolution. For example, it may be the case that the further lens is a single lens with a longer focal length. This lens creates an image of the object on the sensor, which depicts the same image section as each of the individual lenses, but on e.g. four times the sensor area. In this way, a higher-resolution location information of the same motif can be generated with the additional lens.

Depending on the application, the image information particularly preferably relates to the representation through a cylindrical lens. Due to the focal line, this images the motif in one surface dimension of the image sensor, and spectral information can then be measured (by means of the filter element) in the other surface dimension. Regardless of the filter element, however, the image of a cylindrical lens differs from that of spherical individual lenses, for example. In the case where the lenslets are cylindrical lenses, the further lens would have to be, for example, a larger/smaller cylindrical lens or generally a non-cylindrical lens, e.g., a spherical lens. An embodiment with exclusively lenses of the same size in the lens matrix is also possible, but their shape is different. In addition, zoom optics can also be present, by means of which the lens matrix or parts of the lens matrix can be enlarged or reduced, with the lens matrix here also being able to be constructed homogeneously from individual lenses. Combinations of the above cases are also possible.

In the following, some cases are presented again using examples. For example, the first images have a resolution of 500 x 500 pixels on the image sensor and a second image a resolution of 1000 x 1000 (with a further spherical lens with a larger focal length). However, a second image can also have a resolution of 500 x 1 (or even 8000 x 1) if the additional lens is a cylindrical lens. In this case, however, further image points of the image sensor are used in order (with the filter element) to obtain spectral information from the entire area illuminated by the cylindrical lens. A cylindrical lens as an additional lens with the same size and the same focal length as the first lenses could also generate an image with a size of 500 × 500 pixels on the image sensor (total illuminated area), but in contrast to the first images it only has a spatial resolution of 500 ×1 and therefore differs from the first illustrations with regard to the image information of the motif.

The first and second images could also be grouped together (particularly when there are two or more distinct second images). In the case also covered above, in which sequential recordings are viewed, the motif should have remained unchanged for better understanding. Of course, in practice, e.g. when taking aerial photos, the camera can be moved relative to the subject. However, it is important here that this movement is not necessary to obtain the two groups of images, since the imaging optics are designed in such a way that different types of images are generated. According to a preferred embodiment, a second group of images comprising the second images mentioned above has a higher spatial resolution than a first group of images comprising the first images. With the filter element, due to the large number of first images, the first group of images offers a higher spectral resolution than the second group of images.

A second image, at least if it has a comparatively higher spatial resolution than the first images, can also be referred to as a "spatial grid image" for better understanding. According to a preferred embodiment, the purpose of the second images is to display the motif or a region of the motif very precisely (with high resolution). If a cylindrical lens is used as an additional lens, the second image has a lower spatial resolution in one dimension of the surface of the image sensor (due to the focal line). However, you get a very high spectral resolution, at least higher than that of the first images. It is particularly preferred that the second image differs from the first images in that it has a higher spatial resolution of areas of the motif, at least in one surface dimension of the image sensor, or a higher spectral resolution (here viewed after the filter element). The higher spectral resolution can be achieved, for example, by means of a cylindrical lens and the filter element. In a surface dimension of the image sensor (orthogonal to the imaging axis), each sensor pixel can basically provide individual spectral information with a corresponding filter element.

For example, it is advantageous for aerial measurements to have a high spatial resolution, e.g. an image in the range between 3 and 6 megapixels, so that neighboring images can be precisely positioned using (well-resolved) landmarks. The second group of images can include several spatial grid images, but the goal according to the invention can already be achieved with a single (high-resolution) spatial grid image. Several spatial grid images have the advantage that they can be recorded in different parts of the spectrum and the second group of images thus already offers a certain spectral resolution.

At least in the case in which the second image (in both surface dimensions of the image sensor) offers a higher spatial resolution, the first images can also be referred to as "spectral images" for better understanding. In contrast to a second image, in which the high spatial resolution can already be achieved with a single spatial grid image, here the number of spectral images (each recorded at a different wavelength using the filter element) typically contributes to the high spectral resolution of the first images. The purpose of the first images is to show the subject or a part of the subject with a very good quality of the spectrum that can be picked up by the image detector. As mentioned above, good spectral resolution is achieved at the expense of spatial resolution.

• - Bereitstellen eines Motivs. Das Multilinsen-Kamerasystem wird also auf ein Motiv gerichtet.
• - Aufnahme des Motivs mittels des Multilinsen-Kamerasystems. In diesem Rahmen wird eine Vielzahl von rasterförmig angeordneten Abbildungen auf dem Bildsensor erzeugt, wobei in unterschiedlichen Bereichen des Bildsensors und/oder zu unterschiedlichen Aufnahmezeiten eine Vielzahl erster Abbildungen und eine Anzahl zweiter Abbildungen von Bereichen desselben Motivs aufgenommen wird. Dabei unterscheidet sich eine zweite Abbildung von den ersten Abbildungen zumindest in einer Flächendimension des Bildsensors bezüglich der Größe und/oder bezüglich der vor dem Filterelement vorliegenden Bildinformation des Motivs.

A method according to the invention for recording data with a multi-lens camera system according to the invention comprises the following steps:

• - Providing a motive. The multi-lens camera system is thus aimed at a subject.
• - Recording of the subject using the multi-lens camera system. In this framework, a large number of images arranged in a grid pattern is generated on the image sensor, with a large number of first images and a number of second images of areas of the image sensor being in different areas of the image sensor and/or at different recording times same subject is recorded. A second image differs from the first images at least in one surface dimension of the image sensor with regard to the size and/or with regard to the image information of the motif present in front of the filter element.

• - Erzeugen einer Vielzahl von rasterförmig angeordneten ersten Abbildungen eines Motivs in einem ersten Bereich auf dem Bildsensor mittels einer flächigen Linsenmatrix mit einer Vielzahl von Einzellinsen zu einem ersten Aufnahmezeitpunkt.
• - Erzeugen einer zweite Abbildung des Motivs in einem zweiten Bereich des Bildsensors und/oder zu einem zweiten Aufnahmezeitpunkt mittels einer weiteren Linse, die in der Ebene der Linsenmatrix oder abgesetzt von der Ebene der Linsenmatrix angeordnet ist.

In particular, the method according to the invention comprises the steps:

• - Generating a multiplicity of grid-like arranged first images of a motif in a first area on the image sensor by means of a flat lens matrix with a multiplicity of individual lenses at a first recording time.
• - Generating a second image of the motif in a second area of the image sensor and/or at a second recording time by means of a further lens which is arranged in the plane of the lens matrix or offset from the plane of the lens matrix.

The advantage of the present invention is that a motif can be imaged with such a multi-lens camera system both with a high image resolution and with a high spectral resolution. Both groups of images are available together for evaluation. The invention thus achieves a high spectral resolution with a hyperspectral (“full frame snapshot”) multi-lens camera system while also achieving a high spatial resolution. For example, optimal spectral measurement of ground surfaces from the air and control of industrial processes and systems are only possible if high spatial resolution is achieved at the same time as high spectral resolution. This invention fulfills both requirements.

Preferred embodiments of the invention are described below. It is pointed out that a preferred multi-lens camera system can also be configured analogously to the corresponding description of the method and, in particular, individual features of different exemplary embodiments can also be combined with one another.

According to a preferred embodiment, the lens matrix comprises a large number of individual lenses and an additional lens which, compared to the individual lenses, has a different diameter and/or a different focal length and/or a different substrate thickness and/or a different distance from the image sensor. The lens matrix is designed in such a way that it generates images of different sizes on the image sensor. This has the advantage that, with regard to two images with the different lenses, the motif is recorded by a different number of image sensors in each case, with the result that the motif is recorded with different resolutions. Of course, it is assumed here that the density of the pixels of the image sensor is essentially homogeneous over its area. The larger images, ie those with the higher spatial resolution, can be assigned to the second images (local grid images). The smaller images, ie those with the lower spatial resolution, can be assigned to the first images (spectral images). The images can certainly be combined in further groups of images, e.g. if there are three or more systematically different image sizes on the image sensor. However, two different images are sufficient for the invention.

For example, the lens matrix includes two different lens sizes, it also being possible for only a single large lens and a multiplicity of small lenses to be present. It is important here that the number of spectral images is (much) greater than the number of spatial grid images, in particular by an integer multiple. The number of spectral images is preferably more than 5 times, particularly preferably more than 10 times, the spatial grid images. The spectral images are recorded at different wavelengths, e.g. using a filter, as will be described in more detail below. A spatial grid image can be recorded at a single wavelength, but recording in one wavelength interval is preferred, preferably an interval that includes the wavelengths of a plurality of spectral images. A panchromatic spatial grid image is also preferred, in particular if only a single spatial grid image is present on the image sensor.

• Multilinsen-Kamerasystem zur hyperspektralen Aufnahme von Bildern umfassend einen flächigen Bildsensor, ein ortssensitives spektrales Filterelement und ein Abbildungssystem, wobei das Abbildungssystem eine flächige Linsenmatrix mit einer Vielzahl von Einzellinsen umfasst, welche so angeordnet sind, dass sie (zu einem ersten Aufnahmezeitpunkt) eine Vielzahl von rasterförmig angeordneten ersten Abbildungen eines Motivs in einem ersten Bereich auf dem Bildsensor erzeugen, und zusätzlich eine weitere Linse umfasst, die in der Ebene der Linsenmatrix, insbesondere auf der Linsenmatrix, angeordnet ist und dazu ausgestaltet ist, in einem zweiten Bereich des Bildsensors eine zweite Abbildung des Motivs auf dem Bildsensor zu erzeugen.

In the preferred case, in which the lens system has a spherical or cylindrical lens as a further lens, which is larger than the other individual lenses of the lens matrix and which has a different focal length than these individual lenses, it is usually automatically the case that the second image generated by this further lens differs from the first images at least in one surface dimension of the image sensor in terms of size. This preferred embodiment of the invention could therefore be summarized as follows:

• Multi-lens camera system for hyperspectral recording of images, comprising a flat image sensor, a location-sensitive spectral filter element and an imaging system, the imaging system comprising a flat lens matrix with a large number of individual lenses, which are arranged in such a way that they generate (at a first recording time) a large number of first images of a motif arranged in a grid-like manner in a first area on the image sensor, and additionally comprises a further lens which is in the plane of the lens matrix, in particular on the lens matrix, is arranged and is designed to generate a second image of the motif on the image sensor in a second area of the image sensor.

It is preferred that the additional lens takes the place of an integer multiple of the individual lenses in relation to the grid of the lens matrix. On the lens matrix (or in the plane of the lens matrix) there is preferably a number of spherical and/or cylindrical lenses (additional lenses) which are larger than a group of other lenses in the lens matrix (individual lenses). The larger lenses are preferably larger by a factor (in particular an integer) of 2 or more than the other lenses of the lens matrix, at least in one dimension of the lens matrix.

It should be noted that a cylindrical lens does not necessarily contribute to a higher spatial resolution compared to spherical lenses, although this may well be the case in one surface dimension of the image sensor. While the first images of the spherical lenses of the lens matrix have a resolution of e.g. 300x300 pixels on the image sensor, these first images are only imaged with 1 spectral filter per lens. The cylindrical lens, on the other hand, can have a (spatial) resolution of 300x1 pixels (one surface dimension of the image sensor), but with many (e.g. 6000) spectral channels (in the other surface dimension of the image sensor). There can of course be more than one cylindrical lens. Of course, the cylindrical lens does not have to have an exact circular path as the lateral surface. The lens surface could well represent the extrusion of another curve, e.g. an ellipse. The cylindrical lens can be used to advantage for better interpolation of the spectrum of the individual lenses, but it can also display a different image section, e.g. to record the ambient light. This can be used, for example, to correct the change in the ambient light in the spectrum.

According to a preferred embodiment, the optical imaging system includes optics, preferably zoom optics, between the lens matrix and the image sensor, which is designed to display images of at least some of the lenses of the lens matrix at different recording times in a different size and/or different-sized areas of the subject in shape display different images. Depending on the application, zoom optics are preferably designed to zoom all images (i.e. it affects the images of all lenses in the lens matrix) or preferably to zoom some of the images (i.e. it affects the images of some of the lenses in the lens matrix). The lenses of the zoom optics therefore belong to the other lenses. In the first alternative, the larger images (taken at a different point in time than the smaller first images) are preferably considered as second images. In the second alternative, those recordings that can be zoomed with the zoom optics are preferably considered as second images. This embodiment enables different recording modes of the first and second images (the spatial grid images and the spectral images).

The zoom optics are preferably designed in such a way that they track the angle of the filter. Depending on the type of zoom optics or the type of zooming, there could be "gaps" in the measured spectrum that may not be evenly distributed. A tracking that preferably includes a lateral movement parallel to the filter plane and/or a rotation about the surface normal of the filter plane or about the optical axis of the camera has an advantageous effect here. In principle, however, lateral movements in other directions and/or rotations of the filter element around other axes for tracking purposes are also possible. If the filter in the example described above is rotated around the optical axis (or within the plane in which the filter is located), the disadvantage of the "gaps" is compensated. In the case where there is a fixed relationship between the magnification scale of the zoom optics and the angle of the filter, it is preferred that the angle of rotation of the filter element is automatically tracked via a mechanical or electronic compensation unit (e.g., a mechanical connection to the zoom optics). The tracking thus includes in particular an embodiment in which the angle of the filter element to its surface normal (or to the optical axis of the multi-lens camera system) is not fixed, but is designed to be adjustable and particularly preferably depends or is dependent on the setting of the optical imaging system the setting of the optical imaging system can be regulated.

• Multilinsen-Kamerasystem zur hyperspektralen Aufnahme von Bildern umfassend einen flächigen Bildsensor, ein ortssensitives spektrales Filterelement und ein Abbildungssystem, wobei das Abbildungssystem eine flächige Linsenmatrix mit einer Vielzahl von Einzellinsen umfasst, welche so angeordnet sind, dass sie zu einem ersten Aufnahmezeitpunkt eine Vielzahl von rasterförmig angeordneten ersten Abbildungen eines Motivs in einem ersten Bereich auf dem Bildsensor erzeugen, und zusätzlich eine Optik, bevorzugt eine Zoomoptik, zwischen Linsenmatrix und Bildsensor (als weitere Linsen) umfasst,

In the case in which the multi-lens camera system contains the optics described above between the lens matrix and the image sensor, it is usually automatically the case that this has a different focal length than the individual lenses of the lens matrix and that second images differ from the first images at least in an area dimension of the image sensor in terms of size and/or with regard to the image information of the motif present in front of the filter element, since this can be enlarged or reduced at different recording times. This preferred embodiment of the invention could therefore be summarized as follows:

• Multi-lens camera system for hyperspectral recording of images, comprising a flat image sensor, a location-sensitive spectral filter element and an imaging system, the imaging system comprising a flat lens matrix with a large number of individual lenses, which are arranged in such a way that at a first recording time they have a large number of lenses arranged in a grid generate first images of a motif in a first area on the image sensor, and additionally includes optics, preferably zoom optics, between the lens matrix and image sensor (as further lenses),

According to a preferred embodiment, the filter element comprises a mosaic filter, which offers advantages in particular when used together with zoom optics. Preferably, the mosaic of the mosaic filter is arranged such that large wavelength steps are on the inside while smaller intervals are on the outside. In a preferred form of the mosaic filter, a colored mosaic, in particular a colored glass mosaic, is applied, in particular vapor-deposited, to one side of a substrate, preferably glass. According to an advantageous embodiment, the filter element (a mosaic filter or another filter) is applied to the front side of a substrate and the lens matrix (e.g. embossed) on the back side of the substrate. A mosaic filter preferably transmits a different wavelength for each individual lens.

In the case of the filter element, it is preferred (in particular when designed as a mosaic filter) that it is designed to be transparent in the imaging area of a further lens on the image sensor, and that there is preferably a transparent element in the filter element.

According to a preferred alternative or supplement, this imaging system is designed to display images of at least some of the lenses of the lens matrix at different recording times in a different size. This embodiment can be realized in particular with the aforementioned zoom optics. The entire motif is recorded at a first recording time. By means of the lenticular grid, a large number of images of the motif are obtained with a comparatively low spatial resolution but with a high spectral resolution (due to the large number of spectral images). At a second capture time (which may be before or after the first capture time), a larger, higher-resolution image of the subject (second image) is captured. The imaging optics are thus designed in such a way that the individual images on the image sensor are larger at the second recording time than at the first recording time (with the entire motif preferably being imaged in each image). Of course, the images of some lenses in the lens matrix are no longer registered by the image sensor. One thus obtains spatial grid images (the recordings with a higher spatial resolution) and spectral images (the recordings with a lower spatial resolution).

Alternatively or additionally, the imaging system is designed to display areas of the motif of different sizes in different images. This is preferably achieved in that the imaging optics are designed in such a way that individual lenses of the lens matrix are assigned individual zoom optics, the different zoom optics being designed in such a way that different areas of the subject are zoomed in each case. For example, four lenses of a uniform lens matrix can be provided with imaging optics in such a way that the image of one lens shows the lower right quarter of the motif enlarged in its respective image on the image sensor, the second lens shows the upper right quarter, and the other two lenses each the lower left and upper quarter. From the quarters (four spatial grid images), the entire motif then results in a higher spatial resolution, while the remaining images of the lens matrix on the image sensor are assigned to the first images as spectral images.

According to a preferred embodiment, the optical imaging system, preferably the lens matrix, comprises a cylindrical lens as a further lens, e.g. in its top row. The multi-lens camera system is preferably designed in such a way that that part of the image sensor on which the image of the cylindrical lens is generated can be read out independently of the remaining part of the image sensor. This allows the sensor readout to be limited to the image of the cylindrical lens, which has the advantage that the readout rate is increased, since the remaining part of the image sensor is no longer read out in this case. This means that the multi-lens camera system can easily be used as a line scanner with a high read rate, e.g. to identify products in a production line.

• Multilinsen-Kamerasystem zur hyperspektralen Aufnahme von Bildern umfassend einen flächigen Bildsensor, ein ortssensitives spektrales Filterelement und ein Abbildungssystem, wobei das Abbildungssystem eine flächige Linsenmatrix mit einer Vielzahl von Einzellinsen umfasst, welche so angeordnet sind, dass sie zu einem ersten Aufnahmezeitpunkt eine Vielzahl von rasterförmig angeordneten ersten Abbildungen eines Motivs in einem ersten Bereich auf dem Bildsensor erzeugen, und zusätzlich eine Zylinderlinse als weitere Linse umfasst, die in der Ebene der Linsenmatrix, insbesondere auf der Linsenmatrix, vor der Linsenmatrix oder zwischen Linsenmatrix und Bildsensor angeordnet ist und dazu ausgestaltet ist, in einem zweiten Bereich des Bildsensors eine zweite Abbildung des Motivs auf dem Bildsensor zu erzeugen.

In the case in which the multi-lens camera system includes a cylindrical lens as an additional lens, the case automatically exists that this has a focal line and that its second image differs from the first images at least in one area The optical dimension of the image sensor differs with regard to the image information of the motif in front of the filter element. This preferred embodiment of the invention could therefore be summarized as follows:

• Multi-lens camera system for hyperspectral recording of images, comprising a flat image sensor, a location-sensitive spectral filter element and an imaging system, the imaging system comprising a flat lens matrix with a large number of individual lenses, which are arranged in such a way that at a first recording time they have a large number of lenses arranged in a grid generate first images of a motif in a first area on the image sensor, and additionally comprises a cylindrical lens as a further lens, which is arranged in the plane of the lens matrix, in particular on the lens matrix, in front of the lens matrix or between the lens matrix and image sensor and is designed to a second area of the image sensor to generate a second image of the motif on the image sensor.

According to a preferred embodiment, the multi-lens camera system comprises a processing unit which is designed to superimpose and/or register the first images with a second image and/or to link corresponding image areas of the first images with a second image with one another using data technology, with the Processing unit is configured in particular to supplement spectral data from pixels of the second image with spectral data based on the first images and/or to supplement pixels of the first images with spatial data of the second image.

According to a preferred embodiment, the filter element comprises a linearly variable filter with filter lines (“gradient filter”), which is preferably rotated at an angle of between 1° and 45° with regard to the alignment of the filter lines with respect to the lens matrix. Alternatively or additionally, the filter element comprises a filter matrix, particularly preferably a mosaic filter. As mentioned above, the mosaic filter is particularly advantageous for zoom optics. What all filters have in common is that at least each first image (that is, generally each image of the multiplicity of individual lenses) is assigned a different filter characteristic, generally a different spectral range in each case.

According to a preferred embodiment, the multi-lens camera system comprises an aperture mask between the lens matrix and the image sensor, with apertures on the aperture mask being positioned corresponding to the lenses of the lens matrix and the aperture mask being positioned such that light from the images of the individual lenses passes through apertures in the aperture mask. The aperture mask thus has the same pattern as the lens matrix, with apertures being present there instead of lenses. An optical imaging system, such as zoom optics, is preferably positioned between the aperture mask and the image sensor. The aperture mask improves the images because it filters out stray light.

According to a preferred embodiment of the method, a panchromatic image recording is recorded as part of a second image (ie at least one spatial grid image). In this case, multiple panchomatic recordings of different areas of the subject and/or with different spectral resolutions are preferably made at different positions of the image sensor (in particular simultaneously) and/or at different measurement times. A panchromatic image recording is preferably superimposed with (hyperspectral) data from the first recordings. This has the advantage that a new basis for the allocation of this data is created.

According to a preferred embodiment of the method, the data of a second image (at least one spatial grid image) is overlaid with the data of the first images (the spectral images) for the evaluation of the data and/or registered and/or related to one another (see above statements on the processing unit or e.g. "pan sharpening"). The second image can also come from a cylindrical lens.

• - Extrapolation der spektralen Daten der ersten Abbildungen an verschiedenen räumlichen Positionen (unter Verwendung von höher auflösenden zweiten Abbildungen). Damit erhält man eine bessere räumliche Einordnung der spektralen Daten.
• - Ermittlung räumlich präzisierter Bezugspunkte. Dies ist vorteilhaft zur Erstellung von Geokoordinaten für hyperspektrale Karten.
• - Durchführung von Dimensionsmessungen an dem Motiv. Beispielsweise kann eine Längenmessung durchgeführt werden.
• - Ermittlung einer räumlichen Zuordnung mit höherer räumlicher Auflösung als die räumliche Zuordnung der spektralen Daten der ersten Abbildungen. Damit wird die räumliche Zuordnung der spektralen Daten verbessert.

In this case, the data of a second image are preferably used for the following operations as individual operations or as combined operations.

• - Extrapolation of the spectral data of the first images at different spatial positions (using higher resolution second images). This gives a better spatial classification of the spectral data.
• - Determination of spatially precise reference points. This is advantageous for creating geo-coordinates for hyperspectral maps.
• - Carrying out dimensional measurements on the motif. For example, a length measurement can be carried out.
• - Determination of a spatial assignment with a higher spatial resolution than the spatial assignment of the spectral data of the first images. This improves the spatial assignment of the spectral data.

1. a) Eine Verrechnung von ersten Aufnahmen mit einer zweiten Aufnahme, wobei die zweite Aufnahme insbesondere ein hochauflösendes Einzelkanalbild ist. Die Berechnung kann z.B. wie vorangehend beschrieben erfolgen.
2. b) Eine optische Bildkorrektur (räumlich und spektral). Diese erfolgt bevorzugt unter Zuhilfenahme von Umgebungslichtsensoren, z.B. einer Photodiode oder einem Spektrometer.
3. c) Eine Ermittlung von Objekteigenschaften und/oder eine Unterscheidung von Objekten Insbesondere aus den spektralen Informationen kann mittels einer angeschlossenen Recheneinheit Indexbasiert auf Objekteigenschaften wie die chemische Zusammensetzung des Objekts geschlossen werden oder mit Klassifikationsverfahren zwischen verschiedenen Substanzen und Objekten unterschieden werden. Bei der Klassifikation kann neben der spektralen Bildinformation insbesondere die Strukturinformation aus dem hochaufgelösten Sensorbild mitberücksichtigt werden.
4. d) Eine Überlagerung von Bildsequenzen,
   • - zur Erstellung einer Karte, z.B. einer geolokalisierten Messkarte, bevorzugt unter Zuhilfenahme der hochaufgelösten Bildinformation zur automatischen Erzeugung von eindeutigen Passpunkten zwischen überlappenden Bildern verschiedener Orte, welche eine automatisches Stitching der hochaufgelösten Bilddaten ermöglicht und so auch die Verortung der niedriger aufgelösten spektralen Information erlaubt. Hierbei kann die Verwendung von Positionierungsdaten (z.B. GPS-Daten oder Daten eines Beschleunigungssensors) hilfreich sein und die Genauigkeit des Stitchings erhöhen, was der Einordnung der Aufnahmen in eine (geografische) Karte an mit der Realität übereinstimmenden Positionen dient, und/oder
   • - zur Berechnung eines hochauflösenden Bildes, welches insbesondere bei einem Zeilenscanner-Aufbau mit einer Zylinderlinse als eine weitere Linse vorteilhaft ist.

The recorded data are preferably processed as follows. In the case of aerial photographs, the calculation preferably takes place on a computing unit directly in the aircraft (e.g. an airplane, a drone or a satellite) or subsequently on a computing unit on the ground (“ground PC”). This includes:

1. a) Offsetting first recordings with a second recording, the second recording being in particular a high-resolution single-channel image. The calculation can take place, for example, as described above.
2. b) An optical image correction (spatial and spectral). This is preferably done with the help of ambient light sensors, such as a photodiode or a spectrometer.
3. c) Determination of object properties and/or differentiation of objects. In particular from the spectral information, index-based object properties such as the chemical composition of the object can be inferred using a connected processing unit, or different substances and objects can be differentiated using classification methods. In addition to the spectral image information, the structural information from the high-resolution sensor image can also be taken into account in the classification.
4. d) A superimposition of image sequences,
   • - to create a map, e.g. a geolocalized measurement map, preferably with the help of the high-resolution image information for the automatic generation of clear control points between overlapping images of different locations, which enables automatic stitching of the high-resolution image data and thus also allows the localization of the lower-resolution spectral information. The use of positioning data (eg GPS data or data from an acceleration sensor) can be helpful here and increase the accuracy of the stitching, which serves to classify the recordings in a (geographical) map at positions that correspond to reality, and/or
   • - For the calculation of a high-resolution image, which is particularly advantageous in a line scanner structure with a cylindrical lens as a further lens.

The results, e.g. calculated plant protection parameters, can then be stored on the computer unit in the aircraft or on the ground PC, or sent immediately to an executing unit, e.g to perform.

• 1 zeigt schematisch ein Multilinsen-Kamerasystem gemäß dem Stand der Technik.
• 2 zeigt schematisch ein Multilinsen-Kamerasystem gemäß einer bevorzugten Ausführungsform der Erfindung.
• 3 zeigt ein Beispiel für eine Anordnung von Abbildungen.
• 4 zeigt schematisch ein Multilinsen-Kamerasystem gemäß einer weiteren bevorzugten Ausführungsform der Erfindung.
• 5 zeigt ein Beispiel für eine weitere Anordnung von Abbildungen.
• 6 zeigt ein Beispiel für eine bevorzugte Linsenmatrix.
• 7 zeigt ein weiteres Beispiel für eine bevorzugte Linsenmatrix.
• 8 zeigt schematisch ein weiteres Multilinsen-Kamerasystem gemäß einer bevorzugten Ausführungsform der Erfindung.

Examples of preferred embodiments of the device according to the invention are shown in the figures.

• 1 shows schematically a multi-lens camera system according to the prior art.
• 2 FIG. 1 shows schematically a multi-lens camera system according to a preferred embodiment of the invention.
• 3 shows an example of an arrangement of images.
• 4 shows schematically a multi-lens camera system according to a further preferred embodiment of the invention.
• 5 shows an example of another arrangement of images.
• 6 shows an example of a preferred lens array.
• 7 shows another example of a preferred lens array.
• 8th shows schematically another multi-lens camera system according to a preferred embodiment of the invention.

1 shows schematically a multi-lens camera system 1 for hyperspectral recording of images according to the prior art in a perspective view. The multi-lens camera system 1 comprises a planar image sensor 3 and a planar lens matrix 2 made of uniform individual lenses 2a, which is arranged in such a way that from a motif M it produces a large number of first images AS arranged in a grid (see, for example, only the small first images AS in 5 ) generated on the image sensor 3. For the sake of clarity, only one of the individual lenses 2a is provided with a reference symbol.

An aperture mask 5 is arranged between the image sensor 3 and the lens matrix 2 in order to improve the quality of the first images AS. Each aperture 5a of the aperture mask 5 is assigned to an individual lens 2a and is arranged exactly behind it. A filter element 4 is arranged between the aperture mask 5 and the image sensor 3 in order to obtain the spectral information. In other versions, this filter element 4 can also be arranged in front of the lens matrix (see, for example, 8th ). In the case shown here, the filter element 4 is a linearly variable filter that is slightly rotated relative to the image sensor. Each image thus has its center at a different wavelength range of the filter element. Thus, every first mapping yields AS another spectral information on the image sensor and the entirety of the first images AS is used to create an image with spectral information.

2 shows schematically a multi-lens camera system 1 according to a preferred embodiment of the invention in a perspective view. It is similar to the multi-lens camera system 1 constructed with the important difference that the lens matrix 2 has a larger further lens 2b with a different focal length. Correspondingly, the aperture mask 5 also has a larger aperture 5a at this point.

Due to this larger additional lens 2b, the multi-lens camera system 1 is able to position the multi-lens camera system 1 in relation to the motif M (see Fig. 1 ) a second image AO (namely the one that is imaged by the larger additional lens 2b) with a higher spatial resolution than that of the first images AS (the smaller individual lenses 2a) on the image sensor 3. Since the larger additional lens 2b has a greater focal length here, it is arranged somewhat further away from the image sensor 3. For figures AO, AS, click on 3 referred.

3 shows an example of an arrangement of first images AS and second images AO as with a multi-lens camera system 1 2 on whose image sensor 3 could be present. A single second image AO (local raster image) is shown at the top left, as shown by the large additional lens 2b of the multi-lens camera system 1 2 is imaged on the image sensor 3. The remaining images (spectral images) belong to the second group of first images AS. The motif M can be seen on the left, which is imaged by the multi-lens camera system 1, the process of imaging being symbolized by an arrow.

Since the image sensor 3 has a homogeneous pixel distribution, the second image AO and the first images AS show the same motif M in different areas of the image sensor, but the spatial grid image (second image AO) has four times the spatial resolution compared to the spectral images (first images AS) .

4 shows schematically a multi-lens camera system 1 according to a further preferred embodiment with optical imaging optics 6 between the lens matrix 2 in a side view. The multi-lens camera system is shown without an aperture mask 5 in this example. Like the preceding examples, the image sensor 3 has an upstream filter element 4. The optical imaging optics 6 can be zoom optics, for example, which leads to a representation according to FIG 5 can lead. This multi-lens camera system 1 is designed such that, with the positioning of the multi-lens camera system 1 in relation to the subject M remaining unchanged, a plurality of second images AO (or also a single second image AO) with a higher spatial resolution than the first images AS (see 5 ) at different recording times.

5 shows an example of an arrangement of images AO, AS as with a multi-lens camera system 1 4 on whose image sensor 3 could be present. The recordings of the motif M at different recording times are shown on the left and right below the motif M, with the process of imaging being symbolized here again with arrows.

In the picture on the left, the imaging optics 6 are adjusted in such a way that the entire motif M is recorded, but not all images of the imaging optics "fit" on the image sensor (dashed edge). Here, the individual images have a comparatively high spatial resolution, since a comparatively large number of pixels of the image sensor are covered by one image. These recordings can be viewed as second images AO.

In the picture on the right, the imaging optics 6 are set in such a way that all images of the lens matrix “fit” on the image sensor. As a result, they are smaller than the images in the illustration on the left and also have a lower spatial resolution. These recordings can be viewed as the first images AS.

6 shows an example of a preferred lens matrix 2. In addition to the individual lenses 2a in the first line, this includes a cylindrical lens as a further lens 2b. With such a lens, a multi-lens camera system 1 can also be used as a line scanner.

7 shows another example of a preferred lens matrix 2. In the first line, this includes a single cylindrical lens as a further lens 2b, the grid dimension of which corresponds to that of the spherical individual lenses 2a.

8th shows schematically a multi-lens camera system 1 according to a preferred embodiment of the invention in a perspective view. It is similar to the multi-lens camera system 2 structured, but with some special features, which can also be present individually in a multi-lens camera system.

On the one hand there are two lens matrices 2, a flat lens matrix 2 made of uniform individual lenses 2a and a lens matrix 2 with a size Ren further lens 2b, which is slightly further away from the image sensor 3.

Secondly, the filter element 4 is arranged in front of the two lens matrices 2 .

Thirdly, the filter element 4 is a mosaic filter which transmits a different wavelength for each individual lens (indicated by the transparent boxes in the filter element 4). In front of the further lens 2b there is preferably a transparent element in the filter element 4, which is indicated there by the larger transparent box.

Finally, it is noted that the use of the indefinite article, such as "a" or "an", does not rule out the possibility that the relevant characteristics can also be present more than once. So "a" can also be read as "at least one". Terms such as "unit" or "device" do not exclude that the elements in question can consist of several cooperating components, which are not necessarily housed in a common housing, although the case of a comprehensive housing is preferred. In the field of optics, in particular, the element of the lens can consist of a single lens or a system of lenses or an objective without this requiring precise differentiation.

Reference List

1

Multi-lens camera system

2

lens matrix

2a

single lens

2 B

More lens

3

image sensor

4

filter element

5

aperture mask

5a

aperture

6

imaging optics

AS

First figure

oh

Second figure

M

motive

Claims (10)

Multi-lens camera system (1) for hyperspectral recording of images, comprising a planar image sensor (3), a location-sensitive spectral filter element (4) and an imaging system, the spectral filter element (4) - comprises a linear variable filter with filter lines and/or - comprises a mosaic filter, the mosaic of the mosaic filter being arranged such that large wavelength steps are inside while smaller intervals are outside and the imaging system - a flat lens matrix (2) with a multiplicity of individual lenses (2a), which are arranged in such a way that, at a first recording time, they have a multiplicity of first images (AS) of a motif (M) arranged in a grid in a first area on the image sensor (3) generate, and - Additionally comprises a further lens (2b), which is arranged in the plane of the lens matrix (2) or offset from the plane of the lens matrix (2) and is designed to be in a second area of the image sensor (3) and/or to a to generate a second image (AO) of the motif (M) on the image sensor (3) at the second recording time, the further lens (2b) having a focal line and/or a different focal length compared to the plurality of individual lenses, and being positioned in this way that their second image (AO) differs from the first images (AS) at least in one area dimension of the image sensor (3) with regard to the size and/or with regard to the image information of the motif (M) present in front of the filter element (4). multi-lens camera system claim 1 , characterized in that the further lens (2b) compared to the individual lenses (2a) has a different diameter and/or a different focal length and/or a different substrate thickness and/or a different distance from the image sensor (3), the lens matrix ( 2) is designed in such a way that it generates images (AS, AO) of different sizes on the image sensor (3), with the further lens (2b) preferably taking up the space of an integral multiple of the individual lenses ( 2a) takes. Multi-lens camera system according to one of the preceding claims, characterized in that the optical imaging system comprises optics between the lens matrix (2) and the image sensor (3), which are designed to produce images (AS, AO) of at least part of the lenses of the lens matrix (2 ) at different recording times in a different size and/or different sized areas of the subject (M) in the form of different images, the optical imaging system preferably comprising a zoom lens. Multi-lens camera system according to one of the preceding claims, characterized in that the optical imaging system, preferably the lens matrix (2), comprises a cylindrical lens as a further lens (2b), the multi-lens Camera system (1) is designed in such a way that that part of the image sensor (3) on which the image of the cylindrical lens is generated can be read independently of the remaining part of the image sensor (3). Multi-lens camera system according to one of the preceding claims, characterized in that it comprises a processing unit which is designed to overlay and/or register the first images (AS) with a second image (AO) and/or corresponding image areas of the first To link images (AS) with a second image (AO) with one another in terms of data technology, the processing unit being designed in particular to supplement spectral data from pixels of the second image (AO) with spectral data based on the first images (AS) and/or Pixels of the first images (AS) supplemented by spatial data of the second image (AO). Multi-lens camera system according to one of the preceding claims, characterized in that the spectral filter element (4) comprises a linearly variable filter with filter lines which rotates at an angle between 1° and 45° with regard to the orientation of the filter lines with respect to the lens matrix (2). is. Multi-lens camera system according to one of the preceding claims, characterized in that it comprises an aperture mask (5) between lens matrix (2) and image sensor (3), apertures (5a) on the aperture mask (5) corresponding to the lenses of the lens matrix (2) are positioned and the aperture mask (5) is positioned in such a way that light from the images (AS, AO) of the individual lenses passes through apertures (5a) of the aperture mask (5), with an optical imaging system preferably being installed between the aperture mask (5) and the image sensor (3 ) is positioned. Method for recording data with a multi-lens camera system (1) according to one of the preceding claims, characterized by the steps: - providing a motif (M), - recording the motif (M) by means of the multi-lens camera system (1), wherein a A large number of images (AS, AO) arranged in a grid is generated on the image sensor (3) and in different areas of the image sensor (3) and/or at different recording times a large number of first images (AS) and a number of second images (AO) of Areas of the same motif (M) is recorded, with a second image (AO) differing from the first images (AS) at least in one surface dimension of the image sensor (3) with regard to the size and/or with regard to the image information present in front of the filter element (4). Motive (M) differs. procedure after claim 8 , characterized in that a panchromatic image recording is recorded as part of a second image (AO), with several panchromatic recordings of different areas of the motif (M) and/or with different spectral resolution are made, and preferably a panchromatic image recording is superimposed with data from the first recordings. procedure after claim 8 or 9 , characterized in that the data of a second image (AO) are overlaid with the data of the first images (AS) for evaluating the data, registered and/or related to one another, the data of the second image (AO) preferably being used for this purpose - to extrapolate the spectral data of the first images (AS) at different spatial positions, and/or - to obtain spatially precise reference points and/or - to carry out dimensional measurements on the motif (M) and/or - to obtain a spatial To obtain assignment with higher spatial resolution than the spatial assignment of the spectral data of the first images (AS), the recorded data are preferably processed as follows: a) calculation of first recordings with a second recording, b) optical image correction, c) a Determining object properties and/or distinguishing between objects, d) Overlaying image sequences to create a map and/or to calculate a high-resolution image.

Images