DE102012107191A1 - Method for high-resolution digital color imaging in microscopy, involves making additional images using different settings or shutter speed settings of color sensor for generating high-dynamic-range imaging at each scanning position - Google Patents

Abstract

The method involves integrating intermediate scans with scanning steps from a fraction of center distances of sensor elements of the color sensor, as an additional scan positions in the scanning movement of color co-site sampling produced by micro-scanning additional unit image capture. The additional images are made using different settings or shutter speed settings of the color sensor for generating high-dynamic-range imaging at each scanning position, both by Color co-site sampling positions of sensor elements and by the additionally extended intermediate scan sampling positions.

Description

 The invention relates to a method for high-resolution digital image recording with a high dynamic range, in which by means of scanning movement of a two-dimensional color sensor, an increase in resolution of image acquisition by successive coverage of different color-sensitive sensor elements by means of color co-site sampling is performed.

 In particular, CCD or CMOS cameras are used to capture images of any objects. In order to produce color images, it is known to subdivide a CCD or CMOS sensor by means of an applied color mask into three color channels, for example red, green and blue, by means of an applied color mask. It is customary to arrange in one line green and red sensitive pixels and in the immediately adjacent line blue and green sensitive pixels in alternation. Each group of adjacent green / red and blue / green sensitive pixels forms a pixel in the image in the broadest sense. In order to achieve the possible resolution despite the failure pixels compared to the nominal number of sensor pixels, usually a so-called interpolation is performed. The disadvantage of interpolation is that only approximate color values can be provided (no measured colors).

For complete compensation of the color loss, it is known to use the so-called color co-site sampling method, for example, in the DE 38 37 063 C1 is described. For this purpose, the CCD sensor is moved relative to the object to be recorded by a high-precision piezo-mechanics such that in each case an image is created one after the other at four positions which have the same relative position to each other as four pixels of a group. As a result, one and the same pixel of the object is successively detected twice through the green and once through the red and blue color channels, whereby the recording time for an object quadruples, if the time for repositioning is neglected. By concatenating the temporally successively created images, an overall image is then created.

 It is further known that the resolution can be increased by additional microscanning. In this case, additional images are created in positions between the sensor elements. Consequently, an overall picture is then created not only from four, but from 16 or 36 (subpixel pitches) spatially shifted partial images. However, due to the finite aperture of the pixels, this method is limited to a maximum of 50% pixel overlap.

When taking up fluorescent specimens, artifacts appear in the overall image caused by the diminishing fluorescence intensity of the sequentially acquired images. In the US 5,682,567 In order to avoid such artifacts, it is described that the fluorescence intensity decreasing from image to image is compensated by an increasing extension of the exposure time for the individual images. The required correction of the exposure time is calculated in advance on the basis of the knowledge of the decrease in fluorescence. However, this presupposes that the temporal fluorescence profile is known and the decrease in fluorescence, which may differ for preparations of different ingredients, then actually corresponds to this previously known course.

A similar procedure is in the EP 1 235 424 A2 specified for the recording of fluorescence images, in which the fading artifacts due to the increased by the plurality of field recordings time required for a complete high-resolution image acquisition not by a gradually extended recording time, but by a downstream (after image interleaving) intensity compensation depending on the time course of the individual Partial image recordings are eliminated.

 All of the approaches described above for high-resolution color image recording have the disadvantage that the recording time is always increased as a result of the required sensor displacements for recording the partial images, be it to realize the color co-site sampling (CCS) method or to increase the resolution. In order to maintain constant or comparable conditions, exposure time corrections or adapted gain factors may also be required, which prohibit HDR (High Dynamic Range Imaging) image acquisition with further additional partial image recordings, if the correction requirements for n partial exposures recorded with different sensitivity / exposure time are not met still want to multiply.

 The invention is based on the object of finding an improved possibility for high-resolution color image recording, in particular for microscopy, even in image recordings with high color depth and signal dynamics of other high-contrast scenes, in which the dynamic range of image acquisition is significantly improved, without recording time and / or corrections of the different exposure states for the plurality of partial image recordings are unreasonably increased.

According to the invention, the object is achieved with a method for high-resolution digital color image recording with a high dynamic range, in which By means of scanning movement of a two-dimensional color sensor, an increase in resolution of the image acquisition by successive overlapping of all differently color-sensitive sensor elements by means of color co-site sampling is achieved by interim scanning with scanning steps of a fraction of the center distances of the sensor elements of the color sensor as additional scanning positions in the scanning movement of Color-CoSite-Sampling be integrated so that by means of such a defined Microscanning additional partial image recordings generate at least a doubling of the sampling density in each dimension of the color sensor, and that at each sampling position, both provided by the Color-CoSite sampling positions of the sensor elements as well Intermediate scan additionally approached scanning positions, additional partial image recordings with different sensitivity settings or exposure time settings of the color sensor to generate a high-dynamics ic range imaging.

 Advantageously, when using a GRBG Bayer color mask color sensor, an overall image is composed of sub-images of four color co-site sampling positions and a 4 × 4 microscanning having at least one inter-scan per pixel center distance and dimension of the color sensor.

 The microscanning is expediently carried out with a regular line-shaped or column-shaped scanning pattern. It proves to be advantageous to carry out the microscanning with a regular meandering scan pattern. Preferably, the microscanning can be carried out with a regular shortened maander-shaped scanning pattern, in which the color-co-site sampling positions are approached with equalized time interval.

 It proves to be particularly advantageous if the microscanning is carried out with an irregular scanning pattern in which the color co-site sampling positions and the Zwischenabtastpositonen for the individual field images are approached with the closest possible time interval, with adjacent sampling positions, the in a regular meander-shaped scan grid would be scanned directly one after the other, be approached with a time interval of at least two Microscanschritten.

 In order to increase the dynamic range of the image acquisition, it has already proven to be expedient if, for the realization of an HDR imaging, two partial image recordings with different sensitivity settings of the color sensor per micro-scan position are recorded. Particularly advantageous for the HDR imaging three partial image recordings can be recorded with different sensitivity settings of the color sensor per Microscanposition.

 In order to have a particularly flexible camera available, different settings of Microscanmodi of the scanner and drive and read modes of the color sensor are stored in the control electronics for the realization of different levels of resolution of color images.

 The invention is based on the basic idea that professional users of high-resolution color image cameras, such as e.g. In microscopy, photography or in general in the field of surveying or analysis, when digital image recordings are also to be recorded with high dynamic ranges, it is necessary to repeat the same recording several times with different exposure times, so that all requirements are met. A helpful alternative only for macroscopic scenes would be to use an extremely expensive HDR camera with multiple times the number of pixels of the image sensor to at least increase the sensor resolution for color resolution (color co-site sampling and / or microscanning) at a respectable frame rate of the interlaced total images to reach.

 The invention here seeks the practicable compromise to combine the known per se methods of microscanning for general resolution enhancement, CCS for the removal of color artifacts and HDR imaging for increasing the color depth (dynamic range) in a suitable manner by a clever interleaving of partial image recordings for individual steps different of the three methods is chosen. As a result, valuable recording and setting times (overhead time) are saved compared to the usual multiple recordings.

 The combination of RGB subfields in CCS mode (four sensor displacements squared at 1/1 pixel pitch each) with additional (at least one) subpixel increments between the CCS positions and multiple sensor readings at different integration times at each microsan position of the matrix sensor increased (at least twice) resolution a dynamic range of 14, 16 or more bits of color depth of the overall picture. Further, the same principle is applicable to black-and-white and grayscale images, respectively.

The invention makes it possible to produce high-resolution color images, especially in microscopy, but also in image recordings with high color depth and signal dynamics of high-contrast scenes in which the dynamic range of the image acquisition is improved, without unduly increasing the costs for acquisition time, sensor pixel number and / or corrections of the different exposure states for the plurality of partial image recordings.

 The invention will be explained in more detail with reference to embodiments and figures. Show :.

1 : A schematic representation of the method according to the invention with a 16-step microscanning (4 × 4 microscanning) of a matrix sensor for a nested microscanning, CCS and HDR image recording, which consists of 16 × 4 × (2 ... n) partial image recordings is composed (flowchart of 1a in 1b and 1c continued);

2 FIG. 12 is a schematic representation of the 144 field readings of a matrix sensor shown for a color pixel (GRBG Bayermask) in a 12-position microscan CCS scan with integrated triple HDR image sensing;

3 FIG. 4 shows different embodiments of microscan patterns, represented for the local displacement of a sensor element of the matrix sensor for a 16-position microscan CCS scan (plus HDR scan); FIG.

4 : A schematic representation of a camera according to the invention with lens, matrix sensor with Bayer color mask, piezo scanner, sensor and scanner control electronics and computer for image interleaving and normalization.

The method for high-resolution digital color image recording with high dynamic range is - without restriction of generality - in a variant with a micro-scanning 115 in a 3 × 3 microscan screen according to the schematic representation of 2 explained in more detail.

Where is for a pixel group 11 with four sensor elements arranged according to the Bayer type of a color mosaic filter mask arranged directly on the chip of a monochrome CCD or CMOS matrix sensor as a true color picture element of a color sensor 1 is executed, the image recording method according to the invention for a complete (consisting of 27 part recordings) interlaced image recording shown schematically. To identify the spatially seamless image scan with respect to the adjacent true color pixels are pixel groups 12 to 14 drawn with, but left as empty fields, so as not to obscure the presentation. For the pixel groups 12 to 14 run the for the pixel group 11 displayed scan operations in the same way from simultaneously.

The sensor matrix thus regularly has the pixel group in an on-chip color mask of the bayer type 11 with four sensor elements, namely two diagonally arranged green sensitive sensor elements 111 and 114 and a red-sensitive sensor element 112 and a blue-sensitive sensor element 113 on.

For the complete color information of the colors red, green and blue for each of the four positions of the sensor elements 111 to 114 get is for the pixel group 11 according to the conventional color co-site sampling method (hereinafter referred to as: CCS sampling 118 ) are each shifted by a pixel center distance in a square, so that the sensor element 111 successively with the starting positions of the sensor elements 112 to 114 is brought to cover. The coverage for the sensor elements 112 to 114 is done with the arranged in the same color scheme sensor elements of the adjacent pixel groups 12 to 14 , which according to analogous system the numbers 121 . 124 . 131 . 142 and 141 would have to wear (omitted for clarity). All approached CCS sampling positions 119 are in 2 shown with dashed lines. The worn-out pattern of CCS sampling 118 can follow a rectangle outline (not drawn) or - as in 2 for the sensor element 114 displayed - be a crossing zig-zag pattern. The latter scan sequence corresponds to the time sequence, if in addition a meandering microscanning 115 as described below.

To increase the resolution - eg because of the light-insensitive gaps between the sensor elements 111 to 114 and also to those of the adjacent pixel groups 12 . 13 . 14 etc., but also to avoid the use of expensive high-resolution color sensors 1 with multiple number or density of the sensor pixels - takes place in accordance with the representation of 2 an intermediate scan 116 , in which in the light insensitive gaps of the sensor by microscanning 115 more field recordings are performed (in this example exactly one, ie a microscanning 115 with a half pixel pitch). The associated positions of the sensor elements 111 to 114 are shown with fine dotted lines. The step size is in the case of 2 to an intermediate scan 116 (3 × 3 microscan), resulting in a total image of 27 fields given three HDR samples.

The step size of microscanning 115 can be chosen arbitrarily, but will limit the processing effort, but also because of the limited resolution of the camera optics 2 preferably on two intermediate scans 116 (4 × 4 microscan), resulting in a total image of 48 fields given three HDR samples.

In the execution according to 2 describes the microscan pattern 117 for a 4 × 4 microscanning 115 a meandering shape having sixteen sensing positions with respect to a selected sensor element 111 a pixel group 11 for the CCS sampling described above 118 a crossing zigzag pattern between the CCS sampling positions 119 "In a roundabout way" (ie with stored intermediate samples 116 of microsensing 115 ). The 4 × 4 microscanning selected here 115 possible four CCS Samplings 118 are down in 2 with the real scan order and the actual scan positions in the meandering micro scan pattern used 117 shown.

As a result of such 4 × 4 microscanning 115 with sixteen spatially offset sampling positions of the sensor pixel 111 creates a 64 pixel CCS image for the pixel group under consideration 11 from which four complete CCS images can be assembled. Due to the additional per local scanning position of the Microscanning 115 In this example, 192 sub-picture readings (in this case two further) HDR sub-picture recordings result from sixteen local position changes (4 × 4 microscan), each of which has four sensor elements 111 to 114 record four color separations with three different sensitivity settings, ie twelve partial image readings are taken at each of the sixteen scan positions in the image. Here are some color separations of the sensor elements 112 . 113 and 114 that are in the original scan position of the color sensor 1 with the pixels of the neighboring pixel groups 12 . 13 and 14 overlap, discard immediately, because they are not for CCS sampling 118 the pixel group 11 can contribute.

There are different processes of microscanning 115 possible and useful. For the description of the possible microscan patterns 117 becomes - for the sake of simplicity - only the upper left pixel 111 the pixel group 11 , which represents a complete color pixel of the color sensor 1 stands, looks and draws.

To run a microscan 115 As a rule, the shortest scanning path is selected. However, it can also be a kind of chaos micro-scan pattern for special image recordings (eg, those which, given the relatively long recording times for a complete overall picture, are subject to statistical fluctuations, slow changes or vibrations) 117 can be selected in which the scanning order is set so that between all locally directly adjacent microscanning positions as similar as possible intervals of sampling intervals are (szB DE 102 61 665 C1 or EP 1 432 231 A2 ) and / or an equal number of intermediate scans 116 (or pixel-covering main scans) between the CCS sampling positions 119 should lie as they are in 3 are indicated schematically.

3 shows in the partial illustrations a) to c) in each case a 4 × 4 microscan grid of a sensor element group 11 (GRBG Bayer color pixel), of which only the upper left sensor element 111 considered and shown in his scanning motion. In the three illustrated microscan patterns 117 finds the CCS sampling 118 in each case at the known pixel positions, if - in this example: after an intermediate scan 116 in the square 4 × 4 scan grid - the position of the next sensor pixel is congruently set, whereby the entire sensor element group 11 in the same microscan pattern 117 moved.

The samples according to 3a and 3b each take place in shortened meander shapes, in which the scan paths per scan step are each exactly half the pixel pitch. This results in four time-varying nested CCS-Samplings 118 where the usual order of CCS sampling positions 119 along the sensor elements 111 . 112 . 113 and 114 respectively. 111 . 112 . 114 and 113 Although adhered to, but for the Zwischenabtastungen 116 (in 1a - 1c Subpixel Imaging with SubN-CCS Sampling 1-4) and between the CCS sampling positions 119 different number of intermediate scans 116 of the microscan pattern 117 lie. This can be done in non-stationary scenes (objects 3 ) lead to artefacts of the color representation. This becomes clear even with complete meander sampling according to 2 through the (in 2 separately drawn below) CCS sampling positions 119 with the specified scan step numbers for the four CCS samplings shown 118 ,

The partial image b) of 3 shows for the same 4 × 4 microscan grid of a sensor element group 11 a microscan pattern 117 that concerning the CCS seedling 118 is time optimized by adjusting the positions of the sensor elements 111 . 112 . 114 and 113 in this order, with only one or two intermediate scans at a time 116 be approached. The same applies here for all SubN-CCS-Samplings (with N = 1 ... 3).

The partial picture c) of 3 disclosed - in contrast to the ordered microscan patterns 117 An already mentioned, apparently chaotic microscan pattern 117 in which there is a "relatively similar" time interval between adjacent positions of the microscan grid of a sensor element group 11 attention is paid to image blurring due to statistical fluctuations (oscillations) between the object image and the color sensor 1 (or the sensor element group 11 ) to suppress. In addition, the smallest possible number of sampling steps (<3) is made between the CCS sampling positions 119 sought, which - especially when temporal change of the object image - the color separations are obtained in close temporal order. This has an increased significance for the inventive integration of partial image recordings of HDR imaging described below 120 in which the color separations recorded temporally successively for each microscan position with different dynamic range are thus closer together in terms of time and have fewer artifacts in the overlaying (temporal interleaving) of the overall picture.

In order to increase the dynamic range of the image recording, in the method according to the invention, furthermore, a plurality of partial recordings, each with graded different sensitivity settings or exposure times, are made in each approached microscanning position. These field images, which are called HDR imaging (High Dynamic Range Imaging), are spatially perspective pixel outlines with dash-dot lines (for CCS sampling positions 119 ) and dotted lines (for positions of intermediate scans 116 ) in 2 shown. The staggered HDR partial image recordings 121 with different sensitivity settings are carried out in succession for each spatially separated partial image recording position of the selected Microscanmusters 117 , but for all sensor element groups 11 . 12 . 13 . 14 etc. at the same time. This measure produces a high-resolution color image with additionally increased dynamic range, ie with high color depth and high contrast.

This is a significant increase in the dynamic range of a high-resolution scanning color sensor 1 for each overall image capture possible, so that no real HDR sensor matrix with logarithmic sensitivity behavior of the sensor elements is required (cost savings) or successively high-resolution overall images with different exposure settings must be repeated (time savings).

By limiting the number of sensitivity-controlled or exposure-time-controlled partial image recordings to two or three, a good compromise is found in order to reduce the time required for the overall image recording (which in the example shown in FIG 2 consists of 27 partial image recordings) to keep in reasonable frame.

 For 4 × 4 microscanning with two HDR images per scan position, 128 partial image recordings result with only 32 sensor readings, resulting, for example, in T1 = 10 ms and T2 = 140 ms in a total time Ttot <16 s for a complete high-resolution image acquisition with twice the dynamic range ,

The sequence of the method according to the invention for high-resolution digital image recording with a high dynamic range is described in 1a and 1b shown as a flowchart of required for the overall image sub-image scans. It is (as in 2 ) made use of a 3 × 3 microscanning with a concurrent CCS sampling and an HDR imaging (multiple scans with different sensitivity settings or exposure time settings per scanning position). There are several partial recordings of different purpose (in the order given) offset in time by means of a camera with a commercially available color sensor 1 (eg SONY ICX 285AQ) from an object 3 added.

As in 4 shown schematically, becomes the object 3 by means of a camera optics 2 on the color sensor 1 displayed. The color sensor 1 (with Bayer color mask) is on an XY scanner 4 (Preferably, a piezo scanner) applied, the required scanning movements of the color sensor 1 for microscanning 115 and CCS sampling 118 according to the selected microscan pattern 117 performs. However, it is also possible to use mirror scanners (not shown) in order to realize the relative movement of the object image relative to the color sensor. The control of the scanner 4 done by the sensor and scanner control electronics 5 , which also provides the necessary sensor switching for HDR imaging 120 pretends, the different readout modes of the color sensor clocks and the orderly temporary storage of partial image recordings in the computer 6 organized. Advantageously, different sampling and readout modes in the control electronics 5 selectable filed.

In the downstream computer 6 , which can be separate, but also part of the camera, the partial image recordings are then combined into a composite overall image with higher spatial resolution, higher dynamic range and measured or colors. This is done by temporally and spatially subdividing the partial recordings according to the recording modes of microscanning 115 , CCS sampling 118 and HDR imaging 120 ,

This results in limited to only two HDR scans per Microscanning position already brilliant high-resolution color images of high color depth, for their inclusion as a whole image is not required in an assumed 4 × 4 micro scanning with two HDR images of not more than 16 s.

In particular, the combination of RGB color images results in overall images with a high dynamic range of 14-16 bit color depth and additionally at least double resolution in an intermediate scan 116 (in microscanning 115 with CCS sampling 118 ).

 When combining black and white or gray scale images, the same depth of the dynamic range is obtained with the same basic approach.

LIST OF REFERENCE NUMBERS

1

color sensor

11, ..., 14

pixel groups

111 ... 114

sensor elements

115

Microscanning

116

between sampling

117

Microscan pattern

118

CCS Sampling

119

CCS sampling position

120

HDR Imaging

121

HDR image capture part

2

camera optics

3

object

4

scanner

5

Sensor and scanner control electronics

6

computer

QUOTES INCLUDE IN THE DESCRIPTION

This list of the documents listed by the applicant has been generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• DE 3837063 C1 [0003]
• US 5682567 [0005]
• EP 1235424 A2 [0006]
• DE 10261665 C1 [0033]
• EP 1432231 A2 [0033]

Claims (9)

Method for high-resolution digital color image recording with high dynamic range, in which by means of scanning movement of a two-dimensional color sensor, an increase in resolution of the image acquisition by successive coverage of all different color-sensitive sensor elements by color co-site sampling is performed, characterized in that - intermediate samples with sampling steps of a fraction of Center distances of the sensor elements of the color sensor are integrated as additional sampling positions in the scanning movement of the Color-CoSite sampling, so that by means of a defined microscanning additional partial images at least a doubling of the sampling density in each dimension of the color sensor generate, and - at each sampling position, both of the Color -CoSite sampling provided positions of the sensor elements as well as additionally sampled by intermediate scanning scanning positions, additional partial image recordings with different Sensl brightness settings or exposure time settings of the color sensor to produce high-dynamic-range imaging. A method according to claim 1, characterized in that when using a color sensor with Bayer color mask, an overall image composed of sub-images of four color co-site sampling positions and of at least one intermediate sample per pixel pitch and dimension of the color sensor having 4 × 4 microscanning is composed , A method according to claim 1, characterized in that the microscanning is carried out with a regular line-shaped or column-shaped scanning pattern. A method according to claim 3, characterized in that the Micro scanning is carried out with a regular meandering scanning pattern. A method according to claim 1, characterized in that the microscanning is carried out with a regular shortened maanderförmigen scanning pattern in which the color co-site sampling positions are approached with adjusted time interval. Method according to Claim 1, characterized in that the microscanning is carried out with an irregular scanning pattern in which the color co-site sampling positions and the intermediate sampling positions for the individual partial image recordings are approached with as similar a time interval as possible, with adjacent scanning positions, which would be scanned directly in succession in a regular meander-shaped scan grid, are approached with a time interval of at least two Microscanschritten. A method according to claim 1, characterized in that for the HDR imaging two partial image recordings are recorded with different sensitivity settings of the color sensor per Microscanposition. A method according to claim 1, characterized in that for the HDR imaging three field images are recorded with different sensitivity settings of the color sensor per Microscanposition. Method according to one of the preceding claims, characterized in that in the control electronics for the realization of different resolution levels of the color images different settings of Microscanmodi the scanner and drive and readout modes of the color sensor are stored.

Images