EP1438697A2

EP1438697A2 - Method and device for producing light-microscopy, three-dimensional images

Info

Publication number: EP1438697A2
Application number: EP02782898A
Authority: EP
Inventors: Frank Sieckmann
Original assignee: Leica Microsystems Wetzlar GmbH; Leica Microsystems CMS GmbH
Current assignee: Leica Microsystems CMS GmbH
Priority date: 2001-10-22
Filing date: 2002-10-14
Publication date: 2004-07-21
Also published as: JP2005521123A; US20040257360A1; WO2003036566A3; WO2003036566A2

Abstract

The invention relates to a device for imaging a three-dimensional object (22) as an object image (30), which comprises an imaging system that is especially equipped with a microscope for imaging the object (22) and a computer. Actuators change the position of the object (22) in the x, y and z direction in a specific and rapid manner. A recording device records an image stack (26) of individual images (24) in different focal levels of the object (22). A control device controls the hardware of the imaging system, and an analytical device produces a three-dimensional elevation relief image (28) and a texture (29) from the image stack (24). A control device combines the three-dimensional elevation relief image (28) with the texture (29).

Description

Method and device for generating light microscopic, three-dimensional images

The invention relates to a method for displaying a three-dimensional object according to the preamble of claim 1 and a device therefor according to the preamble of claim 17

Known devices of this type, such as microscopes, macroscopes, etc., are used to view an object with the aid of basic physical laws. Limits in sharpness and depth, viewing angle and time dependency have to be accepted despite good technology.

In order to improve the depth of field and the physical limits in microscopic imaging methods, there are already a number of devices and methods. Such devices can be, for example, all types of light microscopes. This includes, for example, a confocal scanning microscope. A sample with the focus of a light beam is scanned point by point in a plane, so that an image, albeit with a shallow depth of field, is obtained on this plane. The object can then be represented three-dimensionally by recording a plurality of different planes and by corresponding image processing. Such a confocal scanning microscopy method is known for example from US Pat. No. 6,128,077. However, the optical components used in confocal scanning microscopy are very expensive and, in addition to a good technical understanding of the operating personnel also a lot of adjustment work.

A method for luminescence microscopy is also known from US Pat. No. 6,055,097. In this case, dyes are introduced into a sample that fluoresce under suitable lighting conditions, so that the irradiation makes it possible to localize the dyes in the sample. To create a spatial image, a number of images are recorded in different focal planes. Each of these images contains image information that originates directly from the focus plane, as does image information that originates from spatial sections of the object that lie outside the focus plane. To determine a sharp image, it is necessary to eliminate the parts of the image that do not originate from the focal plane. For this purpose, it is proposed to provide the microscope with an optical system that makes it possible to illuminate the sample with a special illumination field, for example a standing wave or a non-periodic excitation field. These usual microscopic images are optically limited due to the limited depth of field of the imaging method and because of the way of viewing, i. H. limited by the angle of view, in their representation. Microscopic images can be partially out of focus. This blurring can be explained. a. due to non-planar objects since the object surface is often not completely in the respective focal plane. In conventional imaging systems, due to the direction of view of the object specified by the microscope or the macroscope, no other viewing angle on the object is possible (e.g. tangential to the object surface) without time-consuming re-preparation and new adjustment of the object itself.

In all of these light-optical processes, the imaging accuracy is limited by a limitation of the depth of field.

The unpublished DE 101 49 357.6 describes a method and a device for generating a depth-sharp three-dimensional surface image of microscopic objects. For this purpose, the surface profile of the object becomes optical in a three-dimensional coordinate system (x, y, z) measured. With this method, different focal planes of a microscopic object are recorded digitally or analogously by a CCD camera. This creates an image for each focal plane, so that an “image stack” is created. This image stack is composed of images that come from the different focal planes of an object that is immovably under the microscope during recording. Each of these images in the image stack contains areas of sharp image structures with high detail sharpness and areas that were outside the focal plane when the image was taken and are therefore blurred and without high detail sharpness in the image. In this sense, an image can be understood as a number of image partial areas of high detail sharpness (in focus) and low detail sharpness (Out of focus) By means of image analysis methods, the subsets of the image areas that are present in high detail sharpness can be extracted from each image of the image stack. In a result image, all extracted subsets of each image with high detail sharpness, ie image areas lying in focus, now become one new overall picture d put together. This creates a new, completely detailed picture.

Since the relative position of the focal planes to one another is known, from which the subset of each image comes with a high level of detail, the distance between the images in the image stack is also known. Thus, a three-dimensional surface profile of the microscopic object can also be created.

In order to be able to carry out a deeply focused image and a three-dimensional surface reconstruction of the recorded object area, a previously created sequence of images from different focal planes of the object is required.

So far, the change in the focal plane has been achieved by adjusting the microscope stage height, that is, by adjusting the distance between the object and the objective, by mechanically adjusting the support table. Due to the high table mass and the associated inertia of the overall system, it was not possible to fall below certain speed limits for the image acquisition in different focal planes. For this purpose, the unpublished DE 101 44 709.4 describes an improved method and an improved apparatus for the rapid generation of precise image recordings of the individual images of the image stack in the different focal planes by means of piezo actuators in connection with stepper motor-controlled and / or servomotor-controlled methods. With this method, a focal plane adjustment, through a precise and quick change of the distance between lens and object, as well as an object position adjustment in the xy-plane by different actuators such as piezo lenses, piezo preparation tables, combinations of piezo actuators and standard stepper motor adjustments or but achieved any other table adjustments. The use of piezo actuators increases the precision and fineness of the adjustment. Furthermore, the adjustment speed is accelerated by piezo actuators. It also describes how the image quality and evaluation quality can be further increased by suitable insertion or application of deconvolution techniques.

Surfaces scanned in this way by means of automatically adjustable object holders, however, do not allow a holistic, in-depth view of the object itself. Also, a three-dimensional representation of the entire scan area is not possible. Furthermore, the representation in space cannot be rotated and viewed from different angles

The object of the present invention is therefore to provide a method and a device for generating light-microscopic, three-dimensional images, which works with less technical effort and at the same time provides improved image quality in the three-dimensional representation.

This object is achieved by a method for displaying a three-dimensional object with the features according to claim 1 and by a device with the features according to claim 10.

According to the invention, an image stack is obtained from a real object, which consists of light microscopic images. With a suitable method, in particular a software method, a surface relief image is then obtained from the image stack, which is then combined with a texture is that an image of the object is created. To combine the texture with the elevation relief image, it is particularly advantageous to project a texture onto the elevation relief image. The texture can in turn be obtained from the data of the image stack.

With this method, a virtual image of a real object can be generated that meets all the requirements placed on a virtual image. This object image can also be processed with the manipulations possible with virtual images. In general, attempts are made in virtual reality to represent reality as well as possible by means of suitable methods, which are implemented in particular in a computer program, by means of virtual objects which have been appropriately calculated. Through virtual lamps and shadows, through the simulation of basic physical laws and properties such as refractive index settings, simulation of the elasticity of objects, effects of gravity, tracking of a virtual light beam in virtual space under the influence of matter, so-called ray tracing and many other properties, succeed increasingly real replicas of reality on the computer.

Normally, the scenes and processes are completely re-created by the designer in purely virtual rooms, or existing resources are used. In the present invention, on the other hand, a real imaging system, in particular a microscope, is used to generate the data necessary to generate a virtual image of reality. This data can then be processed so that it is possible to automatically display a virtual three-dimensional structure. A special feature here is that a height relief is determined from the real object and this relief is provided with a texture, which is preferably determined from the data obtained from the object. Particularly good results are achieved with the projection of the texture onto the height relief image.

A major advantage of the invention is therefore to be seen in that

Use of the method according to the invention, the usual light microscopy and light macroscopy is expanded by combining the raw data obtained by real light imaging systems such as light microscopes or light macroscopes, such as static three-dimensional surface information or fuzzy image information, to form a new image. Thus, all or any combination or subset of the real partial information can be displayed simultaneously.

Another advantage is that individual or temporally consecutive, in-depth calculated multi-focus images with associated three-dimensional surface information also obtained are fused together. This amalgamation process is realized by the fact that the depth-sharp multifocus image is understood as a surface texture of an associated three-dimensional surface. The fusion process is achieved by projecting this surface texture onto the three-dimensional surface.

The new three-dimensional virtual image obtained according to the invention thus carries both information at the same time, the three-dimensional surface information and the completely sharp image information. This image representation can be referred to as "virtual reality 3D light microscopy", since the data fusion described cannot be carried out in "real" microscopes.

To generate the image stack, which consists of individual images that are recorded in different focal planes of the object, the method steps already described in more detail above can be carried out. In particular, the method described in DE 101 49 357.6 can be used to generate a three-dimensional surface reconstruction. These provide two data records in the form of an image. One encodes the height information of the microscopic object and is called the mask image below.

The second data set represents a high-contrast, completely deep microscope image and is referred to below as a multi-focus image. This multifocus image is generated with the aid of the mask image by the mask image image - gray values are used to identify the level of a sharpest pixel and to copy the corresponding pixel of the level in the image stack into a summarizing multifocus image.

As described further above, the method steps as described in DE 101 44 709.4 can be used, for example, by means of piezo technology on lenses and / or preparation tables, mask images and multifocus images in higher resolution in the direction of the focal planes (z direction) and by means of a Scan the object over larger areas in the respective focal plane (x, y direction).

The mask image thus carries the height information and the multi-focus image the pure, in-depth image information. A three-dimensional height relief image (pseudo image) is then created from the mask image. This is due to the fact that the mask image is shown as a relief in height. The pseudo image carries no direct image information other than the height information. The three-dimensional pseudo image therefore represents a so-called height relief. In a further step, the three-dimensional pseudo image is provided with the real texture of the sharp image parts of the image stack. For this purpose, the pseudo image and the mask image are suitably aligned, specifically in such a way that the height information of the pseudo image and image information of the mask image, that is to say the texture, are superimposed on one another with pixel accuracy. In this way, each pixel of the multifocus texture image is mapped exactly to its associated pixel in the three-dimensional pseudo image, so that a virtual image of the real object is created.

Previously common light microscopic methods for imaging objects are limited in their display options by an abundance of physical limitations. The invention largely overcomes these limitations and gives the user many new possibilities for examining and displaying microscopic objects.

A suitable user interface can also be defined for the application of the invention, which allows the user to use the invention even without special technical knowledge. Furthermore, the invention is also applicable for three-dimensional display of large areas. By imaging real microscopic or macroscopic image information into a "virtual reality room", conventional light microscopy is given access to the entire technology of virtual worlds. The resulting images provide a much clearer and more meaningful microscopic image than conventional light microscopy and displace it Users are thus able to use any previously known mapping and manipulation methods of virtual reality.

The virtual image has no limitation in the sharpness that arises in conventional object images due to the limited depth of field of the optics used. So it is perfectly sharp. The virtual image also carries the complete depth information. This creates a completely sharp, three-dimensional, lifelike virtual image of a real microscopic object.

In a preferred embodiment of the invention, the mapping can be implemented virtually in a computer. All options for image display and manipulation that can be used with virtual images are now available here. These range from the superimposition of real microscopic surfaces and purely virtual surfaces to the possibility of a view from any angle onto a depth-focused three-dimensional surface. The surfaces can be virtually animated, illuminated or otherwise modified. Time dependencies such as changes in time on the surface of the microscopic object can be mapped simultaneously with in-depth image information and three-dimensional surface topologies.

This opens up completely new possibilities for light microscopy, and limitations in image quality due to physical limitations are compensated for.

In an advantageous embodiment of the invention, the following components are used: 1. Microscope with necessary accessories (lenses, etc.) or another suitable imaging system such as a macroscope

2. Computer with suitable accessories such as a monitor, etc.

3. Actuators for the targeted, rapid change of position of an object in x, y, and z - direction such as piezo, stepper motor table, etc.

4. A camera, in particular an analog or digital CCD camera with necessary or advantageous accessories such as grabber, firewire, hotlink, USB port, Bluetooth for radio data transmission, network card for image data transmission via. Network, etc.

5. A control device for controlling the hardware of the microscope, in particular the preparation table, the camera, and the lighting.

6. An analysis device for creating the multifocus images, the mask images, the mosaic images and the creation of the "virtual reality 3D light microscope images". Control and analysis methods are preferably implemented as software.

7. A device for displaying, calculating and manipulating the “virtual reality 3D light microscope images” such as rotation in the room, changes in lighting, etc. This is in turn preferably implemented as display software.

Software used in a computer controls the microscope, the preparation table in the x, y, and z directions, any piezo actuators, lighting, camera image recording, and other microscope hardware. The procedure for generating the mask and multifocus images and the creation of a "virtual reality 3D microscope image" can also be controlled via this.

By using a piezo-controlled lens or a piezo-controlled object holder or the combination of a piezo-controlled object ves with a piezo-controlled object holder, an object can be positioned very quickly, reproducibly and precisely in all three spatial directions. In combination with the depth of field enhancing image analytical processes and the possibilities for 3D reconstruction, a fast, 3D reconstruction of microscopic surfaces is possible. Furthermore, picture mosaics can be generated quickly, which are both in focus and can create a dimensional surface profile. The individual images are recorded by a suitable CCD camera. Unfolding the individual images with a suitable apparatus profile before the subsequent focus calculation and 3D reconstruction also allows the generation of extremely high-resolution, apparatus profile-corrected microscope images with a high depth of field.

In a further advantageous embodiment of the invention, a plurality of image stacks are recorded in chronological order. The above-described conversion of these temporal individual images of the image stacks into temporally successive virtual reality 3D images enables three-dimensional, completely sharp imaging of time processes in an animation, such as a film.

A further advantageous embodiment of the invention is obtained through the use of so-called morphing, in which several images are converted into one another in an animation. This is an interpolation between images, such that further, previously unknown intermediate images are calculated on the basis of a known start image and a known end image. If you then suitably line up the start, intermediate and end images and then play back the known and interpolated images in succession, you get the impression of a continuous transition between the start and the end image.

Morphing accelerates the described method in that only a few images have to be taken in real time and in space. All other images for a virtual representation are calculated by inter-image interpolation.

It is a particular advantage of the present invention to produce one "Virtual Reality 3D light microscopy image" to use real data from light-optical imaging systems such as light microscopes or light macroscopes. It should be noted that the distortions in light macroscopes caused by the imaging optics are first mathematically corrected. According to the invention, virtual reality becomes automatic, semi-automatic or generated manually from the underlying real data. The possibility of linking the obtained data of "virtual reality 3D light microscopy" with already known techniques of virtual reality, the data generated purely virtually, ie without direct influence of real physical data, is a further possibility Advantage of the invention.

Another advantage of the invention is the possibility of being able to carry out 3D measurements, such as volume measurements, surface measurements, etc., using the data of the "virtual reality 3D light microscopy".

A further advantageous embodiment of the invention offers the possibility of projecting image-analytically influenced and / or changed texture images onto the 3D surface, as described above. This enables a further “expanded perception” through “virtual reality 3D light microscopy”, since the changed textures are projected onto the 3D surface in a location-specific manner. This enables image analysis results to be combined with three-dimensional surface data and displayed simultaneously. This also applies to time series of recordings influenced by image analysis in the above sense.

Another advantage of the invention is to use the method for mosaic images, so that defined partial surface areas of an object are scanned. These partial images are put together in depth and, in addition to the associated 3D object surface data, are calculated into a “virtual reality 3D light microscopy image” of the scanned object surface.

With regard to its advantages, the invention is particularly distinguished by the fact that a considerable expansion of the perception of microscopic facts on the object is possible. This is done by simultaneously displaying a completely sharp image on a microscopic image three-dimensional surface. Through the virtual 3D reality of the microscope image and its compatibility of the virtual representation with standard programs and processes, it is possible to integrate all knowledge and all the possibilities that were previously generated for virtual reality.

The images generated with the method according to the invention correspond more to the actual conditions in the sample than those obtained with conventional microscopes. Because in the "Virtual Reality 3D light microscope image", in addition to complete sharpness, the three-dimensional information of the object is clearly produced.

In addition, "Virtual Reality 3D light microscope image" can be viewed from different solid angles by rotating the image into any position. In addition, the object image can be manipulated by transparencies and other standard methods to amplify or weaken other microscopic details.

The informative value and a three-dimensional representation of a microscopic object that comes much closer to human perception enable completely new analysis methods. Image mosaics, which are displayed as a "Virtual Reality 3D light microscope image", additionally expand the display.

The possibility of fully automating the above-mentioned processes for generating a “virtual reality 3D light microscope image” or several “virtual reality 3D light microscope images” using automatic time series places only minor demands on the technical know-how of the users.

Combinations of the "Virtual Reality 3D light microscope image", which was generated from real recorded basic data, with the possibilities of superimposing purely virtual objects such as Platonic basic bodies or other more complex bodies, result in new didactic possibilities of knowledge transfer. The combination of the data of the "Virtual Reality 3D Light microscope image "with 3D cyberspace glasses allows a view of microscopic see objects with unprecedented precision and wholeness.

Since the data of the "virtual reality 3D light microscope image" can be stored in a computer, the data can be displayed on other systems, data can be transferred via computer networks, such as on the intranet or the Internet, and the "virtual reality 3D light microscope image data" can be displayed via a web browser. A three-dimensional image analysis is also possible.

Virtual microscopy, i.e. microscopy by users "without" a microscope, i.e. only on the basis of the obtained and / or stored "virtual reality 3D light microscope image data", enables a separation between real microscopy and evaluation of the obtained data.

The use of conventional standard light microscopes with standard illumination is possible to generate the 3D image according to the invention and is therefore inexpensive.

Further advantages and advantageous embodiments of the invention are the subject of the following figures and their descriptions, the representation of which is not drawn to scale in the interests of clarity.

They show in detail:

Fig. 1: a schematic flow of the invention

Process.

Fig. 2: a schematic sequence of the invention

Procedure based on an example

3: a schematic sequence of the method according to the invention using an example

4a: Example of a pseudo picture

4b: Example of a structured pseudo image Fig. 5: combination of a texture with a pseudo image based on a

example

Fig. 6: Schematic automated process flow

The basic sequence of the method according to the invention is shown schematically in FIG. 1, which is again illustrated in FIGS. 2 and 3 using a schematic example. Starting from an object 22 (FIG. 2), an image stack 24 is generated in method step 10 in that individual images 26 are recorded manually or fully automatically from a plurality of focal planes of the object 22. The distance between the individual images is suitably dimensioned in order to be able to reconstruct a three-dimensional image with a sharp focus and is preferably kept equidistant. Each of the individual images 26 has sharp and unsharp image areas, the image spacing and the total number of individual images 26 being known. After recording, the images are first stored in uncompressed form or compressed using a lossless compression process in step 12. The individual images 26 can be color images or grayscale images. The color or Gray value resolution (8 bit, 24 bit etc.) can be arbitrary.

When creating the image stack, it can also be done in such a way that several images lie side by side in a focal plane (in the x, y direction) and are reassembled with pixel accuracy so that a so-called mosaic image of the focal plane is created. It is also possible to generate an image stack 24 from the mosaic images. After an individual image has been recorded in each desired focal plane (z plane), an image stack 24 with a series of individual images 26 is available for further image processing. The z-planes are preferably equidistant from one another.

An imaging system can be used to create the image stack 24, in particular using a microscope or a macroscope. However, a suitably attached camera system with a lens can also be used. The entire lighting range of a specimen from near UV to far IR can be used, provided that the education system makes this possible.

The recording system can generally have any analog or digital CCD camera, it being possible for all types of CCD cameras, in particular line cameras, color cameras, gray image cameras, IR cameras, integrating cameras, cameras with multi-channel plates, etc. to be used.

In a further method step 14, a multi-focus image 15 and a mask image 17 are now obtained from the data obtained from the image stack 24, in which case the methods according to DE 101 49 357.6 and DE 101 44 709.4 can be used in particular. Due to the depth of field of the microscope, each of the individual images contains 26 sharp and unsharp areas. The respective sharp areas in the individual images 26 are determined according to certain criteria and assigned to the corresponding coordinate points (x, y) with their plane number. The assignment of level numbers and coordinate points (x, y) is stored in a memory and represents the mask image 17. When the mask image 17 is processed, the level numbers stored in the mask image can be viewed as gray values.

All unsharp image areas of the individual images of the previously recorded and stored image stack 24 are removed in the multifocus image 15, so that a completely sharp, in-depth image is produced. The multifocus image (15) can also be created from a stack of mosaic images in such a way that multiple mosaic images from different focus planes are calculated to form a multifocus image (15).

In the mask image 17, all gray values of the pixels identify the number of the origin level of the sharpest pixel. The mask image can thus also be represented as a three-dimensional height relief 28. The three-dimensionality results from the x, y positions of the mask image pixels and the amount of the gray value of a pixel, which characterizes the focal plane position of the three-dimensional data set. The mask image 17 can also be created from a stack of mosaic images, with a plurality of mosaic images from different focal planes being calculated into the mask image 17. Now that the mask image 17 is available, a so-called three-dimensional pseudo image 28 can be created from it. For this purpose, the mask image 17 is represented as a height relief in method step 16. Apart from the height information, this image does not carry any direct image information. For this purpose, the mask image 17 is represented by a suitable software as a dimensional height relief. The software can be developed, for example, on the basis of the well-known software libraries OpenGL or Direct3D (Microsoft). Other commercially available software packages are also suitable for displaying, creating, animating and manipulating 3D scenes such as Cinema 4D (Maxon), MAYA 3.0, 3D Studio MAX or Povray.

So-called splines are used to generate this representation. Splines are essentially sequences of vertices located in three-dimensional spaces, which are connected with each other by lines. Splines are well known from mathematics in the technology for creating three-dimensional objects. She. to a certain extent represent contour lines on a map. The support points are supplied by the gray values of the mask image, such that the coordinates (X, Y, Z) of the support points for a spline interpolation correspond to the following mask image data

The base coordinate X corresponds to the mask image pixel coordinate X

Base coordinate Y corresponds to the mask image pixel coordinate Y

Base coordinate Z corresponds to the gray value at X, Y of mask image 17.

The course of the spline curves is determined by the so-called interpolation. The course of the spline curves is calculated by an interpolation between the base points of the splines (polynomial fit of an n'th order polynomial by a predetermined number of points in space, such as by Bezier polynomials or Bernstein polynomials etc.), so that the spline curves arise. Depending on the type of interpolation function used and the number of support points, more or less detailed curve adjustments can be made to the given support points. By not all makeup If the points are considered as support points for splines but only a suitably selected subset of the mask image points, the number of support points can be varied. For example, only every fourth pixel of the mask image 17 can be used. A subsequent interpolation between the lower number of reference points would represent the object surface in a lower resolution. Thus, by adjusting the number of reference points, it is possible to display surfaces with a greater or lesser degree of detail and thus to filter out various surface artifacts. Fewer support points therefore lead to a smoothing effect on the three-dimensional surface.

In the present invention, the previously calculated mask image forms the base database. The base points are in a 3D space and must therefore be described by three spatial coordinates. The three spatial coordinates (x, y, z) of each support point for splines is formed by the x, y pixel positions of the mask image pixels and the gray value of each mask pixel (z position). Since the gray values in a mask image anyway correspond to the height information of the underlying microscope image, the 3D pseudo image can be interpreted as a representation of the height profile of the underlying microscope image.

If you therefore specify an array of support points which contains all or a suitable subset of the mask image points and mask image point coordinates, a spline network of selectable density can be placed over the base point array. A three-dimensional pseudo image 28 obtained in this way is shown in FIG. 4a.

As shown in FIG. 4b, a fine structure can then be placed over this surface by suitable triangularization and shading, such as Gouraud shading. With ray tracing algorithms, surface reflections and shadows can also lead to very realistic surfaces 28 '.

Furthermore, it is necessary to link the three-dimensional pseudo image 28 with a texture 29. Under texture 29, a basic element for the

Understanding the surface design of virtual structures, achieving the goal to give surfaces a natural and realistic look. Thus, in step 18, a texture 29 is generated from the previously provided multifocus image 15. For this purpose, for example, the previously calculated depth-of-focus multifocus image 15 is now used as a texture image.

In order to link the information determined in particular, which is present in particular in the multifocus image 15, with the three-dimensional pseudo image 28, the three-dimensional pseudo image 28 is linked with the texture 29 in method step 20, as shown in FIGS. 1-3.

Texture 29 is understood here to mean, as is usual in virtual reality, in particular an image which is suitably projected onto the surface of a virtual three-dimensional object by three-dimensional projection methods. To achieve the desired effect, the texture image must be appropriately aligned on the surface of virtual objects. For a suitable alignment, it is necessary to assign the texture 29 to the three-dimensional pseudo image 28 in such a way that the affiliations of the pixel coordinates (x, y) of the mask image 17 and the multifocus image 15 are not destroyed. Each mask pixel with the gray value at the location (Xj, y _j ) is therefore assigned its corresponding multifocus pixel with the gray value at the exact same location (Xj, y _j ). If the multifocus image 15 has previously been changed analytically or by other image manipulation, care must be taken that the affiliations of the pixel coordinates (x, y) of the mask image and the image analysis or any other manipulation of the multifocus image modified as desired are not lost.

To link the pseudo image 28 with the texture 29, the texture 29 is advantageously appropriately projected onto the three-dimensional pseudo image 28. As a result, the two resources are merged in such a way that a three-dimensional object image 30 of the object 22 is produced. This object image 30 represents a virtual image in the sense of virtual reality.

As shown in the example according to FIG. 5, the basis for the texturing according to the invention is formed by the multifocus image itself, which was previously calculated. The already realistic pseudo image 28 and the mask image 17 is suitably aligned in such a way that the height information of the pseudo image 28 and the image information of the mask image 17, ie the texture, are superimposed on one another with pixel accuracy. The multifocus texture image, that is to say the texture 29, is projected onto the three-dimensional pseudo image 28, so that each pixel of the multifocus texture image 29 is imaged exactly on its associated pixel in the three-dimensional pseudo image 28. As a result, a deeply focused object image 30 of the object 22 is thus generated as a virtual image by merging virtual and real imaging techniques.

With the course of the method shown schematically in FIGS. 1-3, the novel mapping according to the invention is based on real measured values of a real existing object 20, which are brought together in such a way that a virtually real three-dimensional mapping of the light microscopic data is produced. In comparison with conventional virtual techniques, a real image of an object 22 is carried out in the present invention. Data on the image sharpness, the topology of the object and the precise position of sharp partial areas of an image in three-dimensional space are recorded on the real object 22. This real data then serves as the basis for generating a virtual image in a three-dimensional space. The virtual image, which determines data from the real images, such as image information, sharpness and three-dimensionality and at the same time maps them, creates a clear added value compared to conventional light microscopy.

According to the invention, a new type of light microscopy is thus proposed, the core properties of which are the acquisition of real, for example light microscopic, object data and their summarized representation in a three-dimensional virtual space. In this sense, the method according to the invention can be referred to as “virtual reality 3D light microscopy”. In this “virtual reality 3D light microscope”, the generated images of reality (3D, sharpness, etc.) can also be used by all currently known and developed in the future Methods and procedures of virtual imaging techniques are influenced. After the manual and fully automatic generation of a “virtual reality three-dimensional light microscopy image” was described in the previous exemplary embodiment, a method for visualizing, manipulating and analyzing the “virtual reality three-dimensional light microscopy images” is described in a further exemplary embodiment.

In order to visualize the object image data 30 due to the transformation of real microscopic data into a virtual space, the microscopic data of the object image 30 are now available as three-dimensional, in-depth images.

The surface of the object image 30 can now be illuminated by virtual lamps in order to visually highlight certain details of the microscopic data. The virtual lamps can be placed anywhere in the virtual room and the properties of the virtual lamps such as radiation characteristics or light color can be set flexibly.

This process allows the creation of much better and permanently preserved microscope images for teaching and documentation purposes.

The images can be rotated and scaled in space using rotation and translation operators. This operation allows viewing the images from angles that are impossible in a normal microscope.

By gradually changing the orientation of a "virtual reality 3D light microscopy image" and storing these individual images, animation sequences can also be created that simulate a movement of the "virtual reality three-dimensional light microscopy image". These animation sequences can be played back by saving these individual images as a film sequence (e.g. in AVI, MOV data format, etc.).

The data can also be manipulated. The image of the three-dimensional pseudo-image is available as a base for a three-dimensional spline interpolation. With Gouraud shading and ray tracing, these three-dimensional data can finally be assigned a three-dimensional surface. The x, y, z bases can play a central role in data manipulation, which are used, for example, for measurement purposes or to emphasize certain details more clearly.

Multiplying the z-values by a number would mean, for example, stretching or compressing the height relief. By targeted manipulation of individual bases, certain parts of the 3D profile of the three-dimensional pseudo image 28 can be manipulated individually.

Image-analytic manipulations of the projected multifocus texture image can also be used to project image-analytical results such as marking individual image objects, highlighting edges, object classifications, binary images, image improvements, etc. This is done by using an image-analytically modified output image (multifocus texture image) as a new "manipulated" multifocus texture image and projecting the new image as a texture onto the three-dimensional surface of the 3D pseudo image. In this way, image-analytically manipulated images (new textures) can also be used with the three-dimensional pseudo image can be merged.

There are both possibilities of 3D data manipulation such. B. the manipulation of the bases for the spline interpolation as well as the possibility of manipulating the multifocus image by image analysis methods.

A fusion of these two representations can lead to an added value in the microscopic image representation, since the object images 30, in addition to the three-dimensional representation, carry a location-specific overlay of the image-analytically manipulated multifocus images.

Because of the transformation of the data of the real object 22 into data which are present in a virtual space, the three-dimensional data can now be measured with regard to their volume, the surface or the roughness etc.

A further improvement allows the combination of the image analysis measurement results on the multifocus image and the three-dimensional data measurements. By logically linking the three-dimensional data With suitable other three-dimensional objects, a multitude of calculations can also be carried out on three-dimensional data.

The mere type of representation thus extends the two-dimensional image analysis by a third dimension of the image analysis and by a topological dimension of the data analysis.

By recording time series, i.e. H. by taking pictures of the object 22 at different successive times according to the described method, another dimension is added to the data analysis, the temporal dimension. This then enables the representation of a time process, for example the change over time in object 22 in slow motion or in time lapse.

The method according to the invention is also suitable for stereo image generation and stereo image animation. Since the data of the object image 30 are three-dimensional, two views of a virtual microscope image can be calculated from any viewing angle. This allows a visualization of the "Virtual Reality 3D light microscope image" in the sense of a classic stereo image.

In addition to the output on a monitor, printer or plotter, the "virtual reality 3D light microscope image" can also be visualized by means of polarization shutter glasses or in anaglyph technology or by imaging using 3D cyberspace glasses.

By means of an animation of a series of different views of the “virtual reality 3D light microscope image”, which is perspectively separated for the right and left eye, a moving stereo image of a “virtual reality 3D light microscope image” generated from real microscopy data can also be obtained using one of the visualization methods described above. to be generated.

Since the data are available in three dimensions, a perspective-correct view of the "virtual reality 3D light microscope image" can be calculated for the left and right eyes. Thus, the "virtual reality 3D light microscope images" can also be displayed on 3D output units such as 3D stereo LCD monitors or but cyberspace glasses are issued.

With a 3D LCD stereo monitor, the current eye position of the viewer is measured using an image analysis. The respective point of view is then calculated from this data. From this, the data for the perspective view follow on the "Virtual Reality 3D light microscope image" for the right and left observer's eye. These two perspectives are calculated and displayed on the 3D-LCD stereo monitor. This creates the impression for the observer that the "Virtual Reality 3D light microscope image "floating in the room in front of the monitor. This makes it possible to display real microscopy data in such a way that a three-dimensional representation of reality is created. With temporally and perspectively correct image sequences, spatially animated three-dimensional images of real microscope images can also be realized.

In the case of cyberspace glasses, for technical reasons an image with a correct perspective view is shown separately in each eye. The viewer's brain creates a three-dimensional impression. With temporally and perspectively correct image sequences, spatially animated three-dimensional images of real microscope images can also be realized here.

In a further embodiment of the invention, it is possible to combine the data obtained from the “virtual reality 3D light microscopy” with one another in such a way that processes that change over time can also be animated and visualized in the “virtual reality 3D light microscopy”. In addition to the three spatial coordinates X, Y, Z, data manipulation can also be carried out for this, which is based on the texture 29 - pure, sharply calculated image information of the object (multifocus image) or on temporal changes in the surface and / or the texture (time series of images) ) relates.

As in the previously described methods, changes over time in microscopic objects can also be recorded at different times by repeatedly recording the same image stack in the z direction (in the direction of the optical axis of the imaging system). This creates a series of image stacks that are too unrelated to the conditions in object 22. corresponds to different times. Both the three-dimensional microscopic surface structures and the microscopic image data itself can change over time.

Through a time series of always the same microscopic area, a series of temporally successive mask images and associated multifocus images is created in such a way that

Mask [t _. ] -> Mask [t ₂ ] ->-> Mask [t _n ] -> Maskeft _n + ι]

and belonging

Multifocus [t ₁ ] -> Multifocus [t ₂ ] ->->-> Multifocus [t -] -> Multifocus [t _{n + 1} ]

In the event of changes in the topology over time

Mask [t _n ] not equal to mask [t _n + ι] {n = 1, 2,3,4, ...}

and applies to visual changes

Multifocus [t _n ] not equal multifocus [t _{n + 1} ] {n = 1, 2,3,4, ...}

These time series can be generated both manually and automatically.

By recording time sequences of multi-focus mosaic images and the associated mosaic mask images, temporal kinetics of surface changes and / or image changes is also possible.

As shown in FIG. 6, the method sequence for generating an animation can be integrated into the method sequences already known from DE 101 49 357.6 and DE 101 44 709.4, so that a fully automatic sequence can also be implemented. To this end, the process sequence already known from these documents is supplemented by further automatable process steps. If the process sequence for creating a virtual reality object image 30 is started, a virtual reality object image 30 can be performed in step 32 are generated as described above. This object image 30 can be animated as desired in step 34. The animated image is then preferably stored in step 36. In this way, mosaic images, mask images and mosaic multifocus images can be created and saved at specific times. These mask and multifocus images then serve as the starting point for a combination of the associated mask and multifocus images.

In a second step, the associated masks and multifocus images can then be combined to form individual images in "Virtual Reality 3D light microscopy".

A time series of individual “virtual reality 3D light microscopy images” thus arises. Each simultaneously contains the 3D information of the mask image and the projected texture of the multifocus image. However, when the object 22 changes over time, the three-dimensional topological appearance and / or changes may occur distinguish the texture 29.

A sequence of the individual images enables a temporal animation of the recordings with the possibilities of "Virtual Reality 3D light microscopy".

It is thus possible to simultaneously display three-dimensional surface information, changes in the surface over time, multi-focus images calculated in depth and changes in the time of these multi-focus images.

The required mask images 17 and multi-focus images 15 can also be understood as a mosaic mask image and a multi-focus mosaic image created by scanning a surface of the object 22 at certain times.

By rotating these “virtual reality 3D light microscopy” images, the simultaneously imaged features, such as three-dimensional surface information, temporal changes in the surfaces, in-depth calculated multifocus images and temporal changes in these multifocus images, can also be viewed under other viewing angles. The data set, which describes a three-dimensional surface, is subjected to a suitable, three-dimensional transformation.

In summary, the described “virtual reality 3D light microscopy” can thus be understood as a simultaneous mapping of five dimensions of microscopic data of an object 22. The five dimensions are:

X, Y, Z - pure three-dimensional surface information of the object 22

»The texture 29, i.e. pure, sharply calculated image information of the object 22

• The temporal changes in the surface and / or the texture as a time series of images

LIST OF REFERENCE NUMBERS

10 Generation of an image stack of an object 12 Storage of the images of the image stack

14 Creation of a multifocus image and a mask image

15 multifocus image

16 Generation of a three-dimensional pseudo image

17 Mask image 18 Providing a texture 0 Linking the texture with the pseudo image 2 Object 4 image stack 6 Single image of a focal plane 8 Three-dimensional pseudo image 8 'Three-dimensional pseudo image with surface structure 9 Texture 0 Object image 2 Creating a virtual reality image 4 Creating an animation 6 Saving the image

Claims

claims

1. A method for displaying a three-dimensional object (22) as an object image (30), wherein the object (22) first records an image stack (24) from a plurality of images (26) in different focal planes, characterized in that from the image stack (24) a three-dimensional height relief image (28) is generated and the three-dimensional height relief image (28) is combined with a texture (29), in particular the texture (29) is projected onto the three-dimensional height relief image (28).

2. The method according to claim 1, characterized in that the texture and the three-dimensional height relief image (28) are obtained from the data of the individual images (6) of the image stack (24).

3. The method according to claim 1 or 2, characterized in that the three-dimensional height relief image (28) is formed by contour lines which run through support points which are connected to one another by interpolation.

4. The method according to any one of claims 1 to 3, characterized in that the texture (29) is projected onto the three-dimensional height relief image (28).

5. The method according to claim 4, characterized in that the texture (29) for projecting onto the three-dimensional height relief image (28) is precisely aligned to the three-dimensional height relief image (28).

6. The method according to any one of claims 1 to 5, characterized in that the three-dimensional height relief image (28) changes before combining with the texture (29), in particular by triangulation,

Shading and / or ray tracing algorithms is provided with a virtual surface.

7. The method according to any one of claims 1 to 6, characterized in that the texture (29) is obtained from the data of a multifocus image (15) which carries the in-depth information of the object (22).

8. The method according to any one of claims 1 to 7, characterized in that the three-dimensional height relief image (28) from the data

Mask image (17) is obtained, which carries the height information of the focal planes.

9. The method according to claim 3, characterized in that the three-dimensional height relief image (28) by stretching or compressing the

Points of the contour lines are changed before or after the combination with the texture (29).

10. The method according to any one of claims 1 to 9, characterized in that the object image (30) manipulated image analysis, in particular combined with a further texture.

11. The method according to any one of claims 1 to 10, characterized in that the generation of the object image (30) is started manually or automatically after the acquisition of the individual images (26) of the image stack (24).

12. The method according to any one of claims 1 to 7, characterized in that a plurality of temporally successive object images (30) are generated.

13. The method according to any one of claims 1 to 12, characterized in that the object image (30) is output on an output device, in particular a monitor, plotter, printer, LCD monitor or cyberspace glasses.

14. The method as claimed in one of claims 1 to 13, characterized in that the object image (30) changes before being output, in particular illuminated with a virtual lamp, processed by rotation or translation operators and / or is exposed to virtual physical laws.

15. The method according to any one of claims 1 to 14, characterized in that a plurality of object images (30) as a chronological sequence of object images (30), in particular as a film or animation, are output to the output device.

16. The method according to claim 15, characterized in that the plurality of object images (30) are converted into one another by morphing.

17. Device for displaying a three-dimensional object (22) as an object image (30) with an imaging system, in particular a microscope for imaging the object (22), a computer, actuators for targeted, rapid change in position of the object (22) in x, y, and z direction, a recording device, in particular an analog or digital CCD camera for recording an image stack (26) of individual images (24) in different focal planes of the object

(22), a control device for controlling the hardware of the imaging system, characterized in that an analysis device for generating a three-dimensional height relief image (28) and a texture (29) from the individual images (26) of the image stack (24) and a control device for combining the three-dimensional height relief image (28) with the texture (29) is provided.

18. The apparatus according to claim 17, characterized in that an output device for outputting the object image (30), in particular a monitor, plotter, printer, LCD monitor or cyberspace glasses, is provided.

19. The apparatus of claim 17 or 18, characterized in that a device for data analysis of the object image (30) is provided.

20. Device according to one of claims 17-19, characterized in that the analysis, control device and / or the device for

Data analysis of the object image (30) are implemented in the computer.