WO2001027679A1 - Microscope and method for manufacturing a composite image with a high resolution - Google Patents

Microscope and method for manufacturing a composite image with a high resolution

Info

Publication number
WO2001027679A1
WO2001027679A1 PCT/SE2000/001996 SE0001996W WO2001027679A1 WO 2001027679 A1 WO2001027679 A1 WO 2001027679A1 SE 0001996 W SE0001996 W SE 0001996W WO 2001027679 A1 WO2001027679 A1 WO 2001027679A1
Authority
WO
Grant status
Application
Patent type
Prior art keywords
images
image
recorded
area
means
Prior art date
Application number
PCT/SE2000/001996
Other languages
French (fr)
Inventor
Anders Heyden
Johan HÅKANSSON
Björn WALLIN
Adam Karlsson
Original Assignee
Cellavision Ab
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00Geometric image transformation in the plane of the image, e.g. from bit-mapped to bit-mapped creating a different image
    • G06T3/40Scaling the whole image or part thereof
    • G06T3/4038Scaling the whole image or part thereof for image mosaicing, i.e. plane images composed of plane sub-images
    • GPHYSICS
    • G02OPTICS
    • G02BOPTICAL ELEMENTS, SYSTEMS, OR APPARATUS
    • G02B21/00Microscopes
    • G02B21/36Microscopes arranged for photographic purposes or projection purposes or digital imaging or video purposes including associated control and data processing arrangements
    • G02B21/365Control or image processing arrangements for digital or video microscopes
    • G02B21/367Control or image processing arrangements for digital or video microscopes providing an output produced by processing a plurality of individual source images, e.g. image tiling, montage, composite images, depth sectioning, image comparison
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N1/00Scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission; Details thereof
    • H04N1/387Composing, repositioning or otherwise geometrically modifying originals
    • H04N1/3876Recombination of partial images to recreate the original image

Abstract

A microscope comprises an object holder (2), a display, optics (4) which in an image plane forms an image of an object (1) which is placed in the object holder (2), and an image sensor (5) for recording the image. The microscope further comprises a display (9), and an input means (10) for selecting an area in an image and a resolution in the area, a storage means and a calculation means which is connected to the image sensor. The calculation means is adapted (i) to digitally store in the storage means at least two recorded images, each of the images partly overlapping at least one other image, (ii) to join the at least two recorded images so that the composite image is an image of the area which is reproduced on the at least two recorded images, and (iii) to store in the storage means (11) information about the composite image. The calculation means is further adapted to show the selected area with the selected resolution on the display (9).

Description

Microscope and method for manufacturing a composite image with a high resolution.

Field of the Invention

The present invention relates to a microscope with an image sensor and a method therefor. Background Art For studying small structures, microscopes are generally used. There are a plurality of different microscop- ing methods, of which the most widespread one is optically imaging microscoping . A common feature of all microscopy methods is that only a limited part of the object can be examined with high resolution on a separate occasion. This area is usually called the field of view of the microscope. The field of view of an optically imaging microscope is determined among other things by the size of the lenses in the optics. The resolution is determin- ed among other things by the numerical aperture of the optics. It is possible to achieve a large field of view and good resolution by using large lenses with a short focal length. However, it is extremely expensive to produce to large lenses, especially if they should also have a short focal length.

As a result, it has only been possible to study relatively small areas on a separate occasion if at the same time high resolution is desirable.

A further problem of conventional microscoping is that it is necessary to study the object through the ocular and turn control means to move the field of view across the object. To sit studying the object through the ocular for a long time is tiring.

One frequently wants to study old preparations on a subsequent occasion. It has then been necessary to file the preparations and put them into the microscope once more, which however is time-consuming work. Moreover, it is not always possible to file the preparations. This applies in particular to preparations in vitro since the preparations must be dried before being stored, which may cause a change of the preparations.

With a view to simplifying handling, it happens that images of relevant areas of a preparation are recorded and stored. It will consequently be possible to study the areas of the images later on without having to file the preparations. This also renders it possible to study relevant areas of wet preparations a long time after the recording of the images . However, when the recorded images are studied, an image of the preparation in its entirety will not be obtained. Nor is it possible to get an idea of how different images of different parts of the preparations are related to each other. Thus there is a need for a microscope which simplifies handling when studying a preparation on a plurality of subsequent occasions. Summary of the Invention

An object of the present invention is to provide a microscope, which minimises the handling of preparations in repeated studies of an individual preparation.

A further object of the present invention is to provide a microscope which is adapted to store images of an area which is larger than the field of view of the micro- scope while at the same time the relationship between different parts of the preparation can be recovered from the stored images .

One more object of the present invention is to provide a method for microscoping, which facilitates repeat- ed studies of an individual preparation.

A further object of the present invention is to provide a memory medium with a computer program, which facilitates repeated studies of an individual preparation.

These objects are achieved by means of a microscope, a memory medium and a method according to the appended claims . A basic concept of the invention is to form a digital image with high resolution of a larger part of the object than has been possible with conventional equipment . A microscope according to the invention comprises an object holder, a display, optics which in an image plane forms an image of an object placed in the object holder, and an image sensor for recording the image. The microscope is characterised in that it further comprises a storage means, an input means for selecting an area in the composite image and a resolution in the selected area, and a calculation means which is connected to the image sensor and which is adapted to digitally store in the storage means at least two recorded images, each of the images partly overlapping at least one further image, to join the at least two recorded images so that the composite image is an image of the area which was reproduced on the at least two recorded images, and to store in the storage means information about the position of the recorded images in the composite image. The calculation means is further adapted to show the selected area with the selected resolution on the display.

By the microscope being adapted to form a digital image of a plurality of partial images, it is possible to later on study an optional area of the image without the preparation needing to be available.

By joining the recorded images is meant that the pixels in the recorded images are given positions in a common system of coordinates . Preferably, the calculation means is adapted to store in the storage means the composite image, but alternatively each of the recorded images can be given a position which is stored in the storage means, the pixels obtaining a common system of coordinates. According to one aspect of the present invention, a computer-readable memory medium is provided, on which a computer program is stored, which is adapted to be used for storing digital images which have been recorded with an image sensor. The computer program comprises instructions for making the computer read an input signal which represents at least two digital images, to join the digi- tal images to obtain a composite image of the area which is reproduced on the at least two digital images, to read information which defines an area and a resolution for the area in the composite image, and to output a digital output signal which represents the selected area with the selected resolution.

It is advantageous to record the partial images with the highest resolution allowed by the optics since the greatest possible freedom of action will then be obtained when later studying the composite image. Preferably the calculation means is adapted to control the object holder. The calculation means moves the object holder between each recording of a new image and stores at least two recorded images in the storage means. The displacement is adjusted so that the recorded images partly overlap each other.

It is advantageous to record the position of the object holder together with the recorded image since such a position can be used when the images are put together. The position of the object holder is obtained from e.g. stepping motors which are used to move the object holder.

According to a less preferred embodiment, the object holder is moved manually. The calculation means is then adapted to receive an input signal with information about the relationship between the recorded images so that it can join the images in the correct manner. Manual precision displacement of the object holder, however, is very time-consuming, and therefore automatic displacement is preferred.

The joining of the images can be carried out in various ways. According to a preferred embodiment of the invention, the calculation means is adapted to join two recorded images by moving the images in relation to each other so that an edge area of one image is fitted to an edge area of the other image, said edge areas overlapping each other. Consequently, high precision in the motion of the object holder is not necessary, which makes it possible to use a simpler and, thus, less expensive object holder.

The recorded images are stored either as black-and- white images or colour images. In both cases, each pixel is associated with at least one value which describes colour or luminous intensity. It has been found advantageous to fit the edge areas to each other by minimising the sum of squares of the difference between a value for each of the pixels in one edge area and the value of the overlapping pixel in the other edge area. The calculation is carried out for at least two different relative positions between the images, whereupon the position which yields the smallest sum of squares is selected as the correct position.

To minimise the calculation work when the images are to be fitted to each other, it has been found advantageous to carry out the fitting of two images in at least two steps, the resolution for the edge areas increasing for each step. As a starting point for the fitting, use is made of approximate positions which have been recorded with the images. The resolution of the recorded image is reduced digitally, whereupon the images are fitted together. The square of the differences between the pixels is summed up with the images in at least two different relative positions. Subsequently, a higher resolution is used, whereupon the fitting-together is repeated for this higher resolution. The optimal position from the first fitting-together is used as a starting point for the second fitting-together .

To further minimise the calculation work, it has been found advantageous to spatially filter the images before putting them together. This means that the effect of, for example, grains of dust which only appear on one of the images, is minimised. The filtering improves the possibilities of minimising the sum of squares by minimising the sum of squares in one direction at a time. Minimising the sum of squares in one direction at a time reduces the calculation work compared with the case where the sum of squares is calculated for all possible relative positions in a predetermined number of pixels from the starting position. However, with unfiltered images it is not always possible to obtain the minimum sum of squares by minimising the sum of squares in one direction at a time.

When the images are stored as colour images, the difference between two pixels can be defined as the sum of the differences for each of the colour compo- nents included. Alternatively, the difference between two pixels can be defined as the difference between the intensity values of one of the colours included.

The put -together image has areas that are present in more than one of the recorded images . The value of the pixels in the put-together image is advantageously an interpolation between the value of corresponding pixels in the recorded images, the value of each of the pixels being weighted in relation to how close to the edge of the respective images they are located. Thus, pixels located far away from the edge are suitably taken into greater consideration than corresponding pixels located close to the edge of another recorded image.

According to one aspect of the present invention, the calculation means is adapted to select edge areas having high spatial frequencies and a high contrast and to join the recorded images by fitting-together the selected edge areas. By selecting edge areas with high spatial frequencies and a good contrast, a good fitting- together is obtained while at the same time the calcula- tion work is minimised.

When more than two images are to be put together, there are a plurality of possible orders in which the images are put together. It is then advantageous for the microscope to be adapted to recognise high spatial frequencies with a high contrast in the edge areas and to put together pairs of the recorded images in an order determined by the spatial frequencies and the contrast at the edge of the images. Thus, the composite image is increased by a recorded image in each step. Consequently not all edges between the different images are adjusted in the composite image. This gives the advantage that the result in the composite image will be less sensitive to structureless areas in the edge area of an image.

Of course, it is possible within the scope of the invention to adjust all edges between all recorded images at the same time. However, extensive calculations are necessary to achieve simultaneous adjustment of all recorded images .

If all images are to be adjusted to each other at the same time it is, however, advantageous to calculate a comparative value for a plurality of different relative positions for each pair of adjoining images, the comparative value being a measure of how well the edge areas correspond to each other, and subsequently put together the recorded images in relative positions, so that a weighted sum of the comparative values for the pairs of images is maximised.

By this method, the calculation work is minimised while at the same time a better total fitting-together of the images is obtained. By weighted sum is meant that the different comparative values are weighted according to how reliable they are.

It has been found advantageous that the comparative values are based on the covariance of the areas where the images overlap each other for a number of different relative positions between the images. In order to further reduce the calculation work, it is advantageous for the comparative values to be calculated for displacements of the images in two orthogonal directions, the images being projected onto rows and columns respectively in the images before the comparative values are calculated.

By calculating comparative values for one direction at a time, the calculation work is significantly reduced. Projection of the images implies that the intensity values in the rows and columns respectively are added up.

It has also been found advantageous to calculate the sum of the comparative values in each of two orthogonal directions at a time. This significantly reduces the calculation work.

Each of the pixels has at least one intensity value. Preferably the calculation of comparative values is carried out by calculating the covariance of the overlapping areas.

When putting together images and calculating comparative values, it is advantageous to carry out the calculations at a gradually increasing resolution. In a first step, images having a lower resolution are then generated digitally, starting from the recorded images. This reduces the calculation work.

After putting together the images, a total image is obtained which reflects the entire recorded area.

It goes without saying that the above features can be combined in the same embodiment.

With a view to further elucidating the invention, detailed embodiments of the invention will be described below without, however, the invention being considered to be restricted thereto. Brief Description of the Drawings

Fig. 1 illustrates a microscope according to a preferred embodiment of the present invention.

Fig. 2 is a flow chart of the method according to a preferred embodiment of the present invention. Fig. 3 is a schematic view of images recorded by means of a microscope according to an embodiment of the present invention. Fig. 4 shows two images recorded by means of a microscope according to an embodiment of the present invention.

Detailed Description of the Invention Fig. 1 illustrates a microscope according to a preferred embodiment of the present invention. An object 1 is placed on a movable object holder 2 and illuminated from below by a bulb 3. An objective 4 images the object on a CCD 5 which has a number of image elements 6 at a mutual centre distance D. The CCD 5 is intended for recording of images and is connected to a calculation means 7 which in this case is a computer 8 with a display 9 and an input means 10 in the form of a keyboard and a mouse. Moreover, the computer is connected to and can control the object holder 2.

A user of the microscope initiates recording of an image of the object 1. In the initiation, a user inputs via the input means 10 information about how the image is to be recorded, i.e. the desired size of it, the desired resolution and how the object holder is to be arranged in relation to the objective at the beginning of the recording. The object 1 is, for example, a specimen of white blood cells which is to be studied in the microscope. By means of the inputted information, the microscope records images that are stored in a storage means 11 in the computer 8. Between each recording and storing of an image, the computer 8 controls the object holder 2 to be moved so as to image a new area, which partly overlaps the preceding area that has been recorded. The storage means 11 can be a direct access memory or a hard disk in the computer 8. Thus, a number of images are recorded and stored, which correspond to the area that was defined in the inputting via the input means 10.

Fig. 2 is a flow chart of the function of a micro- scope according to a preferred embodiment of the present invention. In step 12, information about which area is to be studied in the microscope is inputted via the input means. With the aid of the inputted information, the microscope records in step 13 a number of images which at least partly overlap each other. In step 14, the recorded images are put together so that a composite image forms, which covers the entire area that was covered with the recorded images. In step 15, the composite image is shown on the display 9 of the calculation means. In step 16, a user inputs information about which area in the shown image is to be shown at higher magnification, the magnified image also being inputted in step 16. In step 17, the selected area at the selected magnification is shown on the display 9. Of course, the image in step 17 cannot be shown with a better resolution than that recorded in step 13. However, in step 17 it is possible to show the image at an optional magnification. According to the preferred embodiment, the images are recorded in step 13 with maximum resolution to allow as great freedom of action as possible in steps 16 and 17. The image shown in step 15 does not always reproduce the maximum resolu- tion that is found in the images recorded in step 13 since the size of the display is limited.

Fig. 3 shows a number of images such as 18 and 19 that have been recorded in step 13. The edge areas are graded in respect of high spatial frequencies and a high contrast. The images recorded in step 13 are put together to a composite image 20. When the recorded images are put together, they are put together in pairs in an order which is determined by how they have been graded with respect to contrast and the presence of high special fre- quencies. For example, one begins with image 18, to which image 19 is added. When images 18 and 19 are to be joined, an edge area of image 18 is compared with an image area of image 19 so that they correspond with each other within the area 21 in which they overlap each other. Sub- sequently, the remaining images are added in an order which is determined by which edge has the best contrast and the greatest presence of high spatial frequencies. When all registered images have been put together, the composite image is shown on the display 9. A user inputs via the input means information about which area in the composite image 20 is to be shown at higher magnifica- tion. The selected area can be, for example, the area 22, which contains information from four recorded images. In the areas of the composite image 20 which contain information from more than one image, there is made in each pixel an interpolation between the values in the pixels from each of the recorded images, weighting being carried out so that pixels located close to the edge of a recorded image obtain less weight than such pixels as are located further away from the edge of another image.

Fig. 4 shows two images 23 and 24 which have been recorded in step 13. With reference to Fig. 4, the putting-together of two recorded images will now be described in more detail. Images 23 and 24 have been recorded with a certain overlap and their fitting-together is carried out by making a comparison within an area 25 so that correspondence is obtained between the two images in the area 25. By means of the computer 8, the area 25 is selected, in which there is a great presence of high special frequencies and a high contrast. In the area 25 there are structures that are to be found on both images, such as the structures 26 on the image 23 and the structures 27 on the image 24. The resolution in the images 23, 24 is reduced electronically by a factor 8, so that the number of pixels in the images is reduced by a factor 64. The image 24 is then moved in relation to the image 23 so that the sum of squares of the differences between the value in one pixel in the image 23 in the area 25 and the overlapping pixel in the image 24 in the area 25 is minimised. The sum of squares of the differences is in most cases minimal when the image 24 has been moved so that the structures 26 have the same position as the structures 27. Then the minimising is repeated for a gradually increasing resolution when the obtained relative position is used as a starting position for the next minimising step. The resolution is increased by a factor 2 in each minimising step until the maximum resolution has been achieved. As a result, the images are placed gradually closer to their correct relative position. In each minimising step, the images are moved to different positions, which represent a combination of the orthogonal displacements from the starting position. The image 23 is composed of a number of pixels which are each asso- ciated with a value describing the luminous intensity of the pixel. If the image 23 is a colour image, it is associated with three different colours, red, green and blue, which each have a value describing the intensity of precisely that colour in the pixel. If the images are colour images, the comparison is made in the edge area between each of the colours or between a converted image, in which case the converted image is a grey-scale image of the original colour image. It is also possible to use only one of the colours when comparing the images . With reference to Fig. 3, an alternative method of putting together the images will now be described. First the optimal fitting between each pair of images 18, 19 is determined. The fitting-together of two images is carried out by calculating the covariance of the area within which the images overlap each other for a plurality of different relative positions between the images 18, 19. The covariance is used as a comparative value to find the best fitting-together of the images. Before calculating the covariance, a projection of the images is made onto each of two orthogonal directions in the images. Thus, the rows and the columns respectively are added up, after which the covariance is calculated in the direction of the columns and rows respectively for a number of different relative positions. The covariance is calculated in the usual fashion except that no dividing by the number of measuring points occurs. For each of the pairs of images a set of covariances is obtained as a function of the displacement. In each of the sets containing covariances for different displacements, its maxima are searched out. This point and two adjoining points are used to form a curve of the second degree, which together with the corresponding curves of the second degree for the other sets is used to put together all the recorded images. When fitting together the pairs of images, a weight W is calculated which is a measure of how reliable the fitting-together is. The weight W is given by:

W S0)- 0.50*max(S)' v 0.50*max(S)

wherein

i e [s(i) ≥ 0.25max(S),|/- max| > 2]- j e [s(j) ≥ 0.50max(S),|./ -max| = l],

wherein S(i) is the covariance at the displacement i, and S (max) = max(S) . If W becomes negative, the fitting- together is considered unusable and W is set at zero. If the images are not square, it is easier to find a good fitting-together of between the images along the long edge. Then the fitting-together is advantageously carried out in this direction first. After determining the relative positions of each pair of images, all the recorded images are put together so that a weighted sum of the comparative values is maximised. This means that a weighted sum of the covariances for the different pairs of images is maximised. This, too, is made first in one of the direction of the columns and the direction of the rows and then in the other direction. The maximisation is carried out by minimising the function

E = -fo* - *MJi , wherein m is the number of images in a column, n the number of images in a row, Cι,k/0 is the optimal fitting- together of image [l,k] and its right neighbour and Cι,k,ι is the optimal fitting-together of image [l,k] and its lower neighbour. Wι;k,0 and Wι;krι are correspondingly the weight of the respective two-image fittings. Xχ;k is the displacement of the image in the x direction or in the y direction depending on the optimisation that has been carried out. A person skilled in the art realises that the invention is limited to the embodiments described, which can be varied within the scope of the invention. For example, the CCD can be replaced by a vidicon. Moreover the user need not necessarily initiate the recording of the par- tial images by specifying the size and resolution of the completed image since this can be carried out according to a predetermined schedule stored in the computer.

Claims

1. A microscope comprising an object holder (2), optics (4) which in an image plane forms an image of an object (1) placed in the object holder (2), and an image sensor (5) for recording the image, c h a r a c t e r i s e d in that the microscope further comprises a display (9) , an input means (10) for selecting an area in an image and a resolution, a storage means (11) , and a calculation means (8) which is connected to the image sensor (5) and which is adapted
(i) to digitally store in the storage means at least two recorded images, each of the recorded images compris - ing a plurality of pixels, and each of the images partly overlapping at least one other image,
(ii) to join the at least two recorded images so that the composite image is an image of the area which is reproduced on the at least two recorded images (18, 19, 23, 24) , and
(iii) to store in the storage means (11) information about the composite image, and to show, on the display (9) , the area having the selected resolution and selected with the aid of the input means.
2. A microscope as claimed in claim 1, c h a r a c t e r i s e d in that the calculation means is adapted to store the composite image in the storage means
(11) .
3. A microscope as claimed in claim 1, c h a r a c t e r i s e d in that the calculation means is adapted to store in the storage means (11) information about the positions of the recorded images in the composite image .
4. A microscope as claimed in any one of the preceding claims, c h a r a c t e r i s e d in that the cal- culation means further is adapted to control the motion of the object holder (2) and to store at least two recorded images in the storage means (11), between which recordings the object holder (2) has been moved so that the recorded images partly overlap each other.
5. A microscope as claimed in any one of the preceding claims, c h a r a c t e r i s e d in that the calculation means (7) is adapted to receive an input signal with information about the relationship between the recorded images.
6. A microscope as claimed in any one of the preceding claims, c h a r a c t e r i s e d in that the calculation means (7) is adapted to join two recorded images by placing the images in relation to each other so that an edge area of one image is fitted to an edge area of the other image, said edge areas overlapping each other.
7. A microscope as claimed in any one of the preceding claims, c h a r a c t e r i s e d in that the calculation means (7) is also adapted to calculate a compa- rative value for a plurality of different relative positions for each pair of recorded images when they are arranged so that an edge area of one image overlaps an edge area of the other image, the comparative value being a measure of how well the edge areas correspond to each other, and subsequently put together the recorded images in relative positions so that a weighted sum of the comparative values for the pairs of images is maximised.
8. A microscope as claimed in claim 7, c h a r a c t e r i s e d in that comparative values are calcu- lated for displacements of the images in two orthogonal directions, the images being projected onto rows and columns respectively in the images before the comparative values are calculated.
9. A microscope as claimed in claim 7 or 8 , c h a r a c t e r i s e d in that the calculation of the sum of the comparative values is carried out in each of two orthogonal directions at a time.
10. A microscope as claimed in claim 7, 8 or 9, c h a r a c t e r i s e d in that each of the pixels has at least one intensity value, and that the calculation of the comparative values is carried out by calculation of the covariance of the overlapping edge areas .
11. A microscope as claimed in claim 6, c h a r a c t e r i s e d in that each of the pixels has at least one intensity value, and that fitting-together of the edge areas results in minimising of the sum of squares of the difference between the intensity value for each of the pixels in one edge area (25) and the overlapping pixel in the other edge area.
12. A microscope as claimed in claim 6 or 11, c h a r a c t e r i s e d in that the calculation means is adapted to select, for said fitting-together, edge areas in the images with a great presence of high spatial frequencies and a high contrast.
13. A microscope as claimed in any one of claims 6, 11 or 12, c h a r a c t e r i s e d in that the calculation means is adapted to recognise structures in the edge areas (25) and to join the recorded images in an order determined by the spatial frequencies and the contrast at the edge of the recorded images.
14. A microscope as claimed in any one of the preceding claims, c h a r a c t e r i s e d in that the recorded images are recorded at maximum resolution of the optics (4) .
15. A method for microscoping comprising the step of recording with an image sensor (15) at least two images, which partly overlap each other and which have been reproduced in a microscope, c h a r a c t e r i s e d in that it further comprises the steps of joining the recorded images (18, 19, 23, 24) to obtain a composite image of the area which is reproduced on the at least two recorded images (18, 19, 23, 24), selecting an area and a resolution for the area in the composite image, and showing the selected area with the selected resolution on a display (9) .
16. A method as claimed in claim 15, c h a r a c t e r i s e d in that it further comprises the steps of selecting on the images pairs of the edge areas which overlap each other, and placing the edge areas in relation to each other so that the sum of squares of the differences between each of the pixels in the edge area of one image and the overlapping pixel in the edge area of the other image is minimised.
17. A method as claimed in claim 15, c h a r a c t e r i s e d in that it further comprises the steps of selecting in pairs on the images the edge areas which overlap each other, calculating a comparative value for a plurality of different relative positions for each pair of images when the edge areas overlap each other, the comparative value being a measure of how well the edge areas correspond to each other, and subsequently joining the recorded images in relative positions so that the sum of the comparative values for the pairs of images is minimised.
18. A method as claimed in claim 17, characterised in that the comparative values are calculated for dis- placements of the images in two orthogonal directions, the images being projected onto the rows and columns respectively before the comparative values are calculated.
19. A method as claimed in claim 17 or 18, c h a r a c t e r i s e d in that the minimising of the sum of the comparative values is made in each of two orthogonal directions at a time.
20. A method as claimed in claim 16, c h a r a c t e r i s e d in that the placing of the images, and the associated calculation of the sum of the differences between their pixels, are carried out in at least two steps, the resolution in the image increasing in each step .
21. A method as claimed in any one of claims 16-20, c h a r a c t e r i s e d in that the recorded images are spatially filtered before being put together.
22. A computer-readable memory medium, on which a computer program is stored, which is intended to be used for storing of digital images which have been recorded with an image sensor, c h a r a c t e r i s e d in that the computer program comprises instructions for making the computer (8) read an input signal which represents at least two digital images, join the digital images (18, 19, 23, 24) to obtain a composite image of the area which is reproduced on the at least two digital images (18, 19, 23, 24), read information which defines an area and a resolution for the area in the composite image, and output a digital output signal which represents the selected area with the selected resolution.
23. A computer-readable memory medium as claimed in claim 22, c h a r a c t e r i s e d in that the computer program also comprises instructions for making the computer (8) select on the images pairs of edge areas (25) which overlap each other, and place the images in relation to each other so that the sum of the differences between each of the pixels in the edge area of one image and the overlapping pixel in the edge area of the other image is minimised.
24. A computer-readable memory medium as claimed in claim 22, c h a r a c t e r i s e d in that the computer program also comprises instructions for making the computer (8) perform the placing of the images and the associated calculation of the sum of the differences between their pixels in at least two steps, the resolution in the image increasing in each step.
PCT/SE2000/001996 1999-10-15 2000-10-16 Microscope and method for manufacturing a composite image with a high resolution WO2001027679A1 (en)

Priority Applications (4)

Application Number Priority Date Filing Date Title
SE9903706-1 1999-10-15
SE9903706 1999-10-15
US16047099 true 1999-10-21 1999-10-21
US60/160,470 1999-10-21

Publications (1)

Publication Number Publication Date
WO2001027679A1 true true WO2001027679A1 (en) 2001-04-19

Family

ID=26663662

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/SE2000/001996 WO2001027679A1 (en) 1999-10-15 2000-10-16 Microscope and method for manufacturing a composite image with a high resolution

Country Status (1)

Country Link
WO (1) WO2001027679A1 (en)

Cited By (48)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2004080076A1 (en) * 2003-02-19 2004-09-16 Carl Zeiss Jena Gmbh Electronic microscope
EP1559059A2 (en) * 2002-10-18 2005-08-03 Leco Corporation Indentation hardness test system
WO2006023675A2 (en) * 2004-08-18 2006-03-02 Tripath Imaging, Inc. Microscopy system having automatic and interactive modes for forming a magnified mosaic image and associated method
WO2007041412A1 (en) * 2005-09-29 2007-04-12 General Hospital Corporation Method and apparatus for method for viewing and analyzing of one or more biological samples with progressively increasing resolutions
EP1794706A2 (en) * 2004-08-16 2007-06-13 Bacus Laboratories, Inc. Method and apparatus of mechanical stage positioning in virtual microscopy image capture
EP1860481A1 (en) * 2006-05-24 2007-11-28 Olympus Corporation Micropscope system and method for synthesiing microscopic images
US8014579B2 (en) 2006-05-24 2011-09-06 Olympus Corporation Microscope system and method for synthesizing microscopic images
EP2200267A3 (en) * 2008-12-17 2012-05-09 Canon Kabushiki Kaisha Image processing apparatus and image processing method
JP2013015665A (en) * 2011-07-04 2013-01-24 Nikon Corp Microscope device and image forming method
US8838213B2 (en) 2006-10-19 2014-09-16 The General Hospital Corporation Apparatus and method for obtaining and providing imaging information associated with at least one portion of a sample, and effecting such portion(s)
US8861910B2 (en) 2008-06-20 2014-10-14 The General Hospital Corporation Fused fiber optic coupler arrangement and method for use thereof
US8896838B2 (en) 2010-03-05 2014-11-25 The General Hospital Corporation Systems, methods and computer-accessible medium which provide microscopic images of at least one anatomical structure at a particular resolution
US8922781B2 (en) 2004-11-29 2014-12-30 The General Hospital Corporation Arrangements, devices, endoscopes, catheters and methods for performing optical imaging by simultaneously illuminating and detecting multiple points on a sample
US8937724B2 (en) 2008-12-10 2015-01-20 The General Hospital Corporation Systems and methods for extending imaging depth range of optical coherence tomography through optical sub-sampling
US8965487B2 (en) 2004-08-24 2015-02-24 The General Hospital Corporation Process, system and software arrangement for measuring a mechanical strain and elastic properties of a sample
USRE45512E1 (en) 2004-09-29 2015-05-12 The General Hospital Corporation System and method for optical coherence imaging
US9060689B2 (en) 2005-06-01 2015-06-23 The General Hospital Corporation Apparatus, method and system for performing phase-resolved optical frequency domain imaging
US9069130B2 (en) 2010-05-03 2015-06-30 The General Hospital Corporation Apparatus, method and system for generating optical radiation from biological gain media
US9087368B2 (en) 2006-01-19 2015-07-21 The General Hospital Corporation Methods and systems for optical imaging or epithelial luminal organs by beam scanning thereof
US9176319B2 (en) 2007-03-23 2015-11-03 The General Hospital Corporation Methods, arrangements and apparatus for utilizing a wavelength-swept laser using angular scanning and dispersion procedures
US9186066B2 (en) 2006-02-01 2015-11-17 The General Hospital Corporation Apparatus for applying a plurality of electro-magnetic radiations to a sample
US9226660B2 (en) 2004-08-06 2016-01-05 The General Hospital Corporation Process, system and software arrangement for determining at least one location in a sample using an optical coherence tomography
US9226665B2 (en) 2003-01-24 2016-01-05 The General Hospital Corporation Speckle reduction in optical coherence tomography by path length encoded angular compounding
US9254102B2 (en) 2004-08-24 2016-02-09 The General Hospital Corporation Method and apparatus for imaging of vessel segments
US9254089B2 (en) 2008-07-14 2016-02-09 The General Hospital Corporation Apparatus and methods for facilitating at least partial overlap of dispersed ration on at least one sample
US9282931B2 (en) 2000-10-30 2016-03-15 The General Hospital Corporation Methods for tissue analysis
US9295391B1 (en) 2000-11-10 2016-03-29 The General Hospital Corporation Spectrally encoded miniature endoscopic imaging probe
US9326682B2 (en) 2005-04-28 2016-05-03 The General Hospital Corporation Systems, processes and software arrangements for evaluating information associated with an anatomical structure by an optical coherence ranging technique
US9330092B2 (en) 2011-07-19 2016-05-03 The General Hospital Corporation Systems, methods, apparatus and computer-accessible-medium for providing polarization-mode dispersion compensation in optical coherence tomography
US9341783B2 (en) 2011-10-18 2016-05-17 The General Hospital Corporation Apparatus and methods for producing and/or providing recirculating optical delay(s)
US9351642B2 (en) 2009-03-12 2016-05-31 The General Hospital Corporation Non-contact optical system, computer-accessible medium and method for measurement at least one mechanical property of tissue using coherent speckle technique(s)
US9364143B2 (en) 2006-05-10 2016-06-14 The General Hospital Corporation Process, arrangements and systems for providing frequency domain imaging of a sample
US9375158B2 (en) 2007-07-31 2016-06-28 The General Hospital Corporation Systems and methods for providing beam scan patterns for high speed doppler optical frequency domain imaging
US9377290B2 (en) 2003-10-27 2016-06-28 The General Hospital Corporation Method and apparatus for performing optical imaging using frequency-domain interferometry
US9441948B2 (en) 2005-08-09 2016-09-13 The General Hospital Corporation Apparatus, methods and storage medium for performing polarization-based quadrature demodulation in optical coherence tomography
US9510758B2 (en) 2010-10-27 2016-12-06 The General Hospital Corporation Apparatus, systems and methods for measuring blood pressure within at least one vessel
US9516997B2 (en) 2006-01-19 2016-12-13 The General Hospital Corporation Spectrally-encoded endoscopy techniques, apparatus and methods
US9557154B2 (en) 2010-05-25 2017-01-31 The General Hospital Corporation Systems, devices, methods, apparatus and computer-accessible media for providing optical imaging of structures and compositions
US9615748B2 (en) 2009-01-20 2017-04-11 The General Hospital Corporation Endoscopic biopsy apparatus, system and method
US9629528B2 (en) 2012-03-30 2017-04-25 The General Hospital Corporation Imaging system, method and distal attachment for multidirectional field of view endoscopy
USRE46412E1 (en) 2006-02-24 2017-05-23 The General Hospital Corporation Methods and systems for performing angle-resolved Fourier-domain optical coherence tomography
US9664615B2 (en) 2004-07-02 2017-05-30 The General Hospital Corporation Imaging system and related techniques
US9668652B2 (en) 2013-07-26 2017-06-06 The General Hospital Corporation System, apparatus and method for utilizing optical dispersion for fourier-domain optical coherence tomography
US9733460B2 (en) 2014-01-08 2017-08-15 The General Hospital Corporation Method and apparatus for microscopic imaging
US9777053B2 (en) 2006-02-08 2017-10-03 The General Hospital Corporation Methods, arrangements and systems for obtaining information associated with an anatomical sample using optical microscopy
US9784681B2 (en) 2013-05-13 2017-10-10 The General Hospital Corporation System and method for efficient detection of the phase and amplitude of a periodic modulation associated with self-interfering fluorescence
US9795301B2 (en) 2010-05-25 2017-10-24 The General Hospital Corporation Apparatus, systems, methods and computer-accessible medium for spectral analysis of optical coherence tomography images
US9968261B2 (en) 2013-01-28 2018-05-15 The General Hospital Corporation Apparatus and method for providing diffuse spectroscopy co-registered with optical frequency domain imaging

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4202037A (en) * 1977-04-22 1980-05-06 Der Loos Hendrik Van Computer microscope apparatus and method for superimposing an electronically-produced image from the computer memory upon the image in the microscope's field of view
US4673988A (en) * 1985-04-22 1987-06-16 E.I. Du Pont De Nemours And Company Electronic mosaic imaging process
US5452105A (en) * 1992-11-19 1995-09-19 Sharp Kabushiki Kaisha Joint-portion processing device for image data for use in an image processing apparatus
EP0731371A1 (en) * 1995-03-06 1996-09-11 Perkin-Elmer Limited Microscope stage control
WO1998009434A1 (en) * 1996-08-26 1998-03-05 Beijing Jinruiya Technology Development Co., Ltd. A method of putting parted images together

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4202037A (en) * 1977-04-22 1980-05-06 Der Loos Hendrik Van Computer microscope apparatus and method for superimposing an electronically-produced image from the computer memory upon the image in the microscope's field of view
US4673988A (en) * 1985-04-22 1987-06-16 E.I. Du Pont De Nemours And Company Electronic mosaic imaging process
US5452105A (en) * 1992-11-19 1995-09-19 Sharp Kabushiki Kaisha Joint-portion processing device for image data for use in an image processing apparatus
EP0731371A1 (en) * 1995-03-06 1996-09-11 Perkin-Elmer Limited Microscope stage control
WO1998009434A1 (en) * 1996-08-26 1998-03-05 Beijing Jinruiya Technology Development Co., Ltd. A method of putting parted images together

Cited By (68)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9282931B2 (en) 2000-10-30 2016-03-15 The General Hospital Corporation Methods for tissue analysis
US9295391B1 (en) 2000-11-10 2016-03-29 The General Hospital Corporation Spectrally encoded miniature endoscopic imaging probe
KR100969413B1 (en) * 2002-10-18 2010-07-14 레코 코포레이션 Indentation hardness test system
EP1559059A2 (en) * 2002-10-18 2005-08-03 Leco Corporation Indentation hardness test system
EP1559059A4 (en) * 2002-10-18 2009-03-25 Leco Corp Indentation hardness test system
EP2386982A1 (en) * 2002-10-18 2011-11-16 Leco Corporation Indentation hardness test system
US9226665B2 (en) 2003-01-24 2016-01-05 The General Hospital Corporation Speckle reduction in optical coherence tomography by path length encoded angular compounding
WO2004080076A1 (en) * 2003-02-19 2004-09-16 Carl Zeiss Jena Gmbh Electronic microscope
US7586688B2 (en) 2003-02-19 2009-09-08 Carl Zeiss Microimaging Gmbh Electronic microscope
US9377290B2 (en) 2003-10-27 2016-06-28 The General Hospital Corporation Method and apparatus for performing optical imaging using frequency-domain interferometry
US9812846B2 (en) 2003-10-27 2017-11-07 The General Hospital Corporation Method and apparatus for performing optical imaging using frequency-domain interferometry
US9664615B2 (en) 2004-07-02 2017-05-30 The General Hospital Corporation Imaging system and related techniques
US9226660B2 (en) 2004-08-06 2016-01-05 The General Hospital Corporation Process, system and software arrangement for determining at least one location in a sample using an optical coherence tomography
EP1794706A2 (en) * 2004-08-16 2007-06-13 Bacus Laboratories, Inc. Method and apparatus of mechanical stage positioning in virtual microscopy image capture
EP1794706A4 (en) * 2004-08-16 2009-10-21 Olympus America Inc Method and apparatus of mechanical stage positioning in virtual microscopy image capture
US7792338B2 (en) 2004-08-16 2010-09-07 Olympus America Inc. Method and apparatus of mechanical stage positioning in virtual microscopy image capture
WO2006023675A2 (en) * 2004-08-18 2006-03-02 Tripath Imaging, Inc. Microscopy system having automatic and interactive modes for forming a magnified mosaic image and associated method
WO2006023675A3 (en) * 2004-08-18 2006-04-20 John Royal Cheesemann Microscopy system having automatic and interactive modes for forming a magnified mosaic image and associated method
US8965487B2 (en) 2004-08-24 2015-02-24 The General Hospital Corporation Process, system and software arrangement for measuring a mechanical strain and elastic properties of a sample
US9254102B2 (en) 2004-08-24 2016-02-09 The General Hospital Corporation Method and apparatus for imaging of vessel segments
US9763623B2 (en) 2004-08-24 2017-09-19 The General Hospital Corporation Method and apparatus for imaging of vessel segments
USRE45512E1 (en) 2004-09-29 2015-05-12 The General Hospital Corporation System and method for optical coherence imaging
US8922781B2 (en) 2004-11-29 2014-12-30 The General Hospital Corporation Arrangements, devices, endoscopes, catheters and methods for performing optical imaging by simultaneously illuminating and detecting multiple points on a sample
US9326682B2 (en) 2005-04-28 2016-05-03 The General Hospital Corporation Systems, processes and software arrangements for evaluating information associated with an anatomical structure by an optical coherence ranging technique
US9060689B2 (en) 2005-06-01 2015-06-23 The General Hospital Corporation Apparatus, method and system for performing phase-resolved optical frequency domain imaging
US9441948B2 (en) 2005-08-09 2016-09-13 The General Hospital Corporation Apparatus, methods and storage medium for performing polarization-based quadrature demodulation in optical coherence tomography
US8928889B2 (en) 2005-09-29 2015-01-06 The General Hospital Corporation Arrangements and methods for providing multimodality microscopic imaging of one or more biological structures
US9513276B2 (en) 2005-09-29 2016-12-06 The General Hospital Corporation Method and apparatus for optical imaging via spectral encoding
US9304121B2 (en) 2005-09-29 2016-04-05 The General Hospital Corporation Method and apparatus for optical imaging via spectral encoding
WO2007041412A1 (en) * 2005-09-29 2007-04-12 General Hospital Corporation Method and apparatus for method for viewing and analyzing of one or more biological samples with progressively increasing resolutions
US9087368B2 (en) 2006-01-19 2015-07-21 The General Hospital Corporation Methods and systems for optical imaging or epithelial luminal organs by beam scanning thereof
US9646377B2 (en) 2006-01-19 2017-05-09 The General Hospital Corporation Methods and systems for optical imaging or epithelial luminal organs by beam scanning thereof
US9516997B2 (en) 2006-01-19 2016-12-13 The General Hospital Corporation Spectrally-encoded endoscopy techniques, apparatus and methods
US9791317B2 (en) 2006-01-19 2017-10-17 The General Hospital Corporation Spectrally-encoded endoscopy techniques and methods
US9186067B2 (en) 2006-02-01 2015-11-17 The General Hospital Corporation Apparatus for applying a plurality of electro-magnetic radiations to a sample
US9186066B2 (en) 2006-02-01 2015-11-17 The General Hospital Corporation Apparatus for applying a plurality of electro-magnetic radiations to a sample
US9777053B2 (en) 2006-02-08 2017-10-03 The General Hospital Corporation Methods, arrangements and systems for obtaining information associated with an anatomical sample using optical microscopy
USRE46412E1 (en) 2006-02-24 2017-05-23 The General Hospital Corporation Methods and systems for performing angle-resolved Fourier-domain optical coherence tomography
US9364143B2 (en) 2006-05-10 2016-06-14 The General Hospital Corporation Process, arrangements and systems for providing frequency domain imaging of a sample
US8014579B2 (en) 2006-05-24 2011-09-06 Olympus Corporation Microscope system and method for synthesizing microscopic images
US7949161B2 (en) 2006-05-24 2011-05-24 Olympus Corporation Microscope system and method for synthesizing microscopic images
EP1860481A1 (en) * 2006-05-24 2007-11-28 Olympus Corporation Micropscope system and method for synthesiing microscopic images
US8838213B2 (en) 2006-10-19 2014-09-16 The General Hospital Corporation Apparatus and method for obtaining and providing imaging information associated with at least one portion of a sample, and effecting such portion(s)
US9176319B2 (en) 2007-03-23 2015-11-03 The General Hospital Corporation Methods, arrangements and apparatus for utilizing a wavelength-swept laser using angular scanning and dispersion procedures
US9375158B2 (en) 2007-07-31 2016-06-28 The General Hospital Corporation Systems and methods for providing beam scan patterns for high speed doppler optical frequency domain imaging
US8861910B2 (en) 2008-06-20 2014-10-14 The General Hospital Corporation Fused fiber optic coupler arrangement and method for use thereof
US9254089B2 (en) 2008-07-14 2016-02-09 The General Hospital Corporation Apparatus and methods for facilitating at least partial overlap of dispersed ration on at least one sample
US8937724B2 (en) 2008-12-10 2015-01-20 The General Hospital Corporation Systems and methods for extending imaging depth range of optical coherence tomography through optical sub-sampling
EP2200267A3 (en) * 2008-12-17 2012-05-09 Canon Kabushiki Kaisha Image processing apparatus and image processing method
US9615748B2 (en) 2009-01-20 2017-04-11 The General Hospital Corporation Endoscopic biopsy apparatus, system and method
US9351642B2 (en) 2009-03-12 2016-05-31 The General Hospital Corporation Non-contact optical system, computer-accessible medium and method for measurement at least one mechanical property of tissue using coherent speckle technique(s)
US8896838B2 (en) 2010-03-05 2014-11-25 The General Hospital Corporation Systems, methods and computer-accessible medium which provide microscopic images of at least one anatomical structure at a particular resolution
US9081148B2 (en) 2010-03-05 2015-07-14 The General Hospital Corporation Systems, methods and computer-accessible medium which provide microscopic images of at least one anatomical structure at a particular resolution
US9408539B2 (en) 2010-03-05 2016-08-09 The General Hospital Corporation Systems, methods and computer-accessible medium which provide microscopic images of at least one anatomical structure at a particular resolution
US9642531B2 (en) 2010-03-05 2017-05-09 The General Hospital Corporation Systems, methods and computer-accessible medium which provide microscopic images of at least one anatomical structure at a particular resolution
US9069130B2 (en) 2010-05-03 2015-06-30 The General Hospital Corporation Apparatus, method and system for generating optical radiation from biological gain media
US9951269B2 (en) 2010-05-03 2018-04-24 The General Hospital Corporation Apparatus, method and system for generating optical radiation from biological gain media
US9557154B2 (en) 2010-05-25 2017-01-31 The General Hospital Corporation Systems, devices, methods, apparatus and computer-accessible media for providing optical imaging of structures and compositions
US9795301B2 (en) 2010-05-25 2017-10-24 The General Hospital Corporation Apparatus, systems, methods and computer-accessible medium for spectral analysis of optical coherence tomography images
US9510758B2 (en) 2010-10-27 2016-12-06 The General Hospital Corporation Apparatus, systems and methods for measuring blood pressure within at least one vessel
JP2013015665A (en) * 2011-07-04 2013-01-24 Nikon Corp Microscope device and image forming method
US9330092B2 (en) 2011-07-19 2016-05-03 The General Hospital Corporation Systems, methods, apparatus and computer-accessible-medium for providing polarization-mode dispersion compensation in optical coherence tomography
US9341783B2 (en) 2011-10-18 2016-05-17 The General Hospital Corporation Apparatus and methods for producing and/or providing recirculating optical delay(s)
US9629528B2 (en) 2012-03-30 2017-04-25 The General Hospital Corporation Imaging system, method and distal attachment for multidirectional field of view endoscopy
US9968261B2 (en) 2013-01-28 2018-05-15 The General Hospital Corporation Apparatus and method for providing diffuse spectroscopy co-registered with optical frequency domain imaging
US9784681B2 (en) 2013-05-13 2017-10-10 The General Hospital Corporation System and method for efficient detection of the phase and amplitude of a periodic modulation associated with self-interfering fluorescence
US9668652B2 (en) 2013-07-26 2017-06-06 The General Hospital Corporation System, apparatus and method for utilizing optical dispersion for fourier-domain optical coherence tomography
US9733460B2 (en) 2014-01-08 2017-08-15 The General Hospital Corporation Method and apparatus for microscopic imaging

Similar Documents

Publication Publication Date Title
US6320979B1 (en) Depth of field enhancement
US5231443A (en) Automatic ranging and automatic focusing
US20020114497A1 (en) Method for maintaining High-quality focus during high-throughput, microscopic digital montage imaging
US4631581A (en) Method and apparatus for microphotometering microscope specimens
US5239170A (en) Autofocus method and apparatus for imaging microscopy using a predetermined visual imaging feature
US5260825A (en) Microscope
US6445415B1 (en) Increased depth of field for photography
US20020150304A1 (en) System for morphological image fusion and change detection
US5235375A (en) Focusing position detecting and automatic focusing apparatus with optimal focusing position calculation method
US5305092A (en) Apparatus for obtaining three-dimensional volume data of an object
US20040165090A1 (en) Auto-focus (AF) lens and process
US20060133657A1 (en) Microscopy system having automatic and interactive modes for forming a magnified mosaic image and associated method
US5757423A (en) Image taking apparatus
US5877809A (en) Method of automatic object detection in image
US20090213214A1 (en) Microscope System, Image Generating Method, and Program for Practising the Same
US20070069106A1 (en) Apparatus and Method for Rapid Microscope Image Focusing
US4786154A (en) Enhanced-image operating microscope
US4852184A (en) Method and system for spectacles frame image processing and recording
US5615398A (en) Optical apparatus with image area sensors for controlling lens focal length
US5751863A (en) Method and system having relaxed front end distortion requirements
US6519359B1 (en) Range camera controller for acquiring 3D models
US20040047033A1 (en) Microscopic image capture apparatus and microscopic image capturing method
US20080095467A1 (en) Large-area imaging by concatenation with array microscope
US5592563A (en) Imaging of 3-dimensional objects
US7729049B2 (en) 3-d optical microscope

Legal Events

Date Code Title Description
AL Designated countries for regional patents

Kind code of ref document: A1

Designated state(s): GH GM KE LS MW MZ SD SL SZ TZ UG ZW AM AZ BY KG KZ MD RU TJ TM AT BE CH CY DE DK ES FI FR GB GR IE IT LU MC NL PT SE BF BJ CF CG CI CM GA GN GW ML MR NE SN TD TG

AK Designated states

Kind code of ref document: A1

Designated state(s): AE AG AL AM AT AT AU AZ BA BB BG BR BY BZ CA CH CN CR CU CZ CZ DE DE DK DK DM DZ EE EE ES FI FI GB GD GE GH GM HR HU ID IL IN IS JP KE KG KP KR KR KZ LC LK LR LS LT LU LV MA MD MG MK MN MW MX MZ NO NZ PL PT RO RU SD SE SG SI SK SK SL TJ TM TR TT TZ UA UG US UZ VN YU ZA ZW

121 Ep: the epo has been informed by wipo that ep was designated in this application
DFPE Request for preliminary examination filed prior to expiration of 19th month from priority date (pct application filed before 20040101)
REG Reference to national code

Ref country code: DE

Ref legal event code: 8642

122 Ep: pct application non-entry in european phase
NENP Non-entry into the national phase in:

Ref country code: JP