EP1830216A1 - Appareil de capture d'images, procédé de capture d'images et programme de capture d'images - Google Patents

Appareil de capture d'images, procédé de capture d'images et programme de capture d'images Download PDF

Info

Publication number
EP1830216A1
EP1830216A1 EP07250728A EP07250728A EP1830216A1 EP 1830216 A1 EP1830216 A1 EP 1830216A1 EP 07250728 A EP07250728 A EP 07250728A EP 07250728 A EP07250728 A EP 07250728A EP 1830216 A1 EP1830216 A1 EP 1830216A1
Authority
EP
European Patent Office
Prior art keywords
image
macro
micro
acquiring
sample
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
EP07250728A
Other languages
German (de)
English (en)
Other versions
EP1830216B1 (fr
Inventor
Jeremy c/o Hamamatsu Photonics UK Limited Cooke
Takayuki Inoue
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hamamatsu Photonics KK
Original Assignee
Hamamatsu Photonics KK
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hamamatsu Photonics KK filed Critical Hamamatsu Photonics KK
Publication of EP1830216A1 publication Critical patent/EP1830216A1/fr
Application granted granted Critical
Publication of EP1830216B1 publication Critical patent/EP1830216B1/fr
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B21/00—Microscopes
    • G02B21/36—Microscopes arranged for photographic purposes or projection purposes or digital imaging or video purposes including associated control and data processing arrangements
    • G02B21/365—Control or image processing arrangements for digital or video microscopes
    • G02B21/367—Control or image processing arrangements for digital or video microscopes providing an output produced by processing a plurality of individual source images, e.g. image tiling, montage, composite images, depth sectioning, image comparison

Definitions

  • the present invention relates to an image acquiring apparatus, an image acquiring method, and an image acquiring program for acquiring images of a sample.
  • an image acquiring apparatus which acquires image data of a sample to be thus used in a virtual microscope acquires an image of a sample at sufficiently high resolution.
  • a structure in which an image of a sample is acquired by using a camera for macroscopic image acquisition and a camera for microscopic image acquisition at high resolution is described in Document 1: US Patent No. 6,816,606 .
  • a method is considered in which a macro image is acquired first by the macro camera, an image pickup condition for the sample is set with reference to the macro image, and then, a micro image is acquired by the micro camera with reference to the set image pickup condition.
  • a virtual microscope there is a case in which many slides in each of which a biological sample or the like is sealed in a slide glass are prepared as samples, and an image acquiring operation is continuously performed with respect to the slides.
  • a macro image acquiring operation and a micro image acquiring operation must be repeatedly performed with respect to the samples.
  • the present invention has been made to solve these problems, and it is therefore an object of the present invention to provide an image acquiring apparatus, an image acquiring method, and an image acquiring program each of which is capable of performing image acquisition with respect to a plurality of samples with high efficiency.
  • an image acquiring apparatus includes (1) sample storage means for storing a plurality of samples, (2) macro image acquiring means for acquiring a macro image of the sample, (3) micro image acquiring means for acquiring a micro image while scanning the sample with higher resolution than that of the macro image, and (4) control means including macro image acquisition control means for controlling an operation to acquire the macro image by the macro image acquiring means, image pickup condition setting means for setting an image pickup condition for the micro image with reference to the macro image, and micro image acquisition control means for controlling an operation to acquire the micro image by the micro image acquiring means, wherein (5) the control means has a semi-automatic mode serving as a control mode, and (6) in the semi-automatic mode, the macro image acquisition control means performs control to place each of the samples from a storage position in the sample storage means at an image acquiring position for the macro image acquiring means so as to acquire the macro image, and then performs control to place the sample at a standby position; the image pickup condition setting means sets the image pickup
  • an image acquiring program is applied to an image acquiring apparatus including (1) sample storage means for storing a plurality of samples, (2) macro image acquiring means for acquiring a macro image of the sample, and (3) micro image acquiring means for acquiring a micro image while scanning the sample with higher resolution than that of the macro image, and includes (4) a control processing including a macro image acquisition control processing for controlling an operation to acquire the macro image by the macro image acquiring means; an image pickup condition setting processing for setting an image pickup condition for the micro image with reference to the macro image; and a micro image acquisition control processing for controlling an operation to acquire the micro image by the micro image acquiring means, wherein (5) the control processing has a semi-automatic mode serving as a control mode, and (6) in the semi-automatic mode, the macro image acquisition control processing performs control to place each of the samples from a storage position in the sample storage means at an image acquiring position for the macro image acquiring means so as to acquire the macro image, and then performs control to place the sample at a standby
  • the macro image acquiring means and the micro image acquiring means are provided for a plurality of samples each of which is an object whose image is acquired, and an image pickup condition is set while referring to a macro image showing the whole of the sample, and then a high-resolution micro image is acquired. Therefore, a high-resolution micro image of the sample that is used as sample image data for use in, for example, a virtual microscope can be suitably acquired.
  • a semi-automatic mode in which an operator makes a necessary confirmation when an image pickup condition for a micro image is set with reference to a macro image is provided as a control mode that controls the acquisition of a macro image, the setting of an image pickup condition, and the acquisition of a micro image with respect to each sample. Therefore, even if there is an unnecessary object, such as dust, in a slide used as a sample, an influence from such an unnecessary object can be reliably excluded, and an image pickup condition for a sample can be correctly set.
  • a sample whose macro image has been acquired by the macro image acquiring means is placed at a standby position without being moved directly to an image acquiring position in the micro image acquiring means.
  • the acquisition of a macro image of the sample and the setting of an image pickup condition using a macro image can be performed independently of the acquisition of a micro image in which a certain amount of time is taken to acquire a high-resolution image. Therefore, after having set an image pickup condition for the sample, the operator has no need to pay constant attention to the image acquiring apparatus until the acquisition of the micro image is completed, and hence an operational burden imposed on the operator can be lightened.
  • the storage position in the sample storage means may be set as the standby position, and the sample whose macro image has been acquired may be again returned to the storage position.
  • sample standby means capable of putting a plurality of samples on standby may be provided independently of the sample storage means.
  • a semi-automatic mode in which an operator makes a necessary confirmation when an image pickup condition for a micro image is set with reference to a macro image, and additionally, a sample whose macro image has been acquired is placed at a standby position without being moved directly to an image acquiring position for the micro image acquiring means; therefore, image acquisition with respect to each of a plurality of samples can be performed with high efficiency.
  • Fig. 1 is a block diagram showing a structure of an embodiment of an image acquiring apparatus.
  • Fig. 2 is a schematic view showing a structure of a microscope unit of the image acquiring apparatus.
  • Fig. 3 is a figure with schematic views showing a method for acquiring a sample image.
  • Fig. 4 is a graph showing a method for setting a threshold to binarize a macro image.
  • Fig. 5 is a block diagram showing a structure of a control unit of the image acquiring apparatus.
  • Fig. 6 is a schematic view showing the production of sample data using a micro image.
  • Fig. 7 is a flowchart showing an image acquiring method in a full-automatic mode.
  • Fig. 8 is a flowchart showing an image acquiring method in a semi-automatic mode.
  • Fig. 9 is a flowchart showing an image acquiring method in a semi-automatic mode.
  • Fig. 10 is a flowchart showing a method for setting an image pickup condition.
  • Fig. 11 is a view showing an example of an operation screen used to set an image pickup condition.
  • Fig. 12 is a view showing an example of an operation screen used to set an image pickup condition.
  • Fig. 13 is a view showing an example of an operation screen used to set an image pickup condition.
  • Fig. 1 is a block diagram showing a structure of an embodiment of an image acquiring apparatus according to the invention.
  • the image acquiring apparatus according to this embodiment is a microscope system used to acquire images of a sample S with high resolution, and is made up of a microscope unit 10 used to acquire images of the sample S and a control unit 60 that controls the microscope unit 10 acquiring the images.
  • a slide (preparation) in which a biological sample, such as a tissue slice, is enclosed in a slide glass when image data to be used in a virtual microscope is acquired can be mentioned as an example of the sample S serving as an object of image acquisition.
  • the microscope unit 10 includes a sample storage unit 11, a macro image acquiring unit 20, and a micro image acquiring unit 30.
  • the sample storage unit 11 is storage means structured so as to store a plurality of samples (for example, a plurality of slides containing biological samples sealed, respectively) S as objects of image acquisition.
  • a door 12 to be used by an operator to store and extract a sample S is provided in this sample storage unit 11.
  • an interlock mechanism 13 for preventing the door 12 from being opened by mistake during image acquisition is attached.
  • the macro image acquiring unit 20 is first image acquiring means for acquiring a macro image as a low-magnification image of a sample S. This image acquiring unit 20 acquires a macro image at low resolution corresponding to a whole image of a sample S.
  • a macro light source 25 is disposed to supply light, which is used to produce an optical image of the sample S when macro images are acquired, for the macro image acquiring unit 20.
  • the micro image acquiring unit 30 is second image acquiring means for acquiring a micro image as a high-magnification image of a sample S.
  • a micro image with high resolution of a target sample S is acquired.
  • a micro light source 35 is disposed to supply light, which is used to produce an optical image of the sample S when micro images are acquired, for the micro image acquiring unit 30.
  • an image correcting unit 38 is disposed to make necessary corrections to image data concerning micro images acquired by the image acquiring unit 30. A dark correction, a shading correction, etc., can be mentioned as corrections performed by the image correcting unit 38.
  • a shading correction can be performed in such a way that a blank image obtained by imaging a reference sample, such as a slide on which no biological sample is placed, is pre-acquired, and then a shading correction is performed while referring to this blank image.
  • Macro images may be corrected in the same way.
  • the sample transfer unit 14 is transferring means for transferring a sample S between a storage position in the sample storage unit 11 and each of the image acquiring positions for the macro image acquiring unit 20 and the micro image acquiring unit 30.
  • the sample stage 15 carries a sample S placed thereon when acquiring a macro image or a micro image, and is used for setting and adjusting an image acquiring position of the sample S.
  • the microscope unit 10 includes a drive control unit 40 that controllably drives each unit provided in the microscope unit 10.
  • the control unit 60 includes a data processing unit 70, a data storage unit 75, and an image acquisition control unit 80.
  • Image data of macro images acquired by the macro image acquiring unit 20 and image data of micro images acquired by the micro image acquiring unit 30 are input into the data processing unit 70, and these image data undergo necessary data processing.
  • the image acquisition control unit 80 controls an operation to acquire an image of the sample S in the microscope unit 10 via the drive control unit 40.
  • Image data input into the data processing unit 70, various data and information obtained by image data processing, or pieces of control information used in the image acquisition control unit 80 are stored and kept in the data storage unit 75 according to need.
  • the control unit 60 is formed of a computer that includes, for example, a CPU and necessary storage devices, such as memories and hard disks.
  • a display device 61 and an input device 62 are connected to the control unit 60.
  • the display device 61 is, for example, a CRT display or a liquid crystal display, and is used to display an operation screen necessary to operate the image acquiring apparatus or to display an acquired image of the sample S.
  • the input device 62 is, for example, a keyboard or a mouse, and is used to input information necessary for image acquisition or to input instructions for an image acquiring operation.
  • Fig. 2 is a schematic view of the structure of the microscope unit 10.
  • the microscope unit 10 according to this embodiment is formed as a transmission type microscope system used to acquire an optical image of a sample S.
  • two directions perpendicular to each other in a horizontal direction are defined as an X-axis direction and a Y-axis direction, respectively, and a vertical direction perpendicular to the horizontal direction is defined as a Z-axis direction, as shown in the figure.
  • the Z-axis direction i.e., the vertical direction is that of an optical axis for image acquisition in this microscope system.
  • the structure of the macro image acquiring unit 20 and the structure of the micro image acquiring unit 30 are mainly shown, and the sample storage unit 11, the sample transfer unit 14, etc., are not shown.
  • the sample S is placed on the sample stage 15 at the time of image acquisition in the image acquiring unit 20 or 30.
  • This sample stage 15 is structured as an XY stage movable in the X-axis direction and the Y-axis direction by using a stepping motor, a DC motor, or a servo motor. With this structure, by driving the sample stage 15 within the XY plane, the image acquiring position for the image acquiring unit 20 or 30 with respect to the sample S is set and adjusted. In this embodiment, this sample stage 15 is movable between an image acquiring position for the macro image acquiring unit 20 and an image acquiring position for the micro image acquiring unit 30.
  • the macro image acquiring unit 20 and the macro light source 25 are provided at respective predetermined positions on an optical axis 20a, with respect to the macro image acquiring position to acquire a macro image of the sample S.
  • the macro light source 25 is a light source from which light used to produce an optical image for macro image acquisition is projected onto the sample S, and is disposed under the sample stage 15.
  • the macro image acquiring unit 20 is configured by using an image pickup device 21, such as a two-dimensional CCD sensor, that is capable of acquiring a two-dimensional image by the optical image of the sample S.
  • An image pickup optical system 22 serving as an optical system that guides the optical image of the sample S is disposed between the macro image acquiring position at which the sample S is placed and the image pickup device 21.
  • the micro image acquiring unit 30 and the micro light source 35 are provided at respective predetermined positions on an optical axis 30a, with respect to the micro image acquiring position to acquire a micro image of the sample S.
  • the micro light source 35 is a light source from which light used to produce an optical image for micro image acquisition is projected onto the sample S, and is, together with a condensing lens 36, disposed under the sample stage 15.
  • the micro image acquiring unit 30 is configured by using an image pickup device 31, such as a one-dimensional CCD sensor, that is capable of acquiring a one-dimensional image by the optical image of the sample S.
  • An objective lens 32 and a light guide optical system 34 serving as an optical system that guides the optical image of the sample S, are disposed between the micro image acquiring position at which the sample S is placed and the image pickup device 31.
  • the objective lens 32 produces an optical image of the sample S by entering light that has been transmitted through the sample S thereinto.
  • the light guide optical system 34 is formed of, for example, a tube lens, and guides the optical image of the sample S to the image pickup device 31.
  • a Z stage 33 using a stepping motor or a piezo-actuator is provided, and by driving the objective lens 32 in the Z-axis direction by this Z stage 33, focusing on the sample S can be performed.
  • the image pickup device 31 in this micro image acquiring unit 30 an image pickup device capable of acquiring a two-dimensional image and TDI driving can also be used as well as the image pickup device capable of acquiring a one-dimensional image.
  • this micro image acquiring image pickup device 31 for example, an image pickup device such as a 3-CCD camera capable of acquiring a color image is preferably used.
  • an image pickup device such as a 3-CCD camera capable of acquiring a color image is preferably used.
  • the macro image acquiring image pickup device 21 either a monochrome image acquiring image pickup device or an image pickup device capable of acquiring a color image can be used as appropriate.
  • a monochrome image acquiring image pickup device when acquisition of a color image is not necessary, a monochrome image acquiring image pickup device can also be used.
  • the drive control unit 40 includes a stage control unit 41, an image pickup device control unit 42, and a light source control unit 43 which are provided for the sample stage 15, the macro image acquiring unit 20, the micro image acquiring unit 30, the light sources 25 and 35.
  • the stage control unit 41 controllably drives the sample stage 15, which is an XY stage, and a Z stage 33 so as to set and adjust image pickup conditions concerning the sample S.
  • the image pickup device control unit 42 controllably drives the image pickup devices 21 and 31 so as to control the image acquisition of the sample S.
  • the light source control unit 43 controllably drives the light sources 25 and 35 so as to control the projection of light used for the image acquisition of the sample S.
  • Concerning the control of each unit of the microscope unit 10 another structure may be used so that each unit is controlled directly by the image acquisition control unit 80 of the control unit 60 without using the drive control unit 40.
  • a macro image as a whole image of the sample S to be used for setting an image pickup condition of a micro image is acquired. For example, if a slide in which a biological sample or the like is enclosed in a slide glass as mentioned above is used as the sample S, an image of the whole of the slide or an image of a predetermined range including the biological sample is acquired as the macro image.
  • a micro image of the sample S at target resolution is acquired with reference to the set image pickup condition.
  • This micro image acquisition is performed by two-dimensionally scanning the sample S at a predetermined resolution higher than that of the macro image as schematically shown in Fig. 3(a).
  • the image pickup device 31 such as a one-dimensional CCD camera
  • the longitudinal direction of an image pickup plane of the image pickup device 31 is defined as an X-axis direction
  • a direction orthogonal to this longitudinal direction is defined as a Y-axis direction.
  • the direction orthogonal to the longitudinal direction of the image pickup plane in the image pickup device 31, that is, the negative direction of the Y axis in Fig. 3(a) is the direction to scan the sample S.
  • the sample S on the sample stage 15 is scanned in the scanning direction (negative direction of the Y axis) by the image pickup device 31 to acquire a strip-like partial image A with the desired resolution. Furthermore, as shown in Fig. 3(a), a plurality of partial images A, B, ..., I are acquired by repeating a plurality of times this partial image acquisition while changing the image pickup position along the longitudinal direction (positive direction of the X axis) of the image pickup plane.
  • the whole micro image of the sample S can be generated.
  • a micro image acquiring method it is possible to preferably acquire image data of the sample S at sufficiently high resolution.
  • the hatched region in the partial image A whose longitudinal direction is along the X-axis direction, shows an image pickup region corresponding to the image pickup plane in the image pickup device 31.
  • an image acquiring range and focus measuring position are set as image pickup conditions of a micro image with reference to the macro image acquired by the image pickup device 21 of the macro image acquiring unit 20.
  • an image acquiring range and focus measuring position are set as image pickup conditions of a micro image with reference to the macro image acquired by the image pickup device 21 of the macro image acquiring unit 20.
  • the image acquiring range with respect to the sample S can be set to a rectangular range R including a biological sample L in the slide as an object of image acquisition.
  • Two-dimensional scanning of the sample S in the micro image acquiring unit 30 is performed within the image acquiring range R thus set.
  • the image acquiring range R is automatically set, for example, it is possible to employ a method in which a range in which an object (for example, a biological sample L) whose image is acquired exists is determined, by binarizing an image with reference to a threshold that has been set with respect to the brightness pattern in a macro image, and, based on a determination result, the image acquiring range R is set.
  • a range in which an object for example, a biological sample L
  • Fig. 4 is a graph showing a method for setting a threshold to binarize a macro image.
  • the abscissa axis represents the brightness value in each pixel of a macro image.
  • a region in which only a slide glass is provided without a biological sample L has the highest brightness resulting from the optically transmissible structure of the macro image acquiring unit 20.
  • a threshold with respect to a macro image, it is possible to employ a method in which two peaks P1 and P2 are found in its brightness distribution, for example, as shown in Fig. 4, and a brightness value at the center therebetween is set as a threshold T. Accordingly, ranges in which objects L exist in the image can be easily determined by binarizing the image according to this threshold. Further, a consecutive range greater than a specified area of the determined ranges is set as a range for one object L, and a minimum, rectangular region containing all of the objects L is set as an image acquiring range R.
  • a threshold may be set at a predetermined, fixed rate with respect to a brightness value of a peak.
  • erosion processing i.e., processing to reduce a pictorial mass
  • dilation processing i.e., processing to enlarge a pictorial mass
  • the focus measuring position is used, in the micro image acquiring unit 30, when acquiring focus information with respect to the sample S prior to acquisition of the micro image of the sample S.
  • focus measurement is performed by using the image pickup device 31 at a set focus measuring position to determine a focus position as focus information on acquisition of the micro image of the sample S.
  • Concerning the focus measuring position for example, when the inclination of the sample S in a horizontal plane, i.e., a deviation in focal position in a horizontal plane is negligible, only one focus measuring position is set for the sample S.
  • a two-dimensional focus map with respect to the image acquiring range R of the sample S can be obtained by setting three or more focus measuring positions in this way and then performing focus measurement.
  • the focal plane can be calculated from a plane including measurement result points at the three focus measuring positions.
  • a focal plane can be calculated from their measurement result points according to a fitting technique, such as a least-squares method.
  • Fig. 3(b) shows an example of setting of the focus measuring positions by using a macro image in the case where nine focus measuring positions are automatically set.
  • this lower left focus measuring position may be set to, for example, a position Q determined by a method in which it is moved toward the center of the image acquiring range R. Alternatively, such a position may be excluded from the focus measuring positions.
  • a focal plane is re-determined by excluding this distant measuring position.
  • the object is regarded as dust and excluded.
  • an image acquiring range R including a biological sample L and a predetermined number of focus measuring positions P are set as the image pickup conditions of a micro image by referring to a macro image acquired by the macro image acquiring unit 20. Thereafter, in the micro image acquiring unit 30, focus information concerning the focal position or the focal plane with respect to the sample S is acquired based on the focus measuring positions P, and the micro image of the sample S is then acquired based on the thus obtained focus information and the set image acquiring range R.
  • Fig. 3(b) shows an example in which the focus measuring positions P are set automatically by using a predetermined setting algorithm, however, to manually set the focus measuring positions, focus measuring positions with an appropriate number and arrangement may be set after an operator confirms the macro image.
  • Fig. 5 is a block diagram showing the structure of control unit 60.
  • the control unit 60 according to this embodiment is made up of the data processing unit 70, the data storage unit 75, and the image acquisition control unit 80.
  • the data processing unit 70 includes an image pickup condition setting unit 71 and a sample data producing unit 72.
  • the image pickup condition setting unit 71 is setting means for setting image pickup conditions for micro images while referring to the macro image acquired by the macro image acquiring unit 20 of the microscope unit 10.
  • the sample data producing unit 72 produces sample data that is image data concerning the sample S by using the micro image acquired by the micro image acquiring unit 30.
  • Fig. 6 is a schematic view for explaining the production of sample data while using micro images.
  • an image data group consisting of pieces of data concerning strip-like partial images A, B, C, ... is input to the control unit 60 as image data concerning the micro images acquired by the micro image acquiring unit 30 of the microscope unit 10 (see Fig. 3(a)).
  • the sample data producing unit 72 arranges and combines these partial images together, and produces image data of micro images with respect to the whole of the sample S so as to be sample data.
  • this sample data can be used as image data for the virtual microscope.
  • the image data concerning the sample S may be subjected to data compression according to need.
  • the image data input from the microscope unit 10 and the sample data produced by the sample data producing unit 72 are stored in the data storage unit 75 according to need.
  • the image acquisition control unit 80 includes a macro image acquisition control unit 81 and a micro image acquisition control unit 82.
  • the macro image acquisition control unit 81 controls an operation to acquire macro images of the sample S by means of the macro image acquiring unit 20.
  • the micro image acquisition control unit 82 controls an operation to acquire micro images by means of the micro image acquiring unit 30.
  • the image acquisition control units 81 and 82 control the acquiring operation of the macro and micro images of the sample S, and the image pickup condition setting unit 71 sets the image pickup condition, according to a control mode selected for the image acquisition processing.
  • the number of control modes provided in the control unit 60 is three, i.e., a full-automatic mode, a manual mode, and a semi-automatic mode.
  • Fig. 7 is a flowchart showing an image acquiring method according to the full-automatic mode.
  • the macro image acquisition control unit 81 of the control unit 60 performs macro image acquisition control by placing the sample S from a storage position in the sample storage unit 11 at an image acquiring position for the macro image acquiring unit 20 (macro image acquisition control step).
  • the image pickup condition setting unit 71 automatically sets an image pickup condition for a micro image corresponding to the macro image (image pickup condition setting step).
  • the micro image acquisition control unit 82 performs micro image acquisition control by placing the sample S whose image pickup condition has been set at an image acquiring position for the micro image acquiring unit 30 while referring to the image pickup condition (micro image acquisition control step).
  • an operator sets a plurality of slides S, which are objects for image acquisition, at the sample storage unit 11 of the microscope unit 10, and then closes a door 12 (step S101).
  • the number of slides that can be stored in the sample storage unit 11 is, for example, several hundred, depending on a specific structure of an apparatus.
  • one of the slides S that is an object for image acquisition is taken out from the sample storage unit 11, is then transferred by the sample transfer unit 14, placed on the sample stage 15, and is set at the image acquiring position for the macro image acquiring unit 20 (see Fig. 1 and Fig. 2) (S102).
  • the image pickup device 21 of the image acquiring unit 20 acquires a macro image of the slide S including the biological sample L (S103, macro image acquiring step).
  • an image acquiring range R and a plurality of focus measuring positions P which are image pickup conditions for the slide S, are automatically set by use of predetermined algorithm while referring to the macro image acquired above (S104, image pickup condition setting step).
  • the slide S that has completed the macro image acquisition is moved from the image acquiring position for the macro image acquiring unit 20 by means of the sample transfer unit 14 or the sample stage 15, and is set at an image acquiring position for the micro image acquiring unit 30 (S105). Thereafter, the image pickup device 31 of the image acquiring unit 30 acquires a micro image of the slide S while referring to the image pickup condition set by the image pickup condition setting unit 71 (S106, micro image acquiring step).
  • focus measurement is first performed in the micro image acquiring unit 30, and a focal plane, which serves as focus information most suitable to acquire an image of the biological sample L, is calculated from the obtained focal positions.
  • a focal plane which serves as focus information most suitable to acquire an image of the biological sample L.
  • the slide S is two-dimensionally scanned by the image pickup device 31 with respect to the image acquiring range R while performing focus control, thus acquiring a plurality of strip-like partial images.
  • the partial images are changed into a high-resolution micro image of the slide S (a digital slide in the virtual microscope) by being subjected to predetermined data combine processing by the sample data producing unit 72 of the control unit 60.
  • the slide S that has completed the micro image acquisition is returned from the image acquiring position for the micro image acquiring unit 30 to the storage position in the sample storage unit 11 by means of the sample transfer unit 14 (S107).
  • the image acquisition control unit 80 a confirmation is made of whether all of the slides S have undergone the image acquisition processing or not (S108). If there is a slide S to be subjected to the image acquisition processing here, steps S102 to S107 mentioned above are repeatedly performed. If all of the slides S have undergone the image acquisition processing, image acquisition with respect to the slides S set in the sample storage unit 11 is ended.
  • the image acquisition procedure according to the manual mode is performed basically in the same way as the image acquisition procedure according to the full-automatic mode, except that each step is manually performed under instructions from the operator.
  • the manual mode it is permissible to automatically perform a step at which instructions from the operator are unnecessary, such as a step at which the slide S that has completed macro image acquisition is moved to the image acquiring position for the micro image acquiring unit 30.
  • a step at which the slide S that has completed macro image acquisition is moved to the image acquiring position for the micro image acquiring unit 30.
  • the image pickup condition can be reset or a micro image can be re-acquired if a problem occurs by allowing the operator to confirm the micro image acquired above.
  • Fig. 8 and Fig. 9 are flowcharts, each showing an image acquiring method according to a semi-automatic mode.
  • the macro image acquisition control unit 81 of the control unit 60 performs control to place each of a plurality of samples S from the storage position in the sample storage unit 11 at an image acquiring position for the macro image acquiring unit 20 so as to acquire a macro image, and then performs control to place the sample S at a standby position (macro image acquisition control step).
  • the image pickup condition setting unit 71 allows the operator to confirm the macro image and a corresponding image pickup condition, and sets an image pickup condition for a micro image (image pickup condition setting step).
  • the micro image acquisition control unit 82 performs control to place the sample S that has been placed at the standby position and for which the image pickup condition has been set at an image acquiring position for the micro image acquiring unit 30, and performs micro image acquisition control while referring to the image pickup condition (micro image acquiring control step).
  • the standby position of the sample S is set at the storage position in the sample storage unit 11.
  • Fig. 8 and Fig. 9 first, the operator sets a plurality of slides S, which are objects for image acquisition, at the sample storage unit 11 of the microscope unit 10 in the same way as in the full-automatic mode, and then closes the door 12 (step S201).
  • the macro image acquisition control unit 81 performs the acquisition of a macro image of each of the slides S (S202, macro image acquiring step).
  • one of the slides S that is an object whose image is acquired is taken out from the storage position in the sample storage unit 11, is then transferred by the sample transfer unit 14, placed on the sample stage 15, and is set at an image acquiring position for the macro image acquiring unit 20 (S301).
  • the image pickup device 21 of the image acquiring unit 20 acquires a macro image of the slide S including the biological sample L (S302).
  • the slide S that has completed the macro image acquisition is returned from the image acquiring position for the macro image acquiring unit 20 to the storage position, which is also its standby position, in the sample storage unit 11 by means of the sample transfer unit 14 (S303). Thereafter, in the macro image acquisition control unit 81, a confirmation is made of whether all of the slides S have undergone the macro image acquisition processing or not (S304). If there is a slide S to be subjected to the macro image acquisition processing here, steps S301 to S303 mentioned above are repeatedly performed. If all of the slides S have undergone the macro image acquisition processing, macro image acquisition with respect to the slides S set in the sample storage unit 11 is ended.
  • an image pickup condition for a micro image is set in parallel with the macro image acquisition processing (S203, image pickup condition setting step).
  • the image pickup condition setting unit 71 of the control unit 60 with respect to a slide S that is an object for which an image pickup condition is set, the operator is allowed to confirm the macro image acquired above and a corresponding image pickup condition through display means, thus setting an image acquiring range R and a plurality of focus measuring positions P, which are image pickup conditions to acquire the micro image.
  • a detailed description of the method for setting the image pickup condition here will be given later.
  • the micro image acquisition control unit 82 performs the acquisition of a micro image of each of the slides S in parallel with a process for setting an image pickup condition (S204, micro image acquiring step).
  • S204 micro image acquiring step
  • one of the slides S for which an image pickup condition has already been set is selected from among the slides S, and image acquisition is performed as shown in Fig. 9.
  • the slide S selected therefrom is taken out from the storage position, which is also its standby position in the sample storage unit 11, is then transferred by the sample transfer unit 14, placed on the sample stage 15, and is set at an image acquiring position for the micro image acquiring unit 30 (S401).
  • the image pickup device 31 of the image acquiring unit 30 acquires a micro image of the slide S while referring to the image pickup condition set by the image pickup condition setting unit 71 (S402).
  • the method for acquiring the micro image is carried out basically in the same way as in the full-automatic mode.
  • the slide S that has completed the micro image acquisition is returned from the image acquiring position for the micro image acquiring unit 30 to the storage position in the sample storage unit 11 by means of the sample transfer unit 14 (S403). Thereafter, in the micro image acquisition control unit 82, a confirmation is made of whether an image pickup condition is to be set for one of the slides S (S404). If there is still a slide S for which an image pickup condition is set and which is subjected to image acquisition processing, steps S401 to S403 mentioned above are repeatedly performed.
  • micro image acquisition control unit 82 If there is not a slide S for which an image pickup condition has been set, in the micro image acquisition control unit 82, a confirmation is further made of whether one of the slides S has already completed undergoing the image acquisition processing or not (S405). If there is a slide S to be further subjected to image acquisition processing here, steps S401 to S403 mentioned above are repeatedly performed while waiting for an image pickup condition to be set. If all of the slides S have completed undergoing the image acquisition processing, image acquisition with respect to the slides S set in the sample storage unit 11 is ended.
  • a session is defined in accordance with a plurality of samples S set in the sample storage unit 11 by the operator, and a sample group of samples S and a corresponding data group of macro images, image pickup conditions, micro images, etc., are associated with each other by this session.
  • the control unit 60 shown in Fig. 5 includes a session managing unit 73 provided in the data processing unit 70 and a session switching unit 83 provided in the image acquisition control unit 80.
  • the session managing unit 73 manages the data group consisting of macro images, image pickup conditions, micro images, etc., in each session, and allows the data storage unit 75 to store the data group if necessary.
  • the data storage unit 75 of Fig. 5 as an example of the data management, a case is shown in which a data group of a plurality of sessions consisting of session 1, session 2, ... is stored.
  • the session switching unit 83 controls switching between an image acquisition processing and another image acquisition processing in association with an exchange for a sample group of a plurality of samples S, and gives instructions to the session managing unit 73 to perform switching to a corresponding data group.
  • the image acquisition processing of the samples S can be suitably controlled, and data corresponding to this can be suitably be managed by associating the sample group of the samples S and the data group in image acquisition with each other by use of the concept of session.
  • FIG. 10 is a flowchart showing one example of a method for setting an image pickup condition.
  • Fig. 11 to Fig. 13 show examples of operation screens used to set an image pickup condition. These operation screens are displayed on the display device 61 in the structure of Fig. 1.
  • An operation screen 100 of Fig. 11 is displayed on the display device 61 in a state in which macro images of one or more of the slides S whose images are to be acquired have been acquired.
  • a list of slides whose macro images have been acquired i.e., slide 1 to slide 3 in Fig. 11
  • thumbnail images of the corresponding macro images are displayed on the display device 61 in a state in which macro images of one or more of the slides S whose images are to be acquired.
  • a desired slide name for example "slide 1”
  • the operator can see a macro image of "slide 1" selected therefrom that is obtained by being enlarged in a macro image display part 102 as shown in Fig. 12, and can confirm the contents thereof.
  • an image pickup condition setting button 103 is clicked in this state, a slide to set an image pickup condition is selected from a slide list displayed thereon (step S501).
  • a macro image of slide 1 selected therefrom is displayed in the macro image display part 111 of the image pickup condition setting screen 110 of Fig. 13.
  • the operator confirms this macro image, and determines whether the image pickup condition is set automatically or manually (S502).
  • a confirmation is made of whether automatic setting has been selected or not (S503), and if automatic setting has been selected, the image pickup condition is automatically set, the setting of the image pickup condition with respect to the slide is then ended, and the process proceeds to the subsequent step to select a next slide and to set an image pickup condition.
  • the operator manually sets an image acquiring range R and a focus measuring position P with respect to the macro image of the slide in consideration of a range in which the biological sample L exists and its shape (S504).
  • a confirmation is made of whether all of the slides S have already completed the setting of the image pickup condition or not (S505). If there is still a slide S for which an image pickup condition must be set, steps S501 to S504 mentioned above are repeatedly performed. If the setting of the image pickup condition for all of the slides S is completed, the setting of the image pickup condition for the slides S set in the sample storage unit 11 is ended.
  • the image pickup condition setting screen 110 shown in Fig. 13 when a macro image of the slide S is displayed in the macro image display part 111, the image acquiring range R and the focus measuring position P that have been automatically set are first calculated, and the results are displayed on the macro image as default values for the image pickup condition.
  • the operator confirms the automatically set image pickup condition displayed here, and selects whether to set the image pickup condition automatically or manually. If the image pickup condition is automatically set, the default values, which have been first displayed, can be used without being changed, therefore, a setting end button 112 is clicked, and the setting of the image pickup condition with respect to this slide S is ended.
  • the automatically set image acquiring range R and the focus measuring position P may be displayed on the thumbnail image of the macro image in the list display part 101 shown in Fig. 11. Since the operator can have a look at the image pickup conditions of the slides by superimposing the image pickup condition on the thumbnail image in this way, the slides to be manually set can be easily extracted.
  • a focus measuring position setting button 113 or an image acquiring range setting button 114 is clicked according to need.
  • the image pickup condition is manually set by changing, adding, deleting, or creating the focus measuring position P or the image acquiring range R on the macro image display part 111. It is possible to again automatically set the image pickup condition by clicking an automatic setting button 115 if necessary.
  • the setting end button 112 is clicked, so that the setting of the image pickup condition with respect to this slide S is ended.
  • a process corresponding to the image acquiring method performed in the image acquiring apparatus shown in Fig. 1 can be carried out by an image acquiring program for allowing the computer to execute the image acquisition processing.
  • the control unit 60 of the image acquiring apparatus can be made up of a CPU that runs each software program necessary for the image acquisition processing, a ROM in which the software programs are stored, and a RAM in which data is temporarily stored during program execution.
  • the image acquiring apparatus and the image acquiring method mentioned above can be realized by executing a predetermined image acquiring program by the CPU in the thus formed structure.
  • recording media include, for example, magnetic media such as hard disks and flexible disks, optical media such as CD-ROMs and DVD-ROMs, magnetic optical media such as floptical disks, and hardware devices such as RAMs, ROMs, and semiconductor nonvolatile memories exclusively installed so as to execute or store program commands.
  • the macro image acquiring unit 20 and the micro image acquiring unit 30 are provided for a plurality of samples S each of which is an object of image acquisition, and an image pickup condition is set while referring to a macro image that shows the whole of a sample S in the image pickup condition setting unit 71, and a high-resolution micro image is acquired. Accordingly, a high-resolution micro image of a sample, which can be used as sample image data for use in, for example, a virtual microscope, can be suitably acquired.
  • a semi-automatic mode is provided as a control mode that controls the acquisition of a macro image of each sample S, the setting of an image pickup condition, and the acquisition of a micro image, in which an operator makes a necessary confirmation when an image pickup condition for a micro image is set with reference to a macro image. Accordingly, even if there is, for example, dust in a slide of a sample S or there is an extra object such as notes on information concerning a biological sample, an influence exerted from such an object in setting an image pickup condition can be reliably excluded, and the image pickup condition for the sample can be correctly set.
  • a sample S whose macro image has been acquired in the macro image acquiring unit 20 is placed at the standby position without moving the sample S directly to the image acquiring position in the micro image acquiring unit 30.
  • the acquisition of the macro image of the sample S (step S202) and the setting of an image pickup condition using the macro image (S203) can be performed independently of the acquisition of a micro image that takes a certain amount of time to scan the sample S so as to acquire a high-resolution image (S204).
  • a micro image is automatically acquired by the micro image acquisition control unit 82 while referring to the set image pickup condition, therefore, there is no need to pay constant attention to the image acquiring apparatus until the acquisition of the micro image is completed, and hence an operational load imposed on the operator can be greatly reduced.
  • the storage position in the sample storage unit 11 is used as the standby position in the above embodiment, and the sample S whose macro image has been acquired is again returned to the storage position. Therefore, the structure of the microscope unit 10 can be simplified.
  • a sample standby unit that can put a plurality of samples S on standby may be provided independently of the sample storage unit 11.
  • the image acquiring apparatus mentioned above uses the sample transfer unit 14 and the sample stage 15 that is an XY stage as sample moving means for moving each of the samples S among the storage position in the sample storage unit 11, the standby position (which is identical with the storage position in the above structure), and the respective image acquiring positions for the macro image acquiring unit 20 and the micro image acquiring unit 30.
  • the acquisition of a macro image, the setting of an image pickup condition, and the acquisition of a micro image, which are performed while moving the sample S thereamong can be suitably realized.
  • the micro image acquiring unit 30 includes the image pickup device 31 that is a one-dimensional sensor capable of acquiring a one-dimensional image or a TDI drive two-dimensional sensor capable of acquiring a two-dimensional image and capable of performing TDI driving, and a direction perpendicular to the longitudinal direction of an image pickup plane in the image pickup device 31 is used as a scanning direction followed when a micro image is acquired. Additionally, the micro image acquisition control unit 82 acquires a partial image while scanning a sample S in the scanning direction with the image pickup device 31, and performs control to acquire a plurality of partial images that are used as micro images by repeatedly performing an operation for acquiring the partial images two or more times while changing the image pickup position along the longitudinal direction of the image pickup plane.
  • strip-like high-resolution partial images by scanning the sample S in one direction are acquired, and are combined together in the other direction, thus making a micro image of the whole of the sample, therefore, image data concerning the sample S can be suitably acquired with fully high resolution.
  • the image pickup condition setting unit 71 displays an image pickup condition automatically obtained in accordance with a macro image on the display device 61 in the form of default values of the image pickup condition, and an operator selects whether to use this image pickup condition or to manually reset the image pickup condition.
  • the operator can manually correct or reset the image pickup condition if there is a problem, for example, if the sample S has extra objects, and if there is not such a problem, the operator can use the automatically-set image pickup condition without changes. Therefore, the operation for acquiring images of a plurality of samples S including the setting of the image pickup condition can be efficiently performed by manually setting the image pickup condition only when necessary.
  • the image pickup condition setting unit 71 sets an image acquiring range R and a focus measuring position P, which are used as image pickup conditions for a micro image, with reference to a macro image
  • the micro image acquisition control unit 82 acquires focus information about a sample S based on the focus measuring position P, and performs control to acquire a micro image based on the obtained focus information and the image acquiring range R. Therefore, parameters used to acquire the micro image can be suitably set from information concerning the sample S obtained from the macro image, and hence sample image data having high resolution and being in an excellent state can be acquired.
  • the full-automatic mode in addition to the semi-automatic mode, is prepared as the image-acquiring operation control mode of the control unit 60.
  • the operator appropriately selects either the semi-automatic mode or the full-automatic mode as the control mode that controls the acquisition of a macro image with respect to each sample, the setting of an image pickup condition, and the acquisition of a micro image, and accordingly, depending on a state of each sample S of the object of image acquisition, for example, depending on the fact that a sample S has much dust or a little dust, image acquisition can be carried out according to a suitable method.
  • switching between the control modes may be performed, for example, switching to the full-automatic mode may be performed during an operation in the semi-automatic mode.
  • switching to the full-automatic mode may be performed during an operation in the semi-automatic mode.
  • an image acquiring operation for a sample S can be again performed if a micro image acquired with respect to the sample S has a problem after the completion of the image acquisition processing.
  • a condition may be manually set for one of the slides, and be then imposed on the other slides.
  • a condition preset in the control unit 60 may be given to one of the slides or to the slides.
  • the image pickup condition may be reset in the semi-automatic mode for only the slide that has failed in image acquisition so as to acquire a new image.
  • the image acquiring apparatus, the image acquiring method, and the image acquiring program of the invention are not limited to those in the embodiment and structure examples, and they can be variously modified.
  • the sample S is moved between the image acquiring position for the macro image acquiring unit 20 and the image acquiring position for the micro image acquiring unit 30 by means of the sample stage 15, however, without being limited to this, a sample stage of the macro image acquiring unit 20 may be individually provided to differ from that of the micro image acquiring unit 30.
  • the sample moving means is the sample transfer unit 14 alone.
  • the macro image acquiring unit 20 can be structured in various manners, for example, the macro image acquiring unit 20 may be disposed at a predetermined position of the sample transfer unit 14, or may be disposed in the sample storage unit 11.
  • the macro image acquiring unit 20 and the micro image acquiring unit 30 are disposed independently of each other, however, the light sources and the image acquiring means may be formed as a single light source and single image acquiring means, respectively, by providing an optical system for image acquisition switchable in accordance with the acquisition of a macro image or a micro image.
  • an optical system provided with a revolver including an objective lens for macro image acquisition and an objective lens for micro image acquisition can be mentioned as an example of the switchable optical system for image acquisition.
  • the image pickup device capable of acquiring a one-dimensional image or the image pickup device capable of acquiring a two-dimensional image and capable of performing TDI driving is shown as the image pickup device 31 in the micro image acquiring unit 30, however, without being limited to this, an image pickup device, such as a two-dimensional CCD camera (area sensor), may be used.
  • an image pickup device such as a two-dimensional CCD camera (area sensor)
  • a micro image of the whole of a sample S can be produced by acquiring a plurality of high-resolution tile images and combining these images together, as is disclosed by US Patent No. 6,816,606 .
  • the image acquiring apparatus is formed by using the transmission type microscope system, however, the present invention is not limited to this.
  • the transmission type light source may be replaced by an epi-illumination type light source so as to form a reflection type microscope system.
  • the image acquiring apparatus includes (1) sample storage means for storing a plurality of samples, (2) macro image acquiring means for acquiring a macro image of the sample, (3) micro image acquiring means for acquiring a micro image while scanning the sample with higher resolution than that of the macro image, and (4) control means including macro image acquisition control means for controlling an operation to acquire the macro image by the macro image acquiring means, image pickup condition setting means for setting an image pickup condition for the micro image with reference to the macro image, and micro image acquisition control means for controlling an operation to acquire the micro image by the micro image acquiring means wherein (5) the control means has a semi-automatic mode serving as a control mode, and (6) in the semi-automatic mode, the macro image acquisition control means performs control to place each of the samples from a storage position in the sample storage means at an image acquiring position for the macro image acquiring means so as to acquire the macro image, and then performs control to place the sample at a standby position; the image pickup condition setting means sets the image pickup condition for the micro image while allowing an image acquisition condition for
  • the image acquiring method (1) uses a plurality of samples stored in sample storage means as objects, and includes (2) a macro image acquiring step of acquiring a macro image of the sample, (3) a micro image acquiring step of acquiring a micro image while scanning the sample with higher resolution than that of the macro image, and (4) a control step including a macro image acquisition control step of controlling an operation to acquire the macro image by the macro image acquiring step, an image pickup condition setting step of setting an image pickup condition for the micro image with reference to the macro image, and a micro image acquisition control step of controlling an operation to acquire the micro image by the micro image acquiring step wherein (5) the control step has a semi-automatic mode serving as a control mode, and (6) in the semi-automatic mode, the macro image acquisition control step performs control to place each of the samples from a storage position in the sample storage means at an image acquiring position for the macro image acquiring step so as to acquire the macro image, and then performs control to place the sample at a standby position; the image pickup condition setting step sets the image
  • the image acquiring program is applied to an image acquiring apparatus including (1) sample storage means for storing a plurality of samples, (2) macro image acquiring means for acquiring a macro image of the sample, and (3) micro image acquiring means for acquiring a micro image while scanning the sample with higher resolution than that of the macro image, and includes (4) a control processing including a macro image acquisition control processing for controlling an operation to acquire the macro image by the macro image acquiring means; an image pickup condition setting processing for setting an image pickup condition for the micro image with reference to the macro image; and a micro image acquisition control processing for controlling an operation to acquire the micro image by the micro image acquiring means wherein (5) the control processing has a semi-automatic mode serving as a control mode, and (6) in the semi-automatic mode, the macro image acquisition control processing performs control to place each of the samples from a storage position in the sample storage means at an image acquiring position for the macro image acquiring means so as to acquire the macro image, and then performs control to place the sample at a standby position; the image pickup condition setting processing
  • the image acquiring apparatus includes sample moving means for moving each of the samples among the storage position, the standby position, and the respective image acquiring positions for the macro image acquiring means and the micro image acquiring means.
  • the image acquiring method includes a sample moving step of moving each of the samples among the storage position, the standby position, and the respective image acquiring positions for the macro image acquiring step and the micro image acquiring step.
  • the image acquiring program includes a sample moving process executed by a computer to move each of the samples among the storage position, the standby position, and the respective image acquiring positions for the macro image acquiring means and the micro image acquiring means. Accordingly, the acquisition of a macro image, the setting of an image pickup condition, and the acquisition of a micro image, which are performed while moving the sample among the above-mentioned positions, can be suitably realized.
  • the micro image acquiring means includes an image pickup device capable of acquiring a one-dimensional image or capable of acquiring a two-dimensional image and performing TDI driving, a direction perpendicular to the longitudinal direction of an image pickup plane in the image pickup device is a scanning direction followed when a micro image is acquired, and the micro image acquisition control means performs control to acquire a partial image while scanning the sample in the scanning direction with the image pickup device and to acquire a plurality of partial images used as a micro image by repeatedly performing the partial image acquisition while changing the image pickup position along the longitudinal direction of the image pickup plane.
  • the micro image acquiring step uses an image pickup device capable of acquiring a one-dimensional image or capable of acquiring a two-dimensional image and performing TDI driving, a direction perpendicular to the longitudinal direction of an image pickup plane in the image pickup device is a scanning direction followed when a micro image is acquired, and the micro image acquisition control step performs control to acquire a partial image while scanning the sample in the scanning direction with the image pickup device and to acquire a plurality of partial images used as a micro image by repeatedly performing the partial image acquisition while changing the image pickup position along the longitudinal direction of the image pickup plane.
  • the micro image acquiring means includes an image pickup device capable of acquiring a one-dimensional image or capable of acquiring a two-dimensional image and performing TDI driving, a direction perpendicular to the longitudinal direction of an image pickup plane in the image pickup device is a scanning direction followed when a micro image is acquired, and the micro image acquisition control processing performs control to acquire a partial image while scanning the sample in the scanning direction with the image pickup device and to acquire a plurality of partial images used as a micro image by repeatedly performing the partial image acquisition while changing the image pickup position along the longitudinal direction of the image pickup plane.
  • strip-like partial images by scanning the sample in one direction with a one-dimensional sensor or a TDI-driven two-dimensional sensor are acquired with high resolution, and the partial images are combined together in another direction so as to be used as a micro image of the whole of the sample, therefore, image data concerning the sample can be suitably acquired with fully high resolution.
  • the image pickup condition setting means allows the display means to display an image pickup condition automatically obtained in accordance with a macro image so that an operator can select whether to use the image pickup condition or to manually reset the image pickup condition.
  • the image pickup condition setting step allows the display means to display an image pickup condition automatically obtained in accordance with a macro image so that an operator can select whether to use the image pickup condition or to manually reset the image pickup condition.
  • the image pickup condition setting processing allows the display means to display an image pickup condition automatically obtained in accordance with a macro image so that an operator can select whether to use the image pickup condition or to manually reset the image pickup condition.
  • a method can be employed in which, if there is a problem, for example, if a sample has an unnecessary object, the operator manually corrects or resets the image pickup condition, and, if there is no problem, the operator uses the automatically-set image pickup condition without changes.
  • an operation to acquire images of a plurality of samples including the setting of an image pickup condition can be efficiently performed by manually setting the image pickup condition only when necessary.
  • the control means may have the full-automatic mode serving as the control mode, and, in the full-automatic mode, the macro image acquisition control means performs control to place the sample from the storage position at the image acquiring position for the macro image acquiring means so as to acquire the macro image; the image pickup condition setting means automatically sets the image pickup condition for the micro image corresponding to the macro image with respect to the sample the macro image of which has been acquired; and the micro image acquisition control means performs control to place the sample for which the image pickup condition has been set at the image acquiring position for the micro image acquiring means, and performs control to acquire the micro image with reference to the image pickup condition.
  • the control step may have the full-automatic mode serving as the control mode, and, in the full-automatic mode, the macro image acquisition control step performs control to place the sample from the storage position at the image acquiring position for the macro image acquiring step so as to acquire the macro image; the image pickup condition setting step automatically sets the image pickup condition for the micro image corresponding to the macro image with respect to the sample the macro image of which has been acquired; and the micro image acquisition control step performs control to place the sample for which the image pickup condition has been set at the image acquiring position for the micro image acquiring step, and performs control to acquire the micro image with reference to the image pickup condition.
  • the control processing may have the full-automatic mode serving as the control mode, and, in the full-automatic mode, the macro image acquisition control processing performs control to place the sample from the storage position at the image acquiring position for the macro image acquiring means so as to acquire the macro image; the image pickup condition setting processing automatically sets the image pickup condition for the micro image corresponding to the macro image with respect to the sample the macro image of which has been acquired; and the micro image acquisition control processing performs control to place the sample for which the image pickup condition has been set at the image acquiring position for the micro image acquiring means, and performs control to acquire the micro image with reference to the image pickup condition, and wherein the control processing is executed by a computer.
  • an operator appropriately selects either the semi-automatic mode or the full-automatic mode as the control mode that controls the acquisition of a macro image, the setting of an image pickup condition, and the acquisition of a micro image with respect to each of the samples, therefore, through a suitable method, image acquisition can be performed in accordance with, for example, the state of the samples whose images are acquired.
  • the image pickup condition setting means sets an image acquiring range and a focus measuring position that are used as image pickup conditions for a micro image while referring to a macro image
  • the micro image acquisition control means performs control to acquire focus information concerning the sample based on the focus measuring position and performs control to acquire a micro image based on the obtained focus information and the image acquiring range.
  • the image pickup condition setting step sets an image acquiring range and a focus measuring position that are used as image pickup conditions for a micro image while referring to a macro image
  • the micro image acquisition control step performs control to acquire focus information concerning the sample based on the focus measuring position, and performs control to acquire a micro image based on the obtained focus information and the image acquiring range.
  • the image pickup condition setting processing sets an image acquiring range and a focus measuring position that are used as image pickup conditions for a micro image while referring to a macro image
  • the micro image acquisition control processing performs control to acquire focus information concerning the sample based on the focus measuring position, and performs control to acquire a micro image based on the obtained focus information and the image acquiring range.
  • parameters used to acquire a micro image can be appropriately set from information concerning the sample obtained by the macro image, and sample image data having high resolution and being in an excellent state can be acquired.
  • the macro image acquiring means and the micro image acquiring means may be single image acquiring means, and the optical system for image acquisition may be configured to be switchable in accordance with the acquisition of a macro image or a micro image.
  • the optical system for image acquisition may be configured to be switchable in accordance with the acquisition of a macro image or a micro image.
  • an optical system provided with a revolver including two kinds of objective lenses can be mentioned as an example of the switchable optical system for image acquisition.
  • the present invention can be employed as an image acquiring apparatus, an image acquiring method, and an image acquiring program each of which is capable of efficiently performing image acquisition with respect to each of a plurality of samples.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Chemical & Material Sciences (AREA)
  • Analytical Chemistry (AREA)
  • General Physics & Mathematics (AREA)
  • Optics & Photonics (AREA)
  • Microscoopes, Condenser (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)
  • Processing Or Creating Images (AREA)
EP07250728A 2006-03-01 2007-02-21 Appareil de capture d'images, procédé de capture d'images et programme de capture d'images Active EP1830216B1 (fr)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
JP2006055456A JP4917329B2 (ja) 2006-03-01 2006-03-01 画像取得装置、画像取得方法、及び画像取得プログラム

Publications (2)

Publication Number Publication Date
EP1830216A1 true EP1830216A1 (fr) 2007-09-05
EP1830216B1 EP1830216B1 (fr) 2011-02-09

Family

ID=38110440

Family Applications (1)

Application Number Title Priority Date Filing Date
EP07250728A Active EP1830216B1 (fr) 2006-03-01 2007-02-21 Appareil de capture d'images, procédé de capture d'images et programme de capture d'images

Country Status (6)

Country Link
US (1) US8126250B2 (fr)
EP (1) EP1830216B1 (fr)
JP (1) JP4917329B2 (fr)
AT (1) ATE498147T1 (fr)
DE (1) DE602007012375D1 (fr)
ES (1) ES2361054T3 (fr)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2010049651A1 (fr) * 2008-10-31 2010-05-06 Novacyt Procedure de preparation d ' une image d ' analyse virtuelle traitee
CN105092582A (zh) * 2015-08-07 2015-11-25 苏州合惠生物科技有限公司 一种用于全自动免疫组化的大视场镜检装置及检测方法
CN112136071A (zh) * 2018-02-26 2020-12-25 凯利博成像和诊断公司 用于对体外组织进行宏观和微观成像的系统和方法

Families Citing this family (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2011001635A1 (fr) * 2009-06-30 2011-01-06 株式会社日立ハイテクノロジーズ Dispositif d'inspection d'un semi-conducteur et procédé d'inspection de semi-conducteur l'utilisant
JP5537281B2 (ja) 2010-06-21 2014-07-02 オリンパス株式会社 顕微鏡装置および画像取得方法
US8388891B2 (en) * 2010-12-28 2013-03-05 Sakura Finetek U.S.A., Inc. Automated system and method of processing biological specimens
WO2012112697A2 (fr) * 2011-02-15 2012-08-23 The Johns Hopkins University Procédé et système destinés à numériser des échantillons de pathologie par étapes pour un examen
JP2013025466A (ja) * 2011-07-19 2013-02-04 Sony Corp 画像処理装置、画像処理システム及び画像処理プログラム
JP5732353B2 (ja) * 2011-08-31 2015-06-10 株式会社キーエンス 拡大観察装置、拡大観察方法および拡大観察プログラム
JP6455829B2 (ja) * 2013-04-01 2019-01-23 キヤノン株式会社 画像処理装置、画像処理方法、およびプログラム
EP3019905B1 (fr) * 2013-04-30 2020-06-03 Molecular Devices, LLC Appareil et procédé permettant de générer des images focalisées à l'aide d'une imagerie parallèle dans un système de microscopie
JP6127926B2 (ja) * 2013-11-11 2017-05-17 ソニー株式会社 画像処理装置及び画像処理方法
WO2015073897A2 (fr) * 2013-11-15 2015-05-21 Mikroscan Technologies, Inc. Scanner géologique
WO2016069794A1 (fr) 2014-10-28 2016-05-06 Mikroscan Technologies, Inc. Système de visualisation de microdissection
JP2016224190A (ja) * 2015-05-28 2016-12-28 オリンパス株式会社 顕微鏡装置、シェーディング補正装置、シェーディング補正方法、及び、プログラム
US11280803B2 (en) * 2016-11-22 2022-03-22 Sakura Finetek U.S.A., Inc. Slide management system
CA3112071A1 (fr) 2018-11-02 2020-05-07 Hologic, Inc. Systeme et procede d'imagerie numerique
JP7506524B2 (ja) 2020-05-20 2024-06-26 株式会社エビデント 顕微鏡システム、制御方法、プログラム
JP7506523B2 (ja) 2020-05-20 2024-06-26 株式会社エビデント 顕微鏡システム、制御方法、プログラム
WO2024176621A1 (fr) * 2023-02-24 2024-08-29 浜松ホトニクス株式会社 Dispositif d'acquisition d'image

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6049421A (en) * 1995-07-19 2000-04-11 Morphometrix Technologies Inc. Automated scanning of microscope slides
WO2001084209A2 (fr) * 2000-05-03 2001-11-08 Dirk Soenksen Numeriseur de lames de microscope rapide et entierement automatique
WO2003012518A2 (fr) * 2001-07-31 2003-02-13 Interscope Technologies, Inc. Systeme servant a creer des images microscopiques de montage numerique
US6816606B2 (en) * 2001-02-21 2004-11-09 Interscope Technologies, Inc. Method for maintaining high-quality focus during high-throughput, microscopic digital montage imaging
US20050105174A1 (en) * 2003-10-03 2005-05-19 Nikon Corporation Microscope system
US20050190437A1 (en) * 2002-09-10 2005-09-01 Olympus Optical Co., Ltd Microscopic image capture apparatus and microscopic image capturing method

Family Cites Families (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4367915A (en) * 1978-06-29 1983-01-11 Georges Michael P Automatic microscope slide
US6272235B1 (en) * 1997-03-03 2001-08-07 Bacus Research Laboratories, Inc. Method and apparatus for creating a virtual microscope slide
US6396941B1 (en) * 1996-08-23 2002-05-28 Bacus Research Laboratories, Inc. Method and apparatus for internet, intranet, and local viewing of virtual microscope slides
US6404906B2 (en) * 1997-03-03 2002-06-11 Bacus Research Laboratories,Inc. Method and apparatus for acquiring and reconstructing magnified specimen images from a computer-controlled microscope
US6847729B1 (en) * 1999-04-21 2005-01-25 Fairfield Imaging Limited Microscopy
GB2348002B (en) * 1999-07-20 2001-02-07 Genpak Ltd Microarrayer
JP2003248176A (ja) * 2001-12-19 2003-09-05 Olympus Optical Co Ltd 顕微鏡画像撮影装置
US7272252B2 (en) * 2002-06-12 2007-09-18 Clarient, Inc. Automated system for combining bright field and fluorescent microscopy
JP4544850B2 (ja) * 2002-11-29 2010-09-15 オリンパス株式会社 顕微鏡画像撮影装置
GB2398196B (en) * 2003-02-05 2005-06-01 Fairfield Imaging Ltd Microscope system and method

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6049421A (en) * 1995-07-19 2000-04-11 Morphometrix Technologies Inc. Automated scanning of microscope slides
WO2001084209A2 (fr) * 2000-05-03 2001-11-08 Dirk Soenksen Numeriseur de lames de microscope rapide et entierement automatique
US6816606B2 (en) * 2001-02-21 2004-11-09 Interscope Technologies, Inc. Method for maintaining high-quality focus during high-throughput, microscopic digital montage imaging
WO2003012518A2 (fr) * 2001-07-31 2003-02-13 Interscope Technologies, Inc. Systeme servant a creer des images microscopiques de montage numerique
US20050190437A1 (en) * 2002-09-10 2005-09-01 Olympus Optical Co., Ltd Microscopic image capture apparatus and microscopic image capturing method
US20050105174A1 (en) * 2003-10-03 2005-05-19 Nikon Corporation Microscope system

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2010049651A1 (fr) * 2008-10-31 2010-05-06 Novacyt Procedure de preparation d ' une image d ' analyse virtuelle traitee
FR2938064A1 (fr) * 2008-10-31 2010-05-07 Novacyt Procede de preparation d'une plaque d'analyse virtuelle traitee.
CN105092582A (zh) * 2015-08-07 2015-11-25 苏州合惠生物科技有限公司 一种用于全自动免疫组化的大视场镜检装置及检测方法
CN112136071A (zh) * 2018-02-26 2020-12-25 凯利博成像和诊断公司 用于对体外组织进行宏观和微观成像的系统和方法
EP3759540A4 (fr) * 2018-02-26 2022-03-16 Caliber Imaging & Diagnostics, Inc. Système et procédé d'imagerie macroscopique et microscopique de tissu ex-vivo
US11280990B2 (en) 2018-02-26 2022-03-22 Caliber Imaging & Diagnostics, Inc. System and method for macroscopic and microscopic imaging ex-vivo tissue
US11681133B2 (en) 2018-02-26 2023-06-20 Caliber Imaging & Diagnostics, Inc. System and method for macroscopic and microscopic imaging ex-vivo tissue
CN112136071B (zh) * 2018-02-26 2023-08-11 凯利博成像和诊断公司 用于对体外组织进行宏观和微观成像的系统和方法

Also Published As

Publication number Publication date
EP1830216B1 (fr) 2011-02-09
ES2361054T3 (es) 2011-06-13
DE602007012375D1 (de) 2011-03-24
US20070206096A1 (en) 2007-09-06
JP2007233093A (ja) 2007-09-13
JP4917329B2 (ja) 2012-04-18
ATE498147T1 (de) 2011-02-15
US8126250B2 (en) 2012-02-28

Similar Documents

Publication Publication Date Title
EP1830216B1 (fr) Appareil de capture d'images, procédé de capture d'images et programme de capture d'images
US7978898B2 (en) Image acquiring apparatus, image acquiring method, and image acquiring program
EP1830218B1 (fr) Appareil de capture d'images, procédé de capture d'images et programme de capture d'images
EP1691230B2 (fr) Dispositif photo-micrographique et procédé de contrôle
EP1764640A2 (fr) Microscope accélérateur pour imagerie multipoint
JP4973162B2 (ja) 画像処理装置、画像処理プログラム及び観察システム
EP4254037A2 (fr) Balayage de mise au point automatique en temps réel
US20050002587A1 (en) Microscope system
JP2008051773A (ja) 蛍光画像取得装置、及び蛍光画像取得方法
KR102135523B1 (ko) 촬영 장치 및 방법과 촬영 제어 프로그램
JP4878815B2 (ja) 顕微鏡装置
US20180081162A1 (en) Microscopy system, microscopy method, and computer-readable recording medium
JP5730696B2 (ja) 画像処理装置および画像表示システム
JP6830548B2 (ja) 顕微鏡装置及びプログラム
JP6864115B2 (ja) 顕微鏡装置及びプログラム
JP4822920B2 (ja) ３次元像構築方法および透過電子顕微鏡

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IS IT LI LT LU LV MC NL PL PT RO SE SI SK TR

AX Request for extension of the european patent

Extension state: AL BA HR MK YU

17P Request for examination filed

Effective date: 20080205

17Q First examination report despatched

Effective date: 20080306

AKX Designation fees paid

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IS IT LI LT LU LV MC NL PL PT RO SE SI SK TR

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

RIN1 Information on inventor provided before grant (corrected)

Inventor name: COOKE, JEREMY,C/O HAMAMATSU PHOTONICS UK LIMITED

Inventor name: INOUE, TAKAYUKI

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IS IT LI LT LU LV MC NL PL PT RO SE SI SK TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REF Corresponds to:

Ref document number: 602007012375

Country of ref document: DE

Date of ref document: 20110324

Kind code of ref document: P

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602007012375

Country of ref document: DE

Effective date: 20110324

REG Reference to a national code

Ref country code: CH

Ref legal event code: NV

Representative=s name: KIRKER & CIE S.A.

REG Reference to a national code

Ref country code: SE

Ref legal event code: TRGR

REG Reference to a national code

Ref country code: NL

Ref legal event code: T3

REG Reference to a national code

Ref country code: ES

Ref legal event code: FG2A

Ref document number: 2361054

Country of ref document: ES

Kind code of ref document: T3

Effective date: 20110613

LTIE Lt: invalidation of european patent or patent extension

Effective date: 20110209

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20110209

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20110510

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20110609

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20110209

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20110209

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20110209

Ref country code: BE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20110209

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20110509

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20110209

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20110209

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20110209

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20110228

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20110209

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20110209

REG Reference to a national code

Ref country code: IE

Ref legal event code: MM4A

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20110209

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20110209

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20110209

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

26N No opposition filed

Effective date: 20111110

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20110221

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602007012375

Country of ref document: DE

Effective date: 20111110

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20110221

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20110209

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20110209

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20110209

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: NL

Payment date: 20150110

Year of fee payment: 9

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: ES

Payment date: 20150113

Year of fee payment: 9

Ref country code: CH

Payment date: 20150213

Year of fee payment: 9

Ref country code: IT

Payment date: 20150218

Year of fee payment: 9

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 10

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20160229

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20160229

REG Reference to a national code

Ref country code: NL

Ref legal event code: MM

Effective date: 20160301

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IT

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20160221

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 11

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: NL

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20160301

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 12

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: ES

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20160222

REG Reference to a national code

Ref country code: ES

Ref legal event code: FD2A

Effective date: 20181207

P01 Opt-out of the competence of the unified patent court (upc) registered

Effective date: 20230517

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20231228

Year of fee payment: 18

Ref country code: GB

Payment date: 20240108

Year of fee payment: 18

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: SE

Payment date: 20240103

Year of fee payment: 18

Ref country code: FR

Payment date: 20240103

Year of fee payment: 18