EP3578955B1 - Imaging system - Google Patents

Imaging system Download PDF

Info

Publication number
EP3578955B1
EP3578955B1 EP18764384.6A EP18764384A EP3578955B1 EP 3578955 B1 EP3578955 B1 EP 3578955B1 EP 18764384 A EP18764384 A EP 18764384A EP 3578955 B1 EP3578955 B1 EP 3578955B1
Authority
EP
European Patent Office
Prior art keywords
imaging
light transmitting
container
tip
image
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
EP18764384.6A
Other languages
German (de)
English (en)
French (fr)
Other versions
EP3578955A1 (en
EP3578955A4 (en
Inventor
Takahiko KUMAGAI
Yohei IZUME
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Yamaha Motor Co Ltd
Original Assignee
Yamaha Motor Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Yamaha Motor Co Ltd filed Critical Yamaha Motor Co Ltd
Publication of EP3578955A1 publication Critical patent/EP3578955A1/en
Publication of EP3578955A4 publication Critical patent/EP3578955A4/en
Application granted granted Critical
Publication of EP3578955B1 publication Critical patent/EP3578955B1/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B21/00—Microscopes
    • G02B21/36—Microscopes arranged for photographic purposes or projection purposes or digital imaging or video purposes including associated control and data processing arrangements
    • G02B21/365—Control or image processing arrangements for digital or video microscopes
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B21/00—Microscopes
    • G02B21/36—Microscopes arranged for photographic purposes or projection purposes or digital imaging or video purposes including associated control and data processing arrangements
    • G02B21/365—Control or image processing arrangements for digital or video microscopes
    • G02B21/367—Control or image processing arrangements for digital or video microscopes providing an output produced by processing a plurality of individual source images, e.g. image tiling, montage, composite images, depth sectioning, image comparison
    • C—CHEMISTRY; METALLURGY
    • C12—BIOCHEMISTRY; BEER; SPIRITS; WINE; VINEGAR; MICROBIOLOGY; ENZYMOLOGY; MUTATION OR GENETIC ENGINEERING
    • C12M—APPARATUS FOR ENZYMOLOGY OR MICROBIOLOGY; APPARATUS FOR CULTURING MICROORGANISMS FOR PRODUCING BIOMASS, FOR GROWING CELLS OR FOR OBTAINING FERMENTATION OR METABOLIC PRODUCTS, i.e. BIOREACTORS OR FERMENTERS
    • C12M1/00—Apparatus for enzymology or microbiology
    • C12M1/34—Measuring or testing with condition measuring or sensing means, e.g. colony counters
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
    • G01B11/00—Measuring arrangements characterised by the use of optical techniques
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
    • G01B11/00—Measuring arrangements characterised by the use of optical techniques
    • G01B11/02—Measuring arrangements characterised by the use of optical techniques for measuring length, width or thickness
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/17—Systems in which incident light is modified in accordance with the properties of the material investigated
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N33/00—Investigating or analysing materials by specific methods not covered by groups G01N1/00 - G01N31/00
    • G01N33/48—Biological material, e.g. blood, urine; Haemocytometers
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B21/00—Microscopes
    • G02B21/24—Base structure
    • G02B21/241—Devices for focusing
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/67—Focus control based on electronic image sensor signals
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B21/00—Microscopes
    • G02B21/06—Means for illuminating specimens
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B6/00—Light guides; Structural details of arrangements comprising light guides and other optical elements, e.g. couplings
    • G02B6/10—Light guides; Structural details of arrangements comprising light guides and other optical elements, e.g. couplings of the optical waveguide type
    • G02B6/12—Light guides; Structural details of arrangements comprising light guides and other optical elements, e.g. couplings of the optical waveguide type of the integrated circuit kind
    • G02B6/12007—Light guides; Structural details of arrangements comprising light guides and other optical elements, e.g. couplings of the optical waveguide type of the integrated circuit kind forming wavelength selective elements, e.g. multiplexer, demultiplexer
    • G02B6/12009—Light guides; Structural details of arrangements comprising light guides and other optical elements, e.g. couplings of the optical waveguide type of the integrated circuit kind forming wavelength selective elements, e.g. multiplexer, demultiplexer comprising arrayed waveguide grating [AWG] devices, i.e. with a phased array of waveguides
    • G02B6/12026—Light guides; Structural details of arrangements comprising light guides and other optical elements, e.g. couplings of the optical waveguide type of the integrated circuit kind forming wavelength selective elements, e.g. multiplexer, demultiplexer comprising arrayed waveguide grating [AWG] devices, i.e. with a phased array of waveguides characterised by means for reducing the temperature dependence
    • G02B6/12028—Light guides; Structural details of arrangements comprising light guides and other optical elements, e.g. couplings of the optical waveguide type of the integrated circuit kind forming wavelength selective elements, e.g. multiplexer, demultiplexer comprising arrayed waveguide grating [AWG] devices, i.e. with a phased array of waveguides characterised by means for reducing the temperature dependence based on a combination of materials having a different refractive index temperature dependence, i.e. the materials are used for transmitting light
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B7/00—Mountings, adjusting means, or light-tight connections, for optical elements
    • G02B7/28—Systems for automatic generation of focusing signals
    • G02B7/36—Systems for automatic generation of focusing signals using image sharpness techniques, e.g. image processing techniques for generating autofocus signals
    • G02B7/38—Systems for automatic generation of focusing signals using image sharpness techniques, e.g. image processing techniques for generating autofocus signals measured at different points on the optical axis, e.g. focussing on two or more planes and comparing image data

Definitions

  • the present invention relates to an imaging system that images an imaging subject with a light transmitting member interposed on an imaging optical axis.
  • a cell or a cell aggregate
  • An operation is carried out in which, for example, cells are scattered on a dish having a lot of accommodation recesses and desired cells are selected and transferred to a microplate.
  • the dish carrying the cells are imaged by an imaging device and the cell are classified into usable cells and unusable cells and foreign matters using, for example, an imaging processing technique.
  • Patent Literature 1 discloses a technique of imaging a nozzle (corresponding to the tip) by an imaging device and determining a position of the tip end of the nozzle using an autofocus mechanism.
  • a light transmitting member is required to be interposed on an imaging optical axis of the imaging device, in some cases.
  • an image of the cells is acquired through the transparent container and the dish.
  • the transparent container is placed on a transparent base, the image of the cells is acquired through the base additionally.
  • the imaging through the base is also required for the imaging of the tip end opening of the tip, sometimes. In these cases, the position and size of the cells and the position of the tip end opening of the tip are not accurately determined because of the interposed light transmitting member such as the container and the base that causes refraction of optical path.
  • Patent Literature 2 describes a calibration method for a photographic device that photographs an object through a transparent body.
  • Patent Literature 3 describes a technique with which distortion of an image of a sample formed by injecting a fluid into a recess can be reduced easily and effectively.
  • Patent Literature 4 describes a defocus amount estimation method for an imaging apparatus that captures, using an image sensor, an image of a specimen formed by an imaging optical system.
  • Non Patent Literature 1 describes a method for correcting spherical aberrations induced by an unknown medium through determination of its refractive index and thickness.
  • Patent Literature 5 describes a computer-controlled automated analysing system based on a microscope controlled by a computer.
  • the microscope comprises a multi-spectral illumination source with a filter wheel to select predefined illumination wavelengths, and a camera with a lens. Due to chromatic aberration of the camera lens when using different wavelengths for illumination, the computer applies a correction by adjusting the focal distance of the lens for each wavelength by shifting the lens position.
  • Non Patent Literature 1 DANIEL IWANIUK ET AL: "Correcting spherical aberrations induced by an unknown medium through determination of its refractive index and thickness", OPTIC EXPRESS, vol. 19, no 20, 26 September 2011 (2011-09-26), pages 19407-2291, XP055659252
  • An object of the present invention is to provide an imaging system that can accurately determine a position and/or a size of an imaging subject even when the imaging subject is imaged with a light transmitting member interposed on an imaging optical axis.
  • One aspect of the present invention provides an imaging system according to claim 1.
  • an exemplified imaging subject for an imaging system is cells or cell aggregates (spheroid) derived from the living body.
  • Each cell aggregate derived from the living body is formed by several to hundreds of thousands of aggregated cells. Thus, the cell aggregates each have a different size.
  • Each cell aggregate formed by living cells has a substantially spherical shape; however, the cell aggregate sometimes has a distorted shape or has an uneven density when part of the cells forming the cell aggregate are denatured or become dead cells.
  • a cell transfer device that picks with a tip thereof a usable cell aggregate among a plurality of cell aggregates that have different shapes and are carried in a dish on a selecting stage and transfers the selected cells to a microplate.
  • a variety of processing is applied to the cell aggregate on the microplate, including observation, drug efficacy check, inspection, culturing etc.
  • the cell aggregates are included in and expressed as cells C, in general.
  • An imaging system is applied to the cell transfer device described above to determine height positions of the tip, dish and cell C or to determine the shape of the cell C based on image information. Exemplified application of an imaging system according to the present invention at each location/section in the cell transfer device will be described below.
  • the cell transfer device S0 includes a base 1 (light transmitting member) having a horizontal mounting surface (top surface), a cell transfer line 10 assembled on the top surface of the base 1, a camera unit 5 (imaging device) disposed below the base 1, and a head unit 61 which is disposed above the base 1 and to which a tip 6 for suctioning and discharging cells C is to be attached.
  • a base 1 light transmitting member
  • a cell transfer line 10 assembled on the top surface of the base 1
  • camera unit 5 imaging device
  • head unit 61 which is disposed above the base 1 and to which a tip 6 for suctioning and discharging cells C is to be attached.
  • the cell transfer device S0 may be provided with a plurality of units 5 and units 61.
  • the base 1 has predetermined rigidity and has a rectangular flat plate partly or wholly made of light transmitting material.
  • the base 1 is made of a glass plate. Forming the base 1 with the light transmitting material such as glass plate enables the camera unit 5 disposed below the base 1 to image each operation section on the cell transfer line 10 disposed on the top surface of the base 1 through the base 1.
  • the cell transfer line 10 includes a plurality of operation sections needed for carrying out a consecutive cell transfer process of suctioning a cell C from a single container by a tip 6 and transporting the suctioned cell C to another container to discharge the cell C from the tip 6.
  • the operation sections are arranged in a left-and-right direction with respect to the base 1 and mounted to the base 1.
  • the plurality of operation sections on the cell transfer line 10 includes a tip stock section 11, a tip calibration section 12, a selecting section 13, a transfer section 14, and a tip disposal section 15.
  • the camera unit 5 includes a lens part 51 and a camera body 52.
  • the lens part 51 is an objective lens used in an optical microscope and includes a lens group forming an optical image of a predetermined magnification and a lens barrel accommodating the lens group.
  • the camera body 52 includes an imaging element such as a CCD image sensor.
  • the lens part 51 forms an optical image of an imaging subject on a light receiving surface of the imaging element.
  • the camera unit 5 can move in a left-and-right direction below the base 1 along a guide rail 53 paralley extending in the left-and-right direction with respect to the base 1.
  • the lens part 51 can move in an up-and-down direction for an in-focus operation.
  • the head unit 61 includes a head body 62 and a plurality of heads 63 that are retained by the head body 62 and are capable of moving in an up-and-down direction with respect to the head body 62.
  • FIG. 1 exemplifies three heads 63 arranged in a single row, the heads 63 are not particularly limited in its number and arrangement.
  • the head unit 61 can move in a left-and-right direction (X direction) above the base 1 along the guide rail 64 parallely extending in the left-and-right direction with respect to the base 1. Note that the head unit 61 can also move in a direction (Y direction) perpendicular to the sheet of FIG. 1 , although not illustrated in FIG. 1 .
  • the head 63 is made of a hollow rod having an open lower end.
  • the tip 6 is mounted to the lower end of the head 63.
  • the tip 6 is made of a tapered tubular member and has a tip end opening 6t for suctioning and discharging a cell C (object).
  • a piston mechanism is mounted in the hollow part of the head 63 and the piston mechanism can operate to generate suctioning and discharging force at a lower end opening of the head 63.
  • the head body 62 incorporates a power unit for the piston mechanism, an elevation mechanism for moving the head 63 in the up-and-down direction, and a power unit of the elevation mechanism.
  • Suctioning and discharging force generated in the head 63 generates suctioning and discharging force in the tip end opening 6t of the tip 6 mounted to the head 63. Accordingly, the tip 6 suctions and discharges a cell C through the tip end opening 6t of the tip 6.
  • the tip stock section 11 is a part to store a lot of non-used tips 6. Disposed in the tip stock section 11 is a stock container 16 that retains tips 6 arranged in a standing and matrix posture. The tip 6 is retained in the stock container 16 such that an upper end opening of the tip 6 faces upward. Specifically, the tip 6 is retained in the stock container 16 so that it can be easily mounted to a lower end of the head 63 that moves in the up-and-down direction.
  • the tip calibration section 12 is a part for determining a position (XYZ coordinates) of the tip end opening 6t of the tip 6 mounted to the head 63.
  • the tip calibration section 12 is provided with an imaging pit 17 for imaging, by the camera unit 5, a tip 6 mounted to the head 63.
  • the position of the tip end opening 6t of the tip 6 in the XYZ coordinates is determined based on an image of the tip 6 and focal position information during imaging.
  • the position in the XYZ coordinates is determined to accurately determine a position of the tip end opening 6t of each tip 6 and accurately determine a transfer amount of the head unit 61 in the XY directions and a transfer amount of the head 63 in the Z direction during suctioning and discharging of the cells C.
  • the height position of the tip end opening 6t varies depending on a mounting error of the tip 6 with respect to the head 63, a dimensional error of the tip 6 itself, and an error in a driving system of the head 63.
  • Imaging of the tip 6 through the tip calibration section 12 determines a deviation of the tip end opening 6t, derived from the errors, with respect to a reference position and a correction value corresponding to the deviation is determined.
  • the selecting section 13 is a part to select cells C targeted to be transferred.
  • a selecting container 18 (light transmitting container) is disposed in the selecting section 13.
  • the selecting container 18 is a container from which cells C are transferred and which stores culture medium L and retains a cell selecting dish 2 (retaining part) in the state that the dish is immersed in the culture medium L.
  • the dish 2 is a plate to retain cells C and includes a plurality of retaining recesses 3 on a top surface thereof, the recesses each being capable of retaining a cell C individually.
  • the culture medium L can be appropriately selected depending on the type of cells C and is not limited to a particular type as long as the cells C are not deteriorated in their characteristics.
  • the selecting container 18 has a columnar or prismatic shape and has a rectangular upper surface opening 18H on an upper surface side thereof.
  • the upper surface opening 18H is an opening used for receiving cells C and for the tip 6 mounted to the head 63 to pick up a selected cell C.
  • the dish 2 is arranged below the upper surface opening 18H.
  • the selecting container 18 and the dish 2 used are made of light transmitting resin material or glass. This is for the camera unit 5 disposed below the selecting container 18 to be able to observe the cells C carried in the dish 2.
  • a plurality of cells C dispersed in the cell culture solution is injected into the selecting container 18 through a dispersing tip not shown.
  • the dispersing tip suctions cells C and cell culture solution from a tube that stores the cell culture solution containing a lot of cells C and retains them inside the dispersing tip. Then, the dispersing tip is moved to a location above the selecting container 18 and is accessed to an upper surface of the dish 2 through the upper surface opening 18H. Subsequently, a tip end opening of the dispersing tip is immersed into the culture medium L of the selecting container 18 to discharge the cells C, together with the cell culture solution, retained inside the tip.
  • the cell transfer device S0 includes the cell stock section in which the tube is disposed and a dispersing tip stock section storing a plurality of dispersing tips, which sections are excluded from FIG. 1 .
  • FIG. 2A is a plan view of the dish 2 and FIG. 2B is a cross section along line IIB-IIB in FIG. 2A .
  • the dish 2 includes a dish body 20 and the retaining recesses 3 formed in the dish body 20.
  • the dish body 20 is made of a flatty plate member having a predetermined thickness and has an upper surface 21 and a lower surface 22.
  • Each retaining recess 3 has an opening (opening 31), at a side of the upper surface 21, for receiving a cell C.
  • the dish 2 is immersed into the culture medium L inside the selecting container 18.
  • the dish body 20 is retained inside the selecting container 18 such that the upper surface 21 of the dish body 20 is immersed into the culture medium L inside the selecting container 18 while the lower surface 22 thereof is positioned with a distance with respect to a bottom plate of the selecting container 18 (see FIG. 1 ).
  • Each of the retaining recesses 3 has an opening 31, a bottom part 32, a tubular wall surface 33, a hole part 34 and a border part 35.
  • This embodiment shows an example in which square retaining recesses 3 are arranged in a matrix form in a plan view.
  • the opening 31 is a square opening provided to the upper surface 21 and has a size to allow entrance of the tip end opening 6t of the tip 6 for selection.
  • the bottom part 32 resides inside the dish body 20 and is located near the lower surface 22.
  • the bottom part 32 forms an inclined surface gently inclining downward toward the center (the center of the square).
  • the tubular wall surface 33 is a wall surface vertically extending downward from the opening 31 toward the bottom part 32.
  • the hole part 34 is a through hole vertically penetrating between the center of the bottom part 32 and the lower surface 22.
  • the border part 35 is located at the upper surface 21, is an area to become an opening edge of each retaining recess 3 and is a ridgeline partitioning the retaining recesses 3.
  • a storing space 3H for storing a cell C is partitioned by the bottom part 32 and the tubular wall surface 33 of each retaining recess 3.
  • the storing space 3H is contemplated to store a single number of cell C.
  • the hole part 34 is provided to discharge a small cell of undesired size or a foreign matter from the storing space 3H. Accordingly, the hole part 34 has a selected size through which a cell C of desired size cannot pass through while a small cell C of undesired size and a foreign matter can pass through.
  • a cell C to be a target for selection is trapped in the retaining recess 3 while a foreign matter or the like drops on the bottom plate of the selecting container 18 through the hole part 34.
  • the cell C retained in the retaining recess 3 is suctioned by the tip 6.
  • the tip end opening 6t of the tip 6 enters inside the retaining recess 3 and approaches closer to the cell C swiftly. Then, suction force generated in the tip end opening 6t causes the tip 6 to suction the cell C into an inside thereof.
  • Such operation of the tip 6 requires accurate determination of the height position of the dish 2 (retaining recess 3).
  • the dish 2 is retained in the selecting container 18 at a height position in accordance with a designed value, but a deviation in the height position can sometimes occur due to manufacturing error or the like. According to the present embodiment, a correction value corresponding to such deviation is determined as the selecting container 18 is imaged by the camera unit 5 in the selecting section 13 and a deviation of the height position of the dish 2 with respect to a reference position is determined from image information acquired.
  • the transfer section 14 is an area to transfer the cells C selected in the selecting section 13.
  • a microplate 4 is arranged in the transfer section 14.
  • the microplate 4 is a container in which cells C are transferred and includes a plurality of wells 41 for receiving the cells C.
  • the tip end opening 6t of the tip 6 in which a cells C is suctioned enters into each well 41 and discharges the cell C.
  • the microplate 4 used is also made of light transmitting resin material or glass. This is for the camera unit 5 disposed below the microplate 4 to be able to observe the cells C carried in the microplate 4.
  • Accurate determination of the height position of the bottom surface of the well 41 is desirable since the tip end opening 6t of the tip 6 enters into the well 41.
  • a container such as a petri dish may be arranged in the transfer section 14 in place of the microplate 4, in which case accurate determination of the height position of the bottom surface of the container is also desirable.
  • the camera unit 5 images the microplate 4 or petri dish to determine, from the acquired image information, a deviation between the height position of the bottom surface and the reference position to determine a correction value corresponding to the deviation.
  • the tip disposal section 15 is an area to collect tips 6, used to complete the suctioning and discharging operations, from the head 63.
  • the tip disposal section 15 includes a tip collecting container 19 for storing the used tips 6.
  • the head unit 61 mounted with the used tips 6 is transferred above the opening of the tip collecting container 19 and an operation for removing the tips 6 from the head 63 is carried out. With this removing operation, the tips 6 drop into the tip collecting container 19.
  • the basic steps of the cell transfer operation includes: mounting of the head 63 to the tip 6 (step 1); positional calibration of the tip end opening 6t of the tip 6 (step 2); picking up of a cell C from the selecting container 18 (dish 2) (step 3); transfer of the cell C to the microplate 4 (step 4); and disposal of the tip 6 (step 5).
  • the head unit 61 is traveled from the left to the right above each section of the cell transfer line 10 along the guide rail 64.
  • the camera unit 5 images the tip 6 mounted to the head 63 to determine the position of the tip end opening 6t in step 2, images the dish 2 to select usable cells C before step 3, and images the microplate 4 to determine the cells C that have been transferred after step 4. Additionally, the camera unit 5 images the dish 2 to determine the height position thereof and images the microplate 4 or petri dish to determine the height position of the bottom surface thereof, before carrying out steps 1 to 5.
  • step 1 the head unit 61 is transferred to a tip mounting position P11 above the tip stock section 11.
  • one head 63 is aligned with a corresponding tip 6 retained in the stock container 16 and is moved downward to fit an upper end part of the tip 6 to the lower end of the head 63. Then, the head 63 is moved upward. Same operation is carried out for the rest of the heads 63 for attachment of the tips 6.
  • the head unit 61 is transferred to a tip calibration position P12 above the tip calibration section 12.
  • the head unit 61 is stopped at a location where one head 63 newly mounted with the tip 6 is aligned with the vertical axis of the imaging pit 17.
  • the camera unit 5 is transferred to a tip imaging position P21 right below the imaging pit 17 of the tip calibration section 12. Then, the camera unit 5 images the tip 6 positioned above the imaging pit 17.
  • the position of the tip end opening 6t is determined by the contrast detection method, for example. Specifically, the tip 6 is imaged sequentially by the camera unit 5 initiating from a predetermined imaging starting point below the tip end opening 6t while the lens part 51 is shifted upward by several tens of microns.
  • the imaging terminal point is a predetermined location that can be determined as above the tip end opening 6t.
  • the height position of the tip end opening 6t is determined based on the height position of the lens part 51 in which a focal position is determined by an in-focus position where a taken image of a line supposed to be the tip end opening 6t has the highest contrast among the taken images.
  • step 3 the head unit 61 is transferred to a cell suction position P13 above the selecting section 13. Note that cell suspension containing cells C is distributed in the dish 2 of the selecting container 18 and the cells C are retained in the dish 2 before step 3 is carried out.
  • the camera unit 5 is transferred to a dish imaging position P22 below the selecting section 13 and images the dish 2 carrying the cells C.
  • a usable cell C is determined based on the image acquired and the coordinates of the retaining recess 3 retaining the usable cell C is specified.
  • a suction sequence where which cell C is to be suctioned with which head 63 (tip 6) and in what order is arranged.
  • a discharge sequence is also arranged in which from which head 63 (tip 6) to which well 41 of the microplate 4 is subjected to the discharge operation.
  • a tip 6 for the first suction is aligned with a retaining recess 3 of a dish 2 of suction target and the head 63 is moved downward with reference to the correction value acquired in step 2.
  • suction force is generated in the head 63.
  • the cell C carried in the retaining recess 3 of the target is suctioned into the tip 6.
  • the head 63 is moved upward.
  • same operation is carried out sequentially between the succeeding tips 6 and retaining recesses 3 for suction of the cells C into the respective tips 6.
  • step 4 the head unit 61 is transferred to a cell discharging position P14 of the transfer section 14, i.e., above the microplate 4.
  • the head unit 61 is stopped so that the tip 6 retaining the cell C is vertically aligned with the well 41 of the microplate 4 that is a discharge target.
  • the head 63 is moved downward until the tip end opening 6t of the tip 6 enters the opening of the well 41.
  • discharge force is generated in the head 63 and the cell C retained inside the tip 6 is discharged from the tip end opening 6t to the well 41.
  • the camera unit 5 is transferred to a microplate imaging position P23 below the transfer section 14.
  • the camera unit 5 captures an image of the microplate 4 carrying the cell C after completion of discharge of the cell C into the well 41. Accordingly, carrying condition of the cell C in the microplate 4 can be determined.
  • step 5 the head unit 61 is transferred to a tip disposal position P15 above the tip disposal section 15.
  • a tip collecting container 19 having an open upper surface is disposed in the tip disposal section 15.
  • the head 63 is moved downward with respect to the tip collecting container 19 and a tip removing rod (not shown) incorporated inside the head 63 is moved downward.
  • the downward movement of the rod pushes and presses the tip 6, so that the tip 6 is removed from the head 63.
  • the removed tip 6 drops inside the tip collecting container 19.
  • the imaging system S includes the camera unit 5 configured to image an imaging subject on an imaging optical axis AX, a controller 7 configured to control operation of the lens part 51 and carry out predetermined process based on image information acquired by the camera body 52, and a servomotor 54 moving the lens part 51 up and down.
  • FIG. 3 illustrates, as an imaging subject, a spherical imaging subject 8 placed on the light transmitting container 81 (light transmitting member).
  • the imaging subject 8 corresponds to a cell C retained in the dish 2 or the microplate 4, for example.
  • the imaging subject includes the tip end opening 6t of the tip 6, the dish 2, and the bottom surface of the container, etc.
  • the servomotor 54 rotates forward or reverse to move the lens part 51 in a up-and-down direction, with the predetermined resolution, via a power transmission mechanism not shown. With this movement, the focal position of the lens part 51 is adjusted to the position of the imaging subject 8. Note that as shown by the dotted line in FIG. 3 , the servomotor 54 may be used instead of the lens part 51 to move up and down the light transmitting container 81 or stage on which the light transmitting container 81 is placed.
  • the controller 7 is composed of a personal computer, for example, and operates a predetermined program to functionally include an imaging control part 71, an image processing part 72, a storing part 73, an input part 74, and a calculation unit 75.
  • the imaging control part 71 controls operation of the camera unit 5. Specifically, the imaging control part 71 controls an imaging operation (exposure amount and/or shutter timing) of the camera body 52. Furthermore, the imaging control part 71 provides a control pulse to cause the servomotor 54 to move the lens part 51 up and down by a predetermined pitch (several tens of ⁇ m pitch, for example).
  • the imaging control part 71 is given from the calculation unit 75 data relating to a regulated height position of the imaging subject 8, i.e., a predetermined position as a height position where the imaging subject 8 is supposed to be positioned. Designating the regulated height position as a target position, the imaging control part 71 instructs the servomotor 54 to cause the lens part 51 to move to the target position. Data relating to the position that the imaging control part 71 has instructed is input to the calculation unit 75. In addition, the imaging control part 71 causes the camera body 52 to capture an image of the imaging subject 8 for every movement pitch of the lens part 51.
  • the image processing part 72 performs predetermined image processing on image data of the imaging subject 8 acquired by the camera body 52.
  • the image processing part 72 performs image processing such as edge detection processing and pattern recognition processing involving feature amount extraction on the image data, etc., to provide processing to extract the shape of the imaging subject 8 from the image data.
  • the image processing part 72 determines contrast between pixels for each image data acquired for every movement pitch of the lens part 51 and specifies an image having greatest contrast as an in-focus image.
  • the height position data of the lens part 51 when the in-focus image is acquired is provided to the calculation unit 75.
  • the storing part 73 stores various data.
  • the stored data includes a reference height of the lens part 51 as a part of the data.
  • the reference height is a height position of a torso surface 55 of the lens part 51 determined on the basis of the regulated height position of the imaging subject 8 and a parfocal distance of the lens part 51.
  • the reference height is corrected corresponding to the focal extension amount when a light transmitting member such as the light transmitting container 81 is interposed on the imaging optical axis AX.
  • the storing part 73 stores the data relating to a change rate of the size as a part of the data.
  • the input part 74 receives from a user input of data relating to the reference height determined from the parfocal distance and the corrected reference height acquired from a theoretical value or actual measurement of the focal extension amount, and input of data relating to the change rate of the size.
  • the storing part 73 stores as a correction value the difference between the reference height (or the corrected reference height) and the actually measured height objected by actual imaging of the imaging subject 8 performed by the camera unit 5. Furthermore, when the tip end opening 6t of the tip 6 and the bottom surface of the dish 2 or the container is regarded as an imaging subject, correction values of these are also stored in the storing part 73. These correction values are applied to control movement of the head 63 (tip 6).
  • the calculation unit 75 determines data relating to the position and/or the size of the imaging subject 8 based on image information acquired from images captured by the camera unit 5. Specifically, the calculation unit 75 reads out the reference height data value from the storing part 73 when data relating to the position of the imaging subject 8 is to be determined. In addition, the calculation unit 75 acquires data relating to the actually measured height of the lens part 51 when an in-focus image is acquired from the image processing part 72. Then, a correction value about the height position of the imaging subject 8 is determined by the comparison between the reference height data and the data relating to the actually measured height. That is, a deviation amount between the theoretical height position of the imaging subject 8 and the actual height position is determined.
  • the calculation unit 75 acquires change information relating to a condition change in an imaging caused by the interposition of the light transmitting container 81 when the light transmitting container 81 is interposed on the imaging optical axis AX in the imaging.
  • the change information is a focal extension amount, for example, and in this case, the calculation unit 75 reads out from the storing part 73 the reference height data value corrected based on the focal extension amount. Then, the calculation unit 75 determines a correction value about the height position of the imaging subject 8 by comparing the corrected reference height data with the data relating to the actually measured height. As a result, it is contemplated that the height position data of the imaging subject 8 is corrected based on the change information (focal extension amount).
  • the calculation unit 75 calculates shape data indicating the sizes of X, Y, and Z of the imaging subject 8 based on an in-focus image selected by the image processing part 72.
  • the calculation unit 75 acquires change information relating to the change in the image of the imaging subject 8 caused by the interposition of the light transmitting container 81. Specifically, a change rate of the size of the imaging subject 8 stored in the storing part 73 in advance is read out.
  • This change rate is a change rate between the actual size of the imaging subject 8 and a size in the image of the imaging subject 8 acquired by the camera unit 5 with the light transmitting container 81 (light transmitting member) being interposed.
  • the calculation unit 75 corrects the shape data with reference to the change rate.
  • FIG. 4 is a view for describing the parfocal distance of the lens part 51.
  • a focal position of the lens part 51 which is an objective lens, configured to image minute subjects such as cells C, is determined by the parfocal distance A from the torso surface 55.
  • the distance between the imaging subject 8 and the torso surface 55 becomes the parfocal distance A when the imaging subject 8 is in an in-focus state. Accordingly, it becomes possible to measure the height position of the imaging subject 8 based on the height position of the torso surface 55 as a reference height.
  • FIG. 5 is a view for describing the focal extension amount ⁇ .
  • the focal position of the lens part 51 extends further when the light transmitting member 82 is interposed on the imaging optical axis AX of the lens part 51. That is, when the light transmitting member 82 is not interposed, a light beam g1 passed through a lens 56 of the lens part 51 forms an image on the focal position f1 defined by the shape of the lens. In contrast, when the light transmitting member 82 is interposed, an optical path refracts in accordance with the refractive index of the light transmitting member 82. Thus, the light beam g2 passing through the light transmitting member 82 forms an image on the focal position f2 that is further than the focal position f1.
  • the distance between the focal positions f1, f2 on the imaging optical axis AX is the focal extension amount a (change information).
  • the distance between the imaging subject 8 and the torso surface 55 becomes a distance obtained by adding the focal extension amount ⁇ to the parfocal distance A.
  • FIG. 6 is a view for describing a method for measuring the focal extension amount ⁇ when the light transmitting member 82 is interposed on the imaging optical axis AX.
  • the light transmitting member 82 interposed on the imaging optical axis AX is made of a single type of member, so that the focal extension amount ⁇ can be calculated based on the Snell's Law if the refractive index thereof is known.
  • the storing part 73 is used to store via the input part 74 the corrected reference height in which the reference height, which is the height position of the torso surface 55 in focusing the imaging subject 8, is added with the focal extension amount ⁇ determined by the calculation.
  • the light transmitting member 82 in FIG. 6 is an example that is a combined composition of different members a, b, c.
  • the selecting container 18 and the culture medium L are to be interposed on the imaging optical axis AX.
  • the lens part 51 takes an in-focus operation for a case in which the light transmitting member 82 is interposed on the imaging optical axis AX and for a case the member 82 is not interposed on the axis AX, while the imaging subject 8 is placed on the same height position for both cases. More specifically, first, the imaging control part 71 moves the lens part 51 to an in-focus position for a case the light transmitting member 82 is not interposed. That is, the imaging control part 71 moves the lens part 51 so that the height position of the torso surface 55 with respect to the imaging subject 8 becomes the parfocal distance A. The height position of the torso surface 55 after it is moved is determined as the reference height.
  • the parfocal distance A of the lens part 51 is generally known by the specification of the lens part 51, so that a movement amount in accordance with the specification is to be set.
  • the torso surface 55 can be set as the reference height by imaging with the light transmitting member 82 excluded from the imaging optical axis AX and by allowing the image processing part 72 to select the in-focus position using the contrast detection system.
  • the imaging control part 71 causes the camera unit 5 to image the imaging subject 8 for a plurality of times while a relative distance with respect to the imaging subject 8 is changed and while the light transmitting member 82 is interposed on the imaging optical axis AX.
  • the imaging control part 71 moves the lens part 51 gradually in the direction away from the imaging subject 8 at a predetermined pitch
  • the camera body 52 captures an image of the imaging subject 8 for every pitch.
  • the image processing part 72 selects the in-focus position using the contrast detection system.
  • the calculation unit 75 calculates the focal extension amount ⁇ for the imaging subject 8 based on the image information acquired by the plurality of imaging operations.
  • the calculation unit 75 calculates the focal extension amount ⁇ from the difference between the height position of the torso surface 55 with the light transmitting member 82 not being interposed and the height position of the torso surface 55 with the in-focus image being acquired while the light transmitting member 82 is interposed.
  • FIG. 7 is a view illustrating a first exemplified application of the imaging system S.
  • the first exemplified application demonstrates an example where the imaging subject is the tip 6 mounted to the head 63 and the height position of the tip end opening 6t of the tip 6 is measured.
  • the reference height II is determined by the height position of the torso surface 55 in the state that the lens part 51 is focused with respect to the tip end opening 6t of the tip 6 for test.
  • the imaging control part 71 controls the servomotor 54 to locate the torso surface 55 of the lens part 51 to be aligned with the reference height H.
  • the value of the reference height H is stored in the storing part 73.
  • the imaging system S is used to measure the focal extension amount ⁇ .
  • the camera unit 5 of the cell transfer device S0 of the present embodiment images the tip end opening 6t of the tip 6 via the transparent base 1 at the tip imaging position P21 right below the imaging pit 17. That is, the imaging control part 71 causes the camera unit 5 to image a zone including at least the tip end opening 6t of the tip 6 for a plurality of times while a vertical relative distance with respect to the test tip 6 is changed and while the base 1 is interposed in the imaging optical axis AX. Alternatively, a dummy tip or the like positioned at a regulated height may be imaged.
  • the imaging control part 71 causes the lens part 51 to move gradually in the direction away from the tip 6 at a predetermined pitch, and causes the camera body 52 to capture an image of the tip 6 for every pitch.
  • the image processing part 72 selects the in-focus position from the images of the tip 6 acquired in the pitches, using the contrast detection system.
  • the calculation unit 75 calculates the focal extension amount ⁇ from the difference between the reference height H and the height position of the torso surface 55 while the base 1 is interposed and the in-focus image is taken. In this case, (H - ⁇ ) is the corrected reference height position, i.e., the theoretical in-focus height.
  • the minus mark in (H - ⁇ ) means that the lens part 51 is moved downward as the focal position moves away.
  • the tip end opening 6t is in-focused at the corrected reference height position (H - ⁇ ) when the head 63 is moved downward with a regular movement amount while the tip of regular size is mounted to the head 63 regularly.
  • step S13 the value of (H - ⁇ ) acquired is stored in the storing part 73 through the input part 74 by the user.
  • the calculation unit 75 reads out the value of (H - ⁇ ) from the storing part 73 and provides the imaging control part 71 with the data as a movement target position for the lens part 51.
  • the head 63 mounted with the tip 6 to be measured is moved downward to a predetermined height position.
  • the imaging control part 71 performs servo control on the servomotor 54 based on the movement target position to move the torso surface 55 to the position of the theoretical in-focus height (H - ⁇ ) while the imaging optical axis AX of the lens part 51 is aligned with the tip 6 to be measured.
  • the actual height position of the tip end opening 6t is determined by the contrast detection system.
  • the imaging control part 71 causes the lens part 51 to move in the up-and-down direction at a predetermined pitch based on the position of (H - ⁇ ) and causes the camera body 52 to capture an image of the tip 6 for every pitch.
  • the image processing part 72 selects the in-focus position by the contrast detection system.
  • the height position of the torso surface 55 at the in-focus position is determined as the actually measured height h corresponding to the actual height position of the tip end opening 6t.
  • the calculation unit 75 determines the correction value for elevation control of the tip 6 (head 63) to be measured based on the actually measured height h and the theoretical in-focus height (H - ⁇ ).
  • steps S13, S14 are repeated and a correction value for each tip 6 is determined in the same manner.
  • the determined correction value is stored in the storing part 73 and referred to at the time of elevation control of the head 63.
  • FIG. 8 is a view illustrating a second exemplified application of the imaging system S.
  • the second exemplified application demonstrates an example where the imaging subject is the dish 2 retained in the selecting container 18 and the height position of the dish 2 is measured.
  • the reference height T is set to the upper surface of the base 1, on which the selecting container 18 is placed, at a location on which the lens part 51 is in-focused.
  • the imaging control part 71 controls the servomotor 54 to place the torso surface 55 of the lens part 51 to the reference height T.
  • the value of the reference height T is stored in the storing part 73.
  • the reference height T is a value that already includes a focal extension amount of the base 1.
  • the imaging system S is used to measure the focal extension amount ⁇ .
  • the selecting container 18 has an upper surface 18U (first surface) and a rear surface 18B (second surface) and the dish 2 (retaining part) is placed on the side of the upper surface 18U.
  • the camera unit 5 images the dish 2 from the side of the rear surface 18B of the selecting container 18 via the transparent base 1 at a dish imaging position P22 below the selecting section 13.
  • the reference height T determined in step S21 is the height of the upper surface of the base 1, i.e., the height of the rear surface 18B of the selecting container 18.
  • the dish 2 is located at a higher position apart from the rear surface 18B (bottom wall) of the selecting container 18.
  • the design value m of the location height of the dish 2 is acquired in advance and the reference height T in step S21 is corrected to (T + m).
  • the imaging control part 71 causes the camera unit 5 to image a zone including at least the dish 2 in the selecting container 18 for a plurality of times while the vertical relative distance with respect to the selecting container 18 is changed and while the base 1, the selecting container 18 and the culture medium L are interposed on the imaging optical axis AX.
  • the imaging control part 71 causes the lens part 51 to move gradually in the direction away from and closer to the selecting container 18 at a predetermined pitch and causes the camera body 52 to capture an image of the selecting container 18 for every pitch.
  • the image processing part 72 selects the in-focus position of the dish 2 from the images of the selecting container 18 acquired in the pitches, using the contrast detection system.
  • the dish 2 has a thickness in the vertical direction, so that an appropriate location is selected as the focal position.
  • the border part 35 of the dish 2 is a ridgeline sectioning the retaining recesses 3 and is readily detected as an edge on the image, and thus the border part 35 is desirably selected as the focal position.
  • the calculation unit 75 calculates the focal extension amount ⁇ from the difference between the reference height (T + m) and the height position of the torso surface 55 in a condition that the in-focus image is acquired while the selecting container 18 is interposed.
  • (T + m - ⁇ ) is the corrected reference height position, i.e., the theoretical in-focus height with respect to the dish 2.
  • step S23 the acquired value of (T + m - ⁇ ) is stored in the storing part 73 from the input part 74 by the user.
  • the calculation unit 75 reads out the value of (T + m - ⁇ ) from the storing part 73 and provides the imaging control part 71 with the data as a movement target position for the lens part 51.
  • the selecting container 18 retaining the dish 2 to be measured is placed on the base 1.
  • the imaging control part 71 performs servo control on the servomotor 54 based on the movement target position to move the torso surface 55 to the position of (T + m - ⁇ ) that is the theoretical in-focus height, in a condition that the imaging optical axis AX of the lens part 51 is aligned with the dish 2 to be measured.
  • the actual height position of the dish 2 is actually measured by the contrast detection system.
  • the imaging control part 71 causes the lens part 51 to move in the up-and-down direction based on the position of (T + m - ⁇ ) at a predetermined pitch and causes the camera body 52 to capture an image of the dish 2 immersed in the culture medium L in the selecting container 18 for every pitch.
  • the image processing part 72 selects the in-focus position by the contrast detection system.
  • the height position of the torso surface 55 in the in-focus position is determined as the actually measured height h that corresponds to the actual height position of the dish 2.
  • the calculation unit 75 determines a correction value to be used in elevation control of the head 63 with respect to the dish 2 based on the actually measured height h and the theoretical in-focus height (T + m - ⁇ ).
  • the determined correction value is stored in the storing part 73.
  • the dish 2 is a thin platy member having a predetermined size of vertical times horizontal.
  • the dish 2 can have tilt and/or warp or can have a thickness variation in the direction of the plate surface. Consequently, an accurate correction value for all of the retaining recesses 3 might not be determined by a mere actual measurement of the position of a single retaining recess 3 in step S24.
  • a border part 35 surrounding a single retaining recess 3 should not be determined as the focal position and desirably border parts 35 for a plurality of retaining recesses 3 are determined as the focal position.
  • the border parts 35 of the retaining recesses 3 at four corners of the dish 2 or the retaining recess 3 in the center of the dish 2 are actually measured in step S24 to determine the respective correction values.
  • the inclination trend of the dish 2 is acquired from the five correction values determined by the actual measurement and the respective correction values can be determined by complementary calculations.
  • FIG. 9 is a view showing a third exemplified application of the imaging system S.
  • the third exemplified application demonstrates an example where the imaging subject is the cell C retained in the light transmitting container 81 such as a petri dish and the height positions of the cell C is measured.
  • the reference height T is set to the upper surface of the light transmitting member 82 (or may be the base 1 in the cell transfer device SO), on which the light transmitting container 81 is placed, at a location on which the lens part 51 is in-focused.
  • the imaging control part 71 controls the servomotor 54 to place the torso surface 55 of the lens part 51 to the reference height T.
  • the value of the reference height T is stored in the storing part 73.
  • the reference height T is a value already including the focal extension amount of the light transmitting member 82.
  • the imaging system S is used to measure the focal extension amount ⁇ .
  • the light transmitting container 81 has an open upper surface and a bottom wall 81B.
  • the camera unit 5 images the cell C through the bottom wall 81B of the light transmitting container 81 via the light transmitting member 82 at a dish imaging position P22 below the selecting section 13.
  • the reference height T determined in step S31 is the height of the upper surface of the light transmitting member 82, i.e., the height of the bottom wall lower surface 812 of the light transmitting container 81.
  • the surface that the cell C makes a contact is the bottom wall upper surface 811 of the bottom wall 81B (retaining part), i.e., the bottom surface of the container.
  • the thickness of the bottom wall 81B, i.e., the height b of the bottom wall upper surface 811 is determined by a design value or actual measurement and the reference height T in step S31 is corrected to (T + b).
  • the imaging control part 71 causes the camera unit 5 to image a zone including the bottom wall 81B of the light transmitting container 81 for a plurality of times in the state that the vertical relative distance with respect to the light transmitting container 81 is changed while the light transmitting container 81 and the light transmitting member 82 are interposed on the imaging optical axis AX. Specifically, the imaging control part 71 causes the lens part 51 to move gradually in the direction away from and closer to the light transmitting container 81 at a predetermined pitch and causes the camera body 52 to capture an image of the light transmitting container 81 for every pitch.
  • the image processing part 72 selects the in-focus position of the bottom wall upper surface 811 from the images of the light transmitting container 81 acquired in the pitches, using the contrast detection system.
  • the calculation unit 75 calculates the focal extension amount ⁇ from the difference between the reference height (T + b) and the height position of the torso surface 55 in the state that the in-focus image of the bottom wall upper surface 811 is determined.
  • (T + b - ⁇ ) is the corrected reference height position, i.e., the theoretical in-focus height of the bottom wall upper surface 811, to which the cell C contacts.
  • step S33 the acquired value (T + b - ⁇ ) is stored in the storing part 73 from the input part 74 by the user.
  • the calculation unit 75 reads out the value (T + b - ⁇ ) from the storing part 73 and provides the imaging control part 71 with the data as a movement target position for the lens part 51. Thereafter, the light transmitting container 81 accommodating the cell C to be measured is placed on the light transmitting member 82.
  • the imaging control part 71 performs servo control on the servomotor 54 based on the movement target position to move the torso surface 55 to the position (T + b - ⁇ ) that is the theoretical in-focus height, while the imaging optical axis AX of the lens part 51 is aligned with the light transmitting container 81 to be measured.
  • step S34 the height position of each of the cells C accommodated in the light transmitting container 81 is measured by the contrast detection system.
  • the imaging control part 71 causes the lens part 51 to move in the up-and-down direction based on the position of (T + b - ⁇ ) at a predetermined pitch and causes the camera body 52 to capture an image of the cell C to be imaged in each pitch.
  • the image processing part 72 selects the in-focus position by the contrast detection system.
  • the height position of the torso surface 55 in the in-focus position is determined as the height h of the cell C to be imaged.
  • the calculation unit 75 determines the correction value of the height position of the cell C based on the actually measured height h and the theoretical in-focus height (T + b - ⁇ ). That is, the correction value indicating how much the cell C is positioned above the bottom wall upper surface 811 is calculated.
  • the determined correction value is stored in the storing part 73.
  • the height positions of the rest of the cells C accommodated in the light transmitting container 81 are determined by the same manner. These correction values are used to pick up the cells C from the light transmitting container 81, etc.
  • FIGs. 10 to 12 are views for describing a fourth exemplified application of the imaging system S, which does not fall within the scope of the claimed invention.
  • the fourth exemplified application demonstrates an example where the imaging subject to be imaged via a light transmitting member having an optical effect to deform the light image can be imaged so that its size is accurately determined. That is, the fourth exemplified application demonstrates an example where the calculation unit 75 determines data relating to the size of the imaging subject based on the image information acquired by the camera unit 5.
  • FIG. 10 is a view showing the imaging subjects 8A, 8B deformed on the image when the imaging subjects 8A, 8B are imaged through the light transmitting member having the optical effect described above.
  • the light transmitting member is made of a semispherical transparent curved member 83, as an example.
  • the transparent curved member 83 has a semispherical bottom wall included in the well 41, for example, and the upper surface (first surface) thereof is a retaining part to retain the imaging subjects 8A, 8B and the lower surface (second surface) thereof is a surface through which the camera unit 5 take images.
  • the imaging subjects 8A, 8B are each in contact with the upper surface side of the transparent curved member 83 in which the imaging subject 8A is in contact with the deepest part of the transparent curved member 83 and the imaging subject 8B is in contact with the side surface of the transparent curved member 83.
  • the transparent curved member 83 may have a cone or pyramid shape.
  • the transparent curved member 83 has a semispherical shape at a lower side thereof, so that it has a lens effect to magnify a light image.
  • the imaging subjects 8A, 8B are imaged as the imaging subjects 80A, 80B each having a magnified size when the imaging subjects 8A, 8B are imaged from the lower surface side of the transparent curved member 83.
  • the imaging subjects 8A, 8B may each have a magnified size of different aspect depending on the location on the surface of the transparent curved member 83 with which the subjects 8A, 8B are in contact.
  • the imaging subject 8A positioned on the deepest part of the transparent curved member 83 forms an isotropic size change since that part is the center of the surface having a lens effect
  • the imaging subject 8B forms an anisotropy size change since it is located on the position near the end edge of the surface having the lens effect.
  • the imaging subjects 8A, 8B are imaged as the imaging subjects 80A, 80B with the size on the image deformed from the actual sizes when the imaging subjects 8A, 8B are imaged via the transparent curved member 83 having the lens effect.
  • data having an accurate size of the imaging subjects 8A, 8B can be acquired in such a manner that the aspect of size change, i.e., the change rate of the size (change information), is measured and acquired in advance and the image data of the imaging subjects 80A, 80B acquired by the imaging is corrected based on the change information.
  • FIG. 11 is a view showing the size change on the image for each retaining positions k1, k2, k3 when the imaging subjects 8 are imaged through the transparent curved member 83.
  • the aspect of the size change can differ depending on the retaining location on the transparent curved member 83 where the imaging subjects 8 are positioned, so that it is desirable to measure the size change rate with the imaging subjects 8 located in different positions.
  • the imaging subjects 8 are located on the respective retaining positions k1, k2, k3 on the upper surface side of the transparent curved member 83 and are imaged from the lower surface side of the transparent curved member 83 to acquire the imaging subjects 8k1, 8k2, 8k3 on the respective positions, from which what size change has formed in each subject can be desirably determined.
  • FIG. 12 is a view for describing the method to acquire the change rate of the size of the imaging subject 8.
  • the imaging control part 71 causes the camera unit 5 to image at least a zone where a test subject 8C with a known size is retained on the transparent curved member 83 with a predetermined distance with respect to the transparent curved member 83 while the test subject 8C is retained by the transparent curved member 83 on the imaging optical axis.
  • the sizes Rx and Ry of the test subject 8C in the x and y directions are known.
  • the sizes rx and ry in the x and y directions of the test subject 8CA imaged via the transparent curved member 83 deform with respect to Rx and Ry depending on the refractive index and surface shape of the transparent curved member 83.
  • the size change rate can be determined swiftly and accurately, and an additional cost for acquiring the size change rate can be avoided as the imaging system S is used.
  • the calculation unit 75 acquires the change information such as the focal extension amount ⁇ and corrects the data relating to the position and/or the size of the imaging subject when an imaging subject such as a cell C is imaged via the light transmitting member.
  • the change information such as the focal extension amount ⁇
  • the focal extension condition change in the imaging
  • the first to third exemplified applications demonstrate examples to determine the focal extension amount ⁇ as the change information while the fourth exemplified application demonstrates an example to determine the size change rate. These may be determined in a combined manner, which, however, is not covered by the scope of the appended claims. For example, in the first to third exemplified applications, the size change rate demonstrated in the fourth exemplified application may additionally be determined.
  • the embodiment specifically described above mainly includes the invention having the following configurations/features.
  • One aspect of the present invention provides an imaging system including the features defined by the independent claim.
  • the change information is a focal extension amount determined by a focal position of the imaging device with respect to the imaging subject when the light transmitting member is not interposed and a focal position of the imaging device with respect to the imaging subject when the light transmitting member is interposed.
  • the focal distance extends by the refraction of light passing through the light transmitting member.
  • the imaging system can correct the data accurately as it handles the focal extension amount as the change information.
  • the focal extension amount may be actually measured using the imaging system or other imaging devices or it may be calculated based on the physical property such as the refractive index of the light transmitting member.
  • the change information is a change rate between an actual size of the imaging subject and a size of the imaging subject on an image acquired by the imaging device when the light transmitting member is interposed.
  • the light transmitting member has a curved surface
  • a light image of the imaging subject passed through the light transmitting member is observed in a deformed shape with respect to the actual shape thereof.
  • the imaging system can correct the data accurately as the change rate of size is handled as the change information. Note that the change rate may be actually measured using the imaging system or other imaging devices or it may be calculated based on the curved surface and the like of the light transmitting member.
  • the imaging system further includes an imaging control part configured to control operation of the imaging device.
  • the imaging control part causes the imaging device to image the imaging subject for a plurality of times while a relative distance with respect to the imaging subject is changed in a condition that the light transmitting member is interposed on the imaging optical axis, and the calculation unit determines the focal extension amount for the imaging subject based on image information acquired by the imaging of a plurality of times.
  • the focal extension amount can be determined with swiftness and accuracy, and additionally an extra cost for determining the focal extension amount can be avoided.
  • the imaging system further includes an imaging control part configured to control operation of the imaging device.
  • the imaging subject is a tip end opening of a tip that suctions and discharges an object
  • the imaging device is configured to image the tip end opening via the light transmitting member
  • the imaging control part causes the imaging device to image a zone including at least the tip end opening of the tip for a plurality of times while a relative distance with respect to the tip is changed in a condition that the light transmitting member is interposed on the imaging optical axis
  • the calculation unit determines the focal extension amount for the tip end opening based on image information acquired by the imaging of a plurality of times.
  • the focal extension amount can be easily determined with the in-focus image of the tip end opening of the tip identified from the image information acquired by the imaging of a plurality of times with the light transmitting member being interposed and with the comparison between the focal distance when the in-focus image is acquired and the focal distance when the light transmitting member is not interposed.
  • the imaging system further includes an imaging control part configured to control operation of the imaging device.
  • the light transmitting member has a first surface and a second surface opposite the first surface and is a light transmitting container including in the first surface side a retaining part for retaining the imaging subject, and the imaging device is configured to image the imaging subject from the second surface side of the container, the imaging control part causes the imaging device to image a zone including at least the retaining part of the container for a plurality of times while a relative distance with respect to the container is changed on the imaging optical axis, and the calculation unit determines the focal extension amount for the retaining part based on image information acquired by the imaging of a plurality of times.
  • the focal extension amount can be easily determined from the image information acquired by the imaging of a plurality of times, in which the in-focus image of the retaining part passed through the container identified as the light transmitting member is acquired and the focal distance of the acquired in-focus image is compared with a focal distance in a state that the container is not interposed.
  • the light transmitting member includes, in addition to the container, a light transmitting base on which the container is mounted, and the imaging device is configured to image the container through the base.
  • the focal extension amount of complexity caused by the container and the base interposed on the imaging optical axis can be easily determined by the actual measurement using the imaging device.
  • the retaining part may be supported apart from a bottom wall of the container, and the light transmitting member further includes light transmitting fluid injected into the container so that the retaining part is immersed.
  • the focal extension amount can be easily determined by the actual measurement using the imaging device, in which the focal extension amount has complexity caused by, on the imaging optical axis, the interposition of two objects, which are the container and the light transmitting fluid, or the interposition of three objects, which are the container, base and the light transmitting fluid.
  • the imaging system further includes an imaging control part configured to control operation of the imaging device.
  • the light transmitting member has a first surface and a second surface opposite the first surface and is a light transmitting container including in the first surface side a retaining part for retaining the imaging subject, the retaining part having an optical effect to deform a light image
  • the imaging device is configured to image the imaging subject from the second surface side of the container
  • the imaging control part causes the imaging device to image a zone including at least the retaining part of the container with a predetermined distance on the imaging optical axis with respect to the container that includes the retaining part in which a test subject of known size is retained, and the calculation unit determines the change rate by a comparison between a size of the test subject determined from an image acquired by the imaging and the known size.
  • the imaging control part causes the imaging device to carry out the imaging in which the test subject in the retaining part is arranged to have different retaining positions, and the calculation unit determines the change rate for each of the retaining positions.
  • the container and the retaining part may have various change rates depending on the retaining positions and their shapes.
  • the imaging system determines the change rate for each of the retaining positions, which allows the correction corresponding to each of the retaining positions.
  • the imaging system of the present invention in which the imaging subject or the object is a cell or a cell aggregate.
  • the present invention described above can provide an imaging system that can determine the position in direction of the imaging optical axis (AX) and the size of the imaging subject with accuracy even when the imaging subject is imaged through the light transmitting member interposed on the imaging optical axis.
  • AX imaging optical axis

Landscapes

  • Physics & Mathematics (AREA)
  • Chemical & Material Sciences (AREA)
  • Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Analytical Chemistry (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Multimedia (AREA)
  • Optics & Photonics (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • General Health & Medical Sciences (AREA)
  • Biochemistry (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Wood Science & Technology (AREA)
  • Organic Chemistry (AREA)
  • Pathology (AREA)
  • Biomedical Technology (AREA)
  • Zoology (AREA)
  • Medicinal Chemistry (AREA)
  • Immunology (AREA)
  • Biotechnology (AREA)
  • Molecular Biology (AREA)
  • Sustainable Development (AREA)
  • Hematology (AREA)
  • Microbiology (AREA)
  • Urology & Nephrology (AREA)
  • Food Science & Technology (AREA)
  • General Engineering & Computer Science (AREA)
  • Genetics & Genomics (AREA)
  • Signal Processing (AREA)
  • Microscoopes, Condenser (AREA)
  • Length Measuring Devices By Optical Means (AREA)
  • Investigating Or Analysing Materials By Optical Means (AREA)
  • Investigating Or Analysing Biological Materials (AREA)
  • Apparatus Associated With Microorganisms And Enzymes (AREA)
EP18764384.6A 2017-03-10 2018-02-02 Imaging system Active EP3578955B1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2017046409 2017-03-10
PCT/JP2018/003705 WO2018163683A1 (ja) 2017-03-10 2018-02-02 撮像システム

Publications (3)

Publication Number Publication Date
EP3578955A1 EP3578955A1 (en) 2019-12-11
EP3578955A4 EP3578955A4 (en) 2020-03-04
EP3578955B1 true EP3578955B1 (en) 2022-04-06

Family

ID=63448644

Family Applications (1)

Application Number Title Priority Date Filing Date
EP18764384.6A Active EP3578955B1 (en) 2017-03-10 2018-02-02 Imaging system

Country Status (5)

Country Link
US (1) US11525995B2 (ja)
EP (1) EP3578955B1 (ja)
JP (1) JP6793244B2 (ja)
CN (1) CN110383042B (ja)
WO (1) WO2018163683A1 (ja)

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111655834B (zh) * 2018-01-29 2024-03-12 雅马哈发动机株式会社 生体对象物处理装置
JP7145051B2 (ja) * 2018-11-21 2022-09-30 株式会社エビデント 倒立型顕微鏡
KR102623341B1 (ko) * 2021-11-30 2024-01-11 연세대학교 산학협력단 스페로이드 관찰장치 및 관찰 시스템
WO2023233827A1 (ja) * 2022-06-03 2023-12-07 富士フイルム株式会社 位相画像取得方法及び定量データ取得方法

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2010126903A1 (en) * 2009-04-27 2010-11-04 Constitution Medical Investors, Inc. Systems and methods for analyzing body fluids

Family Cites Families (31)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH07333668A (ja) * 1994-06-02 1995-12-22 Canon Inc カメラシステム
KR100576526B1 (ko) 1997-01-09 2007-07-09 후지 덴키 가부시끼가이샤 거리 측정 장치
JP3223863B2 (ja) * 1997-01-09 2001-10-29 富士電機株式会社 測距装置
US6067147A (en) 1997-01-09 2000-05-23 Fuji Electric Co., Ltd. Distance-measuring apparatus
JP4396813B2 (ja) * 2003-06-30 2010-01-13 コニカミノルタオプト株式会社 撮像ユニット
JP4578814B2 (ja) * 2004-01-26 2010-11-10 晴夫 高林 標的対象物の自動探索回収装置
IL167193A (en) * 2004-03-18 2009-11-18 Elbit Systems Ltd Method and system for determining optical distortion in a transparent medium
WO2006051813A1 (ja) * 2004-11-09 2006-05-18 Hitachi Medical Corporation 細胞培養装置、画像処理装置及び細胞検出システム
JP5307539B2 (ja) * 2006-05-31 2013-10-02 オリンパス株式会社 生体試料撮像方法および生体試料撮像装置
JP2008311709A (ja) * 2007-06-12 2008-12-25 Nikon Corp 撮像装置
JP2009162974A (ja) * 2008-01-07 2009-07-23 Olympus Corp 顕微鏡システム及びその制御方法
CN102713713B (zh) * 2009-10-13 2016-01-27 佳能株式会社 焦点调节设备和焦点调节方法
JP5644447B2 (ja) 2010-12-06 2014-12-24 ソニー株式会社 顕微鏡、領域判定方法、及びプログラム
JP5762315B2 (ja) * 2012-01-19 2015-08-12 株式会社Ｓｃｒｅｅｎホールディングス 画像処理方法
JP2013170861A (ja) * 2012-02-20 2013-09-02 Dainippon Screen Mfg Co Ltd 撮像装置、試料保持プレートおよび撮像方法
JP6140930B2 (ja) * 2012-03-22 2017-06-07 キヤノン株式会社 撮像装置およびその制御方法、撮影システム
JP6016463B2 (ja) * 2012-06-07 2016-10-26 キヤノン株式会社 デフォーカス量推定方法、撮像装置、および透光性部材
JP5996979B2 (ja) * 2012-09-07 2016-09-21 ヤマハ発動機株式会社 電子部品実装装置および実装位置補正データ作成方法
CA2893590C (en) * 2012-12-14 2023-10-10 The J. David Gladstone Institutes Automated robotic microscopy systems
JP5709829B2 (ja) * 2012-12-17 2015-04-30 キヤノン株式会社 外部装置、外部装置の制御方法、撮像システムおよび撮像システムの制御方法
CA2961223C (en) 2013-12-12 2019-04-30 Yamaha Hatsudoki Kabushiki Kaisha A moving apparatus for moving a subject using tips
JP6506908B2 (ja) * 2014-02-24 2019-04-24 オリンパス株式会社 合焦方法、計測方法、合焦装置、及び計測装置
JP6427900B2 (ja) * 2014-03-07 2018-11-28 株式会社リコー 校正方法、校正システム、プログラム及び移動体
JP6066110B2 (ja) * 2014-06-11 2017-01-25 横河電機株式会社 細胞吸引支援システム
US10429403B2 (en) 2014-06-17 2019-10-01 Yamaha Hatsudoki Kabushiki Kaisha Head device for mounting dispensing tip thereon, and movement device using same
EP3159395B1 (en) 2014-06-17 2018-05-30 Yamaha Hatsudoki Kabushiki Kaisha Cylinder tip mounting head, and head device and movement device each using same
EP3159396B1 (en) 2014-06-17 2019-12-18 Yamaha Hatsudoki Kabushiki Kaisha Object moving device
JP6345001B2 (ja) * 2014-07-01 2018-06-20 株式会社Ｓｃｒｅｅｎホールディングス 画像処理方法および画像処理装置
US10789679B2 (en) * 2016-07-13 2020-09-29 SCREEN Holdings Co., Ltd. Image processing method, image processor, image capturing device, and image capturing method for generating omnifocal image
WO2018047218A1 (ja) * 2016-09-06 2018-03-15 オリンパス株式会社 観察装置
CN110300883B (zh) * 2016-11-29 2022-05-10 光热光谱股份有限公司 用于增强光热成像和光谱的方法和设备

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2010126903A1 (en) * 2009-04-27 2010-11-04 Constitution Medical Investors, Inc. Systems and methods for analyzing body fluids

Also Published As

Publication number Publication date
JP6793244B2 (ja) 2020-12-02
EP3578955A1 (en) 2019-12-11
US20200326525A1 (en) 2020-10-15
EP3578955A4 (en) 2020-03-04
WO2018163683A1 (ja) 2018-09-13
US11525995B2 (en) 2022-12-13
JPWO2018163683A1 (ja) 2020-01-09
CN110383042B (zh) 2023-02-03
CN110383042A (zh) 2019-10-25

Similar Documents

Publication Publication Date Title
EP3578955B1 (en) Imaging system
EP1761817B1 (en) Methods, software, and apparatus for focusing an optical system using computer image analysis
US7769219B2 (en) Method for assessing image focus quality
JP4831972B2 (ja) マイクロマニピュレーションシステム
EP1564575B1 (en) Micromanipulation system operating within the field of view of a microscope
US11572537B2 (en) Cell handling device
JP5826250B2 (ja) マイクロ操作ツールを無衝突で位置決めする方法
CN114002840A (zh) 显微镜和用于确定到样品参考平面的距离的方法
KR102619093B1 (ko) 혈액진단장치
KR20210100181A (ko) 광 현미경검사를 위한 기준 초점 평면을 식별하기 위한 인쇄 커버슬립 및 슬라이드
CN111655836A (zh) 生体对象物的移动方法以及移动装置
JP6694796B2 (ja) 細胞移動装置
JP2018063187A (ja) 容器構造体及びこれを用いた撮像システム
CN112322713B (zh) 成像方法、装置及系统及存储介质
EP3739036B1 (en) Biological subject transfer device
JP2009282357A (ja) 観察装置及び観察方法
JP2020183988A (ja) 微小検体ピッキングシステム及び微小検体ピッキングプログラム
US10591501B2 (en) Automatic structure determination
KR20220038577A (ko) 물리적 캘리브레이션 슬라이드
CN114442296B (zh) 一种用于显微镜的采图方法及医学检测装置、存储介质
EP4253519A1 (en) Cell moving device
CN217638692U (zh) 一种dPCR阅片系统
JP2021083408A (ja) 検査方法および検査装置
JP2024516261A (ja) 液浸液を用いた顕微鏡システムおよび方法
CN118158524A (zh) 成像方法、装置及系统

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20190905

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

A4 Supplementary search report drawn up and despatched

Effective date: 20200204

RIC1 Information provided on ipc code assigned before grant

Ipc: G02B 21/36 20060101ALI20200129BHEP

Ipc: G01B 11/00 20060101ALI20200129BHEP

Ipc: G01N 21/17 20060101AFI20200129BHEP

Ipc: G01B 11/02 20060101ALI20200129BHEP

Ipc: C12M 1/34 20060101ALI20200129BHEP

Ipc: G02B 21/32 20060101ALI20200129BHEP

Ipc: G01N 33/48 20060101ALI20200129BHEP

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

17Q First examination report despatched

Effective date: 20201009

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

RIC1 Information provided on ipc code assigned before grant

Ipc: G02B 21/24 20060101ALI20211005BHEP

Ipc: G02B 21/06 20060101ALI20211005BHEP

Ipc: G02B 21/32 20060101ALI20211005BHEP

Ipc: G02B 21/36 20060101ALI20211005BHEP

Ipc: G01N 33/48 20060101ALI20211005BHEP

Ipc: G01B 11/02 20060101ALI20211005BHEP

Ipc: G01B 11/00 20060101ALI20211005BHEP

Ipc: C12M 1/34 20060101ALI20211005BHEP

Ipc: G01N 21/17 20060101AFI20211005BHEP

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTG Intention to grant announced

Effective date: 20211117

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: SE

Ref legal event code: TRGR

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: AT

Ref legal event code: REF

Ref document number: 1481782

Country of ref document: AT

Kind code of ref document: T

Effective date: 20220415

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602018033445

Country of ref document: DE

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG9D

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20220406

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 1481782

Country of ref document: AT

Kind code of ref document: T

Effective date: 20220406

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220406

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220808

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220706

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220406

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220406

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220707

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220406

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220406

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220706

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220406

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220406

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220406

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220406

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220806

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602018033445

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220406

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220406

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220406

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220406

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220406

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220406

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

26N No opposition filed

Effective date: 20230110

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: AL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220406

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: CH

Payment date: 20230307

Year of fee payment: 6

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220406

P01 Opt-out of the competence of the unified patent court (upc) registered

Effective date: 20230527

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220406

REG Reference to a national code

Ref country code: SE

Ref legal event code: EUG

REG Reference to a national code

Ref country code: BE

Ref legal event code: MM

Effective date: 20230228

GBPC Gb: european patent ceased through non-payment of renewal fee

Effective date: 20230202

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20230203

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20230202

REG Reference to a national code

Ref country code: IE

Ref legal event code: MM4A

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GB

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20230202

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220406

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20230202

Ref country code: GB

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20230202

Ref country code: FR

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20230228

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20230228

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20240219

Year of fee payment: 7