JP2011017620A - Shape measuring method, image processing program, and observation device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a shape measuring method having a constitution capable of measuring a three-dimensional shape of a biosample with high accuracy, and to provide an image processing program and an observation device.SOLUTION: In this shape measuring method, two kinds of images are acquired, namely, a plurality of first images acquired by photographing a fertilized ovum sequentially from one side by a first image capturing apparatus, while changing an optical distance between the fertilized ovum which is a biosample and a first observation optical system for forming an image of the fertilized ovum, and a plurality of second images acquired by photographing the fertilized ovum sequentially from the other side by a second image capturing apparatus, while changing an optical distance between the fertilized ovum and a second observation optical system for forming the image of the fertilized ovum; and a focusing measure is calculated in a pixel unit based on the plurality of first images and second images; and a focusing point of each pixel is determined by applying preferentially focusing measure information acquired from the first images relative to a domain on one side of the fertilized ovum, and by applying preferentially focusing measure information acquired from the second images relative to a domain on the other side of the fertilized ovum; and a three-dimensional shape of the fertilized ovum is organized based on the focusing point position.

Description

  The present invention relates to a shape measuring method, an image processing program, and an observation apparatus having a configuration capable of constructing a three-dimensional shape of a biological sample such as a fertilized egg.

  In recent years, with the development of assisted reproduction technology (ART), it has been practiced to observe the growth state of cultured fertilized eggs by in vitro fertilization. A culture microscope is mentioned as an example of the apparatus which observes the conditions of cultures, such as a fertilized egg (for example, refer patent document 1). The culture microscope includes a culture apparatus (invecutor) that forms a suitable environment for culturing fertilized eggs, and a microscopic observation system that microscopically observes the state of the fertilized eggs in the culture container housed in the culture apparatus. An observation image of a fertilized egg is acquired at regular intervals, and the growth state of the fertilized egg can be automatically observed, recorded, managed, and the like.

  An example of a microscopic observation system that can be observed alive without staining a substantially colorless and transparent biological sample such as a fertilized egg is a phase contrast microscope. In a phase contrast microscope, a biological sample is illuminated with illumination light limited in a ring shape with an aperture stop, the light that has passed through the biological sample is collected by an objective lens, the phase is converted, and the amount of transmitted light is It is well known that a phase ring to be adjusted is provided at a position conjugate with the aperture stop (see, for example, Patent Document 2). The light passing through the biological sample is converted into direct light and diffracted light other than the 0th order to produce a phase difference. By visualizing this phase difference as the brightness of the image, the biological sample can be observed with an increased contrast. ing.

JP 2004-229619 A Japanese Patent No. 3663920

  Now, as one method for evaluating the quality (growth state) of a fertilized egg, it is preferable that all cells in the egg of the fertilized egg undergo cleavage (cell division) at the same timing. In early cleavage, there are few cells overlapping in the observation direction, so the cleavage timing can be easily observed, but as the cleavage proceeds, the number of cells doubles and the cells overlap in the observation direction ( Because of the occurrence of occlusion), it has been difficult to quantitatively evaluate whether or not all the cells in the egg have cracked at the same time in the two-dimensional observation with the phase contrast microscope as described above. As a method for observing a three-dimensional shape, there is a confocal microscopic observation method using autofluorescence, but this observation method takes time for observation (a scan time is long for acquiring one image). ) Above, it is not suitable for observing fertilized eggs because it has a harmful effect on cells.

  The present invention has been made in view of such a problem, and provides a shape measuring method, an image processing program, and an observation apparatus having a configuration capable of measuring a three-dimensional shape of a biological sample with high accuracy. Objective.

  According to the first aspect of the present invention, the biological sample is sequentially moved from one side by the first imaging device while changing the optical distance between the biological sample and the first observation optical system that forms an image of the biological sample. While changing the optical distance between the plurality of first images taken and the second observation optical system that forms an image of the biological sample and the biological sample, the second imaging device causes the biological sample to be opposed to the one side from the other side. A plurality of second images taken sequentially are acquired, a focus measure is calculated in pixel units based on the plurality of first images and the second image, and an area on one side of the biological sample is obtained from the first image. The focus measurement information obtained is preferentially applied, and the focus measure information obtained from the second image is preferentially applied to the other region of the biological sample to obtain the focal point of each pixel. Shape measurement method characterized by constructing 3D shape of sample There is provided.

  According to a second aspect exemplifying the present invention, there is provided an image processing program for causing a computer to function as an image processing device that can be read by a computer and that is captured by an imaging device and acquires an image and performs image processing. A plurality of first images obtained by sequentially photographing the biological sample from one side by the first imaging device while changing an optical distance between the biological sample and the first observation optical system that forms an image of the biological sample; A plurality of second images obtained by sequentially capturing the biological sample from the other side facing one side are acquired by the second imaging device while changing the optical distance from the second observation optical system that forms an image of the biological sample. A step, a step of calculating a focus measure in units of pixels based on the plurality of first images and second images, and a focus measure information obtained from the first image for a region on one side of the biological sample. Applying in advance the focus measurement information obtained from the second image to obtain the focal point of each pixel for the region on the other side of the biological sample, and the tertiary of the biological sample based on the focal position An image processing program is provided that causes a computer to implement the step of constructing an original shape.

  According to a third aspect illustrating the present invention, a sample stage for holding a biological sample, a first imaging device that images a biological sample from one side, and a second imaging device that images from the other side facing the one side, The sample stage is moved relative to the first observation optical system that forms an image of the biological sample on the imaging surface of the first imaging device, and the second observation optical system that forms an image on the imaging surface of the second imaging device. A biological movement sample obtained by imaging with the first and second imaging devices while moving the sample stage relative to the first and second observation optical systems by the movement control unit that moves relative to the first and second observation optical systems. An image processing device that performs image processing on a plurality of first images and second images, and the image treatment device calculates a focus measure in units of pixels based on the plurality of first images and second images, The region on one side of the biological sample is obtained from the first image The focus measure information is preferentially applied, and the focus measure information obtained from the second image is preferentially applied to the other region of the biological sample to obtain the focal point of each pixel, and the biological sample is obtained based on the focus position. An observation apparatus is provided that includes an image analysis unit that constructs a three-dimensional shape.

  According to the present invention, a three-dimensional shape of a biological sample can be constructed with high accuracy.

It is a general | schematic block diagram of the culture observation system shown as an example of application of this invention. It is a block diagram of the said culture observation system. (A) is a top view of a culture container, (B) is a perspective view which shows a dish. It is a figure for demonstrating the process of cleavage of a fertilized egg. It is a figure which shows the structural example of a phase-contrast microscope apparatus. It is a figure for demonstrating the phase difference observation by a 1st phase contrast microscope. It is a figure for demonstrating the phase difference observation by a 2nd phase contrast microscope. 1 is a block diagram illustrating a schematic configuration of an image processing apparatus. It is a flowchart which shows the outline | summary of the shape measuring method. It is a flowchart for demonstrating the parameter acquisition method. It is a figure for demonstrating the conventional SFF system. It is a flowchart for demonstrating the SFF process of this embodiment. It is a figure for demonstrating the focal point calculation by SFF process. It is a figure which shows the modification of a structure of a phase-contrast microscope apparatus.

  Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings. As an example of a system to which an observation unit (observation apparatus) according to this embodiment is applied, a schematic configuration diagram and a block diagram of a culture observation system are shown in FIGS. 1 and 2, respectively.

  The culture observation system BS roughly observes the culture chamber 2 provided in the upper part of the housing 1, the shelf-like stocker 3 that accommodates and holds a plurality of culture containers 10, and the sample in the culture container 10. An observation unit (observation device) 5, a transport unit 4 for transporting the culture vessel 10 between the stocker 3 and the observation unit 5, a control unit 6 for comprehensively controlling the operation of the system, and an image display device. The operation panel 7 and the like.

The culture chamber 2 is a chamber for forming a culture environment, and is kept sealed after the sample is charged in order to prevent environmental changes and contamination. Along with the culture chamber 2, a temperature adjustment device 21 that raises and lowers the temperature in the culture chamber 2, a humidifier 22 that adjusts humidity, and a gas supply device 23 that supplies a gas such as CO ₂ gas or N ₂ gas. A circulation fan 24 for making the entire environment of the culture chamber 2 uniform, an environmental sensor 25 for detecting the temperature, humidity, carbon dioxide concentration, etc. of the culture chamber 2 are provided. The operation of each device is controlled by the control unit 6, and the culture environment defined by the temperature, humidity, carbon dioxide concentration, etc. of the culture chamber 2 is maintained in a state that matches the culture conditions set on the operation panel 7.

  The stocker 3 is formed in a shelf shape that is partitioned into a plurality of parts in the front-rear direction and the up-down direction in FIG. A unique address is set for each shelf. For example, when the front-rear direction is A to C rows and the vertical direction is 1 to 7 rows, the A-row 5 rows shelf is set as A-5. The

  As the culture vessel 10, an appropriate one such as a flask, a dish, or a well plate is selected according to the type and purpose of the culture. In this embodiment, as shown in FIG. The structure provided with the dish 10a and the holder 10b holding the dish 10a is illustrated, and as shown in FIG. 3 (B), the fertilized egg J as a culture is a medium containing a pH indicator such as phenol red. It is injected into each dish 10a together with the drop D. On the bottom surface of the dish 10a, one or more medium drops D of about 20 μl dropped by a pipette or the like are formed (only one is shown in FIG. 3B), and the medium drop D is contained in the dish 10a. It is in a state immersed in colorless and transparent mineral oil O. In each medium drop D, for example, one fertilized egg J collected at the same time from the same mother for external fertilization is inserted. The culture vessel 10 is assigned a code number and is stored in association with the designated address of the stocker 3.

  The transfer unit 4 includes a Z stage 41 that is provided inside the culture chamber 2 and can move up and down, a Y stage 42 that can move back and forth, an X stage 43 that can move left and right, and the culture container on the tip side of the X stage 43. A support arm 45 that lifts and supports 10 is provided. The transport unit 4 has a moving range in which the support arm 45 can move between the entire shelf of the stocker 3 and the observation unit 5.

  The observation unit (observation apparatus) 5 includes a backlight illumination unit 51 that illuminates the sample on the sample stage 15, a transmission illumination unit 52 that illuminates the sample on the sample stage 15 along the optical axis of the microscopic observation system, and macro observation of the sample. A macro observation system 54 for performing the micro observation of the sample, an image processing apparatus 100, and the like are included. The sample stage 15 is made of a light-transmitting material and has a transparent window 16 in the observation area. The sample stage 15 is composed of a fine drive stage that can be moved in the XY direction (horizontal plane direction) and the Z direction (vertical direction) by operation control from the control unit 6, and the culture vessel 10 placed on the upper surface thereof. Is moved in the XY direction so that the culture vessel 10 is inserted on the optical axis of the macro observation system 54 or the microscopic observation system 55 or by moving the culture vessel 10 in the Z direction. It is possible to change the relative position (optical distance) between the observation optical system and the sample in the optical axis direction.

  The backlight illumination unit 51 includes a surface-emitting light source provided on the lower frame 1b side, and backlight-illuminates the entire culture vessel 10 from the lower side of the sample stage 15. The transmitted illumination unit 52 includes light sources 81 and 91 (see FIG. 5) such as LEDs, and illumination optical systems 82a and 92a (see FIG. 5) composed of a phase ring, a condenser lens, and the like. And provided in the lower frame 1b, and illuminates the sample in the culture vessel 10 along the optical axis of the microscopic observation system 55.

  The macro observation system 54 includes an observation optical system 54 a and an imaging device 54 c such as a CCD camera that takes an image of the sample imaged by the observation optical system 54 a and is positioned above the backlight illumination unit 51. Provided in the culture chamber 2. The macro observation system 54 captures an entire observation image (macro image) from above the culture vessel 10 illuminated by the backlight illumination unit 51.

  The microscopic observation system 55 includes observation optical systems 82b and 92b (see FIG. 5) including an objective lens, a phase ring, and the like, and a cooled CCD camera that captures an image of the sample imaged by the observation optical systems 82b and 92b. Imaging cameras 89 and 99 (see FIG. 5) are provided in the culture chamber 2 and the lower frame 1b, respectively. The transmission illumination unit 52 and the microscopic observation system 55 constitute a phase contrast microscope device 56. The microscope observation system 55 captures a microscope observation image (micro image) obtained by observing the transmitted light that is illuminated by the transmission illumination unit 52 and transmitted through the sample.

  The image processing apparatus 100 performs A / D conversion on signals input from the imaging device 54c of the macro observation system 54 and the imaging cameras 89 and 99 of the micro observation system 55, and performs various image processing to perform an entire observation image or microscopic observation. Image data of an observation image is generated. Further, the image processing apparatus 100 performs image analysis on the image data of these observation images (entire observation image and microscopic observation image), and constructs a three-dimensional shape of the fertilized egg J. Specifically, the image processing apparatus 100 is constructed by executing an image processing program stored in the ROM of the control unit 6 described below. The image processing apparatus 100 will be described in detail later.

  The control unit 6 includes a CPU 61 that executes processing, a ROM 62 that stores and stores control programs and control data of the culture observation system BS, a RAM 63 that temporarily stores observation conditions and image data, and the like. Control the operation. Therefore, as shown in FIG. 2, the constituent devices of the culture chamber 2, the transport unit 4, the observation unit 5, and the operation panel 7 are connected to the control unit 6. In the RAM 63, the environmental conditions of the culture chamber 2 according to the observation program, the observation schedule, the observation type and observation position in the observation unit 5, the observation magnification, and the like are set and stored. Further, the RAM 63 is provided with an image data storage area for recording image data photographed by the observation unit 5, and index data including the code number of the culture vessel 10 and photographing date / time are associated with the image data and recorded. Is done.

  The operation panel 7 is provided with an operation panel 71 provided with input / output devices such as a keyboard and a switch, and a display panel 72 for displaying an operation screen, image data, and the like. An operation command or the like is input. The communication unit 65 is configured in accordance with a wired or wireless communication standard, and data can be transmitted to and received from a computer or the like externally connected to the communication unit 65.

  In the culture observation system BS schematically configured as described above, the CPU 61 controls the operation of each part based on the control program stored in the ROM 62 according to the setting conditions of the observation program set on the operation panel 7, and the culture vessel 10 The sample inside is automatically captured. That is, when the observation program is started by a panel operation on the operation panel 71 (or a remote operation via the communication unit 65), the CPU 61 reads each condition value of the environmental conditions stored in the RAM 63, and from the environment sensor 25. The environmental state of the culture chamber 2 to be input is detected, and the temperature adjustment device 21, the humidifier 22, the gas supply device 23, the circulation fan 24, etc. are operated according to the difference between the condition value and the actual measurement value. Feedback control is performed on the culture environment such as temperature, humidity, and carbon dioxide concentration.

  Further, the CPU 61 reads the observation conditions stored in the RAM 63 and operates the X, Y, Z stages 41, 42, 43 of the transport unit 4 based on the observation schedule to observe the culture vessel 10 to be observed from the stocker 3. The sample is transported to the sample stage 15 of the unit 5 and observation by the observation unit 5 is started. For example, when the observation set in the observation program is macro observation, the culture vessel 10 transported from the stocker 3 by the transport unit 4 is positioned on the optical axis of the macro observation system 54 and placed on the sample stage 15. Then, the light source of the backlight illumination unit 51 is turned on, and the entire observation image is taken by the imaging device 54c from above the culture vessel 10 illuminated by the backlight. The signal input from the imaging device 54c to the control unit 6 is processed by the image processing device 100 to generate a whole observation image, and the image data is stored in the image data storage area of the RAM 63 together with index data such as the shooting date and time. Is done.

  In addition, when the observation set in the observation program is micro observation of the sample at a specific position in the culture vessel 10, the specific position in the culture vessel 10 that has been transported by the transport unit 4 is indicated by the light of the microscopic observation system 55. The sample is positioned on the axis and placed on the sample stage 15, the light source of the transmission illumination unit 52 is turned on, and the microscopic observation images by the transmission illumination are taken by the imaging cameras 89 and 99. Signals photographed by the imaging cameras 89 and 99 and input to the control unit 6 are processed by the image processing apparatus 100 to generate a microscopic observation image (phase difference image), and the image data is index data such as photographing date and time. And the like are stored in the image data storage area of the RAM 63.

  The CPU 61 sequentially executes the above-described whole observation image photographing and microscopic observation image photographing based on the observation schedule set in the observation program. The image data stored in the RAM 63 is read from the RAM 63 in response to an image display command input from the operation panel 71. For example, an entire observation image, a microscopic observation image at a specified time, an analysis result of image analysis, and the like are displayed on the display panel 72. Is displayed.

  Here, the time-dependent change of the cleavage of the fertilized egg to be observed (cell division of the fertilized egg) will be described with reference to FIG. FIG. 4 is a diagram schematically showing the state of cleavage of a human fertilized egg. In the figure, examples up to the third generation are illustrated, and time elapses from left to right. And If the embryo H of the fertilized egg J before cleavage is the 0th generation, the surface of the 0th generation embryo H is constricted and divided into two, and two cells C and C of the first generation are generated. . Further, each cell C of the first generation is divided into two to generate four cells C, C,... Of the second generation, and the number of cells C increases twice as it proceeds to the third generation. In this way, the number of cells C doubles as the cleavage occurs, and the size of each cell C itself decreases.

  A method for evaluating the quality of the fertilized egg J is performed based on whether or not cleavage occurs in almost all cells C in the egg at substantially the same timing. That is, regarding the cleavage of the normal fertilized egg J, the cells C of the same generation divide at almost the same timing, and only the cells C of the same generation exist in the embryo H. On the other hand, regarding the cleavage of the fertilized egg J that is abnormal, even when the cells C are of the same generation, the timing of division is shifted, and cells C of different generations are mixed in the embryo H.

  Thus, in order to evaluate the growth state of the fertilized egg J, it is necessary to appropriately observe the timing of cleavage of the fertilized egg J. Since the number of divided cells is relatively small with respect to the initial cleavage, the cell C hardly overlaps in a predetermined observation direction, and the timing of cleavage can be observed, but the number of cells increases as the cleavage proceeds. Therefore, the cell C may overlap with the observation direction or the constriction of the cell C may be hidden (so-called occlusion occurs). Therefore, there is a problem that it is difficult to perform a quantitative evaluation by appropriately observing the timing of cleavage for each cell.

  In order to correct such problems, the culture observation system BS includes two phase contrast microscopes in which the phase difference microscope 56 (the transmission illumination unit 52 and the microscope observation system 55) of the observation unit 5 are arranged substantially symmetrically. Then, the fertilized egg J can be observed from two directions, and a three-dimensional shape of the fertilized egg J is constructed by performing image analysis by the image processing apparatus 100 on the observation images from the two directions. It is comprised so that the cleavage of can be observed appropriately.

  Now, the configuration of the phase-contrast microscope apparatus 56 in the observation unit 5 will be described with reference to FIGS. In the observation unit 5 described above, the phase contrast microscope device 56 including the transmission illumination unit 52 and the microscopic observation system 55 includes two microscopes, a first phase contrast microscope 80 and a second phase contrast microscope 90. These phase-contrast microscopes 80 and 90 are arranged substantially symmetrically with respect to the sample surface of the sample stage 15.

  The first phase contrast microscope 80 includes a light source 81 and an illumination optical system 82a provided in the culture chamber 2 in the transmission illumination unit 52, and an observation optical system 82b and an imaging camera provided in the lower frame 1b in the microscopic observation system 55. 89.

  The illumination optical system 82a includes a collector lens 83, an annular diaphragm 84, a movable folding mirror 85, and a condenser / objective lens 86 in order from the light source side. The observation optical system 82b includes a condenser / objective lens 96, a phase ring 87, and an imaging lens 88 in this order from the light source side.

  On the other hand, the second phase-contrast microscope 90 includes a light source 91 and an illumination optical system 92a provided in the lower frame 1b in the transmission illumination unit 52, and an observation optical system and 92b provided in the culture chamber 2 in the microscope observation unit 55. And an imaging camera 99.

  The illumination optical system 92a includes a collector lens 93, an annular diaphragm 94, a movable folding mirror 95, and a condenser / objective lens 96 in order from the light source side. The observation optical system 92b includes a condenser / objective lens 86, a phase ring 97, and an imaging lens 98 in order from the light source side.

  In such phase difference microscopes 80 and 90, the optical path from one movable folding mirror 85 to the other movable folding mirror 95 is common. As the light sources 81 and 91 of the phase contrast microscopes 80 and 90, for example, an LED, a halogen lamp, a high-pressure mercury lamp, or the like is used. The condenser / objective lens 86 functions as a condenser lens in the illumination optical system 82a, and functions as an objective lens in the observation optical system 92b. Similarly, the condenser / objective lens 96 functions as a condenser lens in the illumination optical system 92 and functions as an objective lens in the observation optical system 82b. Further, in each of the phase contrast microscopes 80 and 90, a mirror driving unit 57 (see FIG. 2) that moves the movable folding mirrors 85 and 95 on the optical axes of the illumination optical systems 82a and 92a and the observation optical systems 82b and 92b, respectively. Is provided). The mirror drive unit 57 is driven in accordance with a drive signal from the control unit 6.

  In the above configuration, a beam splitter may be used instead of the movable folding mirrors 85 and 95. In this case, the mirror drive unit 57 that moves forward and backward to the illumination optical systems 82a and 92a and the observation optical systems 82b and 92b can be omitted, but instead, a compensation optical system is provided immediately before the phase rings 87 and 97. It is necessary to insert.

  In the first phase contrast microscope 80 configured as described above, as shown in FIG. 6, the illumination light emitted from the light source 81 is converted into a substantially parallel light beam by the collector lens 83, and has an annular shape (ring shape). The light enters the annular zone stop 84 provided with a slit. The light that has passed through the slit of the annular diaphragm 84 is reflected substantially vertically downward (so that it can be bent) by the movable folding mirror 85, then condensed by the condenser / objective lens 86, and placed on the sample stage 15. The sample (fertilized egg J) in the culture vessel 10 is irradiated.

  The illumination light applied to the sample is divided into direct light that passes through the sample and diffracted light that is diffracted by the sample that is a phase object. Since this diffraction phenomenon occurs at a part where the refractive index is different, the diffracted light uses the shape information of the phase object such as the boundary part between the phase object (fertilized egg J) and the medium drop D and the internal structure of the phase object. The phase is delayed by ¼ wavelength with respect to the illumination light due to this diffraction. These direct light and diffracted light are condensed on the condenser / objective lens 96 and are incident on the phase ring 87 arranged at a position conjugate with the annular zone stop 84.

  Direct light out of the light incident on the phase ring 87 is incident on the phase plate 87a formed in the same shape (ring shape) as the slit of the annular diaphragm 84, and a part of the light is absorbed by the phase plate 87a. Thus, the amount of light is weakened and the phase is shifted by a quarter wavelength of the illumination light. On the other hand, of the light incident on the phase ring 87, the diffracted light passes through the portion other than the phase plate 87a as it is. The direct light and diffracted light that have passed through the phase ring 87 are imaged by the imaging lens 88 and interfere on the imaging surface of the imaging camera 89. At this time, since the phase of the direct light is shifted by ¼ wavelength by the phase plate 87a, the phase difference between the direct light and the diffracted light is ½ wavelength or 0, and the phase change is caused by the interference. It is visualized as light brightness. As a result, an enlarged image of the sample (fertilized egg J) having a contrast of light and dark is formed on the imaging surface of the imaging camera 89 and is captured by the imaging camera 89. When observing with the first phase-contrast microscope 80, the movable folding mirror 95 of the second phase-contrast microscope 90 is retracted from the optical path of the observation optical system 82b by the mirror driving unit 57.

  At this time, as described above, the sample stage 15 on which the culture vessel 10 is placed can reciprocate in the direction (Z direction) along the optical axis direction of the observation optical system 82b, and the relative relationship between the sample and the observation optical system 82b. As the position (optical distance) changes, the focal position of the optical system changes continuously. On the other hand, since the imaging camera 89 continuously performs imaging from below the sample, a plurality of images whose contrast changes according to the reciprocating movement of the sample stage 15 are displayed at a predetermined imaging pitch (the size of the fertilized egg to be observed). Accordingly, an appropriate pitch is selected. For example, it is possible to take an image at a micron pitch).

  Although the operation of the optical system of the first phase-contrast microscope 80 has been roughly described above, the second phase-contrast microscope 90 is substantially the same (see FIG. 7), so the explanation of the second phase-contrast microscope 90 is as follows. Although omitted, also in the second phase contrast microscope 90, the imaging camera 99 continuously images the sample from above while reciprocating the sample stage 15 in the direction along the optical axis direction of the observation optical system 92b. Thus, it is possible to capture a plurality of images whose contrast changes according to the reciprocation of the sample stage 15 at a predetermined imaging pitch.

  As described above, in the phase contrast microscopes 80 and 90, the observation images of the sample are continuously captured by the imaging cameras 89 and 99 from the two upper and lower directions while changing the relative position between the sample and the observation optical system (focal position of the optical system). Can be taken. Note that simultaneous observation by the first phase contrast microscope 80 and the second phase contrast microscope 90 is not performed, and the light sources 81 and 91 are switched on / off, and images are taken at every predetermined imaging pitch (one pitch feed of the sample stage 15). It is desirable to take images alternately with the cameras 89 and 99. This is because if a transparent sample such as the fertilized egg J is observed simultaneously from above and below with the first and second phase contrast microscopes 80 and 90, the phase contrast may be lowered by the illumination light of the opposing optical system. In order to perform observation by the SFF method, which will be described later, it is preferable that the phase-contrast microscopes 80 and 90 are sequentially switched for observation.

  The imaging cameras 89 and 99 have imaging elements such as a CCD and a CMOS, and output to the image processing apparatus 100 an image signal obtained by capturing an enlarged image of the sample formed on the imaging surface of the imaging element. To do. In addition to the image signals from the imaging cameras 88 and 89, the image processing apparatus 100 receives detection signals from a linear encoder 17 (see FIG. 8) built in the sample stage (fine driving stage) 15. The Therefore, the optical axis direction position (Z direction position) of the sample stage 15 when the surface of the sample is imaged can be known.

  In the present embodiment, the image processing apparatus 100 can obtain the three-dimensional shape of the sample by performing predetermined image processing on a plurality of images on the sample surface input from the imaging cameras 89 and 99. . Here, FIG. 8 shows a schematic configuration of the image processing apparatus 100 as a block diagram. The image processing apparatus 100 includes an image storage unit 110 that acquires and stores a plurality of microscopic observation images obtained by imaging the fertilized eggs J to be observed at a predetermined imaging pitch (sampling interval) by the imaging cameras 89 and 99, and a storage unit 110 in advance. An image correction unit 120 that obtains parameters for image correction based on the images taken by the imaging cameras 89 and 99 and corrects the visual field position and the magnification difference on the observed image of the fertilized egg acquired from the imaging cameras 89 and 99. And image analysis for constructing a three-dimensional shape of a fertilized egg by performing predetermined image processing based on an observation image (microscopic observation image) acquired and acquired and position information of the sample stage 15 when the acquisition and acquisition are performed. Unit 130 and an output unit 140 for outputting the result analyzed by the image analysis unit 130 to the outside, the three-dimensional shape data of the fertilized egg constructed by the image analysis unit 130 For example configured to display and output to the display panel 72. The image processing apparatus 100 is configured such that an image processing program GP preset and stored in the ROM 62 is read by the CPU 61 and processing based on the image processing program GP is sequentially executed by the CPU 61.

  Now, a shape measurement method for obtaining the three-dimensional shape of the fertilized egg J will be described with reference to the flowchart shown in FIG. As described, in the culture observation system BS, fertilized eggs in the culture vessel 10 designated every predetermined time are observed according to the observation conditions set in the observation program.

  In step S1, the image processing apparatus 100 adjusts the observation field of view and the observation magnification difference between the first phase contrast microscope 80 and the second phase contrast microscope 90 (matching the observation images of the phase contrast microscopes 80 and 90). Obtain image correction parameters. Here, details of the parameter acquisition processing will be described with reference to the flowchart of FIG.

  In step S11, a predetermined marker having a phase difference is placed on the sample stage 15 in advance, and the image correction unit 120 causes the imaging camera 89 of the first phase contrast microscope 80 to image the marker from below, The imaging camera 99 of the second phase-contrast microscope 90 is caused to pick up an image of the marker from the upper side, and a marker image is obtained from two directions. As the marker, a transparent bead made of a high-precision sphere serving as a model of an egg cell, a phase grating (a vertically standing phase film) having a known pattern, or the like is used.

  In step S12, the image correction unit 120 compares the two acquired images to calculate the shift amount of the observation visual field position between the first phase-contrast microscope 80 and the second phase-contrast microscope 90. The observation magnification difference between images is calculated. In step S14, an image correction parameter for correcting the observation visual field position deviation and the observation magnification difference in the first phase-contrast microscope 80 and the second phase-contrast microscope 90 is obtained based on the obtained calculated value. The image correction parameters acquired in this way are used for image correction processing performed in step S4 described later.

  Returning to FIG. 9, in step S <b> 2, after the marker is removed from the sample stage 15, the CPU 61 operates each stage of the transport unit 4 to transport the culture container 10 to be observed from the stocker 3 to the observation unit 5. (Positioned on the optical axis of the microscopic observation system 55), and under the control of the image acquisition unit 110, predetermined phase images (observation images) of fertilized eggs by the first and second phase contrast microscopes 80 and 90 Images are taken by the imaging cameras 89 and 99 at a pitch. At this time, in the observation with the first and second phase contrast microscopes 80 and 90, both are alternately switched at predetermined imaging pitches along with the vertical movement of the sample stage 15, and the observation images are taken by the imaging cameras 89 and 99. To do. As a result, a plurality of images with no change in the position or the like of the fertilized egg can be taken in a short time, so that it becomes possible to satisfactorily perform analysis by the SFF method described later.

  In step S <b> 3, the image storage unit 110 acquires a plurality of observation images photographed by the imaging cameras 89 and 99. At this time, the image storage unit 110 continuously captures the fertilized eggs in the culture vessel 10 from below with the imaging camera 89 of the first phase contrast microscope 80 at a predetermined imaging pitch (while changing the focal position of the optical system). A plurality of observed images (referred to as “first observed images”) are acquired, and a fertilized egg is captured at a predetermined imaging pitch (while changing the focal position of the optical system) by the imaging camera 99 of the second phase contrast microscope 90. ) Acquire a plurality of observation images (referred to as “second observation images”) taken continuously from above.

  The image storage unit 110 outputs the first and second observation images acquired from the imaging cameras 89 and 99, and the positional information of the sample stage 15 when the imaging acquisition output from the linear encoder 17 in the sample stage 15 is performed. At the same time, it is stored in association with index data such as the code number, observation position, and observation time of the culture vessel 10.

  In step S4, the image correction unit 120 performs image correction on each of the plurality of first observation images and second observation images stored in the image storage unit 110, based on the image correction parameters obtained in step S2. The observation visual field position of both images is matched with the observation magnification difference. Thereby, a 1st observation image and a 2nd observation image are matched per pixel.

  In step S5, the first and second observation images that have undergone image correction are subjected to a process from Shape From Focus (hereinafter simply referred to as “SFF”). Briefly explaining the principle of the SFF method, this method illuminates the sample to be observed with light, and moves the sample and the observation optical system (objective lens) relative to each other in the Z direction (focus direction). The image of the sample is acquired by the imaging camera at each set distance (imaging pitch) while changing the optical distance to the system, and the focus measurement is performed in pixel units on the obtained image by image processing described later. This is a technique for obtaining the three-dimensional shape information of the sample by using the characteristic that maximizes the focus measure when the sample is at a position conjugate with the imaging camera with respect to the observation optical system. FIG. 11 shows how the focus measure changes, and the surface shape of the sample is obtained by calculating the position at which the focus measure (contrast) is maximized (that is, the position of the sample stage as the focus position). Can do. Here, the details of the image processing by the SSF method will be described with reference to the flowchart of FIG.

  In step S51, the image analysis unit 130 calculates a focus measure for each pixel for each image with respect to the plurality of first observation images and second observation images acquired at each predetermined imaging pitch. At this time, the image analysis unit 130 applies, for example, a differential filter such as a Laplacian filter or a dispersion filter to the corrected first and second observation images acquired from the image correction unit 120, and performs a product-sum operation for each pixel. The obtained differential value or variance value is obtained as a focus measure. Note that the focus measure obtained for each pixel from the first observation image and the second observation image is recorded as one set for each corresponding pixel (pixels in the same region) in the first observation image and the second observation image. The

  In step S <b> 52, the image analysis unit 130 obtains an in-focus position where the in-focus measure reaches a peak for each pixel for the first and second observation images. At this time, in the case of an image of a non-transparent sample, the focus point at which the focus measure shows a peak is basically one for each pixel, but the observation target is a spherical transparent object such as an egg cell. In this case, a plurality of focal points exist for each pixel. For example, as shown in FIG. 13, when cell C1 of embryo H in fertilized egg J (at the 2-cell stage) is divided into cells C11 and C12, and cell C2 is divided into cells C21 and C22 think about. 13A to 13C are schematic diagrams when the fertilized egg is viewed from the horizontal direction.

  For example, considering the case where the points P1, P2, and P3 on the outer periphery of the embryo are scattered in the vertical direction with respect to the target pixel position (the same pixel position in the first and second observation images) in the figure, it is ideal. Specifically, the points P1 to P3 on the embryo H as the texture have a conjugate relationship with the region of the pixel of interest as the optical distance between the sample (fertilized egg J) and the observation optical system changes. The focus measure appears as a peak as a point on the sample.

  Here, as shown in FIG. 13A, when the peak position of the focus measure at the target pixel is detected at the same position in the observation with the first phase contrast microscope 80 and the second phase contrast microscope 90. Performs clustering on the same peak position, and determines that the point appearing as the peak is a focal point. FIG. 13A illustrates a case where all three peak positions appear at the same position, and the image analysis unit 130 has a focal point where the three points appearing as the peaks indicate points P1 to P3. It is judged that. In addition, since the position and size of each pixel area in the first observation image and the second observation image are matched by the image correction of the first observation image and the second observation image in step S4, the focal point is set. It can be detected with high accuracy.

  On the other hand, as shown in FIG. 13B, in the change of the focus measure in the observation from both sides by the phase-contrast microscopes 80 and 90, the peak position of the focus measure appears shifted due to the influence of random noise or the like. In the case, the observation information from the upper side (first phase contrast microscope 80) is preferentially given in the upper region U (upper hemisphere) of the fertilized egg J, and the lower side in the lower region L (lower hemisphere) ( Observation information from the second phase-contrast microscope 90) is preferentially adopted to define the focal point. That is, for the upper region U of the fertilized egg J, using the focus measure obtained by observation with the first phase contrast microscope 80, the focus measure appears as a peak (corresponding to points P1 and P2). ) Is recognized as the focal point. For the lower region L of the fertilized egg, the focus measure obtained by observation with the second phase-contrast microscope 90 is used and the point at which this focus measure appears as a peak is recognized as the focus. Thereby, in the observation from the upper side and the lower side, the in-focus position can be detected with high accuracy while avoiding the overlap of the in-focus points.

  By the way, when there are a plurality of cells in the observation direction (vertical direction in this embodiment), the cell on the near side is the cell on the back side (for example, the cell C22 is on the back side when the cell C11 is on the near side). There is a possibility that occlusion occurs. That is, in the SFF method for unidirectional observation, it is difficult to detect the peak of the focus measure for the structure existing on the back side when viewed from the observation direction due to blurring or scattering due to the structure existing on the front side (peak Therefore, there is a problem that the in-focus point cannot be accurately found. However, in the present embodiment which is the two-way observation, the problem can be solved as follows.

  As shown in FIG. 13C, when the above-described occlusion occurs and a peak appears only in one focus measure change, that is, in the observation from the upper side, the alignment with respect to the lower region L of the fertilized egg J When the focus measure is lowered due to blur or scattering due to the presence of the cell C11 or the like on the near side (no peak appears), or the focus measure with respect to the upper region U of the fertilized egg J is the near side cell in the observation from the lower side When the upper region U of the fertilized egg is lowered due to the presence of C22 or the like, the focus measure appears as a peak in the focus measure change obtained from the first phase contrast microscope 80. (Points with respect to points P1 and P2) are recognized as in-focus points, and for the lower region L of the fertilized egg, the in-focus measure appears as a peak in the in-focus measure change obtained from the second phase contrast microscope 90. (Point with respect to the point P3) is recognized as the focal point. Thus, even when occlusion occurs, it is possible to accurately detect a plurality of in-focus points by applying the SSF method to observation from two directions.

  To summarize this, after detecting all the focus measurement peaks for the same target pixel from each of the first and second observation images (a plurality of each), the upper region U and the lower region L are focused. If the peak position of the measure is the same, the cluster is immediately identified as the focal point, and if the peak position is different or the peak position appears only on one side, the upper region U of the fertilized egg is observed on the upper side. Observation information from the first phase contrast microscope 80 is applied, and observation information from the lower side observation (second phase contrast microscope 80) is applied to the lower region L, giving priority to peaks appearing in the region. To certify the focal point.

  In step S6, the image analysis unit 130 obtains the relative height of the sample surface based on the in-focus information (in-focus position) obtained for each pixel, and constructs a three-dimensional shape of the fertilized egg. Specifically, (X, Y, Z) point cloud coordinates constituting a three-dimensional shape of a fertilized egg are calculated. Further, the image analysis unit 130 generates a two-dimensional slice image obtained by gathering together the portions having the appropriate contrast for each predetermined imaging pitch based on the focal position obtained for each pixel.

  In step S7, the three-dimensional shape data (point cloud data) and the slice image calculated in step S6 are output to the display panel 72 by the output unit and displayed on the display panel 72 as the measurement result of the three-dimensional shape of the fertilized egg. . The display panel 72 can also display a three-dimensional shape based on the point cloud coordinates. Then, from the three-dimensional data constructed by the image processing apparatus 100, an arbitrary position of the fertilized egg J can be observed, and a dimension of the arbitrary position of the fertilized egg J can be calculated. Therefore, the observer can accurately detect the constricted part of the embryo accompanying the cleavage based on the three-dimensional shape data of the fertilized egg J and accurately observe the timing of cleavage of the cells in the egg. It becomes possible.

  As described above, in the culture observation system BS, the microscopic observation images (phase difference images) are sequentially captured by the imaging cameras 89 and 99 of the phase contrast microscope 56 according to the observation schedule at predetermined time intervals based on the observation program. As a result, image data indicating a three-dimensional shape is acquired at each time interval. For this reason, many images taken intermittently are continuously reproduced, and the change state (egg cleavage state) of the fertilized egg J, which is difficult to grasp visually on the actual time axis, is compressed by moving the time axis. By generating a time-lapse image that is visualized as a dynamic change, the growth state of the fertilized egg J can be grasped more accurately, and the growth state of the fertilized egg J can be determined (the cells in the egg are in the same period) It is possible to accurately determine whether or not the data has been split at the timing.

  As described above, according to the observation apparatus, the image processing program GP, and the shape measurement method configured by executing the image processing program GP according to the present embodiment described above, the SFF method is divided into two directions (up and down). By applying to the observation of fertilized eggs from (direction), the three-dimensional shape of the fertilized egg can be constructed with high accuracy, so that the cleavage of the fertilized egg can be judged quantitatively, and the growth status of the fertilized egg can be determined. It becomes possible to evaluate accurately.

  In the above-described embodiment, the first and second phase-contrast microscopes 80 and 90 arranged substantially symmetrically with respect to the sample surface are exemplified as the phase-contrast microscope apparatus 56. However, the present invention is limited to this embodiment. Instead, for example, as shown in FIG. 14, the same effect can be obtained by using a phase contrast microscope apparatus 156 configured by arranging the first phase contrast microscope 180 and the second phase contrast microscope 190 on the left and right. it can. At this time, the first phase-contrast microscope 180 includes a light source 181, an illumination optical system 182a, an observation optical system 182b, and an imaging camera 189. On the other hand, the second phase contrast microscope 190 includes a light source 191, an illumination optical system 192a, an observation optical system 192b, and an imaging camera 199. As described above, in the phase contrast microscope apparatus 156, the first phase contrast microscope 180 and the second phase contrast microscope 190 are moved back and forth in the horizontal direction by a driving device (not shown), so that The fertilized eggs J can be observed alternately from above and below.

  Moreover, in the above-mentioned embodiment, although the phase contrast microscope was illustrated as a microscope apparatus which microscopically observes the fertilized egg J, it is not limited to this, Even if it comprises using a differential phase contrast microscope, the same effect is obtained. Obtainable. Further, the epi-illumination device may be configured by a light source and an illumination optical system to configure a reflection type phase contrast microscope.

  In the above-described embodiment, in the image correction process with the first image and the second image, the amount of shift of the in-focus position in the vertical direction may be further corrected. In this case, the observation magnification is corrected by using an object having a phase gradient in the horizontal direction (for example, a hemisphere, a stair structure having a spatially wider pitch than the lateral resolution, a square pyramid, etc.) as a reference marker. The amount of deviation of the in-focus position in the vertical direction (focus direction) is measured based on the position where the shapes of the first image and the second image match and the contour contrast (focus measure) of the shape is maximized. Thus, the deviation amount can be corrected.

BS Culture observation system GP Image processing program J Fertilized egg 5 Observation unit 6 Control unit 15 Sample stage 52 Transmitted illumination unit 55 Microscopic observation system 56 Phase contrast microscope device 61 CPU
62 ROM 63 RAM
80 First Phase-Contrast Microscope 89 Imaging Camera 90 Second Phase-Contrast Microscope 99 Imaging Camera 100 Image Processing Device 120 Image Correction Unit 130 Image Analysis Unit 140 Output Unit

Claims (9)

A plurality of first images obtained by sequentially photographing the biological sample from one side by a first imaging device while changing an optical distance between the biological sample and a first observation optical system that forms an image of the biological sample; A plurality of second images obtained by sequentially imaging the biological sample from the other side facing the one side by a second imaging device while changing an optical distance between the sample and a second observation optical system that forms an image of the biological sample. Get images and
Calculating a focus measure in pixel units based on the plurality of first and second images;
The focus measure information obtained from the first image is preferentially applied to the one side region of the biological sample, and the focus measure obtained from the second image is applied to the other region of the biological sample. Apply the information preferentially to find the focal point of each pixel
A shape measuring method comprising constructing a three-dimensional shape of the biological sample based on the focal point position.   The shape measurement method according to claim 1, wherein the first and second imaging devices capture a phase difference image obtained by phase difference observation of the biological sample.   The shape measurement method according to claim 1, wherein an observation visual field position and an observation magnification are corrected between the first image and the second image. An image processing program for causing a computer to function as an image processing apparatus that is readable by a computer, acquires an image captured by an imaging device, and performs image processing,
A plurality of first images obtained by sequentially photographing the biological sample from one side by a first imaging device while changing an optical distance between the biological sample and a first observation optical system that forms an image of the biological sample; A plurality of second images obtained by sequentially imaging the biological sample from the other side facing the one side by a second imaging device while changing an optical distance between the sample and a second observation optical system that forms an image of the biological sample. Obtaining an image, and
Calculating a focus measure in pixel units based on the plurality of first and second images;
The focus measure information obtained from the first image is preferentially applied to the one side region of the biological sample, and the focus measure obtained from the second image is applied to the other region of the biological sample. Preferentially applying information to determine the focal point of each pixel;
An image processing program for causing the computer to realize a step of constructing a three-dimensional shape of the biological sample based on the focal point position.   The image processing program according to claim 4, wherein the first and second imaging devices capture a phase difference image obtained by phase difference observation of the biological sample.   6. The image processing program according to claim 4, wherein an observation visual field position and an observation magnification are corrected between the first image and the second image. A sample stage for holding a biological sample;
A first imaging device that images the biological sample from one side and a second imaging device that images from the other side facing the one side;
Second observation optics for causing the sample stage to move relative to a first observation optical system that forms an image of the biological sample on the imaging surface of the first imaging device and to form an image on the imaging surface of the second imaging device. A movement control unit that moves relative to the system;
A plurality of first images of the biological sample obtained by imaging with the first and second imaging devices while the sample stage is moved relative to the first and second observation optical systems by the relative moving unit. And an image processing device that performs image processing on the second image,
The image treatment device calculates a focus measure in units of pixels based on the plurality of first images and second images, and the focus on the one side of the biological sample obtained from the first image is calculated. Priority is applied to the measure information, and the focus measure information obtained from the second image is preferentially applied to the region on the other side of the biological sample to obtain the focus of each pixel, and based on the focus position An observation apparatus comprising an image analysis unit that constructs a three-dimensional shape of the biological sample.   The observation apparatus according to claim 7, wherein the first and second imaging apparatuses capture a phase difference image obtained by phase difference observation of the biological sample.   The observation according to claim 7 or 8, wherein the image processing apparatus includes an image correction unit that corrects an observation visual field position and an observation magnification between the first image and the second image. apparatus.

Images