JP2011004638A - Mthod, program, and apparatus for processing images of observed fertilized egg - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide image processing means for rapidly and toughly detecting fertilized eggs.SOLUTION: The invention includes a step S1 for acquiring a low magnification image obtained by imaging objects positioned in a low magnification observation visual field by an imaging device, steps S3 and 4 for extracting a plurality of the objects in low magnification images, a step S6 for calculating the characteristic amount of low magnification image according to attribution of fertilized egg together with the plurality of the objects contained in the low magnification images and extracting a fertilized egg candidate on the base of the characteristic amount of the low magnification images, steps S7 and 8 for dividing the range in the low magnification observation visual field into a plurality of small ranges according to the size of high magnification observation visual field and successively imaging only small ranges containing the objects selected as a fertilized egg candidate to acquire a plurality of the high magnification image, a step S9 for calculating the characteristic amount of the high magnification image according to the attribution of fertilized egg for each of the objects of fertilized egg candidate contained in the plurality of high magnification images and identifying a fertilized egg from the objects of the fertilized egg candidate on the base of a characteristic amount of high magnification image, and a step S10 for outputting the identified results of the objects.

Description

  The present invention relates to an image processing means for discriminating a fertilized egg and a foreign substance from an observation image acquired in fertilized egg observation.

  In recent years, with the development of assisted reproduction technology (ART), it has been practiced to observe the growth state of cultured fertilized eggs by in vitro fertilization. A culture microscope is mentioned as an example of the apparatus which observes the conditions of cultures, such as a fertilized egg (for example, refer patent document 1). The culture microscope includes a culture device (invecutor) that forms a suitable environment for culturing fertilized eggs, and a microscopic observation system that microscopically observes the state of the fertilized eggs in a culture container housed in the culture device. The observation image of the fertilized egg is acquired at regular intervals, and the user can recognize the fertilized egg by visual observation, and then automatically observe, record, and manage the fertilized egg growth state. The

JP 2004-229619 A

  In such an apparatus, when the growth state of a fertilized egg in a culture container is to be microscopically observed, firstly, the fertilized egg to be observed must be detected from the medium. Since foreign matters such as dust and oil particles are mixed, in order to distinguish a fertilized egg from other foreign matters and recognize the fertilized egg, it is necessary to perform microscopic observation at a magnification as high as possible. However, although higher resolution images can be obtained with higher magnification observation, the observation field (observation range) becomes narrower with higher magnification observation, and the probability that an image of a fertilized egg is reflected in the field of view decreases. In order to detect an egg, it is necessary to take a large number of high-magnification images in a wide field of view, and a complicated process for identifying a fertilized egg from the large number of images is required. It takes a long time to detect a fertilized egg. There was a problem.

  The present invention has been made in view of the problems as described above, and distinguishes between a fertilized egg and other foreign substances based on a feature amount according to the attribute of the fertilized egg by low magnification observation and high magnification observation. An object of the present invention is to provide an image processing means capable of detecting fertilized eggs at high speed and robustly by performing in association with each other.

  According to the first aspect illustrating the present invention, a low-magnification image obtained by photographing a plurality of objects located in a low-magnification observation visual field by an imaging device is acquired, and a plurality of objects imprinted on the low-magnification image are extracted. Then, for each of the plurality of objects included in the low-magnification image, the feature quantity of the low-magnification image is calculated according to the attribute of the fertilized egg, and the fertilized egg is selected from the plurality of objects based on the feature quantity of the low-magnification image. Candidates are extracted, and the area within the low-magnification observation field is divided into a plurality of small areas according to the size of the high-magnification observation field, and only for small areas that contain objects selected as fertilized egg candidates, Obtain multiple high-magnification images by performing sequential shooting, and calculate the feature quantity of the high-magnification image according to the fertilized egg attribute for each fertilized egg candidate object included in the multiple high-magnification images. To identify fertilized eggs from candidates for fertilized eggs based on The image processing method of the embryo observation is provided.

  According to a second aspect of the present invention, there is provided an image processing program that causes a computer to function as an image processing apparatus that can be read by a computer, acquires an image captured by an imaging device, and performs image processing. Included in the low-magnification image are a step of acquiring a low-magnification image obtained by photographing a plurality of objects located in the low-magnification observation field by an imaging device, a step of extracting a plurality of objects imprinted in the low-magnification image, and Calculating a feature amount of a low-magnification image according to the attribute of the fertilized egg for each of the plurality of objects, extracting a fertilized egg candidate from the plurality of objects based on the feature amount of the low-magnification image, The area in the double observation field is divided into a plurality of small areas according to the size of the high magnification observation field, and only the small area including the object selected as a fertilized egg candidate is photographed sequentially by the imaging device. A step of acquiring a plurality of double images and a feature amount of the high magnification image corresponding to the fertilized egg attribute for each fertilized egg candidate object included in the plurality of high magnification images, and based on the feature amount of the high magnification image An image processing program for observing a fertilized egg is provided, which causes a computer to realize a step of identifying a fertilized egg from among fertilized egg candidate objects and a step of outputting an identification result for the object.

  According to the third aspect exemplifying the present invention, an imaging device capable of photographing a plurality of objects at a low magnification and a high magnification, a plurality of objects extracted from a low magnification image photographed by the imaging device, and a plurality of objects An image analysis unit that identifies a fertilized egg from the image, and an output unit that outputs the identification result determined by the image analysis unit to the outside, and the image analysis unit includes a plurality of objects included in the low-magnification image. , Calculate the feature quantity of the low-magnification image according to the attributes of the fertilized egg, extract fertilized egg candidates from a plurality of objects based on the feature quantity of the low-magnification image, and extract the region in the low-magnification observation field By dividing into a plurality of small regions according to the size of the high-magnification observation visual field, only a small region containing the object selected as a fertilized egg candidate is photographed sequentially by the imaging device to obtain a plurality of high-magnification images, For each fertilized egg candidate object included in multiple high-magnification images Fertilized egg observation characterized by calculating the feature quantity of the high-magnification image according to the attributes of the fertilized egg and identifying the fertilized egg from the fertilized egg candidate objects based on the feature quantity of the high-magnification image An image processing apparatus is provided.

  According to such an image processing method, an image processing program, and an image processing apparatus for observing a fertilized egg, even if the observer himself / herself does not visually recognize the fertilized egg to be observed, the object to be observed is selected from a plurality of objects. It becomes possible to detect a fertilized egg fast and robustly.

It is a flowchart which shows the outline | summary of an image processing program. It is a general | schematic block diagram of the culture observation system shown as an example of application of this invention. It is a block diagram of the said culture observation system. (A) is a top view of a culture container, (B) is a perspective view which shows a dish. It is a figure which illustrates the condition of the object extraction process by application of a distributed filter. It is a figure which illustrates the condition of a labeling process. It is a figure which illustrates the condition of the outline extraction process which extracts an object. It is a figure for demonstrating the characteristic of a fertilized egg and other foreign materials. It is a figure for demonstrating the difference of the luminance value of the outline part of a fertilized egg and a foreign material. (A) is a figure which illustrates the state which tiled the high magnification visual field area | region to the low magnification visual field area | region, (B) is a figure which illustrates the high magnification visual field area | region in which a fertilized egg candidate is contained. 1 is a block diagram illustrating a schematic configuration of an image processing apparatus.

  Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings. As an example of a system to which the image processing apparatus according to the present embodiment is applied, a schematic configuration diagram and a block diagram of a culture observation system are shown in FIGS. 2 and 3, respectively.

  The culture observation system BS roughly observes the culture chamber 2 provided in the upper part of the housing 1, the shelf-like stocker 3 that accommodates and holds a plurality of culture containers 10, and the sample in the culture container 10. An observation unit 5, a transfer unit 4 for transferring the culture vessel 10 between the stocker 3 and the observation unit 5, a control unit 6 for comprehensively controlling the operation of the system, and an operation panel 7 provided with an image display device. Etc.

The culture chamber 2 is a chamber for forming a culture environment, and is kept sealed after the sample is charged in order to prevent environmental changes and contamination. Along with the culture chamber 2, a temperature adjustment device 21 that raises and lowers the temperature in the culture chamber 2, a humidifier 22 that adjusts humidity, and a gas supply device 23 that supplies a gas such as CO ₂ gas or N ₂ gas. A circulation fan 24 for making the entire environment of the culture chamber 2 uniform, an environmental sensor 25 for detecting the temperature, humidity, carbon dioxide concentration, etc. of the culture chamber 2 are provided. The operation of each device is controlled by the control unit 6, and the culture environment defined by the temperature, humidity, carbon dioxide concentration, etc. of the culture chamber 2 is maintained in a state that matches the culture conditions set on the operation panel 7.

  The stocker 3 is formed in a shelf shape that is partitioned into a plurality of parts in the front-rear direction and the vertical direction in FIG. A unique address is set for each shelf. For example, when the front-rear direction is A to C rows and the vertical direction is 1 to 7 rows, the A-row 5 rows shelf is set as A-5. The

  As the culture vessel 10, an appropriate one such as a flask, a dish, or a well plate is selected according to the type and purpose of the culture. In this embodiment, as shown in FIG. The structure provided with the dish 10a and the holder 10b which hold | maintains the dish 10a is illustrated, and as shown to FIG. 4 (B), the fertilized egg a which is a culture is a culture medium containing pH indicators, such as phenol red It is injected into each dish 10a together with the drop D. On the bottom surface of the dish 10a, one or more medium drops D of about 20 μl dropped by a pipette or the like are formed (only one is shown in FIG. 4B), and the medium drop D is contained in the dish 10a. It is in a state immersed in colorless and transparent mineral oil O. In each medium drop D, for example, one fertilized egg a collected from the same mother at the same time for external fertilization is inserted. The culture vessel 10 is assigned a code number and is stored in association with the designated address of the stocker 3.

  The transfer unit 4 is provided inside the culture chamber 2 so as to be movable in the vertical direction and is moved up and down by the Z-axis drive mechanism. The transfer unit 4 is attached to the Z stage 41 so as to be movable in the front-rear direction and is moved by the Y-axis drive mechanism. The Y stage 42 that is moved back and forth, the X stage 43 that is attached to the Y stage 42 so as to be movable in the left-right direction and is moved left and right by the X-axis drive mechanism, and the like, is supported by lifting the culture vessel 10 to the tip side of the X stage 43 A support arm 45 is provided. The transport unit 4 has a moving range in which the support arm 45 can move between the entire shelf of the stocker 3 and the observation unit 5. The X-axis drive mechanism, the Y-axis drive mechanism, and the Z-axis drive mechanism are configured by, for example, a servo motor with a ball screw and an encoder, and the operation thereof is controlled by the control unit 6.

  The observation unit 5 includes a first illumination unit 51 that illuminates the sample from the lower side of the sample stage 15, a second illumination unit 52 that illuminates the sample from above the sample stage 15 along the optical axis of the microscopic observation system, and a sample from below. 3, a macro observation system 54 that performs macro observation of the sample, a micro observation system 55 that performs micro observation of the sample, an image processing apparatus 100 (see FIG. 10), and the like. The sample stage 15 is made of a light-transmitting material and has a transparent window 16 in the observation area. The sample stage 15 is composed of a fine drive stage that can be moved in the XY direction (horizontal plane direction) and the Z direction (vertical direction) by operation control from the control unit 6, and the culture vessel 10 placed on the upper surface thereof. Is moved in the XY directions, so that the culture vessel 10 can be inserted on the optical axis of the macro observation system 54 or on the optical axis of the microscopic observation system 55.

  The first illumination unit 51 is a surface-emitting light source provided on the lower frame 1 b side, and backlight-illuminates the entire culture vessel 10 from the lower side of the sample stage 15. The second illumination unit 52 includes a light source such as an LED and an illumination optical system including a phase ring, a condenser lens, and the like. The second illumination unit 52 is provided in the culture chamber 2 and receives light from the microscopic observation system 55 from above the sample stage 15. The sample in the culture vessel 10 is illuminated along the axis. The third illumination unit 53 superimposes light emitted from each of the light sources, such as a plurality of LEDs and mercury that emit light having a wavelength suitable for epi-illumination observation and fluorescence observation, on the optical axis of the microscopic observation system 55. An illumination optical system composed of a beam splitter, a fluorescent filter, and the like to be disposed in the lower frame 1b located below the culture chamber 2, and the light of the microscopic observation system 55 from below the sample stage 15 The sample in the culture vessel 10 is illuminated along the axis.

  The macro observation system 54 includes an observation optical system 54 a and an imaging device 54 c such as a CCD camera that captures an image of the sample imaged by the observation optical system 54 a and is positioned above the first illumination unit 51. Provided in the culture chamber 2. The macro observation system 54 captures an entire observation image (macro image) from above the culture vessel 10 that is backlit by the first illumination unit 51.

  The microscopic observation system 55 includes an observation optical system 55a composed of an objective lens, an intermediate zoom lens, a fluorescent filter, and the like, and an imaging device 55c such as a cooled CCD camera that takes an image of a sample imaged by the observation optical system 55a. And disposed inside the lower frame 1b. The second illumination unit 52 and the microscopic observation system 55 constitute a phase difference observation microscope. A plurality of objective lenses and intermediate zoom lenses are provided, and are configured to be set to a plurality of magnifications using a displacement mechanism such as a revolver or a slider (not shown in detail). In the present embodiment, at least two magnifications for low-magnification observation (for example, for double-magnification observation) and high-magnification observation (for example for ten-times observation) are switched so as to be variable. The microscopic observation system 55 displays the sample in the culture vessel 10 such as a phase difference image by the transmitted light of the sample illuminated by the second illumination unit 52 and a fluorescence image by the fluorescence emitted from the sample illuminated by the third illumination unit 53. A microscopic observation image (micro image) is taken.

  The image processing apparatus 100 performs A / D conversion on signals input from the imaging device 54c of the macro observation system 54 and the imaging device 55c of the microscopic observation system 55, and performs various image processes to perform an entire observation image or a microscopic observation image. Image data is generated. Further, the image processing apparatus 100 performs image analysis on the image data of these observation images (entire observation image and microscopic observation image), calculates the feature amount of an object present in the image, and responds to each feature amount. Image processing such as calculating the score and determining the fertilized egg based on the total score. Specifically, the image processing apparatus 100 is constructed by executing an image processing program stored in the ROM of the control unit 6 described below. The image processing apparatus 100 will be described in detail later.

  The control unit 6 includes a CPU 61 that executes processing, a ROM 62 that stores and stores control programs and control data of the culture observation system BS, a RAM 63 that temporarily stores observation conditions and image data, and the like. Control the operation. Therefore, as shown in FIG. 3, the constituent devices of the culture chamber 2, the transport device 4, the observation unit 5, and the operation panel 7 are connected to the control unit 6. In the RAM 63, the environmental conditions of the culture chamber 2 according to the observation program, the observation schedule, the observation type and observation position in the observation unit 5, the observation magnification, and the like are set and stored. Further, the RAM 63 is provided with an image data storage area for recording image data photographed by the observation unit 5, and index data including the code number of the culture vessel 10 and photographing date / time are associated with the image data and recorded. Is done.

  The operation panel 7 is provided with an operation panel 71 provided with input / output devices such as a keyboard and a switch, and a display panel 72 for displaying an operation screen, image data, and the like. An operation command or the like is input. The communication unit 65 is configured in accordance with a wired or wireless communication standard, and data can be transmitted to and received from a computer or the like externally connected to the communication unit 65.

  In the culture observation system BS schematically configured as described above, the CPU 61 controls the operation of each part based on the control program stored in the ROM 62 according to the setting conditions of the observation program set on the operation panel 7, and the culture vessel 10 The sample inside is automatically captured. That is, when the observation program is started by a panel operation on the operation panel 71 (or a remote operation via the communication unit 65), the CPU 61 reads each condition value of the environmental conditions stored in the RAM 63, and from the environment sensor 25. The environmental state of the culture chamber 2 to be input is detected, and the temperature adjustment device 21, the humidifier 22, the gas supply device 23, the circulation fan 24, etc. are operated according to the difference between the condition value and the actual measurement value. Feedback control is performed on the culture environment such as temperature, humidity, and carbon dioxide concentration.

  Further, the CPU 61 reads the observation conditions stored in the RAM 63 and operates the X, Y, Z stages 41, 42, 43 of the transport unit 4 based on the observation schedule to observe the culture vessel 10 to be observed from the stocker 3. The sample is transported to the sample stage 15 of the unit 5 and observation by the observation unit 5 is started. For example, when the observation set in the observation program is macro observation, the culture vessel 10 transported from the stocker 3 by the transport unit 4 is positioned on the optical axis of the macro observation system 54 and placed on the sample stage 15. Then, the light source of the first illumination unit 51 is turned on, and the entire observation image is taken by the imaging device 54c from above the culture vessel 10 that is backlit. The signal input from the imaging device 54c to the control unit 6 is processed by the image processing device 100 to generate a whole observation image, and the image data is stored in the image data storage area of the RAM 63 together with index data such as the shooting date and time. Is done.

  In addition, when the observation set in the observation program is micro observation of the sample at a specific position in the culture vessel 10, the specific position in the culture vessel 10 that has been transported by the transport unit 4 is indicated by the light of the microscopic observation system 55. Position on the axis and place on the sample stage 15, turn on the light source of the second illumination unit 52 or the third illumination unit 53, and cause the imaging device 55c to take a microscopic observation image by transmitted illumination, epi-illumination, and fluorescence. . A signal photographed by the imaging device 55c and inputted to the control unit 6 is processed by the image processing device 100 to generate a microscopic observation image (phase difference image, fluorescent image, etc.), and the image data is an index such as a photographing date / time. Stored in the image data storage area of the RAM 63 together with the data.

  The CPU 61 sequentially executes the above-described whole observation image photographing and microscopic observation image photographing based on the observation schedule set in the observation program. The image data stored in the RAM 63 is read from the RAM 63 in response to an image display command input from the operation panel 71. For example, an entire observation image, a microscopic observation image at a specified time, an analysis result of image analysis, and the like are displayed on the display panel 72. Is displayed.

  In the culture observation system BS configured as described above, when the growth state of the fertilized egg a in the culture vessel 10 (dish 10a) is to be microscopically observed, the fertilized egg to be observed is first detected from the medium drop D. However, in the medium drop D, foreign matter such as dust and oil particles is mixed in addition to the fertilized egg a. Therefore, the fertilized egg is recognized by distinguishing the fertilized egg from other foreign matters. In order to achieve this, it is necessary to perform microscopic observation at as high a magnification as possible. However, an image with higher resolution can be acquired as the magnification is increased, but the observation field (observation range) is narrowed as the magnification is increased, and the probability that the image of the fertilized egg a is imprinted in the field decreases. In order to detect the image of the fertilized egg a, it is necessary to take a large number of high-magnification images in a wide field of view, and a complicated process for identifying the fertilized egg from the large number of images is required. There was a problem that it took a long time to complete.

  In order to correct such a problem, the image processing apparatus 100 has two-stage images of extraction of fertilized egg candidates in low-magnification observation and identification of fertilized eggs in high-magnification observation with respect to feature amounts corresponding to the attributes of the fertilized egg. It has a function of automatically detecting a fertilized egg to be observed, which is used for analysis processing. Now, an image processing method executed by the image processing apparatus 100 will be described below from a basic concept. In the following description, a case where a fertilized egg is observed based on a phase contrast image (microscopic observation image) photographed by a phase contrast microscope configured by the second illumination unit 52, the microscopic observation system 55, and the like will be exemplified. . At this time, in the microscopic observation system 55, low magnification observation and high magnification observation are possible by switching the observation magnification of the observation optical system 55a.

(Image irregularity correction)
First, in the low-magnification observation image (low-magnification phase-contrast image) photographed using the objective lens for low-magnification observation in the microscopic observation system 55, an image of the medium drop D that is copied as the background of the fertilized egg a Therefore, unevenness correction is performed on the low-magnification observation image. In the smoothing process, an extremely blurred image with respect to the input image is created by applying a Gaussian filter to the captured low-magnification observation image (input image) and performing a convolution operation in units of pixels. Then, as shown in the following equation (1), a blurred image is subtracted from the input image for each corresponding pixel to generate a difference image that employs only positive values. Here, if the input image is I (x, y), the blurred image smoothed by the Gaussian filter is B (x, y), and the difference image is I ′ (x, y), the difference image I ′ (x , Y) is generated by the following equation (1).
I ′ (x, y) = I (x, y) −B (x, y) (1)
However, when I (x, y) −B (x, y) <0, I ′ (x, y) = 0

Here, the weighting of the Gaussian filter is determined by the Gaussian function G (x, y) defined by the following equation (2), where the center coordinate of the pixel of interest is (x, y).
G (x, y) = (1 / (2πσ ² )) exp (− (x ² + y ² ) / 2σ ² ) (2)
Here, the parameter σ is a value indicating the spatial extent of the Gaussian filter, and the higher the value, the higher the degree of smoothing. Therefore, σ is sufficiently large to generate an extreme blurred image. It is desirable. The filter size is set to a radius of 3σ, and the pixel located at the end of the image to which the filter cannot be applied is, for example, not multiplied by the reciprocal of the filter integral value or subjected to filter processing. Thus, by subtracting the blurred image smoothed by a Gaussian filter or the like from the input image, it is possible to generate a difference image in which high frequency components in the input image are extracted.

(Extract objects)
Next, a variance filter is applied to the difference image obtained above to generate a variance value image. The dispersion filter obtains the dispersion value of the luminance value of the pixel of interest and its surrounding pixels, and sets it as the luminance value (luminance dispersion value) of the pixel of interest, and returns a larger value as the variation of the luminance value increases. It is like that. In this variance value image, a pixel having a large luminance variance value corresponds to an edge point or the like. FIG. 5 is a diagram showing the state of object extraction processing by applying this dispersion filter. By applying a dispersion filter to the difference image (A) acquired by low magnification observation and smoothed, (B) is obtained. Like the variance value image shown, an image that appears to be an object (a region of a foreign object such as a fertilized egg and dust, or an object) is extracted.

  Then, a binarization process using a predetermined threshold is performed as a contour extraction method on the variance value image from which an object-like image has been extracted. The binarization process is performed based on, for example, a threshold value based on a predetermined luminance value that can be normally detected or a threshold value obtained by discriminant analysis. By performing the binarization process after applying the dispersion filter in this way, the contour of the object included in the image is extracted, and the closed region surrounded by the contour is specified as the object.

(Object labeling)
Each object thus segmented is labeled to give a unique label. FIG. 6 is a diagram showing the status of the labeling process. As shown in FIG. 6B, the labeling number is set to 1 for the object of the binary image (A) from which the contour is extracted, and the next is set to 1 from the next. Increase and identify. In the labeling process, the area of the object (area surrounded by the extracted contour) is calculated together with labeling. Therefore, the size of an image of a fertilized egg (for example, a diameter of about 100 μm in the case of a human fertilized egg) is almost determined by the image acquisition conditions (observation magnification, etc.), and is therefore subject to discrimination. An upper limit value and a lower limit value of a possible area of the object are set in advance, and an object outside the set range is excluded from a labeling target (fertilized egg discrimination candidate). However, since the size of the fertilized egg is constant and the area of the object to be extracted is close to foreign objects such as dust and oil particles, the area changes, so the above setting range for rejecting unnecessary objects It is desirable to set a large value.

(Extraction of fertilized egg candidate)
A feature amount corresponding to the fertilized egg attribute is calculated for a plurality of labeled objects, and an object that can be a fertilized egg candidate is extracted based on the feature amount. As described above, the medium drop D contains foreign matters in addition to the fertilized egg to be observed. For this reason, the substance of the labeled object is roughly classified as follows. One is a fertilized egg, and in addition to that, it is roughly divided into garbage (remaining residue of the medium), oil grains (the mineral oil O filled in the dish 10a enters the medium), and bubbles (gas). The At this time, as shown in FIG. 7, each of the fertilized egg, dust, oil particles, and bubbles has the following characteristics.

<Characteristics of fertilized eggs>
A fertilized egg has a feature that a contour (a zona pellucida) is present in the contour, the outer shape is spherical, and an egg cell is present in the contour.
<Characteristics of garbage>
The dust has the characteristics that there is no transparent band in the outline, halo appears in the outline in the phase difference image, and the structure inside the outline is indefinite.
<Features of oil particles and bubbles>
In the phase difference image, the oil particles and bubbles have the characteristics that the contour portion is dark and the inside is bright, the outer shape is spherical or elliptical, and the structure inside the contour is small.

  To summarize these characteristics, the fertilized egg has a higher degree of circularity than the foreign object, the fertilized egg has a zona pellucida in the contour, the internal structure of the fertilized egg (Egg cells) are always present.

  Here, the image features in which changes are remarkably observed in the high magnification observation and the low magnification observation based on these attributes are listed below. The high-magnification observation has an advantage that since a high-resolution image is obtained, the image feature is easily digitized and suitable for quantitatively identifying a fertilized egg based on the outer shape feature amount, the texture feature amount, and the like. On the other hand, low-magnification observation has an advantage that the object can be easily discriminated by luminance because it has a wide observation field of view although the resolution of the image is low and changes in luminance appear remarkably. As shown in FIG. 8, the brightness of the outline of the object tends to be brighter due to the halo and the outline of the oil particles and bubbles are darker (black) than when observed at a high magnification. Thus, the outline of the fertilized egg (the zona pellucida) tends to have a brightness close to that of the background (medium drop D) as in the case of high magnification observation.

  Therefore, here, objects that are candidates for fertilized eggs are efficiently narrowed down from a wide range of low-magnification visual fields based on feature quantities that change markedly by low-magnification observation, that is, brightness values of the contours of the objects. As for the brightness value of the contour portion as a feature amount, the brightness value (luminance average value) in the target region of each object in the original image (low-magnification observation image) with the contour portion extracted by binarization processing as the target region. Threshold values are set in advance and rejected from the fertilized egg candidate. Note that the feature amount in low-magnification observation is not limited to the luminance value of the above-described contour portion itself, and for example, the luminance value of the entire object region or the difference between the luminance values inside and outside the contour line may be used.

  In addition to extracting fertilized egg candidates using feature quantities suitable for low-magnification observation in this way, detection accuracy is inferior due to low resolution in low-magnification observation, but using feature quantities suitable for high-magnification observation, Furthermore, it is good also as narrowing down a fertilized egg candidate. For example, using the fact that the fertilized egg to be observed has a spherical shape (close to a perfect circle), the circularity of each object is calculated, and a threshold value is set in advance for an object with a low circularity. , Reject from fertilized egg candidate. Here, the circularity of the object is a scale for determining the circularity of the object. The circularity is determined by, for example, determining the center of gravity of each object in the image plane of the acquired binary image and detecting the edge indicating the contour of each object, and then detecting the distance from the center of gravity to the contour for each object. It is obtained by calculating as a ratio of the minimum value to the maximum value, or by calculating as a moment when it is considered as a two-dimensional object in the image plane. Note that the low-magnification image has poor resolution, so the accuracy of circularity detection is also low, but by setting the threshold level with coarse sensitivity, an object with an outer shape that is not suitable for a fertilized egg candidate can be obtained. It can be eliminated sufficiently.

  In this way, by selecting only those objects that can be fertilized egg candidates from the low-magnification observation images, the number of discriminating targets for fertilized eggs during high-magnification observation is reduced, and the fertilized egg identification process during high-magnification observation is facilitated. .

(Identification of high magnification field position for fertilized egg candidate)
Here, a high-magnification visual field position for observing a fertilized egg candidate object imaged in the low-magnification visual field area at a high magnification is specified. As described above, when the observation magnification in low magnification observation is 2 times and the observation magnification in high magnification observation is 10 times, the size of the high magnification field region is 1/5 with respect to the size of the low magnification field region. Therefore, the low-magnification field area may be divided into 5 × 5 high-magnification field areas vertically and horizontally, and tiling may be performed. However, in this embodiment, in order to prevent a fertilized egg candidate from being overlooked, a margin is provided and adjacent to it. An overlapping region is provided for the high-magnification visual field area to be tiled based on 6 × 6 high-magnification visual field areas.

  FIG. 9 shows a state where the high-magnification field area is tiled on the low-magnification field area, and [m, n] -th (1 ≦ m ≦ 6, 1 ≦ n ≦ 6) in the low-magnification field area T shown in FIG. ), The coordinates (PosXm, PosYn) of the upper left corner are given for each tile area (high magnification visual field area) K. For the tile area K in which the object selected as a fertilized egg candidate is included, a flag indicating the presence of an object is set as an identifier. Then, based on the upper left corner coordinates (PosXm, PosYn) of the tile area for which the flag is set, the sample stage (fine drive stage) 15 is moved in the horizontal plane, and only the tile area K for which this flag is set is imaged. A high-magnification observation image (high-magnification phase difference image) is sequentially acquired by the device 55 and applied to the high-magnification observation algorithm described below.

(Determination of fertilized eggs)
In the high-magnification observation algorithm, for each object of a fertilized egg candidate based on the high-magnification observation image, a feature amount (feature amount suitable for high-magnification observation) corresponding to the above-described fertilized egg attribute is calculated and scored (score calculation) ) And recognize the object with the highest score as a fertilized egg.

  Prior to the fertilized egg discrimination process, the contour of an object that has become a fertilized egg candidate is extracted from the high-magnification observation image by a contour extraction method such as binarization. The feature amount applied in the high magnification observation includes, for example, the circularity of the contour portion and the texture feature amount inside the contour. The feature corresponding to the attribute of the fertilized egg is more quantitatively determined based on the high-resolution image. A suitable one is selected.

  Here, the circularity of the contour portion is synonymous with the circularity described above, and is an external feature based on the fact that the fertilized egg has a spherical shape. For this feature amount, the higher the score as a fertilized egg is set for an object having a contour with higher circularity. Further, the texture feature amount inside the contour is a feature amount utilizing the fact that the fertilized egg has an internal structure (fertilized egg) and thus the internal texture exists on the image. As the texture feature amount, for each object that becomes a fertilized egg candidate, the inside of the contour extracted by the binarization process is used as the target region, and the edge strength (gradation change) in the target region in the original image (high magnification observation image) Average value) is used. For this feature quantity, the higher the edge strength inside the contour, the higher the score as a fertilized egg. Thus, by using a high-resolution observation image with high resolution, the characteristics of the fertilized egg can be captured more quantitatively, and the fertilized egg and other foreign matters can be accurately identified.

  Also, in high magnification observation, the brightness value of the contour part (luminance average value, brightness dispersion value, etc.) is calculated for each object, and the brightness value of the contour part of the object of the same label that has already been calculated in low magnification observation. You may obtain | require a difference value and apply this difference value as a feature-value of high magnification observation. According to this, based on the difference in brightness between low-magnification observation and high-magnification observation, the feature quantity of low-magnification observation can be used as the feature quantity of high-magnification observation, and fertilization based on consistent image features. Egg identification can be performed.

  A fertilized egg is determined on the basis of each characteristic quantity obtained in this way. As a specific method, for example, normalization that converts each feature value to a common scale is performed to obtain each score, a total score is calculated from the sum of each score, and the total of the fertilized egg candidate objects The object with the highest score is determined to be a fertilized egg.

  According to the image processing method for fertilized egg observation as described above, fertilized egg candidate objects in the low-magnification field area are extracted in advance based on the low-magnification observation, and the entire low-magnification field area is virtually expanded into the high-magnification field of view. Take a high-magnification observation image only for the divided areas that contain the fertilized egg candidate object when it is divided into areas corresponding to the area, and finally identify the fertilized egg from the fertilized egg candidate objects based on the high-magnification observation Therefore, it is possible to efficiently detect a fertilized egg in a short time by omitting acquisition of a high-magnification observation image for a useless area that does not include a fertilized egg candidate. At this time, since the fertilized egg is determined based on both the feature quantity that appears prominently in the low magnification observation and the feature quantity that appears prominently in the high magnification observation, the detection accuracy of the fertilized egg can be increased.

(application)
Next, a specific application of image analysis executed in the image processing apparatus 100 of the culture observation system BS will be described with reference to FIGS. Here, FIG. 1 is a flowchart showing an outline of processing in an image processing program GP for fertilized egg observation, and FIG. 10 is a block diagram showing an outline configuration of an image processing apparatus 100 that executes image processing for fertilized egg observation.

  The image processing apparatus 100 acquires and stores an observation image (low-magnification observation image and high-magnification observation image) obtained by photographing the fertilized egg to be observed by the imaging device 55c, and analyzes and observes the observation image. An image analysis unit 120 that identifies a fertilized egg from among objects copied in an image, and a feature amount that is stored for each object by the image analysis unit 120 in association with a label that is assigned to each object. A storage unit 130; a data table 140 having a flag assigned to each tile area within the low-magnification visual field area; and an output unit 150 that outputs the determination result analyzed by the image analysis unit 120 to the outside. The identification result of whether or not the object determined by 120 is a fertilized egg is output on the display panel 72 and displayed, for example. Constructed. The image processing apparatus 100 is configured such that an image processing program GP preset and stored in the ROM 62 is read by the CPU 61 and processing based on the image processing program GP is sequentially executed by the CPU 61.

  As described, in the culture observation system BS, fertilized eggs in the culture vessel 10 designated every predetermined time are observed according to the observation conditions set in the observation program. Specifically, the CPU 61 operates each stage of the transport unit 4 to transport the culture vessel 10 to be observed from the stocker 3 to the observation unit 5 (in this embodiment, it is arranged on the optical axis of the microscopic observation system 55). The objective lens of the observation optical system 55a is set for low-magnification observation, and a low-magnification observation image by the microscopic observation system 55 using the second illumination unit 52 is taken by the imaging device 55c.

  The image processing apparatus 100 first acquires the low-magnification observation image captured by the imaging device 55c in step S1, and uses the acquired low-magnification observation image as an index such as the code number, observation position, and observation time of the culture vessel 10. Save together with the data in the image storage unit 110.

  In step S2, a blur image obtained by performing a smoothing process on the low-magnification observation image acquired from the imaging device 55c using a Gaussian filter or the like is generated in the image analysis unit 120, and image unevenness correction is performed. Accordingly, a difference image between the input image (low-magnification observation image) and the blurred image is generated.

  In step S3, a dispersion value image in which edges and the like in the image are emphasized is generated by applying a dispersion filter to the unevenness-corrected difference image, and a plurality of images that appear to be objects are extracted.

  In step S4, contour extraction processing such as binarization processing using a predetermined threshold is performed on the extracted image that looks like an object obtained by applying the dispersion filter, and the contour of each object is extracted.

  In step S5, labeling is performed on each object of the binary image from which the contour has been extracted. In the labeling process, an appropriate range (upper and lower limit values) is set as the area of the image of the fertilized egg, and the area is calculated for each object region from which the contour is extracted, and has a region area that exceeds this set range. The object is excluded from the fertilized egg discrimination candidates, and a unique label is assigned only to an object having a region area that falls within the set range.

  Next, the feature amount corresponding to the attribute of the fertilized egg is calculated for each labeled object by the image analysis unit 120, and a fertilized egg candidate object is extracted based on the feature amount (step S6). Examples of the feature amount include an image feature amount that changes significantly in low-magnification observation, that is, a luminance value (luminance average value, luminance dispersion value, etc.) of the contour portion. The image analysis unit 120 records the unique label assigned to each object and the feature amount in the feature amount storage unit 130 in association with each other. Further, the image analysis unit 120 compares the feature amount calculated for each object and recorded in the feature amount storage unit 130 with a preset threshold value, rejects the threshold value, and sets the threshold value. Only objects with labels having satisfactory feature quantities are extracted as fertilized egg candidates.

  The image analysis unit 120 sets the flag of the data table 140 only for the tile area including the fertilized egg candidate object for the tile area obtained by dividing the low-magnification visual field area into the area corresponding to the high-magnification visual field, A high-magnification visual field position for observing a fertilized egg candidate object at a high magnification is specified (step S7).

  The CPU 61 accesses the data table 140 and captures the upper left corner coordinates of the high-magnification field area specified by the flag of the data table 140, and the high-magnification field of the imaging device 55c matches the tile area where the fertilized egg candidate object exists. The sample stage 15 is positioned so that the objective lens of the observation optical system 55a is set for high-magnification observation, and high-magnification observation images are sequentially taken by the imaging device 55c.

  In step S8, the image processing apparatus 100 acquires a high-magnification observation image (high-magnification observation image including a fertilized egg candidate) photographed by the imaging device 55c, and uses the acquired observation image as a code number or an observation of the culture vessel 10. It is stored in the image storage unit 110 together with index data such as position and observation time.

  Subsequently, the image analysis unit 120 applies a high-magnification observation algorithm, and for each object that is a fertilized egg candidate, as feature quantities that change significantly in the high-magnification observation image, for example, the circularity of the contour and the internal texture feature quantity Etc., and a unique label assigned to each object is associated with this feature quantity and recorded in the feature quantity storage unit. The image analysis unit 120 calculates a score based on each feature amount, determines that an object having a label with the highest overall score is a fertilized egg to be observed (step S9), and finally the label. A determination result that the object having the symbol is a fertilized egg is output from the output unit (step S10).

  The determination result output from the output unit 130 is displayed on the display panel 72 of the operation panel 7, and a display indicating a fertilized egg is displayed on an object identified as a fertilized egg in the observation image.

  As a specific method, for example, a symbol indicating that the egg is a fertilized egg (for example, “J”) is added, the fertilized egg is displayed with a different hue or luminance, and a foreign object is displayed. Examples of the interface include discriminating and displaying a fertilized egg and other foreign matters by displaying them in a solid color or by displaying an image from which foreign matters have been removed. In order to transmit the discrimination data output from the output unit 150 to an externally connected computer or the like via the communication unit to display a similar image or observe the growth state of a fertilized egg It can be configured to be used as basic data.

  Thus, the observer refers to the image displayed on the display panel 72 and the image displayed on a monitor such as an externally connected computer, so that each of the observers who are observing (or that has already finished obtaining the observed image) It is possible to immediately determine whether or not the object included in the image is a fertilized egg. In addition, by using the data in which the fertilized egg and other foreign matters are discriminated in this way, the growth state of the fertilized egg can be efficiently observed.

  As described above, according to the image processing program GP of the present embodiment, the image processing method for fertilized egg observation configured by executing the image processing program GP, and the image processing apparatus 100, fertilization that is an observation target. It is possible to provide an image processing means capable of detecting an egg at high speed and robustly.

BS culture observation system GP image processing program a fertilized egg 5 observation unit 6 control unit 7 operation panel 54 macro observation system 54c imaging device 55 micro observation system 55c imaging device 61 CPU 62 ROM
63 RAM 100 Image Processing Device 120 Image Analysis Unit 140 Output Unit

Claims (9)

Acquire a low-magnification image obtained by photographing multiple objects located in the low-magnification observation field with an imaging device,
Extracting the plurality of objects imprinted in the low-magnification image;
For each of the plurality of objects included in the low-magnification image, a feature amount of the low-magnification image corresponding to an attribute of a fertilized egg is calculated, and among the plurality of objects based on the feature amount of the low-magnification image Extract fertilized egg candidates from
By dividing the region in the low-magnification observation field into a plurality of small regions according to the size of the high-magnification observation field, only the small region including the object selected as the fertilized egg candidate is sequentially used by the imaging device. Take multiple high-resolution images by shooting,
For each object of the fertilized egg candidate included in the plurality of high-magnification images, calculate a feature amount of the high-magnification image according to an attribute of the fertilized egg, and determine the fertilized egg candidate based on the feature amount of the high-magnification image An image processing method for fertilized egg observation characterized by identifying a fertilized egg from an object. The feature amount of the low-magnification image is a feature amount of an image in which a remarkable change appears in low-magnification observation,
The fertilized egg observation image processing method according to claim 1, wherein the feature amount of the high-magnification image is a feature amount of an image that significantly changes during high-magnification observation.   For the feature quantity of the same target, the difference value between the feature quantity calculation value calculated based on the low-magnification image and the feature quantity calculation value calculated based on the high-magnification image is used as the feature quantity of the high-magnification image. The image processing method for fertilized egg observation according to claim 1. An image processing program that can be read by a computer and that causes the computer to function as an image processing device that acquires an image captured by an imaging device and performs image processing,
Acquiring a low-magnification image obtained by photographing a plurality of objects located in the low-magnification observation visual field by the imaging device;
Extracting the plurality of objects imprinted in the low-magnification image;
For each of the plurality of objects included in the low-magnification image, a feature amount of the low-magnification image corresponding to an attribute of a fertilized egg is calculated, and among the plurality of objects based on the feature amount of the low-magnification image Extracting fertilized egg candidates from
By dividing the region in the low-magnification observation field into a plurality of small regions according to the size of the high-magnification observation field, only the small region including the object selected as the fertilized egg candidate is sequentially used by the imaging device. Taking multiple high-magnification images by shooting,
For each object of the fertilized egg candidate included in the plurality of high-magnification images, calculate a feature amount of the high-magnification image according to an attribute of the fertilized egg, and determine the fertilized egg candidate based on the feature amount of the high-magnification image Identifying a fertilized egg from within the object;
An image processing program for observing a fertilized egg, wherein the computer realizes the step of outputting the identification result for the object. The feature amount of the low-magnification image is a feature amount of an image in which a remarkable change appears in low-magnification observation,
The image processing program for fertilized egg observation according to claim 4, wherein the feature amount of the high-magnification image is a feature amount of an image in which a significant change occurs during high-magnification observation.   For the feature quantity of the same target, the difference value between the feature quantity calculation value calculated based on the low-magnification image and the feature quantity calculation value calculated based on the high-magnification image is used as the feature quantity of the high-magnification image. The image processing program for fertilized egg observation according to claim 4. An imaging device capable of photographing a plurality of objects at a low magnification and a high magnification;
Extracting the plurality of objects from the low-magnification image captured by the imaging device, an image analysis unit for identifying a fertilized egg from the plurality of objects,
An output unit for outputting the identification result determined by the image analysis unit to the outside,
The image analysis unit
For each of the plurality of objects included in the low-magnification image, calculate a feature amount of the low-magnification image according to an attribute of a fertilized egg, and select from the plurality of objects based on the feature amount of the low-magnification image While extracting fertilized egg candidates,
By dividing the region in the low-magnification observation field into a plurality of small regions according to the size of the high-magnification observation field, only the small region including the object selected as the fertilized egg candidate is sequentially used by the imaging device. Take multiple high-resolution images by shooting,
For each object of the fertilized egg candidate included in the plurality of high-magnification images, calculate a feature amount of the high-magnification image according to an attribute of the fertilized egg, and determine the fertilized egg candidate based on the feature amount of the high-magnification image An image processing apparatus for observing a fertilized egg, characterized in that the fertilized egg is identified from an object. The feature amount of the low-magnification image is a feature amount of an image in which a remarkable change appears in low-magnification observation,
The image processing apparatus for fertilized egg observation according to claim 7, wherein the feature amount of the high-magnification image is a feature amount of an image in which a significant change occurs during high-magnification observation.   For the feature quantity of the same target, the difference value between the feature quantity calculation value calculated based on the low-magnification image and the feature quantity calculation value calculated based on the high-magnification image is used as the feature quantity of the high-magnification image. The image processing apparatus for fertilized egg observation according to claim 7.