WO2011004568A1

WO2011004568A1 - Image processing method for observation of fertilized eggs, image processing program, image processing device, and method for producing fertilized eggs

Info

Publication number: WO2011004568A1
Application number: PCT/JP2010/004328
Authority: WO
Inventors: 三村正文; 佐々木秀貴; 伊藤啓
Original assignee: 株式会社ニコン
Priority date: 2009-07-08
Filing date: 2010-07-01
Publication date: 2011-01-13
Also published as: JPWO2011004568A1; TW201102040A

Abstract

Disclosed is a means for automatically recognizing fertilized eggs to be observed by distinguishing fertilized eggs from contaminants during observation of fertilized eggs. A computer executes the following steps: a step (S1) of obtaining an observation image depicting a plurality of objects positioned within a field of view from an imaging device; a step (S2) of extracting the plurality of objects that are depicted in the observation image; steps (S11, S21, S31) of calculating a plurality of feature values, corresponding to the attributes of fertilized eggs, in the image for every object included in the observation image; a step (S8) of identifying fertilized eggs among the plurality of objects on the basis of the calculated feature values; and a step (9) of outputting the identification results for the objects.

Description

Fertilized egg observation image processing method, image processing program, image processing apparatus, and fertilized egg manufacturing method

The present invention relates to an image processing means for automatically discriminating a fertilized egg and a foreign substance from an observation image acquired in fertilized egg observation, and a fertilized egg manufacturing method using the image processing means.

In recent years, with the development of assisted reproduction technology (ART), it has been observed to observe the growth state of cultured fertilized eggs by in vitro fertilization. A culture microscope is mentioned as an example of the apparatus which observes the conditions of cultures, such as a fertilized egg (for example, refer patent document 1). The culture microscope includes a culture apparatus (invecutor) that forms a suitable environment for culturing fertilized eggs, and a microscopic observation system that microscopically observes the state of the fertilized eggs in the culture container housed in the culture apparatus. The observation image of the fertilized egg is acquired at regular intervals, and the user can recognize the fertilized egg by visual observation and automatically perform observation, recording, management, etc. of the fertilized egg. The

JP 2004-229619 A

In such an apparatus, the culture medium in the culture container contains foreign matters such as dust and bubbles in addition to the fertilized egg to be observed. Since these foreign objects may have an appearance similar to a fertilized egg, it is easy to confuse the fertilized egg with the foreign object in the observation image, and it is difficult to automatically recognize the fertilized egg with simple image processing.

The present invention has been made in view of the problems as described above, and provides a means for automatically discriminating a fertilized egg by discriminating a fertilized egg to be observed and other foreign matters in fertilized egg observation. Objective.

According to the first aspect exemplifying the present invention, an observation image obtained by photographing a plurality of objects located in the observation visual field by the imaging device is acquired, and a plurality of objects imprinted in the observation image are extracted, and the observation image is extracted. Calculating a plurality of feature values of an image according to the attributes of a fertilized egg for each object included in the object, and identifying a fertilized egg from the plurality of objects based on the calculated plurality of feature values An image processing method for fertilized egg observation is provided.

According to a second aspect exemplifying the present invention, there is provided an image processing program for causing a computer to function as an image processing device that can be read by a computer and that is captured by an imaging device and acquires an image and performs image processing. A step of acquiring an observation image obtained by photographing a plurality of objects located in a field of view by an imaging device, a step of extracting a plurality of objects imprinted in the observation image, and a fertilized egg for each object included in the observation image Calculating a plurality of image feature amounts according to the attributes of the image, identifying a fertilized egg from the plurality of objects based on the calculated feature amounts, and outputting an identification result for the object An image processing program for observing a fertilized egg is provided.

According to the third aspect exemplifying the present invention, an imaging device that captures a plurality of objects, a plurality of objects are extracted from observation images captured by the imaging device, and a fertilized egg is identified from the plurality of objects. An image analysis unit, and an output unit that outputs an identification result determined by the image analysis unit to the outside, and the image analysis unit determines the feature amount of the image according to the attribute of the fertilized egg for each object included in the observation image A fertilized egg observation image processing apparatus is provided, which is configured to identify a fertilized egg from a plurality of objects based on a plurality of calculated feature values.

According to the fourth aspect illustrating the present invention, the fertilized egg is cultured under predetermined environmental conditions, and the fertilized egg is identified from the culture container in which the fertilized egg exists using the image processing apparatus having the above-described configuration. A featured fertilized egg production method is provided.

According to the fifth aspect illustrating the present invention, a fertilized egg is cultured under a predetermined environmental condition, and an observation in which a plurality of objects located in an observation field are photographed with an imaging device in a culture container in which the fertilized egg exists. Acquire an image, extract a plurality of objects imprinted in the observation image, calculate a plurality of image feature amounts according to the attributes of the fertilized egg for each object included in the observation image, and calculate the plurality of features There is provided a method for producing a fertilized egg characterized by identifying a fertilized egg from a plurality of objects in a culture container based on the amount.

According to the fertilized egg observation image processing method, the image processing program and the image processing apparatus, and the fertilized egg manufacturing method, by image processing that discriminates an object based on a plurality of feature amounts according to the attributes of the fertilized egg. The fertilized egg can be accurately identified from among a plurality of objects included in the observation image.

It is a flowchart which shows the outline | summary of an image processing program. It is a general | schematic block diagram of the culture observation system shown as an example of application of this invention. It is a block diagram of the said culture observation system. (A) is a top view which shows a culture container, (B) is a perspective view which shows a dish. It is a figure for demonstrating the characteristic of a fertilized egg and other foreign materials. It is a figure which illustrates the condition of the outline extraction process which extracts an object. It is a figure for demonstrating the difference of the luminance value inside the outline of a fertilized egg and a foreign material. It is the figure which illustrated the condition of the mask production | generation process when calculating the average luminance value of an outer ring zone and an inner ring zone. It is a figure for demonstrating the contraction mask area | region in an observation image. 1 is a block diagram illustrating a schematic configuration of an image processing apparatus. It is a flowchart which shows the outline | summary of the manufacturing method of a fertilized egg.

Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings. As an example of a system to which the image processing apparatus according to the present embodiment is applied, a schematic configuration diagram and a block diagram of a culture observation system are shown in FIGS. 2 and 3, respectively.

The culture observation system BS roughly observes the culture chamber 2 provided in the upper part of the housing 1, the shelf-like stocker 3 that accommodates and holds a plurality of culture containers 10, and the sample in the culture container 10. An observation unit 5, a transfer unit 4 for transferring the culture vessel 10 between the stocker 3 and the observation unit 5, a control unit 6 for comprehensively controlling the operation of the system, and an operation panel 7 provided with an image display device. Etc.

The culture room 2 is a room that forms a culture environment, and is kept sealed after the sample is charged in order to prevent environmental changes and contamination. Along with the culture chamber 2, a temperature adjustment device 21 that raises and lowers the temperature in the culture chamber 2, a humidifier 22 that adjusts humidity, and a gas supply device 23 that supplies a gas such as CO ₂ gas or N ₂ gas. A circulation fan 24 for making the entire environment of the culture chamber 2 uniform, an environmental sensor 25 for detecting the temperature, humidity, carbon dioxide concentration, etc. of the culture chamber 2 are provided. The operation of each device is controlled by the control unit 6, and the culture environment defined by the temperature, humidity, carbon dioxide concentration, etc. of the culture chamber 2 is maintained in a state that matches the culture conditions set on the operation panel 7.

The stocker 3 is formed in a shelf shape that is partitioned into a plurality of parts in the front-rear direction and the up-down direction in FIG. Each shelf has its own unique address. For example, when the longitudinal direction is A to C rows and the vertical direction is 1 to 7 rows, the A row 5 shelves are set as A-5. The

As the culture vessel 10, an appropriate one such as a flask, a dish, or a well plate is selected according to the type and purpose of the culture. In this embodiment, as shown in FIG. The structure provided with the dish 10a and the holder 10b which hold | maintains the dish 10a is illustrated, and as shown to FIG. 4 (B), the fertilized egg a which is a culture is a culture medium containing pH indicators, such as phenol red It is injected into each dish 10a together with the drop D. On the bottom surface of the dish 10a, one or more medium drops D of about 20 μl dropped by a pipette or the like are formed (only one is shown in FIG. 4B), and the medium drop D is contained in the dish 10a. It is in a state immersed in colorless and transparent mineral oil O. In each medium drop D, for example, one fertilized egg a collected from the same mother at the same time for external fertilization is inserted. The culture vessel 10 is assigned a code number and is stored in association with the designated address of the stocker 3.

The transfer unit 4 is provided inside the culture chamber 2 so as to be movable in the vertical direction and is moved up and down by the Z-axis drive mechanism. The transfer unit 4 is attached to the Z stage 41 so as to be movable in the front-rear direction and by the Y-axis drive mechanism. The Y stage 42 that is moved back and forth, the X stage 43 that is attached to the Y stage 42 so as to be movable in the left-right direction and is moved left and right by the X-axis drive mechanism, and the like, is supported by lifting the culture vessel 10 to the tip side of the X stage 43 A support arm 45 is provided. The transport unit 4 has a moving range in which the support arm 45 can move between the entire shelf of the stocker 3 and the observation unit 5. The X-axis drive mechanism, the Y-axis drive mechanism, and the Z-axis drive mechanism are configured by, for example, a servo motor with a ball screw and an encoder, and the operation thereof is controlled by the control unit 6.

The observation unit 5 includes a first illumination unit 51 that illuminates the sample from the lower side of the sample stage 15, a second illumination unit 52 that illuminates the sample from above the sample stage 15 along the optical axis of the microscopic observation system, and a sample from below. 3, a macro observation system 54 that performs macro observation of the sample, a micro observation system 55 that performs micro observation of the sample, an image processing apparatus 100 (see FIG. 10), and the like. The sample stage 15 is made of a light-transmitting material and has a transparent window 16 in the observation area. The sample stage 15 is composed of a fine drive stage that can be moved in the XY direction (horizontal plane direction) and the Z direction (vertical direction) by operation control from the control unit 6, and the culture vessel 10 placed on the upper surface thereof. Is moved in the XY directions, so that the culture vessel 10 can be inserted on the optical axis of the macro observation system 54 or on the optical axis of the microscopic observation system 55.

The first illumination unit 51 is composed of a surface-emitting light source provided on the lower frame 1b side, and backlight-illuminates the entire culture vessel 10 from the lower side of the sample stage 15. The second illumination unit 52 includes a light source such as an LED and an illumination optical system including a phase ring, a condenser lens, and the like. The second illumination unit 52 is provided in the culture chamber 2 and receives light from the microscopic observation system 55 from above the sample stage 15. The sample in the culture vessel 10 is illuminated along the axis. The third illumination unit 53 superimposes light emitted from each of the light sources, such as a plurality of LEDs and mercury that emit light having a wavelength suitable for epi-illumination observation and fluorescence observation, on the optical axis of the microscopic observation system 55. An illumination optical system composed of a beam splitter, a fluorescent filter, and the like to be disposed in the lower frame 1b located below the culture chamber 2, and the light of the microscopic observation system 55 from below the sample stage 15 The sample in the culture vessel 10 is illuminated along the axis.

The macro observation system 54 includes an observation optical system 54 a and an imaging device 54 c such as a CCD camera that captures an image of the sample imaged by the observation optical system 54 a and is positioned above the first illumination unit 51. Provided in the culture chamber 2. The macro observation system 54 captures an entire observation image (macro image) from above the culture vessel 10 that is backlit by the first illumination unit 51.

The microscopic observation system 55 includes an observation optical system 55a composed of an objective lens, an intermediate zoom lens, a fluorescent filter, and the like, and an imaging device 55c such as a cooled CCD camera that takes an image of a sample imaged by the observation optical system 55a. And disposed inside the lower frame 1b. The second illumination unit 52 and the microscopic observation system 55 constitute a phase difference observation microscope. A plurality of objective lenses and intermediate zoom lenses are provided, and are configured to be set to a plurality of magnifications using a displacement mechanism such as a revolver or a slider (not shown in detail). In the present embodiment, at least two magnifications for low-magnification observation (for example, for double-magnification observation) and high-magnification observation (for example for ten-times observation) are switched so as to be variable. The microscopic observation system 55 displays the sample in the culture vessel 10 such as a phase difference image by the transmitted light of the sample illuminated by the second illumination unit 52 and a fluorescence image by the fluorescence emitted from the sample illuminated by the third illumination unit 53. A microscopic observation image (micro image) is taken.

The image processing apparatus 100 performs A / D conversion on signals input from the imaging device 54c of the macro observation system 54 and the imaging device 55c of the microscopic observation system 55, and performs various image processing to perform an entire observation image or a microscopic observation image. Image data is generated. Further, the image processing apparatus 100 performs image analysis on the image data of these observation images (entire observation image and microscopic observation image), calculates the feature amount of an object present in the image, and responds to each feature amount. Image processing such as calculating the score and determining the fertilized egg based on the total score. Specifically, the image processing apparatus 100 is constructed by executing an image processing program stored in the ROM of the control unit 6 described below. The image processing apparatus 100 will be described in detail later.

The control unit 6 includes a CPU 61 that executes processing, a ROM 62 that stores and stores control programs and control data of the culture observation system BS, a RAM 63 that temporarily stores observation conditions and image data, and the like. Control operation. Therefore, as shown in FIG. 3, the constituent devices of the culture chamber 2, the transport device 4, the observation unit 5, and the operation panel 7 are connected to the control unit 6. In the RAM 63, the environmental conditions of the culture chamber 2 according to the observation program, the observation schedule, the observation type and observation position in the observation unit 5, the observation magnification, and the like are set and stored. Further, the RAM 63 is provided with an image data storage area for recording image data photographed by the observation unit 5, and index data including the code number of the culture vessel 10 and photographing date / time are associated with the image data and recorded. Is done.

The operation panel 7 is provided with an operation panel 71 provided with input / output devices such as a keyboard and a switch, and a display panel 72 for displaying an operation screen, image data, and the like. An operation command or the like is input. The communication unit 65 is configured in accordance with a wired or wireless communication standard, and data can be transmitted to and received from a computer or the like externally connected to the communication unit 65.

In the culture observation system BS schematically configured as described above, the CPU 61 controls the operation of each part based on the control program stored in the ROM 62 according to the setting conditions of the observation program set on the operation panel 7, and the culture vessel 10 The sample inside is automatically captured. That is, when the observation program is started by a panel operation on the operation panel 71 (or a remote operation via the communication unit 65), the CPU 61 reads each condition value of the environmental conditions stored in the RAM 63, and from the environment sensor 25. The environmental state of the culture chamber 2 to be input is detected, and the temperature adjustment device 21, the humidifier 22, the gas supply device 23, the circulation fan 24, etc. are operated according to the difference between the condition value and the actual measurement value. Feedback control is performed on the culture environment such as temperature, humidity, and carbon dioxide concentration.

Further, the CPU 61 reads the observation conditions stored in the RAM 63 and operates the X, Y, Z stages 41, 42, 43 of the transport unit 4 based on the observation schedule to observe the culture vessel 10 to be observed from the stocker 3. The sample is transported to the sample stage 15 of the unit 5 and observation by the observation unit 5 is started. For example, when the observation set in the observation program is macro observation, the culture vessel 10 transported from the stocker 3 by the transport unit 4 is positioned on the optical axis of the macro observation system 54 and placed on the sample stage 15. Then, the light source of the first illumination unit 51 is turned on, and the entire observation image is taken by the imaging device 54c from above the culture vessel 10 that is backlit. The signal input from the imaging device 54c to the control unit 6 is processed by the image processing device 100 to generate a whole observation image, and the image data is stored in the image data storage area of the RAM 63 together with index data such as the shooting date and time. Is done.

In addition, when the observation set in the observation program is micro observation of the sample at a specific position in the culture vessel 10, the specific position in the culture vessel 10 that has been transported by the transport unit 4 is indicated by the light of the microscopic observation system 55. Position on the axis and place on the sample stage 15, turn on the light source of the second illumination unit 52 or the third illumination unit 53, and cause the imaging device 55c to take a microscopic observation image by transmitted illumination, epi-illumination, and fluorescence. . A signal photographed by the imaging device 55c and inputted to the control unit 6 is processed by the image processing device 100 to generate a microscopic observation image (phase difference image, fluorescent image, etc.), and the image data is an index such as a photographing date / time. Stored in the image data storage area of the RAM 63 together with the data.

The CPU 61 sequentially executes the above-described whole observation image photographing and microscopic observation image photographing based on the observation schedule set in the observation program. The image data stored in the RAM 63 is read from the RAM 63 in response to an image display command input from the operation panel 71. For example, an entire observation image, a microscopic observation image at a specified time, an analysis result of image analysis, and the like are displayed on the display panel 72. Is displayed.

In the culture observation system BS configured as described above, the image processing apparatus 100 performs observation of the growth state and the like of the fertilized egg in the culture container 10, so that the fertilized egg to be observed and other garbage or the like It has a function of discriminating foreign substances (asymmetrical substances) such as bubbles. That is, the medium drop D in the culture container 10 is mixed with other objects such as dust and bubbles in addition to the fertilized egg, which may have a similar appearance to the fertilized egg and are confused in observation. Since it is easy, the image processing apparatus 100 has a function of automatically discriminating a fertilized egg to be observed by discriminating a fertilized egg from other foreign substances.

Here, the contaminants in the medium drop D are roughly classified into garbage (remaining residue of the medium), oil particles (the mineral oil O filled in the dish 10a enters the medium), and bubbles (gas). The At this time, as shown in FIG. 5, each of the fertilized egg, dust, oil particles, and bubbles has the following characteristics.

(Characteristics of fertilized eggs)
A fertilized egg has a feature that a contour (a zona pellucida) is present in the contour, the outer shape is spherical, and an egg cell is present in the contour.

(Characteristics of garbage)
The dust has the characteristics that there is no transparent band in the outline, halo appears in the outline in the phase difference image, and the structure inside the outline is indefinite.

(Features of oil particles and bubbles)
In the phase difference image, the oil particles and bubbles have the characteristics that the contour portion is dark and the inside is bright, the outer shape is spherical or elliptical, and the structure inside the contour is small.

To summarize these characteristics, the fertilized egg has a higher degree of circularity than the foreign object, the fertilized egg has a zona pellucida in the contour, the internal structure of the fertilized egg (Egg cells) are always present.

Therefore, in order to recognize a fertilized egg as an observation target from the acquired observation image, three feature amounts (that is, I. Circularity of contour, II. Outside and inside of contour, which will be described in detail below) are used as image feature amounts. Luminance difference, III. Texture feature amount in the contour) is used for discrimination between fertilized eggs and foreign objects.

The image processing method for observing a fertilized egg by the image processing apparatus 100 acquires images obtained by photographing an object located in the observation field of view of the observation unit 5 with an imaging device, and images of the objects (objects) imprinted in these images. As a result of calculating the above three feature quantities and scoring (score calculation) based on these feature quantities, it is determined that the fertilized egg is the one that has the highest score among these three feature quantities. Configured. In the following, this image processing method will be described from the basic concept. In the following description, a case where a fertilized egg is observed based on a phase contrast image (microscopic observation image) photographed by a phase contrast microscope configured by the second illumination unit 52, the microscopic observation system 55, and the like will be exemplified. .

(Extract objects)
Prior to discrimination processing of fertilized eggs, an object (fertilized egg, dust, etc.) included in the image by applying a contour extraction method such as the LevelSet method to the observation image (phase difference image) taken by the imaging device 55c. The outermost contour of the foreign object (also referred to as object) is extracted, and the closed region surrounded by the contour is extracted as the object. FIG. 6 is a diagram showing a state of the contour extraction process. A plurality of objects are extracted from the observation image (A) photographed and acquired by the imaging device 55c as in the binary image shown in (B).

(Object labeling)
Each segmented object (binary image) is labeled to give a unique label. In the labeling process, the area of the object (area surrounded by the extracted contour) is calculated together with labeling. Therefore, the size of an image of a fertilized egg (for example, a diameter of about 100 μm in the case of a human fertilized egg) is almost determined by the image acquisition conditions (observation magnification, etc.), and is therefore subject to discrimination. An upper limit value and a lower limit value of a possible area of the object are set in advance, and an object outside the set range is excluded from a labeling target (fertilized egg discrimination candidate).

(Extraction of object features)
Each object that has been labeled will be described in detail below. Contour circularity, II. Brightness difference between outside and inside of the contour, III. The three feature amounts of the texture feature amount inside the contour are calculated.

1. Contour circularity (feature 1)
Utilizing the fact that the fertilized egg to be observed has a spherical shape (the cross section is close to a perfect circle), the circularity of each object is used as the feature amount 1. The degree of circularity of an object is a scale for determining the degree of circularity of an object. For example, after determining the center of gravity of each object in the image plane and detecting the edge indicating the outline of each object, Then, the shortest distance and the longest distance from the center of gravity to the contour are calculated and obtained from the following equation (α).
Contour circularity = (shortest distance from centroid to contour) / (longest distance from centroid to contour)… (α)
The circularity obtained in this way is calculated as a numerical value within the range of 0 to 1, and indicates a numerical value closer to 1 as the distance from the center of gravity to the contour becomes a uniform circular shape. That is, an object having a circularity close to 1 has a higher score as a fertilized egg (highly likely to be a fertilized egg). In addition, as another index indicating the degree of circularity of the contour, the complexity of the shape (contour circumference ² / area), the eccentricity using the second moment, and the like are exemplified.

2. Brightness difference between outside and inside contour (feature value 2)
A transparent thin film called a zona pellucida exists in the outline of the fertilized egg to be observed. In the observation image by the microscopic observation system 55 shown in FIG. 7, since the image of the zona pellucida has a luminance value close to the background such as the medium drop D, the difference in luminance value between the outside and inside of the contour is small. On the other hand, the inside of the outline of oil particles and bubbles is darker than the background outside, and in the phase contrast image, the inside of the outline of dust tends to be brightened by halo, so the difference in brightness value between the outside and inside of the outline is large. Therefore, using the characteristics of the zona pellucida, as the feature amount 2, an average luminance value (for example, 8-bit gradation) between the outer ring zone (outer ring zone) and the inner ring zone (inner ring zone) of the contour in the object is used. In this case, the difference in brightness is obtained by the following equation (β) using the difference of 0 to 255).
Brightness difference between outside and inside contour = (255− | inner annular average luminance value−outer annular average luminance value |) / 255 (β)

FIG. 8 is a diagram illustrating the state of mask generation processing when calculating the average luminance value of the outer annular zone and the inner annular zone. When calculating these luminance values, the images (labeling) obtained by the above-described processing are illustrated. A binary image) is used. First, the binary image (label image) of each object is subjected to expansion and contraction processing, and two types of mask images (expansion mask M1 and contraction mask M2) showing the expanded and contracted regions are prepared for each object. To do. Next, for each object, the difference between the areas of the two mask images M1 and M2 and the original original label image is calculated in a state where the centers of gravity coincide with each other, and these substantially ring-shaped difference images are used as the outline of the object. On the other hand, they are generated as an outer difference mask M3 and an inner difference mask M4 indicating outer and inner regions. The average of the luminance values in the region of the observation image (the phase difference image captured by the imaging device 55c) corresponding to the regions of the two difference masks M3 and M4 is calculated, and these are calculated as the outer annular average luminance value and the inner annular zone. The average luminance value. By substituting these into the above equation (β), the brightness difference between the outside and inside of the contour is obtained as the feature amount 2. An object with a smaller difference between the inner ring zone average brightness value and the outer zone average brightness value has a numerical value closer to 1, and it can be said that the score as a fertilized egg is higher (highly likely to be a fertilized egg).

3. Texture feature amount inside contour (feature amount 3)
Egg cells are contained inside the fertilized egg to be observed, and it is considered that a texture (internal texture) exists on the image. On the other hand, oil grains and bubbles do not have an internal structure. Therefore, by detecting the internal structure of each object, it is possible to differentiate a fertilized egg from oil particles and bubbles. As for the region of the object for detecting the internal structure, as shown in FIG. 9, using the contraction mask M2 generated by the above processing, an image region corresponding to its contraction mask M2 in the observation image (hereinafter, referred to as “image region”). Also referred to as “shrink mask area”. As the texture feature amount inside the object, an average value of edge strength (gradation change strength) in the entire image in the contraction mask region is used (see the following equation (γ)). The edge strength is obtained on a pixel basis by a convolution operation in which a differential filter such as a Laplacian filter represented by, for example, a 3 × 3 matrix or a 5 × 5 matrix is applied to the image of the contraction mask region of each object.
Texture feature amount inside contour = total edge intensity in shrink mask area / number of pixels in shrink mask area (γ)
Here, as the edge strength, only the edge strength having a positive value is used in consideration of the direction of luminance gradation change, and the edge strength having a negative value is treated as an invalid edge (edge strength = 0). In addition, the number of pixels included in the contraction mask region is used to obtain the average edge intensity. Then, an average value obtained by calculating the edge strength in the contraction mask area for each object in units of pixels in this way is obtained as the texture feature quantity inside the object, which is the feature quantity 3 by the number of pixels in the contraction mask area. .

The above three feature values are given for each labeled object, and the fertilized egg is determined based on this.

(Determination of fertilized eggs)
First, (I) an object having clearly different characteristics from a fertilized egg is rejected with a threshold value. Next, (II) normalizing the three feature values in consideration of differences in units and scaling of each feature value, and summing the

scores

1, 2, and 3 obtained by normalization, the total score is used for each object. Scoring for. Since one fertilized egg is injected for one medium drop D, it is determined that the object having the highest overall score is a fertilized egg. This specific method is exemplified below.

(I: Reject by threshold)
Features of objects that are not fertilized eggs tend to be as follows. For oil particles and bubbles, the feature amount 2 (internal / external luminance difference) and the feature amount 3 (internal texture) tend to be small. As for dust, feature amount 1 (circularity) and feature amount 2 (internal / external luminance difference) tend to be small. Therefore, a threshold value is set in advance, and an object having a feature quantity that is clearly deviated is rejected by this threshold value to prevent erroneous detection due to scoring (score). At this time, with respect to the texture feature amount inside the outline of the feature amount 3, the value (average value of edge strength) itself is not used, but the ratio (edge strength) to the maximum value of the average value of edge strength in all labeled objects. Average value / maximum value of edge strength average value) is used. This is because, for the feature amount 3, the edge strength changes extremely depending on the contrast of the image, so that it is meaningless to provide a certain threshold value. Normalize between images of all objects to remove. Further, when there is no internal texture (for example, when only bubbles are captured), the edge strength originally appears to be small as a whole, but by performing normalization, the feature amount 3 is large for all objects. Since there is a risk of becoming too much, a permissible value is set in advance, and when the maximum value of the edge strength average value falls below this permissible value, all of the feature amount 3 is rejected.

(II: Feature scale normalization)
Scale normalization is performed on the feature quantities 1 to 3, and each feature quantity is converted to a common scale (for example, a scale of 0 to 1). Specifically, the following upper and lower limits are set for each feature amount,
Feature amount 1 (circularity): lower limit = threshold of feature amount 1, upper limit = maximum value of feature amount 1 Feature amount 2 (internal / external luminance difference): lower limit = threshold value of feature amount 2, upper limit = feature amount 2 Feature value 3 (internal texture): lower limit = 0, upper limit = maximum value of feature value 3 The range within the upper and lower limits thereof is normalized as a full scale. Since the feature amount 3 has a characteristic that it depends on the contrast of the observation image, it is difficult to provide a threshold value. Therefore, the lower limit value is set to 0, and the upper limit value is less than the above allowable value. Is set to 0, and scoring of the feature amount 3 is not performed.

Then, the feature quantities normalized and converted to a common scale are set as

scores

1, 2, and 3, and a total score that is the sum of them is calculated.
Overall score = Score 1 + Score 2 + Score 3
As a result, the object having the maximum total score in the medium drop D is determined as a fertilized egg.

According to the method for discriminating a fertilized egg as described above, even when there are a plurality of objects other than the fertilized egg to be observed in the observation image, a plurality of pieces for distinguishing the fertilized egg from other objects Using this feature amount, a score indicating the degree of fertility is calculated based on the feature amount, and the fertilized egg can be recognized clearly by separating the fertilized egg from the foreign substance based on the height of this score. .

Regarding the method of determining a fertilized egg based on each score, in addition to ranking based on the total score as a simple sum of the scores 1 to 3, for example, a total score obtained by changing the weights for the scores 1 to 3 respectively. Calculate and determine the fertilized egg, rank each score 1 to 3 and judge that the object with the lowest total rank is a fertilized egg, Among them, a method of determining that the highest score is 3 is a fertilized egg. Moreover, the structure which recognizes that a total score exceeded the threshold value as a fertilized egg may be sufficient.

(application)
Next, a specific application of image analysis executed in the image processing apparatus 100 of the culture observation system BS will be described with reference to FIGS. Here, FIG. 1 is a flowchart showing an outline of processing in the image processing program GP for fertilized egg discrimination, and FIG. 10 is a block diagram showing a schematic configuration of an image processing apparatus 100 that executes image processing for fertilized egg discrimination.

The image processing apparatus 100 includes an image storage unit 110 that acquires and stores an observation image in which a fertilized egg to be observed is captured by the imaging device 55c, and an object that analyzes the observation image and is imprinted on the observation image. An image analysis unit 120 that determines whether or not the image analysis unit 120, a feature amount storage unit 130 that stores the three feature amounts calculated for each object by the image analysis unit 120 in association with labels assigned to the objects, An output unit 140 that outputs the determination result analyzed by the image analysis unit 120 to the outside, and outputs, for example, to the display panel 72 the identification result as to whether or not the object determined by the image analysis unit 120 is a fertilized egg Configured to be displayed. The image processing apparatus 100 is configured such that an image processing program GP preset and stored in the ROM 62 is read by the CPU 61 and processing based on the image processing program GP is sequentially executed by the CPU 61.

As described above, in the culture observation system BS, the fertilized egg in the culture vessel 10 designated every predetermined time is observed according to the observation conditions set in the observation program. Specifically, the CPU 61 operates each stage of the transport unit 4 to transport the culture vessel 10 to be observed from the stocker 3 to the observation unit 5 (in this embodiment, it is arranged on the optical axis of the microscopic observation system 55). Then, an observation image (phase difference image) by the microscopic observation system 55 using the second illumination unit 52 is photographed by the imaging device 55c.

The image processing apparatus 100 first acquires an observation image (phase difference image) taken by the imaging device 55c in step S1, and acquires the acquired observation image such as the code number, the observation position, and the observation time of the culture vessel 10. It is stored in the image storage unit 110 together with the index data.

In step S2, contour extraction processing such as the LevelSet method is performed on the observation image (phase difference image) acquired from the imaging device 55c in the image analysis unit 120, and as shown in FIG. Objects included in are extracted.

In step S3, each object of the observation image (binary image) from which the contour is extracted is labeled. In the labeling process, an appropriate range (upper and lower limit values) is set as the area of the image of the fertilized egg, and the area is calculated for each object region from which the contour is extracted, and has a region area that exceeds this set range. Objects are excluded from the fertilized egg discrimination candidates, and only objects having a region area that falls within the set range are given unique labels as fertilized egg discrimination candidates.

Next, the image analysis unit 120 calculates the circularity of the contour, the brightness difference between the outside and inside of the contour, and the texture feature amount inside the contour as the image feature amount of the fertilized egg (step S4). The image analysis unit 120 records the unique label assigned to each object and the three feature amounts calculated below in the feature amount storage unit 130 in association with each other.

In the processing flow F10 for calculating the circularity (feature value 1) of the contour, the image analysis unit 120 calculates the center of gravity of the object from the label image, and the longest distance from the center of gravity to the contour is the distance from the center of gravity to the contour (edge). The ratio of the shortest distance to is calculated as the circularity of the contour (step S11).

In the processing flow F20 for obtaining the luminance difference (feature value 2) between the outer side and the inner side of the contour, the image analysis unit 120 generates two mask images of the expansion mask M1 and the contraction mask M2 for each object from the label image. Thus, difference masks M3 and M4 are respectively generated from the difference between the two mask images M1 and M2 and the original label image. Then, the average of the luminance values in the regions corresponding to the region of the outer differential mask M3 and the region of the inner differential mask M4 in the observation image captured by the imaging device 55c is obtained, and the contour is calculated using the above equation (β). The brightness difference between the outside and the inside is calculated (step S21).

In the processing flow F30 for obtaining the texture feature amount (feature amount 3) inside the contour, the contraction mask M2 generated in step S21 is used, and a differential filter is applied to the observation image captured by the imaging device 55c, so that the contraction mask M2 is applied. The average edge strength in the corresponding region (shrink mask region) is calculated as the texture feature amount inside the contour (step S31).

The image analysis unit 120 compares each feature quantity with a preset threshold value for each feature quantity calculated for each object and recorded in the feature quantity storage unit 130, and rejects each threshold value. Perform (Steps S12, S22, S32). Only objects having three feature quantities that satisfy the threshold are regarded as fertilized egg candidates. As described above, the texture feature amount (feature amount 3) inside the contour is treated as a ratio to the maximum value of the average luminance in all objects so that the feature amount does not change greatly due to the influence of contrast.

Subsequently, normalization of the scale is performed on each feature amount with a predetermined upper and lower limit range as a full scale (step S5), and each feature amount converted into a common scale is set as

scores

1, 2, and 3. Obtained (step S6). In step S7, a total score composed of the sum of

scores

1, 2, and 3 is calculated for each object, and the total scores are sorted (rearranged in descending order of scores). The object with the label having the maximum total score is recognized as a fertilized egg to be observed (step S8). Finally, a determination result that the object with this label is a fertilized egg is output from the output unit 140. (Step S9).

The determination result output from the output unit 130 is displayed on the display panel 72 of the operation panel 7, and a display indicating a fertilized egg is displayed on the object having the highest overall score in the observation image.

As a specific display method, for example, a symbol indicating that the egg is a fertilized egg (for example, “J”) is added, the fertilized egg is displayed with a different hue or brightness, and a foreign object is displayed. The interface is exemplified such that a fertilized egg and other foreign matters are discriminated and displayed by, for example, painting and displaying an image from which foreign matters have been removed. The discrimination data output from the output unit 140 is transmitted to a computer or the like externally connected via the communication unit 65 to display the same image or observe the growth state of the fertilized egg. For use as basic data.

Thus, the observer refers to the image displayed on the display panel 72 and the image displayed on a monitor such as an externally connected computer, so that each of the observers who are observing (or that has already finished obtaining the observed image) It is possible to immediately determine whether or not the object included in the image is a fertilized egg. In addition, by using the data in which the fertilized egg and other foreign matters are discriminated in this way, the growth state of the fertilized egg can be efficiently observed.

Next, a method for producing a fertilized egg will be briefly described with reference to FIG. First, in step S110, the fertilized egg is injected into the culture container 10 (dish 10a) together with the medium drop D, and the culture container 10 is stored in the culture chamber 2 maintained at environmental conditions suitable for culturing the fertilized egg. Then, fertilized eggs are cultured under the environmental conditions. The environmental conditions are adjusted in the control unit 6 according to the culture environment of the fertilized egg such as the temperature, humidity, and carbon dioxide concentration in the culture chamber 2.

In step S120, as the observation of the fertilized egg in the culture container 10, the above-described image processing steps S1 to S9 (see FIG. 1) are executed, and a fertilized egg is selected from a plurality of objects copied in the observation image. Identify. At this time, one fertilized egg is identified for each medium drop D in the culture vessel 10 (dish 10a).

Subsequently, in step S130, a plurality of fertilized eggs identified for each medium drop D are selected based on a predetermined selection criterion. As a selection standard for a fertilized egg, the grade of the fertilized egg is determined based on the timing of cleavage, the shape of the blastomere, and the like, and a good one satisfying this selection standard is selected. For example, it is performed based on whether or not the timing at which cleavage occurs in all egg cells in the egg is in the same period as having passed through a good growth state. That is, regarding the cleavage of normal fertilized eggs, cells of the same generation divide at almost the same timing, and only cells of the same generation exist in the embryo. On the other hand, regarding the cleavage of an abnormal fertilized egg, even when cells are of the same generation, the division timing is shifted, and cells of different generations are mixed in the embryo.

In step S140, the selected fertilized eggs (good fertilized eggs that have grown to a state called a blastocyst) are collected and stored frozen, for example, in liquid nitrogen at minus 196 ° C. The fertilized egg (blastocyst) is returned to the mother (embryo transfer) at a predetermined cycle. The fertilized eggs to be cultured may be fertilized eggs such as humans, cows, horses, pigs and mice. In addition, the fertilized egg may be stored in a blastocyst state or may be stored in a divided phase (4-cell stage embryo, 8-cell stage embryo).

As described above, according to the image processing program GP of the present embodiment, the image processing method and image processing apparatus 100 for observing a fertilized egg configured by executing the image processing program GP, and the method for manufacturing a fertilized egg For example, it is possible to accurately recognize a fertilized egg that is an observation target from among a plurality of objects copied in the captured observation image.

In the above-described embodiment, the processing method for recognizing a fertilized egg based on the three feature amounts of the circularity of the contour, the luminance difference between the outside and inside of the contour, and the texture feature amount inside the contour is illustrated. The present invention is not limited to this embodiment, and a method for recognizing a fertilized egg based on other feature quantities (such as shape feature quantities and texture feature quantities), and four additional feature quantities are added. Even when applied to a method for recognizing a fertilized egg based on the five feature quantities, the same effect can be obtained. For example, based on an image with a relatively low observation magnification, the brightness value of the outline of the object, the variance value of the brightness inside the object, and the like are used as new feature values, replacing one of the above three feature values. The configuration may be configured as four or five feature amounts in addition to the above three feature amounts.

Further, as described above, in the microscopic observation system 55 of the observation unit 5 exemplified in this embodiment, the observation magnification is configured to be variable according to lens settings such as an objective lens, and is obtained from the observation image. The feature amount includes a feature amount that appears more prominently in a low-magnification image (low-magnification phase difference image) and a feature amount that appears more prominently in a high-magnification image (high-magnification phase difference image). The three feature quantities exemplified in the present embodiment are preferably obtained based on a high-resolution high-magnification image so that numerical values and textures can be detected, and the brightness inside the contour as the new feature quantities described above. The value is preferably obtained based on a low-magnification image in which a luminance change is noticeable. Here, the high-magnification image is an image having an observation magnification of 10 or 20 times, for example, and the low-magnification image is an image having an observation magnification of about 2 times, for example, the size of a fertilized egg, egg cell, embryo, or the like to be observed An appropriate magnification can be used according to the above.

Furthermore, in the above-described embodiment, the configuration for identifying and processing one fertilized egg a injected into the medium drop D is exemplified, but the present invention is not limited to this embodiment, You may comprise so that all the several fertilized eggs a inject | poured may be identified. For example, a threshold value for identifying a fertilized egg is set in advance for the total score, and an object exceeding this threshold value is determined to be a fertilized egg, or the fertilized egg a injected into the medium drop D A configuration may be adopted in which only the number is set in advance, and an object having a higher overall score corresponding to the number is recognized as a fertilized egg.

BS culture observation system GP image processing program a fertilized egg 5 observation unit 6 control unit 7 operation panel 54 macro observation system 54c imaging device 55 micro observation system 55c imaging device 61 CPU 62 ROM
63 RAM 100 Image Processing Device 120 Image Analysis Unit 140 Output Unit

Claims

Acquire an observation image obtained by photographing a plurality of objects located in the observation visual field with an imaging device,
Extracting the plurality of objects imprinted in the observation image;
For each of the objects included in the observed image, calculate a plurality of image feature amounts according to the fertilized egg attributes,
A fertilized egg observation image processing method, wherein a fertilized egg is identified from the plurality of objects based on the plurality of calculated feature quantities.
2. The fertilized egg observation image processing method according to claim 1, wherein the attribute of the fertilized egg relates to a contour portion and a structure in the contour region of the fertilized egg.
The image processing method for fertilized egg observation according to claim 1 or 2, wherein the plurality of feature amounts include an image feature amount indicating a degree of a circular shape of the contour portion of the object.
The image processing method for fertilized egg observation according to claim 1 or 2, wherein the plurality of feature amounts include an image feature amount based on luminance of a band-shaped portion in a contour region of the object.
3. The fertilized egg observation image processing method according to claim 1, wherein the plurality of feature amounts include image feature amounts based on a texture structure in a contour region of the object.
An image processing program for causing a computer to function as an image processing apparatus that is readable by a computer, acquires an image captured by an imaging device, and performs image processing,
Obtaining an observation image obtained by photographing a plurality of objects located within a visual field by the imaging device;
Extracting the plurality of objects imprinted in the observation image;
For each of the objects included in the observation image, calculating a plurality of image feature amounts according to fertilized egg attributes;
Identifying a fertilized egg from the plurality of objects based on the calculated feature quantities;
An image processing program for observing a fertilized egg, wherein the computer realizes the step of outputting the identification result for the object.
The fertilized egg observation image processing program according to claim 6, wherein the attributes of the fertilized egg relate to a structure in a contour portion and a contour region of the fertilized egg.
The image processing program for fertilized egg observation according to claim 6 or 7, wherein the plurality of feature amounts include an image feature amount indicating a degree of a circular shape of a contour portion of the object.
The image processing program for fertilized egg observation according to claim 6 or 7, wherein the plurality of feature amounts include an image feature amount based on luminance of a band-shaped portion in a contour region of the object.
The image processing program for fertilized egg observation according to claim 6 or 7, wherein the plurality of feature amounts include image feature amounts based on a texture structure in a contour region of the object.
An imaging device for photographing a plurality of objects;
Extracting the plurality of objects from the observation image photographed by the imaging device, and an image analysis unit for identifying a fertilized egg from the plurality of objects;
An output unit for outputting the identification result determined by the image analysis unit to the outside,
The image analysis unit calculates a plurality of image feature amounts according to the attributes of the fertilized egg for each object included in the observation image, and based on the calculated feature amounts, the plurality of object features An image processing apparatus for observing a fertilized egg, characterized in that the fertilized egg is identified from the inside.
12. The fertilized egg observation image processing apparatus according to claim 11, wherein the attribute of the fertilized egg relates to a structure in a contour portion and a contour region of the fertilized egg.
The image processing apparatus for fertilized egg observation according to claim 11 or 12, wherein the plurality of feature amounts include an image feature amount indicating a degree of a circular shape of a contour portion of the object.
The image processing apparatus for fertilized egg observation according to claim 11 or 12, wherein the plurality of feature quantities include an image feature quantity based on a luminance of a band-shaped portion in a contour region of the object.
The image processing apparatus for fertilized egg observation according to claim 11 or 12, wherein the plurality of feature amounts include image feature amounts based on a texture structure in a contour region of the object.
Culturing fertilized eggs under the specified environmental conditions,
A method for producing a fertilized egg, wherein the fertilized egg is identified from the culture container in which the fertilized egg exists using the image processing apparatus according to any one of claims 11 to 15.
Culturing fertilized eggs under the specified environmental conditions,
In the culture container in which the fertilized egg exists, obtain an observation image obtained by photographing a plurality of objects located in the observation visual field with an imaging device,
Extracting the plurality of objects imprinted in the observation image;
For each of the objects included in the observed image, calculate a plurality of image feature amounts according to the fertilized egg attributes,
A fertilized egg manufacturing method, wherein a fertilized egg is identified from the plurality of objects in the culture container based on the plurality of calculated feature quantities.
Screening the identified fertilized eggs based on a predetermined screening standard,
18. The method for producing a fertilized egg according to claim 16 or 17, wherein the selected fertilized egg is collected from the culture container and stored.