JP6975418B2 - Area judgment method - Google Patents

Description

The present invention relates to a region determination method for determining a plurality of types of regions contained in an image of a biological sample.

In the fields of medicine and biochemistry, images of cells to be observed are taken. When observing cells housed in an observation vessel, transparent cells may have difficulty observing their characteristics. Therefore, conventionally, in order to improve the visibility of cells, cells are stained with a dye. However, there is a problem that the cells are damaged when staining is performed. Therefore, it is necessary to improve the visibility of cells without staining. A conventional method for imaging cells in a bright field is described in, for example, Patent Document 1. In the imaging method described in Patent Document 1, the contrast is improved by shifting the focal length of the imaging system from the focal length.

Japanese Unexamined Patent Publication No. 2008-20298

In addition, in the image obtained by photographing the cells, there are a plurality of types of regions such as a region where the cytoplasm of a large cell exists, a region where a nucleus of a large cell exists, a region where a small cell exists, a region where a cell does not exist, and the like. Exists. Traditionally, the observer has determined these multiple types of regions by photographing the stained cells. However, it was difficult to observe for a long period of time because the cells were damaged by staining. Further, depending on the judgment of the observer, the burden on the observer becomes large when comparing and analyzing a large number of images. In addition, the judgment may vary depending on the observer, or a judgment error may occur.

The present invention has been made in view of such circumstances, and is a technique capable of photographing a biological sample such as a cell in a bright field and automatically determining a plurality of types of regions included in the obtained observation target image. The purpose is to provide.

In order to solve the above problems, the first invention of the present application is a region determination method for determining a plurality of types of regions included in an image of a biological sample, which includes a learning step and an execution step. a-1) A step of acquiring a plurality of learning images by photographing a known biological sample in a plurality of wavelength bands having different center wavelengths, and a-2) a plurality of steps acquired in the step a-1). For the learning image, the step of obtaining a wavelength profile representing the change in brightness with respect to the change in the wavelength band for each pixel and the step of a-3) the step of obtaining the wavelength profile obtained in the step a-2) are used for the learning. A step of classifying each pixel of the image into a plurality of clusters, a-4) a step of designating a plurality of types of regions included in the learning image, and a-5) information of the plurality of clusters for each of the regions. The execution step includes b-1) a step of acquiring a plurality of observation target images by photographing a biological sample to be observed in a plurality of wavelength bands having different center wavelengths. b-2) A step of obtaining a wavelength profile representing a change in brightness with respect to a change in the wavelength band for each of the plurality of observation target images acquired in the step b-1), and b-3) the step b. -2) A step of classifying each pixel of the observation target image into a plurality of clusters according to the wavelength profile obtained in the observation target image, b-4) Information of the cluster in the observation target image, and the step a-5. A step of determining a plurality of types of regions included in the observation target image based on the information of the cluster learned in)), and in the step a-3), the clusters are classified for a plurality of visual fields. The processes are collectively performed, and in the step b-3), each pixel of the observation image is classified into a plurality of clusters based on the model configured in the step a-3).
The second invention of the present application is a region determination method for determining a plurality of types of regions contained in an image of a biological sample, which comprises a learning step and an execution step, and the learning step is a-1) known living body. A step of acquiring a plurality of learning images by photographing a sample in a plurality of wavelength bands having different center wavelengths, and a-2) a plurality of the learning images acquired in the step a-1) for each pixel. In addition, a plurality of pixels of the learning image are used according to the step of obtaining a wavelength profile representing the change in brightness with respect to the change in the wavelength band and the wavelength profile obtained in a-3) the step a-2). A step of classifying into clusters, a-4) a step of designating a plurality of types of regions included in the learning image, and a-5) a step of learning information on the plurality of clusters for each of the regions. The execution step includes b-1) a step of acquiring a plurality of observation target images by photographing a biological sample to be observed in a plurality of wavelength bands having different center wavelengths, and b-2) the step b. For each of the plurality of observation target images acquired in -1), a step of obtaining a wavelength profile showing a change in brightness with respect to a change in the wavelength band, and b-3) the step obtained in the step b-2). The steps of classifying each pixel of the observation target image into a plurality of clusters according to the wavelength profile, b-4) the information of the cluster in the observation target image, and the cluster learned in the step a-5). It has a step of determining a plurality of types of regions included in the observation target image based on the information, and in the step a-3), the cluster classification process is collectively or individually for a plurality of visual fields. Then, in the step b-3), the cluster classification process is individually performed for the plurality of visual fields.
The third invention of the present application is a region determination method for determining a plurality of types of regions contained in an image of a biological sample, which comprises a learning step and an execution step, and the learning step is a-1) known living body. A step of acquiring a plurality of learning images by photographing a sample in a plurality of wavelength bands having different center wavelengths, and a-2) a plurality of the learning images acquired in the step a-1) for each pixel. In addition, a plurality of pixels of the learning image are used according to the step of obtaining a wavelength profile representing the change in brightness with respect to the change in the wavelength band and the wavelength profile obtained in a-3) the step a-2). A step of classifying into clusters, a-4) a step of designating a plurality of types of regions included in the learning image, and a-5) a step of learning information on the plurality of clusters for each of the regions. The execution step includes b-1) a step of acquiring a plurality of observation target images by photographing a biological sample to be observed in a plurality of wavelength bands having different center wavelengths, and b-2) the step b. For each of the plurality of observation target images acquired in -1), a step of obtaining a wavelength profile showing a change in brightness with respect to a change in the wavelength band, and b-3) the step obtained in the step b-2). The steps of classifying each pixel of the observation target image into a plurality of clusters according to the wavelength profile, b-4) the information of the cluster in the observation target image, and the cluster learned in the step a-5). It has a step of determining a plurality of types of regions included in the observation target image based on the information, and the information of the plurality of clusters is a composition ratio of the plurality of clusters.
The fourth invention of the present application is the region determination method according to any one of the first invention to the third invention, wherein the biological sample is transparent, and in the step a-1) and the step b-1), the living body is described. While irradiating the biological sample with illumination light from either the upper side or the lower side of the sample, the biological sample is photographed from either the upper side or the lower side of the biological sample.

The fifth invention of the present application is the region determination method according to any one of the first invention to the fourth invention, and in the learning step, the steps a-1) to a-of a plurality of visual fields including a known biological sample. 5) is executed.

The sixth invention of the present application is the area determination method according to any one of the first invention to the fifth invention, and in the step a-3) and the step b-3), each pixel is used by the k-means method. Is classified into multiple clusters.

The seventh invention of the present application is the region determination method according to any one of the first invention to the sixth invention, and in the step a-5), the information of the plurality of clusters is obtained for each region by the random forest method. learn.

The eighth invention of the present application is the region determination method according to any one of the first invention to the seventh invention, and the biological sample is a cell.

According to the first to eighth inventions of the present application, by learning the information of the cluster of the known learning image and comparing it with the information of the cluster of the observation target image, a plurality of types of regions included in the observation target image are included. Can be determined automatically.

In particular, according to the fifth invention of the present application, the features of a plurality of types of regions included in an image can be learned more accurately. Therefore, in the execution step, it is possible to more accurately determine a plurality of types of regions included in the observation target image.

In particular, according to the first invention of the present application, pixels having the same wavelength profile in the learning image and the observation target image are classified into the same cluster. This makes it possible to improve the accuracy of classification.

In particular, according to the second invention of the present application, when the wavelength profile of the same type of region is different due to the difference in imaging conditions or the like, the difference in imaging conditions can be absorbed by individually performing the cluster classification processing.

It is a schematic diagram of a cell observation device. It is a perspective view of a culture vessel. It is a flowchart which showed the flow of the observation process using the cell observation apparatus. It is a flowchart which showed the flow of a learning process. It is a flowchart which showed the flow of a shooting process. It is a figure which conceptually showed a plurality of photographed images. It is a graph which showed the example of the wavelength profile. It is a figure which showed the example of the photographed image. It is a figure which showed the example of the cluster image. It is a figure which showed the example of the area designation. It is a figure which showed the example of the histogram. It is a flowchart which showed the flow of an execution process. It is a figure which showed the change of the photographed image by a coherence factor. It is a figure which showed the change of the photographed image by the half width of a wavelength band. It is a figure which showed schematically the relationship between the focal position and the appearance of a cell.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

<1. Configuration of cell observation device>
FIG. 1 is a schematic view showing the configuration of a cell observation device 1 which is an example of a device for executing the region determination method of the present invention. The cell observation device 1 is a device that acquires an image of the cell 92 and analyzes the obtained image. When the cell observation device 1 is used, the culture vessel 9 is arranged at a predetermined imaging position P in the device. In the culture vessel 9, the cells 92, which are biological samples, are held together with the culture solution 91.

FIG. 2 is a perspective view of the culture vessel 9. The culture vessel 9 is a well plate having a plurality of recesses 900. The plurality of recesses 900 are arranged two-dimensionally and regularly. Further, each recess 900 has a light-transmitting bottom. The culture container 9 houses the culture solution 91 in each recess 900, and holds the cells 92 to be observed in the culture solution 91. By culturing the cells 92 inside each depression 900, the cells 92 are cultured along the bottom of the depression 900. Hereinafter, the culture vessel 9, the culture solution 91 and the cells 92 in the culture vessel 9 are collectively referred to as a sample 90. The culture vessel 9 may be a light-transmitting petri dish made of glass, resin, or the like, instead of the well plate.

As shown in FIG. 1, the cell observation device 1 includes an illumination unit 20, an image pickup optical system 30, an image pickup unit 40, and a computer 60. The illumination unit 20, the image pickup optical system 30, and the image pickup unit 40 are supported by a device main body (not shown).

The illumination unit 20 includes a light source 21 and an illumination optical system 22. The illumination unit 20 is arranged above the shooting position P. When the cells 92 are photographed, the illumination unit 20 irradiates the sample 90 to be imaged with illumination light from above.

The light source 21 of this embodiment is a halogen lamp. Therefore, the light source 21 emits white light in which light having a large number of wavelengths is mixed.

The illumination optical system 22 includes a bandpass filter 50, a collector lens 221, a field diaphragm 222, an aperture diaphragm 223, and a condenser lens 224. The illumination light emitted from the light source 21 passes through the collector lens 221, the field diaphragm 222, the bandpass filter 50, the aperture diaphragm 223, and the condenser lens 224 in this order, and then irradiates the sample 90.

The bandpass filter 50 is an optical filter having a specific narrow wavelength band as a pass band. The bandpass filter 50 passes only the light of a narrow wavelength band (monochromatic light) among the white light having a wide wavelength band emitted from the light source 21. Since the illumination optical system 22 includes the bandpass filter 50, the illumination unit 20 irradiates the sample 90 with illumination light in a narrow wavelength band. Further, as the bandpass filter 50 of the present embodiment, a filter capable of changing the pass band (for example, a liquid crystal tunable filter) is used.

The collector lens 221 is a lens that adjusts the direction of the illumination light emitted from the light source 21. The field diaphragm 222 adjusts the illumination area at the shooting position P. The aperture stop 223 adjusts the numerical aperture NA of the illumination optical system 22. The condenser lens 224 collects the illumination light. The numerical aperture NA of the illumination optical system 22 is preferably 0.3 or less, for example.

The image pickup optical system 30 is arranged between the shooting position P and the image pickup unit 40. The imaging optical system 30 includes an objective lens 31, an aperture diaphragm 32, an imaging lens 33, a beam splitter 34, and an eyepiece lens 35. The objective lens 31 is a lens for enlarging an observation target and projecting an image. The aperture stop 32 adjusts the numerical aperture NA of the imaging optical system 30. The imaging lens 33 forms an image of an image to be imaged on the light receiving surface of the imaging unit 40 and the eyepiece 35. The beam splitter 34 splits the light that has passed through the imaging lens 33 into the image pickup unit 40 side and the eyepiece lens 35 side.

The light irradiated by the illumination unit 20 and transmitted through the sample 90 travels through the objective lens 31, the aperture stop 32, and the imaging lens 33. Then, the beam splitter 34 separates the light toward the eyepiece 35 and the light toward the image pickup unit 40. The beam splitter 34 may be able to switch the ratio of the light toward the eyepiece 35 and the light toward the image pickup unit 40. The eyepiece 35 is arranged at a position farther from the beam splitter 34 than the imaging position 350. The eyepiece 35 is a lens that the observer looks into when observing. The image magnified by the objective lens 31 can be further magnified and observed by the eyepiece lens 35.

In FIG. 1, the parts constituting the illumination optical system 22 and the image pickup optical system 30 are arranged in a row along the optical axis. However, the optical paths in the illumination optical system 22 and the image pickup optical system 30 may be folded back by a reflecting mirror or the like.

The image pickup unit 40 acquires a two-dimensional image of the sample 90 imaged on the light receiving surface as digital image data. That is, the imaging unit 40 images the cells 92 arranged at the imaging position P. For the image pickup unit 40, for example, a CCD camera or a CMOS camera is used.

In this cell observation device 1, the illumination unit 20 irradiates the sample 90 with illumination light in a narrow wavelength band having a wavelength band smaller than that of white light by the bandpass filter 50 described above. Further, the cell observation device 1 can change the pass band of the bandpass filter 50. Therefore, the image pickup unit 40 can acquire an image of the cell 92 in a plurality of wavelength bands having different center wavelengths.

In the present embodiment, the illumination unit 20 irradiates the illumination light from above the sample 90, and the image pickup unit 40 captures an image from below the sample 90. However, the cell observation device of the present invention is not limited to such a structure. The illumination unit 20 may irradiate the illumination light from below the sample 90, and the image pickup unit 40 may capture an image from above the sample 90.

The computer 60 functions as a control unit that controls the operation of each unit in the cell observation device 1, and as an image processing unit that processes an image acquired by the image pickup unit 40 to determine a plurality of types of regions included in the image. Has the function of. As conceptually shown in FIG. 1, the computer 60 has a processor 601 such as a CPU, a memory 602 such as a RAM, and a storage unit 603 such as a hard disk drive. The operation control program P1 and the image processing program P2 are stored in the storage unit 603. Further, the computer 60 is electrically connected to the above-mentioned light source 21, bandpass filter 50, and image pickup unit 40.

The computer 60 controls the operation of the light source 21, the bandpass filter 50, and the image pickup unit 40 described above by operating according to the operation control program P1. As a result, the imaging process of the cells 92 in the cell observation device 1 proceeds. As will be described later, in this cell observation device 1, a plurality of times of imaging are performed while changing the pass band of the bandpass filter 50. As a result, the computer 60 acquires a plurality of captured images I1 to In from the image pickup unit 40. Further, the computer 60 performs image processing based on the image processing program P2 on the plurality of captured images I1 to In. As a result, one cluster image Io is generated, and a plurality of types of regions included in the cluster image Io are determined.

<2. About cell observation using a cell observation device>
Subsequently, the procedure for observing the cells 92 using the above-mentioned cell observation device 1 will be described in more detail.

FIG. 3 is a flowchart showing a rough flow of the observation process using the cell observation device 1. As shown in FIG. 3, when observing the cells 92 using the cell observation device 1, a learning step (step S1) and an execution step (step S2) are performed. In the learning step, an image of a known cell 92 is acquired and the characteristics of the image are learned. In the execution step, an image of the unknown cell 92 is acquired, and a plurality of types of regions included in the image of the unknown cell 92 are determined based on the information obtained by the learning in step S1.

<2-1. About the learning process>
FIG. 4 is a flowchart showing the flow of the learning process. In the learning step, first, the culture vessel 9 in which the known cells 92 are held is set at the imaging position P of the cell observation device 1. Then, an instruction to start shooting is input to the computer 60. Then, the computer 60 turns on the light source 21 and irradiates the cells 92 with the illumination light. Then, the computer 60 operates the bandpass filter 50 and the imaging unit 40 to perform multispectral photography (multi-wavelength photography) of the cells 92 (step S11). That is, the cells 92 are photographed a plurality of times by the imaging unit 40 while changing the pass band of the bandpass filter 50.

FIG. 5 is a flowchart showing the flow of the shooting process in step S11 in more detail. As shown in FIG. 5, in step S11, after performing one imaging of the cells 92 by the imaging unit 40 (step S111), the computer 60 determines whether or not there is a next wavelength band to be imaged. (Step S112). The number of wavelength bands to be photographed may be, for example, four or more. If there is a next wavelength band to be imaged (yes in step S112), the pass band of the bandpass filter 50 is switched to that wavelength band (step S113), and the cells 92 are photographed again (step S111). .. Then, the processes of steps S111 to S113 are repeated until the captured images of all the preset wavelength bands are acquired. Eventually, when the wavelength band to be imaged disappears (no in step S112), the cell observing apparatus 1 ends the photographing of the cells 92. As a result, a plurality of captured images I1 to In (n: an integer of 2 or more) captured in a plurality of wavelength bands having different center wavelengths can be obtained. In the following, a plurality of captured images I1 to In (learning images) obtained by multispectral imaging in this learning step are represented by reference numerals IAn to IAn.

Note that FIG. 5 is an imaging procedure for one imaging field of view in the culture vessel 9. When it is desired to acquire the photographed images IA1 to IAn for learning in a plurality of imaging fields of view, the cell observation device 1 may execute the process of FIG. 5 for each imaging field of view.

FIG. 6 is a diagram conceptually showing a plurality of captured images IA1 to IAn acquired in step S11. Each of the plurality of captured images IA1 to IAn is composed of a plurality of pixels p. First, the computer 60 compares the pixels p located at the same coordinates (for example, the pixels p shown in FIG. 5) with respect to the plurality of captured images IA1 to IAn. Then, a wavelength profile W is created for each coordinate in which the pixel p exists (hereinafter, referred to as “pixel coordinate C”) (step S12). Specifically, pixels p located at the same pixel coordinate C are compared with each other to create a wavelength profile W representing a change in luminance. Then, such a wavelength profile W is created for all the pixel coordinates C.

As a result, for example, as shown in FIG. 7, a wavelength profile W representing a change in luminance with respect to a change in wavelength band is obtained for each pixel coordinate C. The horizontal axis of FIG. 7 indicates the central wavelength of the wavelength band at the time of photographing. The vertical axis of FIG. 7 indicates the brightness.

Return to FIG. When a plurality of wavelength profiles W are obtained, the computer 60 subsequently classifies each pixel coordinate C into a plurality of clusters (step S13). Here, the wavelength profile W obtained in step S12 is referred to for each pixel coordinate C. Then, each pixel coordinate C is classified into one of a plurality of clusters prepared in advance according to the characteristics of the wavelength profile W. At this time, the computer 60 classifies the pixel coordinates C having similar characteristics of the wavelength profile W into the same cluster. For example, a plurality of pixel coordinates C are set in consideration of a plurality of factors such as the slope of the wavelength profile W, the number of inflection points, the position of the inflection point, the average value of luminance, the maximum value of luminance, and the minimum value of luminance. Classify into clusters of.

Further, the k-means method may be used for the classification of the cluster in step S13. When the k-means method is used, first, a plurality of pixel coordinates C are classified into a predetermined number of clusters at random or according to a predetermined rule. Next, the center of the wavelength profile W belonging to each cluster is obtained. At this time, the element of the wavelength profile W of each pixel coordinate C is included in the calculation of the center position. After that, the plurality of pixel coordinates C are reclassified into the clusters closest to the center of the wavelength profile W. The computer 60 repeats such a process until the classification result of each pixel coordinate C does not change or a predetermined stop condition is satisfied. By using such a k-means method, each pixel coordinate C can be appropriately and automatically classified into a plurality of clusters according to the wavelength profile W.

After that, the computer 60 generates a cluster image Io based on the classification result of step S13 (step S14). In the following, the cluster image Io (learning image) generated in this learning step is represented by the reference numeral IAo. Here, a gradation value corresponding to the classified cluster is assigned to each pixel of the cluster image IAo (that is, to each pixel coordinate C). The gradation value may be a value predetermined for each cluster, or may be a value calculated for each cluster using the luminance values of the captured images IA1 to IAn. This gives a cluster image IAo in one field of view.

FIG. 8 is a diagram showing an example of the captured images IA1 to IAn acquired in step S11. FIG. 9 is a diagram showing an example of the cluster image IAo generated in step S14. Comparing FIGS. 8 and 9, the cluster image IAo of FIG. 9 can clearly grasp the position and shape of the cells 92 than the captured image of FIG. That is, it can be seen that the cluster image IAo that is more suitable for the classification of the region than the photographed images IA1 to IAn is generated by the cluster division based on the wavelength profile W described above.

Subsequently, a plurality of known types of regions included in the cluster image IAo are designated (step S15). FIG. 10 is a diagram showing an example of region designation in step S15. Here, the user of the cell observation device 1 specifies each region existing in the cluster image IAo. For example, in the cluster image IAo, a plurality of types of regions such as a region in which the cytoplasm of a large cell exists, a region in which a nucleus of a large cell exists, a region in which a small cell exists, a region in which a cell does not exist, and the like are designated. In the example of FIG. 10, three types of regions T1, T2, and T3 are designated for the cluster image IAo. The area may be designated, for example, by the user operating the input unit of the computer 60 while viewing the cluster image IAo displayed on the display of the computer 60.

When multispectral imaging is performed in a plurality of imaging fields of view in step S11 described above, a cluster image IAo is generated for each imaging field of view in step S14 described above. In that case, the user specifies a plurality of types of regions for each of the generated cluster images IAo in step S15.

When the user's designation of the known area is completed, the computer 60 learns the cluster configuration ratio for each designated area (step S16). Here, first, a histogram H showing the composition ratio of the cluster is created for each region. FIG. 11 is a diagram showing an example of a histogram H created for each region. The horizontal axis of the histogram H shows the clusters classified in step S13. The vertical axis of FIG. 11 shows the number of pixels. The histogram H is information indicating how many pixels belong to which cluster in the region.

The computer 60 creates such a histogram H for each region designated in step S15. Then, the characteristics of the histogram H are learned for each region. That is, the characteristics of the histogram H are generalized so that one area can be output when arbitrary information is input as the configuration ratio of the cluster. For the processing of step S16, for example, a machine learning method such as a random forest can be used. By using a random forest, it is possible to automatically create multiple conditional branches to reach each region from the cluster composition ratio.

In the learning step of step S1, it is preferable to perform the above steps S11 to S16 for a plurality of visual fields including known cells 92. The more the field of view is learned, the more accurately the characteristics of the plurality of types of regions contained in the image can be obtained. Therefore, in the execution step described later, it is possible to more accurately determine a plurality of types of regions included in the image to be observed.

<2-2. About the execution process>
When the above learning step (step S1) is completed, the computer 60 next performs the execution step (step S2).

FIG. 12 is a flowchart showing the flow of the execution process. In the execution step, first, the culture vessel 9 in which the cells 92 to be observed are held is set at the imaging position P of the cell observation device 1. Then, an instruction to start shooting is input to the computer 60. Then, the computer 60 turns on the light source 21 and irradiates the cells 92 with the illumination light. Then, the computer 60 operates the bandpass filter 50 and the imaging unit 40 to perform multispectral imaging of the cells 92 (step S21). That is, the cells 92 are photographed a plurality of times by the imaging unit 40 while changing the pass band of the bandpass filter 50. As a result, a plurality of captured images I1 to In (n: an integer of 2 or more) captured in a plurality of wavelength bands having different center wavelengths can be obtained. In the following, a plurality of captured images I1 to In (observation target images) obtained by multispectral imaging in this execution step are represented by reference numerals IBn to IBn.

Since the detailed procedure for multispectral imaging is the same as in step S11, duplicate description will be omitted.

Next, the computer 60 creates a wavelength profile W for each pixel coordinate C for the plurality of captured images IB1 to IBn in the same manner as in step S12 described above (step S22). That is, the pixels p located at the same pixel coordinate C are compared with each other to create a wavelength profile W representing the change in luminance with respect to the change in the wavelength band. Then, such a wavelength profile W is created for all the pixel coordinates C.

When the plurality of wavelength profiles W are obtained, the computer 60 subsequently classifies each pixel coordinate C into a plurality of clusters in the same manner as in step S13 described above (step S23). Here, the wavelength profile W obtained in step S22 is referred to for each pixel coordinate C. Then, each pixel coordinate C is classified into one of a plurality of clusters prepared in advance according to the characteristics of the wavelength profile W. As in step S13, the k-means method may be used for the classification of the cluster in step S23.

After that, the computer 60 generates a cluster image Io based on the classification result of step S23 in the same manner as in step S14 described above (step S24). In the following, the cluster image Io (observation target image) generated in this execution step is represented by the reference numeral IBo. Here, a gradation value corresponding to the classified cluster is assigned to each pixel of the cluster image IBo (that is, to each pixel coordinate C). The gradation value may be a value predetermined for each cluster, or may be a value calculated for each cluster using the luminance values of the captured images IB1 to IBn. As a result, a cluster image IBo in the field of view of the observation target can be obtained.

Subsequently, the computer 60 determines a plurality of types of regions included in the cluster image IBo (step S25). Here, the region is determined based on the distribution of the clusters in the cluster image IBo to be observed and the learning result of step S16 described above. Specifically, the cluster image IBo to be observed is divided into a plurality of small regions, and the learning result obtained in step S16 is applied to the cluster composition ratio of each small region, and each small region is described above. Of the plurality of regions specified in step S15, which region most closely matches is determined. For example, when the conditional branch by the random forest is acquired in step S16, the distribution of the clusters in each small region of the cluster image IBo is applied to the conditional branch, and the region to be reached is used as the determination result.

After that, the computer 60 displays the determination result of the area on the screen. For example, each region is visually shown to the user by coloring the cluster image IBo according to the region.

As described above, in this cell observation device 1, by learning the cluster distribution of the image of the known cell 92 and comparing it with the cluster distribution of the cell 92 to be observed, a plurality of types of regions included in the image to be observed are included. To judge. As a result, it is possible to automatically and accurately determine a plurality of types of regions included in the image to be observed.

In particular, in this cell observation device 1, the region is not determined by the brightness of the captured image itself, but a cluster distribution is derived from a plurality of captured images obtained by multispectral imaging, and a plurality of types are derived based on the cluster distribution. Judge the area of. Thereby, a plurality of types of regions contained in the transparent cell 92 image can be determined more accurately.

<3. About shooting conditions>
Subsequently, preferable shooting conditions in step S11 and step S21 described above will be described. In this cell observation device 1, it is assumed that unstained cells 92 are observed. In general, since cells 92 are almost colorless and transparent, it is difficult to identify the contours of the components when observing unstained cells 92 in a bright field. Therefore, in the multispectral imaging in steps S11 and S21, it is preferable to appropriately set the coherence factor σ, the half width at half maximum of the wavelength band of the illumination light, and the focal position to enhance the contrast of the captured image.

<3-1. About Coherence Factor>
First, the condition of the coherence factor σ will be described with reference to FIG. FIG. 13 is a diagram showing a photographed image acquired by the imaging unit 40 by changing the value of the coherence factor σ.

The coherence factor σ is the ratio between the numerical aperture NA of the illumination optical system 22 and the numerical aperture NA of the imaging optical system 30. Specifically, the coherence factor σ is a value obtained by dividing the numerical aperture NA of the illumination optical system 22 by the numerical aperture NA of the imaging optical system 30. By reducing the coherence factor σ, the contrast of the image that can be acquired by the image pickup unit 40 can be improved.

FIG. 13 shows the case where the values of the coherence factor σ are σ = 1.00, σ = 0.83, σ = 0.58, σ = 0.36, σ = 0.26, and σ = 0.14. An example of a photographed image of the cell 92 of the cell 92 is shown. The cell 92 of the example of FIG. 13 has a cell body 921, a protrusion 922 protruding from the cell body 921, and an internal microstructure 923 of the cell body 921. The center wavelength of the bandpass filter 50 at this time is 550 nm, and the half width is 39 nm. Therefore, the central wavelength of the illumination light in the narrow wavelength band irradiated to the cell 92 is 550 nm, and the half width is 39 nm.

As shown in FIG. 13, as the value of the coherence factor σ becomes smaller, the contrast of the captured image of the cell 92 improves. In particular, the outline of the cell body 921, which was blurred when σ = 0.83, can be clearly seen when σ = 0.58. Further, the existence of the contour of the protrusion 922, which is difficult to see when σ = 0.83, and the contour of the microstructure 923 of the cell body 921 can be visually recognized when σ = 0.58. By setting the value of the coherence factor σ to σ = 0.6 or less in this way, the contrast of the captured image can be improved to the extent that each part of the transparent cell 92 can be visually recognized in the bright field.

Further, when σ = 0.58, the outline of the protrusion 922 which was blurred and the outline of the fine structure 923 of the cell body 921 can be visually recognized when σ = 0.36. By setting the value of the coherence factor σ to σ = 0.4 or less in this way, the contrast of the captured image can be further improved to the extent that each part of the transparent cell 92 can be clearly seen in the bright field.

When σ = 0.26, the contour of the cell body 921 can be clearly seen as in the case of σ = 0.36. Further, when σ = 0.26, the contrast of the captured image is further improved as compared with the case of σ = 0.36. On the other hand, in the case of σ = 0.14, no clear improvement in visibility and contrast was observed as compared with the case of σ = 0.26.

From the above results, in steps S11 and S21 described above, it is preferable to adjust the illumination optical system 22 and the imaging optical system 30 so that the coherence factor σ is 0.6 or less, and perform multispectral photography. It can be said that. Further, it can be said that it is more preferable that the coherence factor σ is 0.4 or less. Further, it can be said that it is more preferable if the coherence factor σ is 0.3 or less.

<3-2. About half-value width of wavelength band of illumination light>
Next, the half width of the wavelength band of the illumination light will be described. The illumination light emitted from the illumination unit 20 is refracted on the surface of the cell 92. At this time, when the illumination light is light having a narrow wavelength band smaller than that of white light, the spread of light at the refraction point is smaller than that of white light. That is, the illumination light in the narrow wavelength band travels in substantially the same direction even after refraction. Therefore, if the illumination light in a narrow wavelength band is used, it is possible to obtain a photographed image having high visibility and including highly accurate information necessary for determining a region.

FIG. 14 shows an example of a photographed image of the cell 92 when the bandpass filter 50 is not used and when the half width of the bandpass filter 50 is 153 nm, 124 nm, and 10 nm. The smaller the half-value width of the bandpass filter 50, the smaller the wavelength band of the illumination light that has passed through the bandpass filter 50. The cell 92 in the example of FIG. 14 has a cell body 921 and a protrusion 922 protruding from the cell body 921. The value of the coherence factor σ at this time is σ = 0.26, and the center wavelength of the bandpass filter 50 is 550 nm.

As shown in FIG. 14, it is easier to visually recognize each part of the cell 92 in the three captured images in the narrow wavelength band using the bandpass filter 50 as compared with the captured image in white light without the bandpass filter 50. In particular, it is difficult to visually recognize the protrusion 922 extending from the cell body 921 in the photographed image in white light, whereas the protrusion 922 can be visually recognized in the three photographed images in the narrow wavelength band. By imaging the image of the cell 92 in the narrow wavelength band in this way, it is possible to obtain a photographed image having high contrast. As described above, due to the intervention of the bandpass filter 50, the imaging unit 40 captures only the light in the narrow wavelength band in which the cells 92 are unlikely to spread. Therefore, it is possible to acquire a captured image with high contrast.

Further, as shown in FIG. 14, when three captured images in the narrow wavelength band are compared, the contrast of the images is improved as the half width of the narrow wavelength band becomes smaller. In particular, when the half-value width is 124 nm, the visibility of the contour of the protrusion 922 is greatly improved as compared with the case where the half-value width is 153 nm. As described above, the half width of the narrow wavelength band is preferably 150 nm or less. On the other hand, when the half-value width is 10 nm, the visibility of the contour of the protrusion 922 is further improved as compared with the case where the half-value width is 124 nm. Therefore, the half width of the narrow wavelength band is more preferably 100 nm or less. That is, by setting the half width of the narrow wavelength band to 150 nm or less or 100 nm or less, a cell image with higher contrast can be obtained.

From the above results, it can be said that in step S11 and step S21 described above, it is preferable that the half width of each wavelength band used for multispectral imaging is 150 nm or less. Further, it can be said that it is more preferable that the half width of each wavelength band used for multispectral photography is 100 nm or less. Thereby, in the above-mentioned steps S12 and S22, the wavelength profile W having a large change can be obtained. As a result, in the above-mentioned steps S13 and S23, each pixel can be classified more appropriately.

<3-3. Focus position>
Finally, the condition of the focal position will be described with reference to FIG. FIG. 15 is a diagram schematically showing the relationship between the focal position and the appearance of the cell 92. As schematically shown in FIG. 15, since the cells 92 are transparent, if the focal position of the imaging optical system 30 is set to the focused position aligned with the cells 92, the cells in the image acquired by the imaging unit 40 The brightness inside and outside the cell is similar. Therefore, it becomes difficult to visually recognize the cells 92 at the in-focus position.

When the focal position of the image pickup optical system 30 is shifted from the in-focus position to the short distance direction (on the side of the image pickup unit 40), the inside of the cell looks whiter than the outside of the cell. Then, at the first position where the focal position of the imaging optical system 30 is shifted from the focal focus position in the short distance direction by a certain distance, the edge strength takes a maximum value. Therefore, at the first position, the visibility of each part of the cell 92 and the cell 92 is improved. When the focal position of the imaging optical system 30 is further shifted in the short distance direction from the first position, the image is blurred, the edges are weakened, and the visibility of the cells 92 is lowered.

On the other hand, when the focal length of the imaging optical system 30 is shifted from the focal length position to the long distance direction (illumination unit 20 side), the inside of the cell looks blacker than the outside of the cell. Then, at the second position where the focal position of the imaging optical system 30 is shifted from the focal focus position in a certain distance in a long distance direction, the edge strength takes a maximum value. Therefore, at the second position, the visibility of each part of the cell 92 and the cell 92 is improved. When the focal position of the imaging optical system 30 is further shifted in the far distance direction from the second position, the image is blurred, the edges are weakened, and the visibility of the cells 92 is deteriorated.

Therefore, in step S11 and step S21 described above, it is preferable to perform multispectral photography with the focal position of the imaging optical system 30 as the first position or the second position shifted by a certain distance from the focal focus position. As a result, the captured images I1 to In with higher contrast can be acquired. The distance from the in-focus position to the first position or the second position may be a predetermined distance predetermined for each type of cell 92. Further, the distance from the in-focus position to the first position or the second position may be adjusted and determined while observing.

<4. Modification example>
Although one embodiment of the present invention has been described above, the present invention is not limited to the above embodiment.

In the above embodiment, the cluster classification process (step S13) in the learning process and the cluster classification process (step S23) in the execution process are performed by individual processes. However, the cluster classification process in the learning process and the cluster classification process in the execution process may be performed by a common process. For example, the k-means method may be collectively applied to the learning images of a plurality of fields of view, and the cluster classification processing may be performed on the observation images based on the constructed model. In this way, pixels having the same wavelength profile W are classified into the same cluster. Therefore, the accuracy of classification can be improved.

However, the wavelength profile W in the same region may differ between the captured images IA1 to IAn for learning and the captured images IB1 to IBn for observation due to differences in imaging conditions and the like. In such a case, as in the above embodiment, the cluster classification process in the learning process and the cluster classification process in the execution process are performed individually to make the difference in the biological sample and the difference in the imaging conditions. Can be absorbed. Therefore, it may be possible to obtain more accurate results by individually performing the cluster classification process in the learning process and the cluster classification process in the execution process. The entire visual field may be processed individually through the learning process and the execution process. Further, in the learning process, a plurality of visual fields may be processed individually, and in the execution process, a plurality of visual fields may be processed individually.

Further, in the above embodiment, the region is determined based on the cluster composition ratio in each region. However, for each area of the image for learning, information other than the composition ratio such as the arrangement state of the cluster is learned, and based on the information and the information of the cluster in the image to be observed, a plurality of types included in the image to be observed are included. The area of may be determined.

Further, the configurations of the illumination optical system 22 and the image pickup optical system 30 may be different from those in the above embodiment. For example, the illumination optical system 22 and the image pickup optical system 30 may have other elements, or the elements included in the above embodiment may be omitted. Further, the position of the bandpass filter 50 in the illumination optical system 22 or the image pickup optical system 30 may be different from that of the above embodiment.

In the above embodiment, the bandpass filter 50 is provided in the illumination optical system 22. However, the bandpass filter 50 may be provided in the imaging optical system 30. In that case, the cells 92 are irradiated with white light as illumination light, but the wavelength band of the light reaching the image pickup unit 40 can be changed by the bandpass filter 50 of the image pickup optical system 30. Therefore, the image pickup unit 40 can acquire the captured images I1 to In in a plurality of wavelength bands.

Further, in the above embodiment, a halogen lamp is used as the light source 21, and the wavelength band of the illumination light is changed by the bandpass filter 50. However, as the light source 21, a laser oscillator or LED capable of wavelength modulation may be used. Then, since the wavelength band of the light emitted from the light source 21 itself can be changed, the bandpass filter 50 can be omitted.

Further, the cell observation device 1 of the above embodiment has an eyepiece 35 for the observer to directly observe. However, the eyepiece 35 may be omitted.

Further, the elements appearing in the above-described embodiments and modifications may be appropriately combined as long as there is no contradiction.

1 Cell observation device 9 Culture vessel 20 Illumination unit 21 Light source 22 Illumination optical system 30 Imaging optical system 31 Objective lens 32 Aperture aperture 33 Imaging lens 34 Beam splitter 35 Eyepiece lens 40 Imaging unit 50 Band pass filter 60 Computer 90 Sample 91 Culture solution 92 Cell 221 Collector lens 222 Field aperture 223 Aperture aperture 224 Aperture lens C Pixel coordinates H histogram I1, IA1, IB1 Photographed image Io, IAo, IBo Cluster image P Photographed position T1, T2, T3 Region W Wavelength profile

Claims (8)

It is a region determination method for determining a plurality of types of regions contained in an image of a biological sample.
It has a learning process and an execution process.
The learning process is
a-1) A process of acquiring a plurality of learning images by photographing a known biological sample in a plurality of wavelength bands having different center wavelengths.
a-2) A step of obtaining a wavelength profile representing a change in luminance with respect to a change in the wavelength band for each pixel of the plurality of learning images acquired in the step a-1).
a-3) A step of classifying each pixel of the learning image into a plurality of clusters according to the wavelength profile obtained in the step a-2).
a-4) A step of designating a plurality of types of regions included in the learning image, and
a-5) The process of learning the information of the plurality of clusters for each of the regions, and
Have,
The execution step is
b-1) A process of acquiring a plurality of observation target images by photographing a biological sample to be observed in a plurality of wavelength bands having different center wavelengths.
b-2) A step of obtaining a wavelength profile representing a change in luminance with respect to a change in the wavelength band for each pixel of the plurality of observation target images acquired in the step b-1).
b-3) A step of classifying each pixel of the observation target image into a plurality of clusters according to the wavelength profile obtained in the step b-2).
b-4) A step of determining a plurality of types of regions included in the observation target image based on the information of the cluster in the observation target image and the information of the cluster learned in the step a-5).
Have,
In the step a-3), the cluster classification process is collectively performed for a plurality of fields of view, and the clusters are classified.
In the step b-3), a region determination method for classifying each pixel of the observation image into a plurality of clusters based on the model configured in the step a-3). It is a region determination method for determining a plurality of types of regions contained in an image of a biological sample.
It has a learning process and an execution process.
The learning process is
a-1) A process of acquiring a plurality of learning images by photographing a known biological sample in a plurality of wavelength bands having different center wavelengths.
a-2) A step of obtaining a wavelength profile representing a change in luminance with respect to a change in the wavelength band for each pixel of the plurality of learning images acquired in the step a-1).
a-3) A step of classifying each pixel of the learning image into a plurality of clusters according to the wavelength profile obtained in the step a-2).
a-4) A step of designating a plurality of types of regions included in the learning image, and
a-5) The process of learning the information of the plurality of clusters for each of the regions, and
Have,
The execution step is
b-1) A process of acquiring a plurality of observation target images by photographing a biological sample to be observed in a plurality of wavelength bands having different center wavelengths.
b-2) A step of obtaining a wavelength profile representing a change in luminance with respect to a change in the wavelength band for each pixel of the plurality of observation target images acquired in the step b-1).
b-3) A step of classifying each pixel of the observation target image into a plurality of clusters according to the wavelength profile obtained in the step b-2).
b-4) A step of determining a plurality of types of regions included in the observation target image based on the information of the cluster in the observation target image and the information of the cluster learned in the step a-5).
Have,
In step a-3), cluster classification processing is performed collectively or individually for a plurality of fields of view.
In the step b-3), there is a region determination method in which cluster classification processing is individually performed for a plurality of visual fields. It is a region determination method for determining a plurality of types of regions contained in an image of a biological sample.
It has a learning process and an execution process.
The learning process is
a-1) A process of acquiring a plurality of learning images by photographing a known biological sample in a plurality of wavelength bands having different center wavelengths.
a-2) A step of obtaining a wavelength profile representing a change in luminance with respect to a change in the wavelength band for each pixel of the plurality of learning images acquired in the step a-1).
a-3) A step of classifying each pixel of the learning image into a plurality of clusters according to the wavelength profile obtained in the step a-2).
a-4) A step of designating a plurality of types of regions included in the learning image, and
a-5) The process of learning the information of the plurality of clusters for each of the regions, and
Have,
The execution step is
b-1) A process of acquiring a plurality of observation target images by photographing a biological sample to be observed in a plurality of wavelength bands having different center wavelengths.
b-2) A step of obtaining a wavelength profile representing a change in luminance with respect to a change in the wavelength band for each pixel of the plurality of observation target images acquired in the step b-1).
b-3) A step of classifying each pixel of the observation target image into a plurality of clusters according to the wavelength profile obtained in the step b-2).
b-4) A step of determining a plurality of types of regions included in the observation target image based on the information of the cluster in the observation target image and the information of the cluster learned in the step a-5).
Have,
The area determination method in which the information of the plurality of clusters is the composition ratio of the plurality of clusters. The area determination method according to any one of claims 1 to 3.
The biological sample is transparent and
In the steps a-1) and b-1), the biological sample is irradiated with illumination light from either the upper side or the lower side of the biological sample, and from the upper side or the lower side of the biological sample. A method for determining an area for photographing a biological sample. The area determination method according to any one of claims 1 to 4.
In the learning step, a region determination method in which the steps a-1) to a-5) are executed for a plurality of visual fields including a known biological sample. The area determination method according to any one of claims 1 to 5.
In the step a-3) and the step b-3), a region determination method for classifying each pixel into a plurality of clusters by using the k-means method. The area determination method according to any one of claims 1 to 6.
In the step a-5), a region determination method for learning information on the plurality of clusters for each region by the random forest method. The area determination method according to any one of claims 1 to 7.
A method for determining a region in which the biological sample is a cell.

Images