WO2022113367A1

WO2022113367A1 - Image conversion method, program, and image processing device

Info

Publication number: WO2022113367A1
Application number: PCT/JP2020/044565
Authority: WO
Inventors: 隆彦吉田; ちか坂本; 千枝子中田
Original assignee: 株式会社ニコン
Priority date: 2020-11-30
Filing date: 2020-11-30
Publication date: 2022-06-02

Abstract

A method is for performing conversion of an image that is not a phase contrast image, by using a trained model, the method comprising: a first step for acquiring a first image obtained by imaging a biological sample and that is not a phase contrast image; a second step for generating a second image by performing smoothing processing on at least a partial region of the first image; and a third step for converting the second image to a third image by using the trained model, wherein the trained model has been trained by using, as learning data, a learning source image group composed of images that are not phase contrast images, and a learning target image group composed of phase contrast images.

Description

Image conversion method, program, image processing device

The present invention relates to an image conversion method, a program, and an image processing apparatus.

Conventionally, a technique of converting a bright-field image of a cell into an image like a phase-difference image by AI is known. (See, for example, Patent Document 1).

Japanese Unexamined Patent Publication No. 2020-60822

The technique of the present disclosure provides a novel image conversion method.

One embodiment of the present disclosure is a method for image-transforming an image of an image of a biological sample using a trained model, wherein the image conversion method is an image of a biological sample, phase difference. The first step of acquiring a first image that is not an image, the second step of performing a smoothing process on at least a part of the region of the first image to generate a second image, and the learning. The trained model includes a training source image group consisting of images that are not phase-difference images and a phase-difference image, including a third step of converting the second image into a third image using the trained model. It is a method that is a trained model in which a training target image group consisting of is trained as training data.

Another embodiment of the present disclosure is a method of image-transforming an image obtained by capturing an image of a biological sample using a trained model, wherein the first image obtained by capturing the biological sample by the first method is obtained. The process of selecting a trained model suitable for the first method from a plurality of different trained models based on the acquisition process and the information regarding the first method, and the selected trained model are used. Then, the learned image includes a step of converting the original image into an image similar to an image captured by a second method different from the first method, and a step of outputting the converted image. The model is a model in which a learning source image group consisting of images captured by the first method and a learning target image group consisting of images captured by the second method are trained as learning data. The method.

It is a schematic diagram of the image processing system which concerns on one Embodiment of this invention. It is a flowchart of the image processing method which concerns on one Embodiment of this invention. In one embodiment of the present invention, it is an image before and after conversion when a bright field image captured by a camera is converted into an artificial retardation image by a trained model. The same image was converted after (A) denoising (B) without denoising. In one embodiment of the present invention, it is an image before and after conversion when a bright field image captured by a transmission detector is converted into an artificial retardation image by a trained model. The same image was converted after (A) denoising (B) without denoising. In one embodiment of the present invention, in order to determine that the imaging condition is equal to or higher than a predetermined magnification, the original image (A) is processed by a dispersion filter (B), and then the image (C) is binarized. ). In one embodiment of the present invention, it is an image for determining whether or not the magnification of an objective lens is equal to or higher than a predetermined magnification.

Hereinafter, embodiments of the technique disclosed in the present disclosure will be described in detail with reference to the attached drawings. The embodiments and specific examples of the invention described below are shown for illustration purposes only, and the present invention is not limited thereto.

In addition, the image processing described in the examples is merely an example, and when the technique of the present disclosure is implemented, unnecessary steps are deleted, new steps are added, and processing is performed within a range that does not deviate from the gist. Needless to say, the order may be changed.

In addition, all documents (patented documents and non-patented documents) and technical standards described in the present specification are referred to in the present specification as if they were specifically and individually described in the present specification. It is captured.

One embodiment of the technique of the present disclosure is a method for transforming an image of an image of a biological sample using a trained model, not a phase-difference image of an image of the biological sample. The first step of acquiring the image of the above, the second step of performing the smoothing process on at least a part of the region of the first image to generate the second image, and the second step using the trained model. Including a third step of converting the image of It is a method that is a trained model trained as training data.

Hereinafter, each process will be described in detail with reference to the system configuration of FIG. 1 and the flowchart of FIG.

<Image acquisition>
The image conversion system of the present disclosure includes an image acquisition device 101 and an image conversion device 103 (FIG. 1).

First, the image conversion device 103 acquires an image obtained by capturing a biological sample by an imaging means from the image acquisition device 101 by the image acquisition unit 111 (S11).

Here, the image acquisition device 101 has a biological sample observation means 106 and an image pickup means 107. The observation means 106 is a microscope suitable for observing a biological sample, and is, for example, a bright-field microscope, a fluorescence microscope, a differential interference microscope, and the like. The image pickup means 107 is an image pickup element such as a CCD. The images captured by the method of the imaging means 107 are a bright field image, a fluorescence image, and a differential interference contrast image, respectively. Since the image acquired by the image acquisition device 101 is an image before image processing described later, it may be referred to as an original image.

When taking an image, if the amount of light is low, the subsequent image conversion may not be successful. Therefore, when imaging at a high magnification, for example, 40 to 100 times, the exposure time may be lengthened or the light source power may be increased.

<Increased contrast>
First, the image preprocessing unit 113 may perform a process of enhancing the contrast of the image (S12). Specifically, a known method may be used, and examples thereof include tone mapping, histogram equalization, and local histogram equalization.

<Possibility of artifacts>
As shown in FIGS. 3 and 4, particularly in a high-magnification image, image conversion may result in a structure or pattern (hereinafter referred to as an artifact) that cannot be recognized in the original image in the background region where cells do not exist.

Therefore, the noise reduction unit 115 determines whether or not there is a possibility that an artifact may occur in the image to be converted (S13). For example, if the background area occupies a predetermined ratio or more in the image, or if the imaging conditions are high magnification or the predetermined magnification or more, an artifact may occur in the background portion. Can be judged.

When the background area is equal to or more than a predetermined ratio in the image, for example, the position where the observation object exists is detected, the area where the observation object exists is calculated, and the area where the observation object exists is calculated from the total area. The background area is calculated by subtraction, the ratio of the background area to the total area is calculated, and the judgment can be made by determining whether or not the ratio is equal to or more than a predetermined value. Here, the predetermined ratio is not particularly limited, but may be 40%, preferably 50%, and more preferably 60%.

If the imaging condition is at least a predetermined magnification, it can be determined, for example, whether or not the magnification of the objective lens is at least a predetermined magnification from the binarized image obtained by binarizing the image. First, the image is processed with a dispersion filter (FIG. 5). Next, the image processed by the dispersion filter is binarized to generate a binarized image (FIG. 5). Using this binarized image, it is determined whether or not the magnification of the objective lens is equal to or higher than a predetermined magnification (FIG. 6). Here, based on whether or not there is a region that does not include an object in the partial region of the binarized image, if there is such a region, it can be determined that the magnification is high (FIG. 6, right). Alternatively, the magnification information of the image may be acquired from the data incidental to the image, and this data may be included in the metadata. Here, the predetermined value is not particularly limited, but when the magnification information of the image is the magnification of the objective lens, it may be 20 times, preferably 30 times, and more preferably 40 times.

<Preprocessing>
The noise reduction unit 115 performs a process of reducing noise that may cause an artifact in a region where an artifact may occur. Specifically, for an image of an object region in which an object is imaged and a background region not including the object, which is determined to have a possibility of causing an artifact, a part or all of the background area is taken. The region of No. 1 is smoothed (S14).

When processing at least a part of the background area, the object area may be specified in advance and only at least a part of the background area may be processed. As a result, the object area to be observed is maintained as the original image, and noise is reduced only in the background portion.

Alternatively, the entire image may be processed for each object area without specifying the object area. This makes the process simple. In that case, the smoothing of the image affects not only the background area but also the object area. In this case, it is preferable to adjust the smoothing parameter so that the resolution of the object region after image conversion becomes a desired resolution. Alternatively, an image that has been subjected to noise reduction processing and image-converted may be combined with an image that has been image-converted without noise reduction processing. For example, the same two images are prepared, the first image is entirely smoothed and converted, the second image is converted without smoothing, and the background area is removed from the converted first image. By cutting out an object area from the second image that has been cut out and converted and combining them to generate a converted image, it is possible to create a converted image in which only the background area is processed.

As an example of smoothing processing, smoothing in the spatial direction or the luminance direction can be given.

As an example of smoothing in the spatial direction, there is a process of sampling an image in the spatial direction, and this process reduces the image size. As the interpolation process when reducing the image, a known method such as a bicubic method can be used. The image reduction may be performed only in the background area as described above. As a result, the resolution of the object area to be observed is maintained, and the resolution is lowered only in the processed background area. Image reduction may include, for example, reducing the image to 90%, 80%, 75%, 70%, 60%, 50%, 40%, 30%, 25%, 20%. The larger the reduction ratio, the higher the effect of removing the artifacts. Therefore, when creating a converted image in which only the background area is processed, it is preferable that the reduction ratio is large. On the other hand, since the resolution of the object area after image conversion is lowered due to the reduction of the image, the reduction ratio of the image is small when the entire image is processed for each object area without specifying the object area. Is preferable. The image reduction ratio may be changed between the background area and the object area.

As an example of smoothing in the luminance direction, there is a process of correcting the luminance value of the portion to be processed to make it constant. Further, as an example of the smoothing process in the luminance direction, there is a filter process such as a generally well-known bilateral filter. Also, other well-known noise reduction processing may be utilized. When correcting the luminance value, it is desirable to divide the image into a cell region and a background region, and perform this processing only on the background region.

Since these smoothing processes can reduce noise before image conversion, it is a noise reduction process (also called noise reduction process). And the artifacts after image conversion are reduced.

After the pre-processing and before starting the conversion processing, the image may be divided into small areas (S15). If you split the image without duplication, the borders will be noticeable when combined after image conversion. Therefore, when dividing into small areas, it is preferable to divide by overlapping the boundaries. Then, when the images are combined after the image conversion, a uniform image can be obtained as a whole by combining the images with weighting at the overlapping portion.

<Image conversion>

The smoothed image or a small area obtained by further dividing the smoothed image is input to the trained model of the image conversion unit and converted (S16).

In the technique of the present disclosure, an image conversion network is used, an image belonging to the first image group is used as a source image, and an image of a second image group different from the first image group is used as a target image. It is preferable to learn the mapping from the source to the target and generate a trained model. The image conversion network is not particularly limited, and is "convolutional neural networks (CNN)", "generative adversarial networks (GAN)", "CGAN (Conditional GAN)", and "DCGAN (Deep Conventional GAN)". ) ”,“ Pix2Pix ”,“ CycleGAN ”, etc. Is preferable.

When a small area divided into inputs is used, the image post-processing unit combines the obtained small areas after conversion (S17) to generate a third image.

Here, the third image is a training model trained using an image belonging to the first image group as a source image and an image of the second image group as a target image, and is used as a second image group. It refers to an image obtained by inputting an image that does not belong to the first image group and converting it. It can be said that the third image is an image converted into the style of the second image group while the position of the object captured in the input image does not change. When the target image is a phase difference image, it may be referred to as an artificial phase difference image or a pseudo phase difference image. It was

At the time of this connection, it is preferable to weight the overlapping portion and combine them. As a result, the brightness of the entire image can be made uniform and the boundaries of division can be made inconspicuous.

As the weighting, for example, when weighting the divided area 1 and the divided area 2, the distance to the divided area 1 and the distance to the divided area 2 are a (pixel) and b (pixel), respectively, for the pixels of the overlapping portion. At this time, it is conceivable to add the brightness of the original image with an inversely proportional weight according to the distance. That is, with the weight w1 = b / (a + b) of the divided area 1 and the weight w2 = a / (a + b) of the divided area 2, the luminance value of the corresponding pixel of the combined image is I = w1 × I1 + w2 × I2 (I1, I2 refers to the corresponding pixel luminance value of the divided area 1 and the divided area 2).

The weighting calculation method is not limited to this, and different calculation formulas may be used, but even in that case, the distance to the overlapping portion of the pixels is short according to the distance to the division area 1 and the division area 2. It is preferable to increase the weight corresponding to the brightness of the divided region.

As an example of image conversion, a microscope observation image other than the phase difference image is input to a trained model trained using a microscope observation image other than the phase difference image as a source image and a phase difference microscope observation image as a target image. Therefore, an artificial phase difference image similar to the phase difference image can be output. The microscope-observed image other than the phase difference image may be an image captured at a magnification of 20 times or more, but it is preferably a magnification of 10 times or more, and more preferably a magnification of 4 times or more.

Here, the microscope observation image other than the phase contrast image as the source image includes a bright field image, a fluorescence image, a differential interference contrast image, and the like, and a trained model can be created separately for each. From this, it is preferable to select the most suitable trained model for the image to be converted, and for that purpose, select a trained model that uses the image captured by the same method as when it was captured as the source image. It is preferable to do.

Further, as the training data of the source image and the target image, images captured at different magnifications of the objective lens may be used, and a trained model trained by each may be created. In that case, it is preferable to select a trained model that uses an image having the same magnification as the image to be converted as training data. It was

<Post-processing>
The image post-processing unit determines whether or not the third image obtained by combining the images has been reduced in the pre-processing (S18). If the image is not reduced, the third image is output as it is (S20). When the image is reduced, the reduced portion is enlarged to generate a fourth image (S19), and the fourth image is output (S20).

<Output of artificial phase difference image>
Finally, the image output unit 121 outputs the generated artificial phase difference image and causes the image display means 123 to display it. At this time, the image display means 123 may display the original image and / or an image of only the region where the cells exist.

Note that the image conversion process and the output of the artificial phase difference image may be performed while observing the sample in the image pickup apparatus. In this case, when the original image is acquired, preprocessing and image conversion are performed immediately, and the artificial phase difference image is displayed in real time on the image display means 123. As a result, the user can operate the microscope while looking at the image display means 123 on which the artificial retardation image is displayed.

<Programs and storage media>
A program for causing a computer to execute the image processing method described so far, and a non-transient computer-readable storage medium for storing the program are also embodiments of the technique of the present disclosure.

HEK293 cells were seeded on a glass bottom dish with a diameter of 35 mm, and a bright field image (using a camera or transmission detector for imaging) and a phase difference image were imaged in a 50% confluent state (objective lens was 60 times).

A trained model trained by CycleGAN was prepared with multiple bright-field images or transparent bright-field images as sources and multiple phase-difference images as targets.

On the other hand, a bright-field image or a transparent bright-field image different from the source was reduced by 50%, input to the trained model, and converted into an artificial phase difference image using a generator. The obtained image was magnified twice using linear interpolation to obtain a converted image. As a comparative example, the bright field image was converted into an artificial retardation image without reduction-enlargement processing. The images before and after the conversion are shown in FIGS. 3 and 4.

As is clear from FIGS. 3 and 4, when a bright-field image or a transparent bright-field image is converted into a phase-difference image using an image conversion network, white dot-like noise, which is an artifact, is generated in the background portion. In such cases, the techniques of the present disclosure have made it possible to remove noise. The

Claims

A method for transforming an image that is not a phase difference image using a trained model.
The method is
The first step of acquiring a first non-phase difference image of a biological sample,
A second step of performing a smoothing process on at least a part of the first image to generate a second image, and a second step.
A third step of converting the second image into a third image using the trained model, and the like.
The trained model is a trained model in which a learning source image group consisting of images that are not retardation images and a learning target image group consisting of retardation images are trained as training data.
The method according to claim 1, wherein the smoothing process is a process for correcting a luminance value in at least a part of the first image.
The method according to claim 2, wherein the smoothing process is a process of reducing at least a part of the area of the first image.
The method according to any one of claims 1 to 3, further comprising a fourth step of outputting the generated third image.
The method of claim 3, comprising a fifth step of enlarging at least a portion of the third image to generate a fourth image.
The method according to claim 5, further comprising the step of outputting the fourth image.
The third step is
A sub-step that divides the second image into smaller areas,
A sub-step in which each of the small areas is input to the learning model and converted.
A sub-step that combines the converted subregions to generate the third image,
The method according to any one of claims 1 to 6, comprising the method according to claim 1.
In the second step, the smoothing process is performed on the entire first image, and the smoothing process is performed.
In the fifth step, the entire third image is enlarged.
The method is
A sixth step of inputting the first image into the learning model and converting it into a fifth image,
A seventh step of synthesizing a first region in the fourth image and a second region different from the first region in the fifth image to generate the fourth image.
5. The method of claim 5.
The first image includes an object region in which the biological sample is imaged and a background region that does not include the biological sample.
The image conversion method according to any one of claims 1 to 8, wherein the partial area is a part or all of the background area.
The image conversion method according to claim 9, further comprising a step of detecting the object region and the background region from the first image.
The image conversion method according to any one of claims 1 to 10, wherein the biological sample is a cell.
The method according to any one of claims 1 to 11, wherein the first image and the source image are a bright field image, a fluorescent image, or a differential interference image.
A program that causes a computer to execute the method according to any one of claims 1 to 12.
An image processing apparatus having a processing unit that executes the method according to any one of claims 1 to 13.
A method of transforming an image of a biological sample using a trained model.
The process of acquiring the first image of the biological sample taken by the first method,
A step of selecting a trained model suitable for the first method from a plurality of different trained models based on the information regarding the first method.
A step of converting the original image into an image similar to an image captured by a second method different from the first method by using the selected trained model.
Including the step of outputting the converted image.
The trained model is a model in which a learning source image group consisting of images captured by the first method and a learning target image group consisting of images captured by the second method are trained as training data. Is the way.
The method according to claim 15, wherein the plurality of different trained models have different first methods from each other.
The method according to claim 15 or 16, wherein the plurality of different image conversion networks have different magnifications at the time of capturing an image included in the learning data.