WO2021095256A1 - 画像処理システム、画像処理方法、及び、プログラム - Google Patents

画像処理システム、画像処理方法、及び、プログラム Download PDF

Info

Publication number
WO2021095256A1
WO2021095256A1 PCT/JP2019/044942 JP2019044942W WO2021095256A1 WO 2021095256 A1 WO2021095256 A1 WO 2021095256A1 JP 2019044942 W JP2019044942 W JP 2019044942W WO 2021095256 A1 WO2021095256 A1 WO 2021095256A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
trained
model
image processing
models
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2019/044942
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
啓悟 森
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Olympus Corp
Original Assignee
Olympus Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Olympus Corp filed Critical Olympus Corp
Priority to JP2021555765A priority Critical patent/JPWO2021095256A1/ja
Priority to PCT/JP2019/044942 priority patent/WO2021095256A1/ja
Publication of WO2021095256A1 publication Critical patent/WO2021095256A1/ja
Priority to US17/731,407 priority patent/US12345870B2/en
Anticipated expiration legal-status Critical
Priority to JP2024116698A priority patent/JP7810759B2/ja
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B21/00—Microscopes
    • G02B21/36—Microscopes arranged for photographic purposes or projection purposes or digital imaging or video purposes including associated control and data processing arrangements
    • G02B21/365—Control or image processing arrangements for digital or video microscopes
    • G02B21/367—Control or image processing arrangements for digital or video microscopes providing an output produced by processing a plurality of individual source images, e.g. image tiling, montage, composite images, depth sectioning, image comparison
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0475—Generative networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/09—Supervised learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/094—Adversarial learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4046—Scaling of whole images or parts thereof, e.g. expanding or contracting using neural networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/60—Image enhancement or restoration using machine learning, e.g. neural networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0012—Biomedical image inspection
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10056—Microscopic image
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30168—Image quality inspection
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/617—Upgrading or updating of programs or applications for camera control
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/95—Computational photography systems, e.g. light-field imaging systems
    • H04N23/951—Computational photography systems, e.g. light-field imaging systems by using two or more images to influence resolution, frame rate or aspect ratio

Definitions

  • the disclosure of this specification relates to an image processing system, an image processing method, and a program.
  • Non-Patent Document 1 describes a technique for converting an image constrained by a diffraction limit into a super-resolution image by using a hostile generation network (GAN) model.
  • Non-Patent Document 2 describes a technique for expanding the observable range to biological phenomena that were conventionally difficult to observe by image restoration based on deep learning.
  • the trained model of machine learning has high generalization performance.
  • the performance tends to deteriorate for an image that is significantly different from the image used for learning.
  • an object of one aspect of the present invention is to provide a technique for realizing high generalization performance in image processing for improving image quality.
  • the image processing system is a selection unit that selects a trained model from a plurality of trained models that have learned image conversion to convert an input image into an output image having higher image quality than the input image. And an image conversion unit that performs the image conversion using the trained model selected by the selection unit, and each of the plurality of trained models has at least a sample type from the other trained models. It is a trained model trained with different images, or a trained model trained with images having at least a different image quality range from other trained models.
  • the image processing method selects and selects a trained model from a plurality of trained models that have learned image conversion to convert an input image into an output image having higher image quality than the input image.
  • the image conversion is performed using the trained model, and each of the plurality of trained models is a trained model trained with an image having at least a different sample type from the other trained models, or another trained model.
  • a model is a trained model trained with images having at least a different image quality range.
  • the program selects a trained model from a plurality of trained models that have learned image conversion to convert an input image into an output image having higher image quality than the input image on a computer.
  • the image transformation is performed using the selected trained model, and each of the plurality of trained models is a trained model trained with an image having at least a different sample type from the other trained models, or another trained model.
  • the trained model is a trained model trained with images having at least a different image quality range, and the process is executed.
  • FIG. 1 It is a figure which illustrated the structure of the system 1. It is a figure which illustrated the physical structure of the image processing unit 20. It is a figure which illustrated the functional structure of the image processing unit 20 which concerns on 1st Embodiment. It is a figure for demonstrating an example of a plurality of trained models. It is a flowchart which shows an example of the processing performed by the image processing unit 20. It is a figure which showed an example of the screen displayed by the image processing unit 20. It is a figure for demonstrating another example of a plurality of trained models. It is a figure which showed another example of the screen which the image processing unit 20 displays. It is a figure which showed still another example of the screen displayed by the image processing unit 20.
  • FIG. 1 is a diagram illustrating the configuration of the system 1.
  • FIG. 2 is a diagram illustrating the physical configuration of the image processing unit 20.
  • the configuration of the system 1 will be described with reference to FIGS. 1 and 2.
  • the system 1 shown in FIG. 1 is an example of an image processing system for improving image quality, and includes an image processing unit 20 for performing image processing.
  • the system 1 may further include an image acquisition unit 10 that acquires an image to be input to the image processing unit 20.
  • the image processing unit 20 communicates with the image acquisition unit 10 and the terminal unit 30.
  • the terminal unit 30 includes, for example, a notebook-type terminal device 31 and a tablet-type terminal device 32.
  • the system 1 may include a terminal unit 30.
  • the image acquisition unit 10 is a device or system that acquires a digital image of a sample by imaging the sample.
  • the image acquisition unit 10 includes, for example, a digital camera 11, an endoscope system 12, a microscope system 13, and the like.
  • the image acquisition unit 10 outputs the acquired image to the image processing unit 20.
  • the image acquired by the image acquisition unit 10 may be sent directly from the image acquisition unit 10 to the image processing unit 20, and indirectly sent from the image acquisition unit 10 to the image processing unit 20 via other devices. May be done.
  • the image processing unit 20 is a device or system that performs image processing using a trained model of machine learning, especially deep learning.
  • the image processing performed by the image processing unit 20 is an image conversion that converts an input image into an output image having a higher image quality than the input image, and for example, improvement of image quality such as noise reduction, resolution improvement, and aberration correction is achieved.
  • the image processing unit 20 may include one or more electric circuits, and may be a dedicated or general-purpose computer. Specifically, the image processing unit 20 includes, for example, a processor 21 and a memory 22 as shown in FIG. The image processing unit 20 may further include an auxiliary storage device 23, an input device 24, an output device 25, a portable recording medium driving device 26 for driving the portable recording medium 29, a communication module 27, and a bus 28. .. The auxiliary storage device 23 and the portable recording medium 29 are examples of non-transient computer-readable recording media on which programs are recorded.
  • the processor 21 is, for example, an electric circuit (circuitry) including a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and the like.
  • the processor 21 expands the program stored in the auxiliary storage device 23 or the portable recording medium 29 into the memory 22, and then executes the program to perform programmed processing such as an image processing method described later.
  • the memory 22 is, for example, an arbitrary semiconductor memory such as a RAM (Random Access Memory).
  • the memory 22 functions as a work memory for storing the program or data stored in the auxiliary storage device 23 or the portable recording medium 29 when the program is executed.
  • the auxiliary storage device 23 is, for example, a non-volatile memory such as a hard disk or a flash memory.
  • the auxiliary storage device 23 is mainly used for storing various data and programs.
  • the portable recording medium drive device 26 accommodates the portable recording medium 29.
  • the portable recording medium driving device 26 can output the data stored in the memory 22 or the auxiliary storage device 23 to the portable recording medium 29, and reads out programs, data, and the like from the portable recording medium 29. Can be done.
  • the portable recording medium 29 is any portable recording medium.
  • the portable recording medium 29 includes, for example, an SD card, a USB (Universal Serial Bus) flash memory, a CD (Compact Disc), a DVD (Digital Versatile Disc), and the like.
  • the input device 24 is, for example, a keyboard, a mouse, or the like.
  • the output device 25 is, for example, a display device, a printer, or the like.
  • the communication module 27 is, for example, a wired communication module that communicates with the image acquisition unit 10 and the terminal unit 30 connected via an external port.
  • the communication module 27 may be a wireless communication module.
  • the bus 28 connects the processor 21, the memory 22, the auxiliary storage device 23, and the like to each other so that data can be exchanged.
  • FIG. 3 is a diagram illustrating the functional configuration of the image processing unit 20 according to this embodiment.
  • FIG. 4 is a diagram for explaining an example of a plurality of trained models.
  • FIG. 5 is a flowchart showing an example of processing performed by the image processing unit 20.
  • FIG. 6 is a diagram showing an example of a screen displayed by the image processing unit 20.
  • the image processing method performed by the system 1 will be described with reference to FIGS. 3 to 6.
  • the image processing unit 20 selects a trained model from a plurality of trained models as a configuration related to the image processing method performed by the system 1, and the selection unit 41 selects the trained model. It includes an image conversion unit 42 that performs image conversion using the trained model, and a storage unit 43 that stores a plurality of trained models.
  • the selection unit 41, the image conversion unit 42, and the storage unit 43 have a functional configuration realized by one or more electric circuits included in the image processing unit 20. More specifically, the selection unit 41 and the image conversion unit 42 have, for example, a functional configuration realized by the processor 21 and the memory 22 shown in FIG. 2, and the processor 21 executes a program stored in the memory 22. May be realized by.
  • the storage unit 43 may have, for example, a functional configuration realized by the memory 22 and the auxiliary storage device 23 shown in FIG.
  • the selection unit 41 selects a trained model from a plurality of trained models stored in the storage unit 43 in response to an input. Further, the selection unit 41 outputs information for identifying the selected trained model to the image conversion unit 42.
  • the image conversion unit 42 applies the trained model selected by the selection unit 41 to the input image, performs image conversion, and generates an output image.
  • the output image may be stored in the auxiliary storage device 23, for example. Further, the output image may be displayed on the output device 25, for example. Further, the output image may be output to the terminal unit 30 via the communication module 27, for example, or may be displayed by the display device of the terminal unit 30.
  • Each of the plurality of trained models stored in the storage unit 43 is a trained model that has learned image conversion for converting an input image into an output image having a higher image quality than the input image.
  • a trained model M when a plurality of trained models M1, M2, M3 ... Are not particularly distinguished, each or a set thereof is referred to as a trained model M.
  • the trained model M stored in the storage unit 43 may be generated, for example, by taking the following procedure in advance. If the content of the image quality improved by the trained model M is noise, first, a plurality of pairs of images with less noise and images with more noise are prepared. The images that make up the pair are images of the same object. Specifically, the same sample is imaged at the same imaging position, and an image with less noise and an image with more noise are acquired as a pair. This is repeated with a plurality of imaging positions or a plurality of samples of the same type. Then, the trained model M is generated by training the model to perform image conversion that converts a noisy image into a noiseless image by deep learning using a plurality of image pairs.
  • the amount of noise contained in the image can be adjusted, for example, by changing the length of the exposure time and the illumination intensity at the time of imaging. Therefore, the model may be trained using a pair of images in which the image captured with a relatively long exposure time is regarded as an image with less noise and the image captured with a relatively short exposure time is regarded as a noisy image.
  • the content of the image quality improved by the trained model M is the resolution
  • the images that make up the pair are images of the same object. Specifically, the same sample is imaged at the same imaging position, and a low-resolution image and a high-resolution image are acquired as a pair. This is repeated with a plurality of imaging positions or a plurality of samples of the same type.
  • the trained model M is generated by training the model to perform image conversion that converts a low-resolution image into a high-resolution image by deep learning using a plurality of image pairs.
  • the image resolution can be adjusted by changing the pixel resolution and the optical resolution.
  • the image resolution may be adjusted by changing the objective lens used at the time of imaging. Therefore, the image captured by using the objective lens having a relatively low numerical aperture is regarded as a low resolution image, and the image captured by using an objective lens having a relatively high numerical aperture is regarded as a high resolution image.
  • the model may be trained using a pair of images.
  • the trained model M is generated by training the model to perform image conversion that converts an image whose aberration is not sufficiently corrected by deep learning using a plurality of image pairs into an image whose aberration is sufficiently corrected. To do.
  • the degree of aberration correction can be adjusted by changing the objective lens used at the time of imaging, for example. Therefore, an image captured by using an objective lens having a relatively large aberration is regarded as an image in which the aberration is not sufficiently corrected, and an image captured by using an objective lens having a relatively small aberration has sufficient aberration.
  • the model may be trained using a pair of images as corrected images.
  • each of the plurality of trained models M is a trained model trained using images having at least a different sample type from the other trained models.
  • the trained model M1 is a trained model trained using an image of a cell nucleus.
  • the trained model M2 is a trained model trained using an image of an actin filament.
  • the trained model M3 is a trained model trained using an image of mitochondria.
  • the image processing unit 20 configured as described above is selected when the user selects an image to be improved in image quality from the images acquired by the image acquisition unit 10 using the input device 24.
  • the obtained image is acquired as the input image 50 (step S1).
  • the image processing unit 20 acquires the image selected by the user as the input image 50, and displays the screen G1 including the input image 50 on the output device 25 as shown in FIG. In this example, the cell nucleus is shown in the input image 50.
  • the image processing unit 20 receives a plurality of trained models corresponding to the selected image quality improvement content. Is displayed (step S2).
  • FIG. 6 shows an example in which the user selects “noise reduction” using the tab 61.
  • three trained models (M1 to M3) for cell nuclei, actin filaments, and mitochondria are displayed.
  • the image processing unit 20 selects the trained model 20 by the user.
  • the trained model is selected (step S3).
  • the selection unit 41 of the image processing unit 20 selects the trained model M1 for the cell nucleus from the plurality of trained models M according to the user's selection.
  • the image processing unit 20 converts the input image 50 using the trained model selected in step S3 (step S4).
  • the image conversion unit 42 of the image processing unit 20 applies the trained model M1 selected in step S3 to the input image 50 acquired in step S1 to generate an output image with less noise than the input image 50. To do.
  • the image processing unit 20 displays the converted image obtained in step S4 (step S5).
  • the image processing unit 20 displays the output image generated in step S4 on the output device 25.
  • the output image may be displayed side by side with the input image 50, or may be displayed in a pop-up format.
  • the trained model M is prepared in advance. That is, a trained model M optimized for each sample is prepared. Then, the user is made to select a trained model according to the type of the sample shown in the input image 50 from the plurality of trained models M generated for each type of the sample.
  • the type of sample corresponding to the trained model selected by the user does not necessarily have to match the type of sample shown in the input image 50.
  • the user may select a trained model trained using an image of a sample having a shape similar to the sample shown in the input image 50.
  • the image quality improvement performance of the trained model differs depending on the sample, mainly because the shape differs for each sample.
  • the system 1 can apply a trained model learned using an image similar to the input image 50, regardless of the sample shown in the input image 50. Therefore, even when images of various samples having different shapes are input as the input image 50, stable image quality improvement performance can be exhibited regardless of the input image 50. Therefore, according to the system 1, high generalization performance can be realized in the image processing for improving the image quality of the system 1 as a whole.
  • the image quality can be improved by image processing.
  • the image acquisition unit 10 acquires an image having a higher image quality
  • a trade-off with the image quality may occur in terms other than the image quality. For example, if the exposure time is long to reduce noise, it takes time to acquire an image. In addition, increasing the illumination intensity increases the damage to the sample.
  • the numerical aperture is increased to improve the resolution, it becomes necessary to use an immersion liquid. Especially when the numerical aperture is high, it is necessary to use oil as an immersion liquid, which increases the amount of work performed by the user such as cleaning.
  • the magnification of the objective lens is increased, the observation range becomes narrower. Further, if a higher performance objective lens is used for aberration correction, the cost of the device increases. In the system 1, these disadvantages can be avoided by improving the image quality by image processing.
  • the image quality improvement performance of a single trained model differs depending on the image quality range of the input image 50.
  • the image quality improvement performance of the trained model differs depending on the image quality range mainly because the appearance of the image differs depending on the image quality range.
  • inputting a high noise level image into a trained model capable of converting a medium noise level image into a low noise level image may not sufficiently reduce the noise level.
  • inputting a medium noise level image into a trained model that can convert a high noise level image to a low noise level image can have the effect of weakening the signal.
  • each of the plurality of trained models stored in the storage unit 43 is a trained model trained using images having at least a different image quality range from the other trained models. It is different from the embodiment of.
  • FIG. 7 is a diagram for explaining another example of a plurality of trained models.
  • 8 to 10 are views showing another example of the screen displayed by the image processing unit 20, respectively.
  • a modified example of the first embodiment will be described with reference to FIGS. 7 to 10.
  • each of the plurality of trained models for improving noise stored in the storage unit 43 is a trained model trained using an image having a noise level range different from that of the other trained models. is there. Specifically, each of the plurality of trained models has a different exposure time of the image used for input to the model among the images forming the pair from that of the other trained models.
  • the trained model M4 is a model using an image captured with a relatively short exposure time as an image used for input to the model among the images constituting the pair.
  • the trained model M5 is a model using an image captured at a medium exposure time as an image used for input to the model among the images constituting the pair.
  • the trained model M6 is a model using an image captured with a relatively long exposure time as an image used for input to the model among the images constituting the pair.
  • each of the plurality of trained models for improving the resolution stored in the storage unit 43 is a trained model trained using an image having a resolution range different from that of the other trained models.
  • the numerical aperture of the objective lens used at the time of capturing the image used for input to the model among the images forming the pair is the same as that of the other trained model. Is different. That is, each of the plurality of trained models is a trained model trained with an image acquired by using an objective lens having a numerical aperture different from that of the other trained models.
  • each of the plurality of trained models for improving the aberration level stored in the storage unit 43 is a trained model learned using an image having an aberration level range different from that of the other trained models. .. Specifically, in each of the plurality of trained models, for example, the aberration performance of the objective lens used at the time of capturing the image used for input to the model among the images forming the pair is different from that of the other trained models. Is different. That is, each of the plurality of trained models is a trained model trained with an image acquired by using an objective lens having an aberration performance different from that of the other trained models.
  • the image processing unit 20 displays the screen G2 shown in FIG. 8 on the output device 25.
  • the user can grasp the noise level to be dealt with by each trained model.
  • the trained model M4 can deal with an image having a short exposure time, that is, an image having a large noise. Therefore, the user can select the trained model from the trained model M4 and the trained model M6 according to the noise level of the input image 50. Therefore, the system 1 can exhibit stable noise reduction performance regardless of the noise level of the input image 50.
  • the image processing unit 20 displays the screen G3 shown in FIG. 9 on the output device 25.
  • the user can grasp the resolution to be dealt with by each trained model.
  • the trained model M7 it can be understood that an image having an NA of about 0.8 can be converted into an image having an NA of about 1.1. Therefore, the user can select the trained model from the trained model M7 to the trained model M9 according to the resolution of the input image 50, so that the resolution is stably improved regardless of the resolution of the input image 50. It can demonstrate its performance.
  • the image processing unit 20 displays the screen G4 shown in FIG. 10 on the output device 25.
  • the user can grasp the aberration level to be dealt with by each trained model.
  • the learned model M10 can further improve the achromatic level aberration correction state. Therefore, the user can select the trained model from the trained model M10 and the trained model M12 according to the aberration level of the input image 50. Therefore, the system 1 can exhibit stable aberration correction performance regardless of the aberration level of the input image 50.
  • the system 1 can realize high generalization performance in the image processing for improving the image quality of the system 1 as a whole, as in the first embodiment. Further, since the image quality improvement is achieved by the image processing, it is possible to avoid the disadvantage caused by the image acquisition unit 10 acquiring the image with higher image quality.
  • the exposure time as a method of classifying the noise level of the image has been shown.
  • the illumination intensity may be used, and the combination of the exposure time and the illumination intensity may be used. May be used.
  • the noise level and the appearance of noise differ depending on the type of image acquisition device. Specifically, for example, the noise level and the appearance of noise are different between the image acquired by the laser scanning microscope and the image acquired by the wide-field microscope having a digital camera. Therefore, the type of image acquisition device may be used for classifying the noise level.
  • each of the plurality of trained models may be a trained model trained with an image acquired by a laser scanning microscope having a confocal pinhole diameter different from that of the other trained models, and may be another trained model. It may be a trained model trained with an image acquired with a pixel resolution different from that of the model.
  • the image quality range of the image included in the learning data is classified by using the setting of the image acquisition unit 10 at the time of imaging (for example, the device itself such as the objective lens to be used, the setting for the device such as the exposure time).
  • the image quality range may be classified from other than the setting of the image acquisition unit 10.
  • the person who observes the image may subjectively classify it.
  • FIG. 11 is a diagram for explaining still another example of the plurality of trained models.
  • FIG. 12 is a diagram showing still another example of the screen displayed by the image processing unit 20.
  • the second embodiment will be described with reference to FIGS. 11 and 12.
  • each of the plurality of trained models for improving noise stored in the storage unit 43 uses an image in which at least the noise level range and the combination of sample types are different from those of the other trained models. It is a trained model that has been learned. That is, as shown in FIG. 11, trained models are prepared for each combination of noise level range and sample type. Specifically, each of the plurality of trained models differs from that of the other trained models in the combination of the exposure time and the sample type of the images used for input to the model among the images forming the pair. There is.
  • the trained models M41 to M43 are models using images captured with a relatively short exposure time as the images used for input to the model among the images constituting the pair, and are different from each other.
  • the trained models M51 to M53 are models using images captured at a medium exposure time as images used for input to the model among the images constituting the pair, and images of samples different from each other are used. It is a model.
  • the trained models M61 to M63 are models using images captured with a relatively long exposure time for the images used for input to the model among the images constituting the pair, and images of samples different from each other are used. It is a model.
  • each of the plurality of trained models for improving the resolution stored in the storage unit 43 is learned by using an image in which the combination of the resolution range and the sample type is different from that of the other trained models. It is a finished model. Further, each of the plurality of trained models for improving the aberration level stored in the storage unit 43 was trained using images having a different combination of aberration level range and sample type from the other trained models. It is a trained model.
  • the image processing unit 20 displays the screen G5 shown in FIG. 12 on the output device 25.
  • the user can grasp the combination of the noise level and the sample to be dealt with by each trained model.
  • the trained model M41 it can be understood that the sample is a cell nucleus and can deal with an image having a short exposure time, that is, an image having a large noise. Therefore, the user can select the trained model from the trained model M41 and the trained model M63 according to the combination of the sample of the input image 50 and the noise level. Therefore, the system 1 can exhibit stable noise reduction performance regardless of the combination of the sample of the input image 50 and the noise level.
  • the system 1 can realize higher generalization performance than the first embodiment in the image processing for improving the image quality of the system 1 as a whole. Further, since the image quality improvement is achieved by the image processing, it is possible to avoid the disadvantage caused by the image acquisition unit 10 acquiring the image with higher image quality.
  • FIG. 13 is a diagram illustrating the functional configuration of the image processing unit 20 according to the present embodiment.
  • FIG. 14 is a diagram for explaining the selection of the trained model based on the similarity.
  • the third embodiment will be described with reference to FIGS. 13 and 14.
  • the image processing unit 20 has a selection unit 71 that selects a trained model from a plurality of trained models M11 to M13, and a trained model selected by the selection unit 71. It includes an image conversion unit 72 that performs image conversion using the above, and a storage unit 73 that stores a plurality of trained models M11 to M13.
  • the image processing unit 20 according to the present embodiment is different from the image processing unit 20 according to the first embodiment in the following two points.
  • the storage unit 73 stores a plurality of model supplementary information MS11 to MS13 that show the characteristics of the plurality of trained models in association with the plurality of trained models M11 to M13.
  • the plurality of model supplementary information MS11 to MS13 may be, for example, a plurality of trained models. It is a plurality of representative images corresponding to the sample types of the trained models M11 to M13.
  • the representative image is preferably an image that captures the shape characteristics of the sample well.
  • the second point is that the selection unit 71 selects a trained model to be used for image conversion from the plurality of trained models M11 to M13 based on the input image 50 and the plurality of model supplementary information MS11 to MS13.
  • the selection unit 71 may compare the input image 50 with each of the plurality of representative images (model supplementary information MS11 to MS13), and select the trained model based on the comparison result. More specifically, as shown in FIG. 14, the selection unit 71 compares each of the input image 50 and the plurality of representative images (model supplementary information MS11 to MS13) to calculate the similarity, and has the highest degree of similarity. A trained model corresponding to a high representative image may be selected.
  • the similarity may be calculated by using any known algorithm such as an algorithm for calculating the local feature amount. Further, the degree of similarity may be calculated using a trained model in which similar images are learned in advance. In the example shown in FIG. 14, the trained model M11 corresponding to the representative diagram, which is the model supplementary information MS11, is selected as the trained model used for the image conversion.
  • the image processing unit 20 automatically selects the trained model according to the input image 50. Therefore, the user can obtain an image with improved image quality simply by selecting an image to be improved in image quality. Further, since the image processing unit 20 appropriately selects the trained model, it is possible to realize high generalization performance in the image processing for improving the image quality of the system 1 as a whole.
  • the model supplementary information MS11 to MS13 is not limited to the image.
  • the plurality of model supplementary information MS11 to MS13 may be a plurality of trained models. It may be a plurality of model image quality information corresponding to the image quality range of the images used for learning M11 to M13, and more specifically, the model image quality information may be, for example, a quantified noise level. May be good.
  • the selection unit 71 calculates the noise level of the input image 50 from the input image 50 as the input image quality information.
  • the noise level of the input image 50 is not particularly limited, but may be calculated as, for example, the dispersion of the brightness in the background portion in the image. This is because it can be evaluated that the larger the luminance dispersion is, the larger the noise is. Further, the selection unit 71 selects a trained model to be used for image conversion from the plurality of trained models M11 to M13 based on the comparison result between the input image quality information and the plurality of model image quality information.
  • the selection unit 71 may select, for example, a trained model corresponding to the noise level closest to the noise level of the input image 50 as the trained model used for the image conversion. In this case as well, the user can obtain an image with improved image quality simply by selecting an image to be improved in image quality.
  • FIG. 15 is a diagram illustrating the functional configuration of the image processing unit 20 according to the present embodiment.
  • the fourth embodiment will be described with reference to FIG.
  • the image processing unit 20 has a selection unit 81 that selects a trained model from a plurality of trained models M21 to M23, and a trained model selected by the selection unit 81. It includes an image conversion unit 82 that performs image conversion using the above, and a storage unit 83 that stores a plurality of trained models M21 to M23.
  • the image processing unit 20 according to the present embodiment is different from the image processing unit 20 according to the first embodiment in the following two points.
  • the storage unit 83 stores a plurality of model supplementary information MS21 to MS23 indicating the characteristics of the plurality of trained models in association with the plurality of trained models M21 to M23.
  • the plurality of trained models M21 to M23 stored in the storage unit 83 are trained models trained with images having at least a different image quality range from the other trained models.
  • the second point is that the selection unit 81 acquires the setting information of the image acquisition unit 10 and uses it for image conversion from the plurality of trained models M21 to M23 based on the acquired setting information and the plurality of model supplementary information MS21 to MS23.
  • the point is to select the trained model to be used.
  • the selection unit 81 acquires, for example, the resolution information of the microscope system 13 from the microscope system 13 that has acquired the input image 50. If the microscope system 13 is a laser scanning microscope, for example, the numerical aperture of the objective lens, the magnification of the objective lens, the scan size of the galvano scanner (for example, 512 ⁇ 512, 1024 ⁇ 1024), the pinhole diameter, etc. are acquired. To do.
  • the selection unit 81 calculates the resolution of the input image 50 from this information and compares it with the resolution specified by the plurality of model supplementary information MS21 to MS23 read from the storage unit 83. As a result of the comparison, the selection unit 81 selects, for example, model supplementary information indicating the resolution closest to the resolution of the input image 50, and uses the trained model corresponding to the model supplementary information as the trained model used for the image conversion. select.
  • the image processing unit 20 automatically selects the trained model according to the input image 50, as in the third embodiment. Therefore, the user can obtain an image with improved image quality simply by selecting an image to be improved in image quality. Further, as in the third embodiment, the image processing unit 20 can realize high generalization performance in image processing for improving the image quality of the system 1 as a whole by selecting an appropriate trained model. Is.
  • a trained model suitable for the input image is specified as one
  • the image processing unit 20 has a plurality of trained models suitable for the input image to improve the image quality. Those with different degrees of may be stored.
  • a control radio button in this example
  • 1 is based on the degree of image quality improvement desired by the user.
  • One trained model may be identified. This allows the user to select the balance between the image quality and the side effects associated with the image processing.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • Computing Systems (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Biomedical Technology (AREA)
  • Molecular Biology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Multimedia (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Chemical & Material Sciences (AREA)
  • Analytical Chemistry (AREA)
  • Optics & Photonics (AREA)
  • Medical Informatics (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Radiology & Medical Imaging (AREA)
  • Quality & Reliability (AREA)
  • Image Analysis (AREA)
  • Microscoopes, Condenser (AREA)
  • Image Processing (AREA)
PCT/JP2019/044942 2019-11-15 2019-11-15 画像処理システム、画像処理方法、及び、プログラム Ceased WO2021095256A1 (ja)

Priority Applications (4)

Application Number Priority Date Filing Date Title
JP2021555765A JPWO2021095256A1 (https=) 2019-11-15 2019-11-15
PCT/JP2019/044942 WO2021095256A1 (ja) 2019-11-15 2019-11-15 画像処理システム、画像処理方法、及び、プログラム
US17/731,407 US12345870B2 (en) 2019-11-15 2022-04-28 Image processing system selecting learned model based on setting information of a microscope system, and image processing method and computer-readable medium
JP2024116698A JP7810759B2 (ja) 2019-11-15 2024-07-22 画像処理システム、画像処理方法、及び、プログラム

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2019/044942 WO2021095256A1 (ja) 2019-11-15 2019-11-15 画像処理システム、画像処理方法、及び、プログラム

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/731,407 Continuation US12345870B2 (en) 2019-11-15 2022-04-28 Image processing system selecting learned model based on setting information of a microscope system, and image processing method and computer-readable medium

Publications (1)

Publication Number Publication Date
WO2021095256A1 true WO2021095256A1 (ja) 2021-05-20

Family

ID=75912095

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2019/044942 Ceased WO2021095256A1 (ja) 2019-11-15 2019-11-15 画像処理システム、画像処理方法、及び、プログラム

Country Status (3)

Country Link
US (1) US12345870B2 (https=)
JP (2) JPWO2021095256A1 (https=)
WO (1) WO2021095256A1 (https=)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2023276327A1 (ja) * 2021-06-29 2023-01-05 浜松ホトニクス株式会社 光学画像処理方法、機械学習方法、学習済みモデル、機械学習の前処理方法、光学画像処理モジュール、光学画像処理プログラム、及び光学画像処理システム
WO2023162394A1 (ja) * 2022-02-25 2023-08-31 株式会社Ｓｃｒｅｅｎホールディングス モデル選定方法および画像処理方法
JPWO2023248355A1 (https=) * 2022-06-21 2023-12-28
KR20250140559A (ko) 2023-04-25 2025-09-25 주식회사 히타치하이테크 학습 모델 적합 영역 검지 장치, 학습 모델 적합 영역 검지 방법, 그리고 학습 모델 운용 방법

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
TW202143119A (zh) * 2020-05-07 2021-11-16 瑞典商安訊士有限公司 在具有光強度廣泛範圍的場景中利用神經網路之物件偵測
US11887313B2 (en) * 2020-09-30 2024-01-30 Splitmedialabs Limited Computing platform using machine learning for foreground mask estimation
US20250078214A1 (en) * 2023-09-01 2025-03-06 Nvidia Corporation Noise augmentation for computer vision and machine learning systems and applications
US20250285237A1 (en) * 2024-03-08 2025-09-11 University Of Southern California Processing multiplex images and analysis of immune enriched spatial proteomic data

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2007189589A (ja) * 2006-01-16 2007-07-26 Sony Corp 情報処理装置および情報処理方法、学習装置および学習方法、並びにプログラム
WO2018037521A1 (ja) * 2016-08-25 2018-03-01 キヤノン株式会社 画像処理方法、画像処理装置、撮像装置、画像処理プログラム、記憶媒体
JP2018195069A (ja) * 2017-05-17 2018-12-06 キヤノン株式会社 画像処理装置および画像処理方法
WO2018225133A1 (ja) * 2017-06-05 2018-12-13 楽天株式会社 画像処理装置、画像処理方法、および画像処理プログラム
JP2018206371A (ja) * 2017-06-02 2018-12-27 三菱電機株式会社 コンピューター実施画像再構成システムおよび画像再構成方法

Family Cites Families (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP6019998B2 (ja) * 2012-02-17 2016-11-02 ソニー株式会社 撮像装置、撮像制御プログラム及び撮像方法
WO2015041177A1 (ja) * 2013-09-18 2015-03-26 株式会社ニコン 画像解析装置、画像解析方法、画像解析プログラム、細胞の製造方法、細胞の培養方法、および細胞の製造装置
JP6363890B2 (ja) * 2014-07-04 2018-07-25 オリンパス株式会社 走査型顕微鏡装置および超解像画像の生成方法
US10379329B2 (en) * 2014-12-15 2019-08-13 Olympus Corporation Microscope system and setting value calculation method
WO2018052788A1 (en) * 2016-09-13 2018-03-22 Inscopix, Inc. Adapter for microscopic imaging
GB201619637D0 (en) * 2016-11-21 2017-01-04 Pekna Marcela And Pekny Milos And Stokowska Anna C3a receptor agonists
JP6422616B1 (ja) * 2016-12-22 2018-11-14 国立大学法人 筑波大学 データ作成方法及びデータ使用方法
JP6668278B2 (ja) * 2017-02-20 2020-03-18 株式会社日立ハイテク 試料観察装置および試料観察方法
CN110476101B (zh) 2017-06-13 2022-02-18 谷歌有限责任公司 用于病理学的增强现实显微镜
JP2019063146A (ja) * 2017-09-29 2019-04-25 キヤノン株式会社 画像処理装置、眼科撮影装置、画像処理方法、及びプログラム
JP7243718B2 (ja) * 2018-05-25 2023-03-22 ソニーグループ株式会社 制御装置、制御方法、およびプログラム
WO2019229151A2 (de) * 2018-05-30 2019-12-05 Carl Zeiss Microscopy Gmbh Verfahren zum scannen von teilbereichen einer probe mittels eines rastermikroskops, computerprogrammprodukt, computerlesbares medium und system zum scannen von teilbereichen einer probe mittels eines rastermikroskops
US11946854B2 (en) * 2018-12-26 2024-04-02 The Regents Of The University Of California Systems and methods for two-dimensional fluorescence wave propagation onto surfaces using deep learning
US11829885B2 (en) * 2020-06-17 2023-11-28 Canon Kabushiki Kaisha Image processing apparatus that performs machine learning of learning model, method of controlling image processing apparatus, and storage medium

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2007189589A (ja) * 2006-01-16 2007-07-26 Sony Corp 情報処理装置および情報処理方法、学習装置および学習方法、並びにプログラム
WO2018037521A1 (ja) * 2016-08-25 2018-03-01 キヤノン株式会社 画像処理方法、画像処理装置、撮像装置、画像処理プログラム、記憶媒体
JP2018195069A (ja) * 2017-05-17 2018-12-06 キヤノン株式会社 画像処理装置および画像処理方法
JP2018206371A (ja) * 2017-06-02 2018-12-27 三菱電機株式会社 コンピューター実施画像再構成システムおよび画像再構成方法
WO2018225133A1 (ja) * 2017-06-05 2018-12-13 楽天株式会社 画像処理装置、画像処理方法、および画像処理プログラム

Cited By (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2023276327A1 (ja) * 2021-06-29 2023-01-05 浜松ホトニクス株式会社 光学画像処理方法、機械学習方法、学習済みモデル、機械学習の前処理方法、光学画像処理モジュール、光学画像処理プログラム、及び光学画像処理システム
JPWO2023276327A1 (https=) * 2021-06-29 2023-01-05
EP4343681A4 (en) * 2021-06-29 2025-05-21 Hamamatsu Photonics K.K. OPTICAL IMAGE PROCESSING METHOD, MACHINE LEARNING METHOD, LEARNED MODEL, MACHINE LEARNING PREPROCESSING METHOD, OPTICAL IMAGE PROCESSING MODULE, OPTICAL IMAGE PROCESSING PROGRAM, AND OPTICAL IMAGE PROCESSING SYSTEM
JP7784429B2 (ja) 2021-06-29 2025-12-11 浜松ホトニクス株式会社 光学画像処理方法、機械学習方法、学習済みモデル、機械学習の前処理方法、光学画像処理モジュール、光学画像処理プログラム、及び光学画像処理システム
US12579621B2 (en) 2021-06-29 2026-03-17 Hamamatsu Photonics K.K. Optical image processing method, machine learning method, trained model, machine learning pre-processing method, optical image processing module, optical image processing program, and optical image processing system
WO2023162394A1 (ja) * 2022-02-25 2023-08-31 株式会社Ｓｃｒｅｅｎホールディングス モデル選定方法および画像処理方法
JP2023123987A (ja) * 2022-02-25 2023-09-06 株式会社Ｓｃｒｅｅｎホールディングス モデル選定方法および画像処理方法
JP7818985B2 (ja) 2022-02-25 2026-02-24 株式会社Ｓｃｒｅｅｎホールディングス モデル選定方法および画像処理方法
JPWO2023248355A1 (https=) * 2022-06-21 2023-12-28
KR20240163122A (ko) 2022-06-21 2024-11-18 주식회사 히타치하이테크 부적합 검출 장치, 및, 부적합 검출 방법
KR20250140559A (ko) 2023-04-25 2025-09-25 주식회사 히타치하이테크 학습 모델 적합 영역 검지 장치, 학습 모델 적합 영역 검지 방법, 그리고 학습 모델 운용 방법

Also Published As

Publication number Publication date
JP2024133338A (ja) 2024-10-01
JP7810759B2 (ja) 2026-02-03
US20220252857A1 (en) 2022-08-11
JPWO2021095256A1 (https=) 2021-05-20
US12345870B2 (en) 2025-07-01

Similar Documents

Publication Publication Date Title
JP7810759B2 (ja) 画像処理システム、画像処理方法、及び、プログラム
JP5589366B2 (ja) 情報処理装置、情報処理方法及びそのプログラム
CN104285175B (zh) 使用小波分析进行单颗粒定位的方法和装置
CN102687006B (zh) 图像诊断方法、图像诊断装置
JPWO2021095256A5 (https=)
JP6020950B1 (ja) 内視鏡システム、および内視鏡システムの作動方法、並びにプログラム
WO2007097170A1 (ja) スペクトル画像処理方法、コンピュータ実行可能なスペクトル画像処理プログラム、スペクトルイメージングシステム
JP6099477B2 (ja) 撮像装置、顕微鏡システム及び撮像方法
JP2013134574A (ja) 画像データ生成装置、画像データ表示システムおよび画像データ生成方法
JP5940288B2 (ja) 画像処理装置、顕微鏡システム、画像処理方法、及び画像処理プログラム
JP2021071936A (ja) 画像処理装置、画像処理方法、画像処理プログラムおよび記録媒体
EP3232181A1 (en) Information processing device, information acquisition system, information processing method, image information acquisition method, and program
WO2019181072A1 (ja) 画像処理方法、コンピュータプログラムおよび記録媒体
Fan et al. High-fidelity reconstruction of structured illumination microscopy by an amplitude-phase channel attention network with multitemporal information
JP2016139397A (ja) 画像処理装置、画像処理方法、画像表示装置およびコンピュータプログラム
JP2013125519A (ja) 画像処理装置、データ構造、および画像生成装置
JP2017049686A (ja) 画像処理装置
CN112955929B (zh) 图像处理装置、图像处理方法以及程序
WO2022113367A1 (ja) 画像変換方法、プログラム、画像処理装置
JP5119111B2 (ja) 顕微鏡装置
JP5326881B2 (ja) 画像処理装置、画像処理方法、および画像処理プログラム
JP2007010340A (ja) カメラ装置
RU2009134147A (ru) Способ поддержки различения корнеоцитов
JP5176747B2 (ja) 画像フィルタ生成装置、画像フィルタ生成方法およびプログラム
JP5103436B2 (ja) 画像処理装置、画像処理方法及び画像処理プログラム

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19952449

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2021555765

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19952449

Country of ref document: EP

Kind code of ref document: A1