WO2021090469A1 - 情報処理システム、内視鏡システム、学習済みモデル、情報記憶媒体及び情報処理方法 - Google Patents

情報処理システム、内視鏡システム、学習済みモデル、情報記憶媒体及び情報処理方法 Download PDF

Info

Publication number
WO2021090469A1
WO2021090469A1 PCT/JP2019/043806 JP2019043806W WO2021090469A1 WO 2021090469 A1 WO2021090469 A1 WO 2021090469A1 JP 2019043806 W JP2019043806 W JP 2019043806W WO 2021090469 A1 WO2021090469 A1 WO 2021090469A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
resolution
processing
imaging system
learning
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2019/043806
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
直 菊地
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Olympus Corp
Original Assignee
Olympus Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Olympus Corp filed Critical Olympus Corp
Priority to PCT/JP2019/043806 priority Critical patent/WO2021090469A1/ja
Priority to CN201980101970.1A priority patent/CN114651439B/zh
Priority to JP2021554529A priority patent/JP7303896B2/ja
Publication of WO2021090469A1 publication Critical patent/WO2021090469A1/ja
Priority to US17/731,627 priority patent/US12347066B2/en
Anticipated expiration legal-status Critical
Priority to US19/230,291 priority patent/US20250299297A1/en
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/00002—Operational features of endoscopes
    • A61B1/00004—Operational features of endoscopes characterised by electronic signal processing
    • A61B1/00009—Operational features of endoscopes characterised by electronic signal processing of image signals during a use of endoscope
    • A61B1/000095—Operational features of endoscopes characterised by electronic signal processing of image signals during a use of endoscope for image enhancement
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/00002—Operational features of endoscopes
    • A61B1/00004—Operational features of endoscopes characterised by electronic signal processing
    • A61B1/00009—Operational features of endoscopes characterised by electronic signal processing of image signals during a use of endoscope
    • A61B1/000096—Operational features of endoscopes characterised by electronic signal processing of image signals during a use of endoscope using artificial intelligence
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4046—Scaling of whole images or parts thereof, e.g. expanding or contracting using neural networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/60—Image enhancement or restoration using machine learning, e.g. neural networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/70—Denoising; Smoothing
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/73—Deblurring; Sharpening
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/10—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from different wavelengths
    • H04N23/13—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from different wavelengths with multiple sensors
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/50—Constructional details
    • H04N23/555—Constructional details for picking-up images in sites, inaccessible due to their dimensions or hazardous conditions, e.g. endoscopes or borescopes
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/95—Computational photography systems, e.g. light-field imaging systems
    • H04N23/951—Computational photography systems, e.g. light-field imaging systems by using two or more images to influence resolution, frame rate or aspect ratio
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10068—Endoscopic image
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]

Definitions

  • the present invention relates to an information processing system, an endoscope system, a learned model, an information storage medium, an information processing method, and the like.
  • Patent Document 1 discloses a technique for super-resolution of an input image using a trained model obtained by deep learning.
  • the high-resolution image actually captured by the high-resolution image sensor is used as the teacher data, and the high-resolution image is simply reduced to generate a low-resolution image corresponding to the input image at the time of inference.
  • a low-resolution image is input to the learning model, and deep learning is performed based on the output image output by the learning model and the high-resolution image which is the teacher data.
  • a low resolution image is generated by simply reducing a high resolution image.
  • This low-resolution image has different properties from the image actually taken by the low-resolution image sensor.
  • the resolution is different between the low-resolution image generated by simple reduction and the image taken by the low-resolution image sensor because the optical system is not taken into consideration.
  • the low-resolution image generated by simple reduction cannot accurately reproduce the resolution equivalent to the input image at the time of inference, so the trained model learned using the low-resolution image.
  • One aspect of the present invention is to process a processing target image captured by a storage unit that stores a trained model, a processing unit, and a second imaging system that captures images having a lower resolution than the first imaging system.
  • the trained model includes an input unit to be input to, and the trained model is a trained model trained to restore a low-resolution learning image to a high-resolution learning image.
  • the image is a high-resolution image of a predetermined subject taken by the first imaging system, and the low-resolution learning image is generated by lowering the resolution of the high-resolution learning image.
  • the low-resolution processing is a process of generating a low-resolution image as if the predetermined subject was imaged by the second imaging system, and the resolution characteristics of the optical system of the second imaging system are improved.
  • An information processing system that includes an optical system simulating process to simulate, and the processing unit uses the trained model to resolve and restore the image to be processed to the resolution when the image to be processed is captured by the first imaging system.
  • Another aspect of the present invention is a processor unit having the information processing system described above, and an endoscope scope connected to the processor unit to capture an image to be processed and transmit it to the input unit. Including endoscopic systems.
  • the image to be processed captured by the second imaging system that captures images having a lower resolution than that of the first imaging system is resolved to the resolution when the image is captured by the first imaging system.
  • a trained model that causes a computer to function so as to restore an image.
  • the trained model is trained to restore a low-resolution learning image to a high-resolution learning image, and the high-resolution learning image is restored.
  • the learning image is a high-resolution image of a predetermined subject taken by the first imaging system, and the low-resolution learning image is obtained by processing the high-resolution learning image to a low resolution.
  • the generated low-resolution processing is a process of generating a low-resolution image as if the predetermined subject was imaged by the second imaging system, and is a resolution characteristic of the optical system of the second imaging system. It pertains to a trained model that includes an optical system simulation process that simulates.
  • Yet another aspect of the present invention relates to an information storage medium that stores the trained model described above.
  • Still another aspect of the present invention is an information processing method that performs resolution restoration using a trained model, in which the trained model resolves a low-resolution learning image into a high-resolution learning image.
  • the high-resolution learning image learned to be restored is a high-resolution image of a predetermined subject taken by the first imaging system
  • the low-resolution learning image is the high-resolution learning image.
  • the image is generated by low-resolution processing, and the low-resolution processing is a process for generating a low-resolution image as if the predetermined subject was captured by the second imaging system.
  • the second imaging system which includes an optical system simulating process for simulating the resolution characteristics of the optical system of the imaging system and uses the trained model to perform imaging with a lower resolution than the first imaging system. It relates to an information processing method for resolving and restoring an image to be processed to be imaged to the resolution when the image is imaged by the first imaging system.
  • the configuration example of the information processing system in the first modification and the processing flow of the model creation process An example of a complementary color image sensor. An example of a mixed image sensor.
  • the configuration example of the information processing system in the sixth modification and the processing flow of the model creation process Configuration example of the learning device.
  • a second configuration example of an endoscope system. The processing flow of the blur processing of the first method.
  • the present embodiment described below does not unreasonably limit the content of the present invention described in the claims. Moreover, not all of the configurations described in the present embodiment are essential constituent requirements of the present invention.
  • the case where the information processing system is applied to a medical endoscope will be described below as an example, but the present invention is not limited to this, and the information processing system of the present invention can be applied to various imaging systems or video display systems.
  • the information processing system of the present invention can be applied to a still camera, a video camera, a television receiver, a microscope, or an industrial endoscope.
  • the endoscope has the advantage that the smaller the probe diameter, the less invasive the patient can be examined.
  • An example of an endoscope with a small probe diameter is a nasal endoscope.
  • the smaller the probe system the smaller the size of the imager, and the lower the resolution. Therefore, it is conceivable to improve the resolution of the nasal endoscope by using super-resolution technology, which is a kind of image processing, and generate an image as if it was captured by an endoscope having a large probe system. ..
  • FIG. 1 shows a configuration example of the information processing system 100 according to the first embodiment and a processing flow of the model creation process S300.
  • the information processing system 100 includes an input unit 1 for inputting a processing target image 10 to the processing unit 3, a storage unit 2 for storing the learned model 20, and a processing unit 3 for performing resolution restoration processing.
  • the input unit 1, the storage unit 2, and the processing unit 3 are also referred to as an input device, a storage device, and a processing device, respectively.
  • the information processing system 100 is a system that performs inference using the trained model 20.
  • the inference in the present embodiment is a process of resolving and restoring a high-resolution image from the image 10 to be processed.
  • the trained model 20 is generated by the model creation process S300 and stored in the storage unit 2.
  • the model creation process S300 is executed by, for example, a learning device different from the information processing system 100.
  • the information processing system 100 may execute the model creation process S300 in the learning stage and perform inference using the trained model 20 in the inference stage.
  • the information processing system 100 also serves as a learning device, and for example, the processing unit 3 executes the learning process.
  • the input unit 1 is, for example, an image data interface that receives image data from an imaging system, a storage interface that reads image data from storage, a communication interface that receives image data from the outside of the information processing system 100, and the like.
  • the input unit 1 inputs the acquired image data to the processing unit 3 as a processing target image 10.
  • the frame image of the moving image is input to the processing unit 3 as the processing target image 10.
  • the storage unit 2 is a storage device, for example, a semiconductor memory, a hard disk drive, an optical disk drive, or the like.
  • the learned model 20 generated by the model creation process S300 is stored in the storage unit 2 in advance.
  • the learned model 20 may be input to the information processing system 100 from an external device such as a server via a network, and the learned model 20 may be stored in the storage unit 2.
  • the processing unit 3 restores the high-resolution image from the processing target image 10 by performing the resolution restoration processing S200 on the processing target image 10 using the learned model 20 stored in the storage unit 2.
  • the restored high-resolution image is an image in which the same subject as the processing target image 10 is captured, and is an image having a higher resolution than the processing target image 10.
  • the resolution is an index showing how finely the subject in the image is resolved. The resolution depends on, for example, the number of pixels of the image, the performance of the optical system used for imaging, the type of the image sensor used for imaging, the content of image processing applied to the image, and the like.
  • the first imaging system is the imaging system with the resolution that is the target of resolution restoration.
  • the image 10 to be processed is imaged by a second imaging system having a lower resolution than the first imaging system.
  • the high-resolution image restored from the processing target image 10 corresponds to an image as if the same subject as the processing target image 10 was captured by the first imaging system.
  • the image pickup system includes an optical system for imaging a subject and an image pickup element for imaging a subject imaged by the optical system.
  • the image sensor is also called an image sensor. As the image sensor, various types such as monochrome, Bayer type, complementary color type and the like can be adopted.
  • the first imaging system is an imaging system of a first endoscope equipped with a large-diameter scope
  • the second imaging system is a second endoscope having a scope smaller in diameter than the scope of the first endoscope. It is an imaging system of a mirror.
  • the hardware that constitutes the processing unit 3 is, for example, a general-purpose processor such as a CPU.
  • the storage unit 2 stores the program in which the inference algorithm is described and the parameters used in the inference algorithm as the trained model 20.
  • the processing unit 3 may be a dedicated processor in which the inference algorithm is hardwareized.
  • the dedicated processor is, for example, an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).
  • the storage unit 2 stores the parameters used in the inference algorithm as the trained model 20.
  • a neural network can be applied to the inference algorithm.
  • the weighting factor of the connection between nodes in the neural network is a parameter.
  • the neural network includes an input layer into which image data is input, an intermediate layer in which arithmetic processing is performed on the data input through the input layer, and an output layer in which image data is output based on the arithmetic results output from the intermediate layer.
  • CNN Convolutional Neural Network
  • CNN is suitable as the neural network used for the resolution restoration process S203.
  • various AI Artificial Intelligence
  • FIG. 2 shows the processing flow of the resolution restoration processing S200.
  • step S201 the processing unit 3 reads the processing target image 10 from the input unit 1.
  • step S202 the processing unit 3 reads the learned model 20 used for resolution restoration from the storage unit 2.
  • step S203 the processing target image 10 acquired in step S201 is subjected to resolution restoration processing using the learned model 20 acquired in step S202 to generate a high-resolution image.
  • the order of S201 and S202 may be interchanged.
  • FIG. 3 shows the processing flow of the model creation processing S300.
  • the learning device includes a processing unit that executes the model creation process S300.
  • this processing unit will be referred to as a learning processing unit.
  • step S301 the learning processing unit reads the high-resolution learning image 30.
  • the high-resolution learning image 30 is an image captured by the first imaging system described above.
  • steps S302 and S303 the learning processing unit reads the optical system information 50 and the image sensor information 60 used when generating the low-resolution learning image 40.
  • the optical system information 50 is information related to the resolution of the optical system of the first imaging system and the second imaging system.
  • the image sensor information 60 is information related to the resolution of the image sensor included in the first image sensor and the second image sensor.
  • step S304 the learning processing unit uses at least one of the optical system information acquired in step S302 and the image sensor information acquired in step S303 to be lower than the high-resolution learning image 30 acquired in step S301.
  • the resolution processing is performed to generate a low-resolution learning image 40.
  • the low-resolution learning image 40 corresponds to an image as if the same subject as the high-resolution learning image 30 was captured by the second imaging system, and has the same number of pixels as the processing target image 10 in the inference.
  • step S305 the learning processing unit performs resolution restoration learning processing on the learning model using the high-resolution learning image 30 acquired in step S301 and the low-resolution learning image 40 acquired in step S304.
  • the learning processing unit repeatedly executes the learning processes of steps S301 to S305 using the plurality of high-resolution learning images 30, and outputs the learned learning model as the learned model 20.
  • the learning model used for the learning process is the same algorithm as the trained model 20 used for inference.
  • the learning model is a CNN, and the learning processing unit calculates the weight value and the bias value of each layer of the CNN, and stores these values as the trained model 20.
  • Various known learning algorithms can be adopted as machine learning algorithms in neural networks. For example, a supervised learning algorithm using the backpropagation method can be adopted.
  • FIG. 4 shows the processing flow of the low resolution processing S304.
  • the learning processing unit uses the optical system information 50 and the image sensor information 60 to generate a low-resolution learning image 40 from the high-resolution learning image 30.
  • step S401 the learning processing unit acquires the optical system information 50 when the high-resolution learning image 30 is acquired and the optical system information 50 used in the low-resolution learning image 40.
  • the optical system information 50 refers to the focal length and aperture of the optical system.
  • the learning processing unit acquires a PSF (Point Spread Function) or an OTF (Optical Transfer Function) under these conditions. That is, the learning processing unit acquires the point spread function of the first imaging system and the point spread function of the second imaging system, or the optical transfer function of the first imaging system and the optical transfer function of the second imaging system. To get.
  • PSF Point Spread Function
  • OTF Optical Transfer Function
  • the learning processing unit uses information such as the number of pixels of the high-resolution learning image 30, the number of pixels of the low-resolution learning image 40, the imaging method, and the like to lower the image 30 for high-resolution learning.
  • the reduction magnification of the resolution learning image 40 is set. For example, when the high-resolution learning image 30 is 640 ⁇ 480 [pixels] and the low-resolution learning image 40 is 320 ⁇ 240 [pixels], the reduction magnification is 1/2 in both vertical and horizontal directions.
  • the order of S401 and S402 may be interchanged.
  • step S403 the learning processing unit adds blur to the high-resolution learning image 30 using the optical system information calculated in step S401.
  • the performance of the optical system of the second imaging system corresponding to the low-resolution learning image 40 is inferior to that of the optical system of the first imaging system that has acquired the high-resolution learning image 30.
  • a process of removing the high frequency band of the image is performed in advance by using a filter such as bicubic.
  • the band of the reduced image is smaller than that of the image before reduction, but this process alone does not reproduce the difference in band when the optical system is different. Therefore, blurring is added to the image so as to compensate for the difference in the optical system.
  • the learning processing unit uses the PSF of the first imaging system and the second imaging system acquired in step S401, or the OTF of the first imaging system and the second imaging system, to secondly capture the blur of the high-resolution learning image. Compensate so that it resembles the blur of the image captured by the system. The details of the blurring process will be described later.
  • step S404 the learning processing unit performs reduction processing on the image generated in step S403 at the reduction magnification calculated in step S402.
  • the learning processing unit performs reduction processing such as bicubic or bilinear.
  • the learning processing unit executes the resolution restoration learning process in step S305 using the image after the reduction process as the low-resolution learning image 40.
  • the order of the blurring process in step S403 and the reduction process in step S404 may be reversed. That is, the learning processing unit may generate the low-resolution learning image 40 by reducing the high-resolution learning image 30 and performing the blurring process on the image after the reduction processing.
  • the trained model 20 is trained to resolve the low-resolution learning image 40 into the high-resolution learning image 30.
  • the high-resolution learning image 30 is a high-resolution image of a predetermined subject taken by the first imaging system.
  • the low-resolution learning image 40 is generated.
  • the low resolution process is a process of generating a low resolution image as if a predetermined subject was captured by the second imaging system.
  • the resolution reduction process includes an optical system simulation process that simulates the resolution characteristics of the optical system of the second imaging system.
  • the optical system simulation process is a process of convolving the PSF into the high-resolution learning image 30 so as to compensate for the difference between the PSF of the optical system of the first imaging system and the PSF of the optical system of the second imaging system, for example. Is.
  • step S403 The blurring process in step S403 will be described.
  • the first and second methods will be described as an example of the blurring process.
  • FIG. 20 shows the processing flow of the blur processing of the first method.
  • the learning processing unit adds blur to the high-resolution learning image 30 by using the PSFs of the high-resolution first imaging system and the low-resolution second imaging system.
  • the PSF is a function that expresses a unit impulse response to the optical system, that is, a function that expresses an image distribution when the optical system forms an image of a point light source.
  • the learning processing unit performs reduction processing on the image obtained in step S410, and outputs the result as a low-resolution learning image 40.
  • step S410 The details of step S410 will be described.
  • the learning processing unit deconvolves the PSF of the first imaging system with respect to the high-resolution learning image 30 as shown in step S411, and the second image 30 with respect to the high-resolution learning image 30 as shown in step S412. Convolution the PSF of the imaging system. Specifically, the learning processing unit adds blur by the following equation (1).
  • h 1 and h 2 are the PSF of the optical system of the first imaging system and the PSF of the optical system of the second imaging system acquired in step S401, respectively
  • f the image 30 for high resolution learning.
  • G is an image with blur added
  • n is a noise term. * Represents a convolution operation.
  • the noise term n may be omitted.
  • FIG. 21 shows the processing flow of the blur processing of the second method.
  • the learning processing unit blurs the high-resolution learning image 30 by using the OTFs of the optical systems of the high-resolution first imaging system and the low-resolution second imaging system.
  • OTF is a function that expresses the unit impulse frequency response to the optical system, that is, a function that expresses the frequency characteristic of the image distribution when the optical system images a point light source.
  • the learning processing unit performs reduction processing on the image obtained in step S420, and outputs the result as a low-resolution learning image 40.
  • the learning processing unit reads the OTFs of the optical system of the first imaging system and the optical system of the second imaging system as the optical system information 50 of FIG. As shown in step S421 of FIG. 21, the learning processing unit obtains the frequency characteristics of the high-resolution learning image 30 by performing an FFT (Fast Fourier Transform) on the high-resolution learning image. In steps S422 and S423, the learning processing unit calculates the result of dividing the frequency characteristic of the high-resolution learning image 30 by the OTF of the first imaging system and multiplying by the OTF of the second imaging system. In step S424, the learning processing unit performs an inverse FFT on the frequency characteristic which is the calculation result.
  • FFT Fast Fourier Transform
  • OTF and PSF are in a Fourier transform relationship. That is, the frequency response of OTF coincides with the unit impulse response (PSF) of the optical system in real space. Therefore, by performing the blurring process using the OTF shown in step S420 of FIG. 21, substantially the same result as the blurring process using the PSF shown in step S410 of FIG. 20 can be obtained.
  • the resolution characteristics of the optical system of the second imaging system are simulated using a transfer function such as PSF or OTF, but the present invention is not limited to this, and the design values of known optical systems are used.
  • the resolution characteristics of the optical system of the second imaging system may be simulated by using based calculation or machine learning.
  • the second image pickup system includes a simultaneous image pickup element, and performs a resolution reduction process in consideration of the type of the image pickup element. Specifically, the image to be processed is photographed by an image sensor having a Bayer array, and the resolution is reduced in consideration of the resolution reduction due to the demosaic processing.
  • the Bayer type image sensor will be described as an example, but the simultaneous image sensor is not limited to the Bayer type image sensor. The description of the same configuration and processing as in the first embodiment will be omitted.
  • FIG. 5 shows a configuration example of the information processing system 100 according to the second embodiment and a processing flow of the model creation process S600.
  • the configuration of the information processing system 100 and the resolution restoration process are the same as those of the first embodiment shown in FIGS. 1 and 2.
  • FIG. 6 shows the processing flow of the model creation processing S600. Steps S601 to S603 are the same as steps S301 to S303 of the first embodiment.
  • Steps S604 to S606 correspond to the resolution reduction process shown in step S610 of FIG.
  • the reduction and blurring processing in step S604 corresponds to steps S401 to S404 of the first embodiment shown in FIG. That is, in step S604, the learning processing unit adds blur to the high-resolution learning image 30 using the optical system information, and reduces the blurred image using the image sensor information.
  • the image after the reduction process is a low-resolution color image 41 in which each pixel has an RGB pixel value.
  • step S605 the learning processing unit performs mosaic arrangement processing on the low-resolution color image 41 generated in step S604. That is, the learning processing unit generates the low-resolution Bayer image 42 by mosaic-arranging the pixel values of the low-resolution color image 41 in a Bayer shape using the image sensor information 60 acquired in step S603. Any one of RGB colors is assigned to each pixel of the low-resolution Bayer image 42. For example, taking the R pixel of the low resolution bayer image 42 as an example, the R pixel value is extracted from the pixel of the low resolution color image 41 at the same position as the R pixel, and the R pixel value is used as the low resolution bayer image. Allocate to 42 R pixels. The same applies to the G and B pixels of the low-resolution bayer image 42.
  • step S606 the learning processing unit performs the mosaic processing on the mosaic-like low-resolution bayer image 42 to colorize the low-resolution bayer image 42 again.
  • the image after this demosaic processing becomes the low-resolution learning image 40.
  • existing processing such as interpolation using color correlation or bilinear interpolation can be adopted.
  • the demosaic process when the image 10 to be processed is generated is known, it is desirable to perform the same process as the demosaic process in step S606.
  • step S607 the resolution restoration learning process is performed by using the high resolution learning image 30 acquired in step S601 and the low resolution learning image 40 acquired in step S606, as in step S305 of the first embodiment. To generate the trained model 20.
  • the first image pickup system includes the first image pickup device of the first image pickup method.
  • the second image pickup system includes a second image pickup device having a second imaging method, which has a lower number of pixels than the first image pickup device and is different from the first image pickup method.
  • the resolution reduction process S610 further includes an image pickup method simulation process that simulates the second image pickup method.
  • the image pickup method simulation process simulates the image process when the image to be processed 10 is generated from the image signal acquired by the second image sensor. For example, as shown in FIGS. 5 and 6, the mosaic arrangement process S605 and the demosaic process S606 are performed.
  • the image processing when generating a color image from the image signal acquired by the image sensor differs depending on the image pickup method. This image processing affects the resolution of a color image.
  • the resolution of the image 10 to be processed can be reproduced in the low-resolution learning image 40 by performing the imaging method simulation process in the low-resolution processing S610. As a result, high-performance super-resolution can be realized.
  • the low resolution processing S610 is a process of reducing the image 30 for high resolution learning of the first imaging method and performing an imaging method simulation process on the image 41 after the reduction processing.
  • the image 42 of the second imaging method is generated from the image 41 after the reduction process (S605), and the image 10 to be processed is generated with respect to the generated image 42 of the second imaging method.
  • This is a process of generating a low-resolution learning image 40 by performing the same image processing as the image processing (S606).
  • the same image processing as that of the image processing when generating the image 10 to be processed is performed on the image 42. carry out. This makes it possible to simulate image processing when the image to be processed 10 is generated from the image signal acquired by the second image sensor.
  • the first image pickup system captures a plurality of images by a monochrome image sensor at the timing when the light of each wavelength band is irradiated when the light of a plurality of wavelength bands is sequentially irradiated. get.
  • the high-resolution learning image 30 is a surface-sequential image in which a plurality of images are combined.
  • the second image pickup system acquires a Bayer image in which one color is assigned to each pixel by using a simultaneous image pickup element having a plurality of pixels having different colors from each other and one color is assigned to each pixel.
  • the low-resolution processing S610 is a process of performing a reduction process of reducing the number of pixels of the surface-sequential image (30) and performing an imaging method simulation process on the image 41 after the reduction process.
  • the imaging method simulation process constitutes a low-resolution bayer image 42 from the image 41 after the reduction process (S605).
  • the low-resolution bayer image 42 corresponds to a bayer image acquired by the second imaging system.
  • the imaging method simulation process generates a low-resolution learning image 40 by demosacing the low-resolution bayer image 42 (S606).
  • the second image sensor has a plurality of types of pixels having different spectral sensitivity characteristics.
  • the high-resolution learning image 30 is a surface generated by synthesizing an image pickup signal obtained by sequentially irradiating and imaging light of a plurality of wavelength bands using a monochrome image sensor and using the light of the plurality of wavelength bands. It is a sequential image.
  • the low-resolution learning image 40 simulates an image captured by a second image pickup system including a simultaneous image pickup device having a plurality of types of pixels having different spectral sensitivities.
  • a low resolution processing S610 when simulating the second image pickup method, a plurality of signals of each pixel of the simultaneous image pickup element are acquired by the pixels of the monochrome image pickup element at the positions corresponding to the positions of the pixels.
  • This is a process in which a low-resolution learning image 40 is generated by performing demosaic processing (S606) after generating based on a signal generated by light in at least one wavelength band among the signals generated by light in the wavelength band of.
  • the simultaneous system which is the second imaging system can be simulated in the resolution reduction processing.
  • the demosaic process in the simultaneous method affects the resolution of the image 10 to be processed.
  • the resolution of the image 10 to be processed can be reproduced in the low-resolution learning image 40 by performing the demosaic process in the imaging method simulation process. As a result, high-performance super-resolution can be realized.
  • a modified example of the second embodiment will be described.
  • the resolution reduction process is performed in consideration of the resolution reduction due to the noise reduction process.
  • the description of the configuration and processing similar to those of the first and second embodiments will be omitted.
  • FIG. 7 shows a configuration example of the information processing system 100 in the modified example of the second embodiment and a processing flow of the model creation process S800.
  • the configuration of the information processing system 100 and the resolution restoration process are the same as those of the first embodiment shown in FIGS. 1 and 2.
  • FIG. 8 shows the processing flow of the model creation processing S800.
  • the resolution reduction process in step S810 includes steps S804 to S807.
  • Steps S801 to S805 are the same as steps S601 to S605 of the second embodiment.
  • step S806 the learning processing unit performs noise reduction processing on the mosaic-like low-resolution bayer image 42 generated in step S805. For example, a known noise reduction process for the bayer image can be adopted.
  • step S807 the learning processing unit generates a low-resolution learning image 40 by performing demosaic processing on the image after the noise reduction processing, as in step S606 of the second embodiment.
  • step S808 as in step S305 of the first embodiment, the resolution restoration learning process using the high-resolution learning image 30 acquired in step S801 and the low-resolution learning image 40 acquired in step S807. To generate the trained model 20.
  • the imaging method simulation process further includes a noise reduction process (S806) for the low resolution bayer image 42.
  • the noise reduction processing affects the resolution of the image 10 to be processed.
  • the resolution of the image to be processed 10 can be reproduced in the low resolution learning image 40.
  • high-performance super-resolution can be realized.
  • the type of the imaging system that has captured the image to be processed is detected, and the resolution is restored according to the detection result.
  • the imaging method differs depending on the type of the imaging system will be described as an example, but the present invention is not limited to this, and the resolution of the image to be processed may differ depending on the type of the imaging system.
  • the number of pixels, the optical system, and the like may differ depending on the type of the imaging system.
  • the description of the configuration and processing similar to those of the first and second embodiments will be omitted.
  • FIG. 9 shows a configuration example of the information processing system 100 according to the third embodiment and a processing flow of the model creation processes S300 and S600.
  • the configuration and processing of the input unit 1 are the same as those in the first embodiment.
  • the model creation processes S300 and S600 are the same as those in the first and second embodiments.
  • the storage unit 2 stores the first trained model 21 and the second trained model 22.
  • the first trained model 21 is generated by the model creation process S300 described in the first embodiment.
  • the second trained model 22 is generated by the model creation process S600 described in the second embodiment.
  • the processing unit 3 performs an imaging information detection process S250 for detecting an imaging method and a resolution restoration process S200 for performing resolution restoration using the detection result.
  • FIG. 10 shows the processing flow of the imaging information detection processing S250 and the resolution restoration processing S200.
  • step S1001 the processing unit 3 reads the processing target image 10 from the input unit 1.
  • step S1002 the processing unit 3 detects the imaging information from the image to be processed 10 and determines the imaging method based on the imaging information.
  • the imaging information is, for example, image information such as the color distribution of the image 10 to be processed, the amount of color shift of the edge portion by the surface sequential method, or the number of pixels.
  • the imaging method is a surface-sequential method or a simultaneous method. When the processing target image 10 is captured by the surface sequential method, the processing target image 10 has imaging information corresponding to the surface sequential method, and when the processing target image 10 is captured by the surface sequential method, the processing target image 10 is , Has imaging information corresponding to the simultaneous method.
  • the light source sequentially emits R light, G light, and B light.
  • the imaging system has a monochrome image sensor, captures an R image at the timing when the light source emits R light, images the G image at the timing when the light source emits G light, and B at the timing when the light source emits B light. Take an image. A color image is generated by synthesizing these three color images.
  • the light source emits white light.
  • White light has, for example, a continuous spectrum covering the visible light region.
  • the image pickup system has a Bayer type image pickup device and captures an image of a Bayer array. A color image is generated by demosacing the image of this Bayer array.
  • the processing unit 3 determines the imaging method by determining the hue of the image using hue or the like.
  • the amount of color shift of the edge portion will be described.
  • the RGB imaging timings are different, so that the subject position is deviated in RGB.
  • the processing unit 3 detects the amount of color shift of the edge portion by using a matching process or the like, and determines that the imaging method is a surface-sequential method when the amount of color shift is larger than a predetermined value.
  • the number of pixels will be described. As described above, since the image sensor is different between the surface sequential method and the simultaneous method, the number of pixels of the captured image may be different.
  • the processing unit 3 determines the imaging method from the number of pixels of the image to be processed 10.
  • step S1003 the processing unit 3 reads out the learned model corresponding to the imaging method determined in step S1002 from the storage unit 2. That is, when the processing unit 3 determines that the imaging method is the surface sequential method, the first learned model 21 is read from the storage unit 2, and when it is determined that the imaging method is the simultaneous method, the storage unit 2 to the first. 2 Read the trained model 22.
  • step S1004 the processing unit 3 performs resolution restoration processing on the processing target image 10 acquired in step S1001 using the learned model acquired in step S1003 to generate a high resolution image.
  • the storage unit 2 corresponds to the first trained model 21, which is a trained model corresponding to the second imaging system, and the third imaging system, which performs imaging with a lower resolution than the first imaging system.
  • the second trained model 22 is stored.
  • the input unit 1 inputs the first processing target image captured by the second imaging system or the second processing target image captured by the third imaging system to the processing unit 3 as the processing target image 10.
  • the processing unit 3 uses the first trained model 21 to restore the resolution of the first processed image, and uses the second trained model 22 to restore the resolution of the second processed image.
  • the trained model corresponding to the imaging system in which the image 10 to be processed is captured is selected.
  • high-performance super-resolution compatible with a plurality of imaging systems can be realized. That is, since a trained model having appropriate restoration parameters is selected according to each imaging system, high-performance super-resolution can be realized regardless of the imaging system.
  • the processing unit 3 detects the type of the imaging system that captured the processing target image 10 from the processing target image 10. When the processing unit 3 determines that the first processing target image has been input based on the detection result, the processing unit 3 selects the first trained model 21, and determines that the second processing target image has been input based on the detection result. , Select the second trained model 22.
  • the type of the imaging system can be determined from the image 10 to be processed, and the trained model corresponding to the type of the imaging system can be selected.
  • the information processing system is used for the endoscope system.
  • various endoscopic scopes can be attached and detached.
  • resolution restoration is performed according to the endoscope scope mounted on the endoscope system. Specifically, the type of imaging system of the endoscope is detected, and the resolution is restored according to the detection result.
  • the imaging method differs depending on the type of the imaging system will be described as an example, but the present invention is not limited to this, and the resolution of the image to be processed may differ depending on the type of the imaging system. For example, the number of pixels, the optical system, and the like may differ depending on the type of the imaging system. The description of the same configuration and processing as those of the first to third embodiments will be omitted.
  • FIG. 11 shows a configuration example of the endoscope system 200 according to the fourth embodiment and a processing flow of the model creation processes S300 and S600.
  • the endoscope system 200 includes a processor unit 4 and an endoscope scope 5 connected to the processor unit 4.
  • the processor unit 4 includes an input unit 1, a storage unit 2, and a processing unit 3.
  • the input unit 1 is the same as that of the first embodiment
  • the storage unit 2 is the same as that of the third embodiment
  • the model creation processes S300 and S600 are the same as those of the first and second embodiments.
  • the processing unit 3 performs an imaging information detection process S260 for detecting the imaging method of the endoscope scope 5 connected to the processor unit 4 and a resolution restoration process S200 for performing resolution restoration using the detection result. ..
  • FIG. 12 shows the processing flow of the imaging information detection processing S260 and the resolution restoration processing S200.
  • step S1201 the processing unit 3 reads the processing target image 10 from the input unit 1.
  • step S1202 the processing unit 3 detects the ID of the endoscope scope 5.
  • the ID includes the optical system information of the imaging system and the imaging method information included in the endoscope scope 5.
  • step S1203 the processing unit 3 reads the learned model corresponding to the ID detected in step S1202 from the storage unit 2. That is, the processing unit 3 reads the first trained model 21 from the storage unit 2 when the ID indicates the surface sequential method, and reads the second trained model 22 from the storage unit 2 when the ID indicates the simultaneous method. read out.
  • step S1204 the processing unit 3 performs resolution restoration processing on the processing target image 10 acquired in step S1201 using the learned model acquired in step S1203 to generate a high resolution image.
  • the processing unit 3 detects the ID information of the endoscope scope 5 connected to the processor unit 4.
  • the processing unit 3 determines that the endoscope scope 5 includes the second imaging system based on the ID information
  • the processing unit 3 selects the first trained model 21, and the endoscope scope 5 is the third based on the ID information.
  • the second trained model 22 is selected.
  • the trained model corresponding to the ID information of the endoscopic scope is selected.
  • high-performance super-resolution compatible with a plurality of types of endoscopic scopes can be realized. That is, since a trained model having appropriate restoration parameters is selected according to each type of endoscopic scope, high-performance super-resolution can be realized regardless of the type of endoscopic scope.
  • the first modification example will be described.
  • at least one of the optical system information and the image sensor information is used for the low resolution processing, but the information of the light source may be further used.
  • the resolution reduction processing is switched according to the light source when capturing the image to be processed. Specifically, since the light source differs depending on the observation method, switching according to the light source can be said to be switching according to the observation method.
  • the observation method is also called the observation mode.
  • Examples of the observation method include WLI (White Light Imaging) using white illumination light, special light observation using special light other than white light, and the like.
  • WLI White Light Imaging
  • NBI Near Band Imaging
  • the two narrow-band lights are narrow-band light included in the blue wavelength band and narrow-band light included in the green wavelength band.
  • WLI and NBI differ in image processing when generating a color image from an image signal output by an image sensor. For example, the content of demosaic processing or the parameters in image processing are different.
  • FIG. 13 is a configuration example of the information processing system 100 in the first modification and a processing flow of the model creation process S500.
  • the learning processing unit acquires the observation method information 70 and switches the processing according to the observation method information 70.
  • the observation method information 70 is information indicating an observation method when the image to be processed 10 is imaged.
  • the learning processing unit selects the WLI processing S511 when the observation method information 70 indicates WLI, and selects the NBI processing S512 when the observation method information indicates NBI.
  • the WLI processing S511 and the NBI processing S512 are processes for reducing the resolution of the high-resolution learning image 30 into the low-resolution learning image 40. For example, the WLI processing S511 interpolates the R image and the B image using the G image in the demosaic processing. On the other hand, the NBI process S512 interpolates the G image and the B image independently in the demosaic process.
  • the image to be processed 10 is generated by image processing corresponding to the light source used when the first imaging system takes an image.
  • the low resolution processing S510 includes image processing (S511, S512) corresponding to the light source.
  • the image processing for converting the bayer image into a low-resolution learning image is demosaic processing, or demosaic processing and noise reduction processing, but the image processing is not limited thereto.
  • the image processing for converting the bayer image into a low-resolution learning image may include various image processing such as correction of defective pixels or edge enhancement processing.
  • the third modification example will be described.
  • blurring is added to the image using PSF or OTF of two optical systems, but the method of adding blurring is limited to this. Not done.
  • the amount of blur added to the image may be experimentally determined by using the image captured by using the optical system of the first imaging system and the image captured by using the optical system of the second imaging system.
  • a plurality of two-dimensional filters that approximate the blur are generated by Gaussian or the like. The plurality of two-dimensional filters approximate different amounts of blur.
  • a plurality of two-dimensional filters are generated for an image captured by using the optical system of the first imaging system, and the obtained plurality of images are compared with an image captured by using the optical system of the second imaging system. .. Based on the comparison result, a two-dimensional filter that approximates the optimum amount of blur is selected.
  • the third modification it is possible to easily reduce the resolution by reflecting the optical system information. That is, even when the PSF or OTF of the imaging system is unknown, the resolution characteristics of the optical system can be approximated by a two-dimensional filter such as Gaussian.
  • the simultaneous image sensor may have a complementary color array shown in FIG. 14, or may have an array in which primary color pixels and complementary color pixels shown in FIG. 15 are mixed.
  • the complementary color imaging element of FIG. 14 cyan (Cy) pixels, magenta (Mg) pixels, yellow (Ye) pixels, and green (G) pixels are arranged in a 2 ⁇ 2 pixel unit, and the units are repeatedly arranged. Will be done.
  • red (R) pixels, green (G) pixels, blue (B) pixels, and cyan (Cy) pixels are arranged in a 2 ⁇ 2 pixel unit, and the units are repeatedly arranged. Will be done.
  • the simultaneous image sensor has the following configuration, for example.
  • the simultaneous imaging element may have at least two color pixels out of four color pixels of Cy pixel, Mg pixel, Ye pixel, and G pixel provided with color filters of Cy, Mg, and Ye.
  • the simultaneous imaging element may have a complementary color array composed of four color pixels of Cy pixel, Mg pixel, Ye pixel, and G pixel provided with Cy, Mg, Ye, and G color filters.
  • the simultaneous imaging element may have at least two color pixels out of the three color pixels of the R pixel, the G pixel, and the B pixel provided with the three color filters of R, G, and B.
  • the simultaneous image sensor may have a Bayer array.
  • the imaging information indicating the type of the imaging system is detected from the image to be processed, but the method for detecting the imaging information is not limited to this.
  • the user may input the information into the information processing system 100.
  • a low-resolution bayer image is generated by arranging mosaics of high-resolution learning images taken in a plane-sequential manner.
  • the low-resolution bayer image assumes images taken simultaneously.
  • color shift occurs in the surface-sequential formula, but color shift does not occur in the simultaneous formula. Therefore, it is from the viewpoint of super-resolution accuracy that a low-resolution bayer image is generated from an image of a scene with many color shifts. Is not desirable.
  • FIG. 16 shows a configuration example of the information processing system 100 in the sixth modification and a processing flow of the model creation process S700.
  • the configuration and operation of the information processing system 100 are the same as those in the second embodiment.
  • the learning process unit performs the color shift determination process S710 on the high-resolution learning image 30.
  • the learning processing unit performs the low resolution processing S610 on the high resolution learning image 30 whose color shift amount is smaller than a predetermined value, and generates the low resolution learning image 40.
  • the color shift determination process S710 for example, the amount of coloring around the saturated portion of the image is compared with a predetermined threshold value.
  • the sixth modification in the first imaging system, when light in a plurality of wavelength bands is sequentially irradiated, a plurality of images are taken by a monochrome image sensor at the timing when the light in each wavelength band is irradiated. Get an image.
  • the high-resolution learning image 30 is a surface-sequential image in which a plurality of images are combined.
  • the trained model 20 is trained using the surface-sequential image (30) in which the amount of color shift in the surface-sequential image is equal to or less than a preset threshold value.
  • the low resolution learning image 40 is output by performing the low resolution processing on the high resolution learning image 30, but the high resolution learning image 30 is subjected to the low resolution processing.
  • a known grayscale processing may be performed to output a monochrome low-resolution learning image.
  • the trained model uses the processing target image 10 as the processing target image 10. Not only can the resolution be restored, but the actual color of the subject of the image 10 to be processed can be reproduced.
  • FIG. 17 is a configuration example of a learning device 350 that executes the above-mentioned model creation process.
  • the learning device 350 includes a processing unit 351, a storage unit 352, an operation unit 353, and a display unit 354.
  • the learning device 350 is an information processing device such as a PC or a server.
  • the processing unit 351 is a processor such as a CPU.
  • the processing unit 351 performs machine learning on the learning model to generate a trained model.
  • the storage unit 352 is a storage device such as a semiconductor memory or a hard disk drive.
  • the operation unit 353 is various operation input devices such as a mouse, a touch panel, and a keyboard.
  • the display unit 354 is a display device such as a liquid crystal display.
  • the learning device 350 may be a cloud system in which a plurality of information processing devices connected by a network perform parallel processing.
  • the information processing system 100 shown in FIG. 1 or the like may also serve as a learning device.
  • the processing unit 3 and the storage unit 2 also serve as the processing unit 351 and the storage unit 352 of the learning device 350, respectively.
  • the learned model generated by the learning device 350 is stored in the storage unit 2 of the information processing system 100.
  • the trained model may be stored in an information storage medium that is a medium that can be read by a computer.
  • the information storage medium can be realized by, for example, an optical disk, a memory card, an HDD, a semiconductor memory, or the like.
  • the semiconductor memory is, for example, a ROM.
  • the information processing system 100 reads out a program and data stored in the information storage medium, and performs various processes of the present embodiment based on the program and data. That is, the information storage medium stores a program and parameters for causing the computer to execute the trained model of the present embodiment.
  • a computer is a device including an input device, a processing unit, a storage unit, and an output unit.
  • the program is a program for causing a computer to execute an inference algorithm of a trained model.
  • the parameter is a parameter used in an inference algorithm, for example, a weighting coefficient of a node-to-node connection in a neural network.
  • various recording media that can be read by a computer, such as an optical disk such as a DVD or a CD, a magneto-optical disk, a hard disk, and a memory such as a non-volatile memory or a RAM, can be assumed.
  • FIG. 18 is a first configuration example of the endoscope system 200 including the information processing system 100.
  • the endoscope system 200 includes a processor unit 4, an endoscope scope 5, an operation unit 230, and a display unit 240.
  • An imaging device is provided at the tip of the endoscope scope 5, and the tip is inserted into the body cavity.
  • the imaging device is the second imaging system described above.
  • the imaging device captures an image in the abdominal cavity, and the imaging data is transmitted from the endoscope scope 5 to the processor unit 4.
  • the processor unit 4 is a device that performs various processes in the endoscope system 200.
  • the processor unit 4 controls the endoscope system 200, performs image processing, and the like.
  • the processor unit 4 includes an information processing system 100, a storage unit 211, a control unit 212, and an output unit 213.
  • the control unit 212 controls each unit of the endoscope system 200. For example, the mode of the endoscope system 200, the zoom operation, the display switching, and the like are performed based on the information input from the operation unit 230.
  • the operation unit 230 is a device for the user to operate the endoscope system 200.
  • the operation unit 230 is a button, a dial, a foot switch, a touch panel, or the like. The connection between the control unit 212 and each unit is not shown.
  • the storage unit 211 records the image captured by the endoscope scope 5.
  • the storage unit 211 is, for example, a semiconductor memory, a hard disk drive, an optical drive, or the like.
  • the information processing system 100 includes an imaging data receiving unit 110, a storage interface 120, a processing unit 3, and a storage unit 2.
  • the image pickup data receiving unit 110 receives the image pickup data from the endoscope scope 5.
  • the image pickup data receiving unit 110 is, for example, a connector to which a cable of the endoscope scope 5 is connected, an interface circuit for receiving image pickup data, or the like.
  • the storage interface 120 is an interface for accessing the storage unit 211.
  • the storage interface 120 records the image data received by the imaging data receiving unit 110 in the storage unit 211. When reproducing the recorded image data, the storage interface 120 reads the image data from the storage unit 211 and transmits the image data to the processing unit 3.
  • the processing unit 3 performs resolution restoration processing using the image data from the image pickup data receiving unit 110 or the storage interface 120 as the image to be processed.
  • the processing unit 3 outputs a high-resolution image after restoration.
  • the input unit 1 in the first to fourth embodiments corresponds to the image pickup data receiving unit 110 or the storage interface 120 in FIG.
  • the output unit 213 is a display controller that controls the image display on the display unit 240, and causes the display unit 240 to display the high-resolution image output from the processing unit 3.
  • the display unit 240 is a monitor that displays an image output from the output unit 213, and is a display device such as a liquid crystal display or an organic EL display.
  • the endoscope system 200 can include a light source (not shown).
  • the light source produces illumination light.
  • the endoscope scope 5 includes a light guide that guides the illumination light generated by the light source to the tip of the scope, and an illumination lens that diffuses the guided illumination light.
  • FIG. 19 is a second configuration example of the endoscope system 200 including the information processing system 100.
  • the endoscope system 200 includes an information processing system 100, a processor unit 4, an endoscope scope 5, an operation unit 230, and a display unit 240.
  • the information processing system 100 is provided outside the processor unit 4.
  • the information processing system 100 and the processor unit 4 may be connected by, for example, inter-device communication such as USB, or may be connected by network communication such as LAN or WAN.
  • the information processing system 100 is composed of one or a plurality of information processing devices.
  • the information processing system 100 may be a cloud system in which a plurality of PCs or a plurality of servers connected via a network perform parallel processing.
  • the processor unit 4 includes an imaging data receiving unit 110, a processing unit 214, an external interface 215, a control unit 212, and an output unit 213.
  • the processing unit 214 transmits the image data received by the imaging data receiving unit 110 to the information processing system 100 via the external interface 215.
  • the information processing system 100 generates a high-resolution image by super-resolution processing the received image data.
  • the external interface 215 receives the high-resolution image transmitted from the information processing system 100, and outputs the high-resolution image to the processing unit 214.
  • the processing unit 214 outputs a high-resolution image to the output unit 213, and the output unit 213 displays the high-resolution image on the display unit 240.
  • the information processing system 100 includes an external interface 130, a processing unit 3, a storage unit 2, a storage interface 120, and a storage unit 211.
  • the external interface 130 receives the image data transmitted from the processor unit 4.
  • the storage interface 120 and the storage unit 211 are the same as those in the first configuration example.
  • the processing unit 3 performs resolution restoration processing using the image data from the external interface 130 or the storage interface 120 as the image to be processed.
  • the processing unit 3 outputs the restored high-resolution image to the external interface 130, and the external interface 130 transmits the high-resolution image to the processor unit 4.
  • the input unit 1 in the first to fourth embodiments corresponds to the external interface 130 or the storage interface 120 in FIG.
  • the present invention is not limited to the respective embodiments and the modified examples as they are, and the present invention is within a range that does not deviate from the gist of the invention at the embodiment.
  • the components can be transformed and embodied with.
  • various inventions can be formed by appropriately combining a plurality of components disclosed in the above-described embodiments and modifications. For example, some components may be deleted from all the components described in each embodiment or modification. Further, the components described in different embodiments and modifications may be combined as appropriate. In this way, various modifications and applications are possible within a range that does not deviate from the gist of the invention.
  • a term described at least once in the specification or drawing together with a different term having a broader meaning or a synonym may be replaced with the different term at any part of the specification or drawing.

Landscapes

  • Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Surgery (AREA)
  • General Physics & Mathematics (AREA)
  • Signal Processing (AREA)
  • Medical Informatics (AREA)
  • Optics & Photonics (AREA)
  • General Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Biophysics (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Veterinary Medicine (AREA)
  • Pathology (AREA)
  • Radiology & Medical Imaging (AREA)
  • Public Health (AREA)
  • Biomedical Technology (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Molecular Biology (AREA)
  • Animal Behavior & Ethology (AREA)
  • Evolutionary Computation (AREA)
  • Multimedia (AREA)
  • Computing Systems (AREA)
  • Image Processing (AREA)
  • Studio Devices (AREA)
  • Image Analysis (AREA)
PCT/JP2019/043806 2019-11-08 2019-11-08 情報処理システム、内視鏡システム、学習済みモデル、情報記憶媒体及び情報処理方法 Ceased WO2021090469A1 (ja)

Priority Applications (5)

Application Number Priority Date Filing Date Title
PCT/JP2019/043806 WO2021090469A1 (ja) 2019-11-08 2019-11-08 情報処理システム、内視鏡システム、学習済みモデル、情報記憶媒体及び情報処理方法
CN201980101970.1A CN114651439B (zh) 2019-11-08 2019-11-08 信息处理系统、内窥镜系统、信息存储介质及信息处理方法
JP2021554529A JP7303896B2 (ja) 2019-11-08 2019-11-08 情報処理システム、内視鏡システム、学習済みモデル、情報記憶媒体及び情報処理方法
US17/731,627 US12347066B2 (en) 2019-11-08 2022-04-28 Information processing system, endoscope system, and information storage medium
US19/230,291 US20250299297A1 (en) 2019-11-08 2025-06-06 Information processing system, endoscope system, and information storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2019/043806 WO2021090469A1 (ja) 2019-11-08 2019-11-08 情報処理システム、内視鏡システム、学習済みモデル、情報記憶媒体及び情報処理方法

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/731,627 Continuation US12347066B2 (en) 2019-11-08 2022-04-28 Information processing system, endoscope system, and information storage medium

Publications (1)

Publication Number Publication Date
WO2021090469A1 true WO2021090469A1 (ja) 2021-05-14

Family

ID=75848306

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2019/043806 Ceased WO2021090469A1 (ja) 2019-11-08 2019-11-08 情報処理システム、内視鏡システム、学習済みモデル、情報記憶媒体及び情報処理方法

Country Status (4)

Country Link
US (2) US12347066B2 (https=)
JP (1) JP7303896B2 (https=)
CN (1) CN114651439B (https=)
WO (1) WO2021090469A1 (https=)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2022249934A1 (ja) * 2021-05-26 2022-12-01 キヤノン株式会社 画像処理方法、画像処理装置、プログラム、訓練済み機械学習モデルの製造方法、処理装置、画像処理システム
WO2024203900A1 (ja) * 2023-03-24 2024-10-03 モルゲンロット株式会社 情報処理装置、情報処理方法及びプログラム
WO2026004136A1 (ja) * 2024-06-28 2026-01-02 オリンパスメディカルシステムズ株式会社 機械学習用画像処理方法、機械学習用画像生成装置、機械学習方法、機械学習用画像生成用プログラム及び内視鏡装置
WO2026004143A1 (ja) * 2024-06-28 2026-01-02 オリンパスメディカルシステムズ株式会社 機械学習用画像処理方法、機械学習用画像生成装置、機械学習方法、機械学習用画像生成プログラム及び内視鏡装置

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20230334626A1 (en) * 2022-04-14 2023-10-19 Disney Enterprises, Inc. Techniques for denoising videos
CN115689896B (zh) * 2023-01-05 2023-06-27 荣耀终端有限公司 图像处理方法和图像处理装置
WO2025171486A1 (en) * 2024-02-16 2025-08-21 Vope Medical Inc. Systems and methods for improving image quality through an endoscope
WO2026016042A1 (zh) * 2024-07-16 2026-01-22 中国科学院深圳先进技术研究院 图像增强方法、装置、电子设备及存储介质

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2010250774A (ja) * 2009-04-20 2010-11-04 Fujifilm Corp 画像処理装置、画像処理方法およびプログラム
JP2016092596A (ja) * 2014-11-04 2016-05-23 キヤノン株式会社 撮像装置、画像処理装置及びそれらの制御方法
JP2016100795A (ja) * 2014-11-21 2016-05-30 キヤノン株式会社 画像処理装置、撮像装置、画像処理方法および画像処理プログラム
JP2018005841A (ja) * 2016-07-08 2018-01-11 株式会社トプコン 医用画像処理方法及び医用画像処理装置
JP2018195069A (ja) * 2017-05-17 2018-12-06 キヤノン株式会社 画像処理装置および画像処理方法

Family Cites Families (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
AU2003272936A1 (en) * 2003-01-31 2004-08-23 The Circle For The Promotion Of Science And Engineering Method for creating high resolution color image, system for creating high resolution color image and program for creating high resolution color image
JP4701187B2 (ja) * 2004-11-30 2011-06-15 パナソニック株式会社 画像処理方法、画像処理装置、および画像処理プログラム
JP5243432B2 (ja) * 2008-02-06 2013-07-24 パナソニック株式会社 画像処理装置、及び画像処理方法
JP2012244395A (ja) * 2011-05-19 2012-12-10 Sony Corp 学習装置および方法、画像処理装置および方法、プログラム、並びに記録媒体
JP6614843B2 (ja) * 2015-07-30 2019-12-04 キヤノン株式会社 画像処理装置、撮像装置および画像処理プログラム
JP6236731B1 (ja) 2017-03-10 2017-11-29 株式会社ツバサファクトリー 超解像度処理装置、超解像度処理方法およびコンピュータプログラム
FR3071344B1 (fr) * 2017-09-20 2019-09-13 Idemia Identity And Security Procede de determination d'une fonction d'etalement de point d'un systeme d'imagerie
JP7151109B2 (ja) * 2018-03-19 2022-10-12 ソニーグループ株式会社 医療用撮像装置及び医療用観察システム
JP7258604B2 (ja) * 2019-03-05 2023-04-17 キヤノン株式会社 画像処理方法、画像処理装置、プログラム、および学習済みモデルの製造方法
US12136487B2 (en) * 2019-11-15 2024-11-05 Sony Group Corporation Medical imaging system, medical imaging method, and image processing device
CN119768825A (zh) * 2022-09-08 2025-04-04 奥林巴斯医疗株式会社 信息处理系统、内窥镜系统、学习完毕模型、信息存储介质和信息处理方法

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2010250774A (ja) * 2009-04-20 2010-11-04 Fujifilm Corp 画像処理装置、画像処理方法およびプログラム
JP2016092596A (ja) * 2014-11-04 2016-05-23 キヤノン株式会社 撮像装置、画像処理装置及びそれらの制御方法
JP2016100795A (ja) * 2014-11-21 2016-05-30 キヤノン株式会社 画像処理装置、撮像装置、画像処理方法および画像処理プログラム
JP2018005841A (ja) * 2016-07-08 2018-01-11 株式会社トプコン 医用画像処理方法及び医用画像処理装置
JP2018195069A (ja) * 2017-05-17 2018-12-06 キヤノン株式会社 画像処理装置および画像処理方法

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2022249934A1 (ja) * 2021-05-26 2022-12-01 キヤノン株式会社 画像処理方法、画像処理装置、プログラム、訓練済み機械学習モデルの製造方法、処理装置、画像処理システム
JP2022181572A (ja) * 2021-05-26 2022-12-08 キヤノン株式会社 画像処理方法、画像処理装置、プログラム、訓練済み機械学習モデルの製造方法、処理装置、画像処理システム
JP7558890B2 (ja) 2021-05-26 2024-10-01 キヤノン株式会社 画像処理方法、画像処理装置、プログラム、訓練済み機械学習モデルの製造方法、処理装置、画像処理システム
JP2024175114A (ja) * 2021-05-26 2024-12-17 キヤノン株式会社 画像処理方法、画像処理装置、プログラム、訓練済み機械学習モデルの製造方法、処理装置、画像処理システム
WO2024203900A1 (ja) * 2023-03-24 2024-10-03 モルゲンロット株式会社 情報処理装置、情報処理方法及びプログラム
WO2026004136A1 (ja) * 2024-06-28 2026-01-02 オリンパスメディカルシステムズ株式会社 機械学習用画像処理方法、機械学習用画像生成装置、機械学習方法、機械学習用画像生成用プログラム及び内視鏡装置
WO2026004143A1 (ja) * 2024-06-28 2026-01-02 オリンパスメディカルシステムズ株式会社 機械学習用画像処理方法、機械学習用画像生成装置、機械学習方法、機械学習用画像生成プログラム及び内視鏡装置

Also Published As

Publication number Publication date
US12347066B2 (en) 2025-07-01
JP7303896B2 (ja) 2023-07-05
US20220253979A1 (en) 2022-08-11
JPWO2021090469A1 (https=) 2021-05-14
US20250299297A1 (en) 2025-09-25
CN114651439A (zh) 2022-06-21
CN114651439B (zh) 2024-03-05

Similar Documents

Publication Publication Date Title
JP7303896B2 (ja) 情報処理システム、内視鏡システム、学習済みモデル、情報記憶媒体及び情報処理方法
US8885067B2 (en) Multocular image pickup apparatus and multocular image pickup method
JP7297470B2 (ja) 画像処理方法、画像処理装置、プログラム、画像処理システム、および、学習済みモデルの製造方法
CN100481937C (zh) 一种高动态、超分辨率图像重建装置
CN102111544B (zh) 摄像模块、图像处理装置及图像记录方法
JP5306563B2 (ja) 撮像装置及び画像生成方法
JP5006814B2 (ja) 撮像装置
JPWO2004068862A1 (ja) 高解像度カラー画像生成方法、高解像度カラー画像生成装置及び高解像度カラー画像生成プログラム
JPWO2014125659A1 (ja) 画像処理装置、撮像装置、フィルタ生成装置、画像復元方法及びプログラム
CN114556897B (zh) 原始到rgb的图像转换
JP2009194896A (ja) 画像処理装置及び方法並びに撮像装置
KR102083721B1 (ko) 딥 러닝을 이용한 양안기반 초해상 이미징 방법 및 그 장치
CN101512600A (zh) 应用于运动分析的稀疏积分图像描述符
TW201220830A (en) Imaging apparatus, imaging method, and program
WO2011148760A1 (ja) 画像処理装置、撮像装置、プログラム及び画像処理方法
Song et al. Real-scene reflection removal with RAW-RGB image pairs
JPWO2021090469A5 (https=)
Pan et al. Joint demosaicking and denoising for CFA and MSFA images using a mosaic-adaptive dense residual network
US12294789B2 (en) Complementing wobulation in stereo cameras
JP2012003454A (ja) 画像処理装置、撮像装置および画像処理プログラム
US20250220315A1 (en) High-dynamic range imaging using partial polarisation mask
KR20220040025A (ko) 딥러닝 기반 열 영상 재구성 장치 및 방법
JP2023069527A (ja) 画像処理装置および画像処理方法
US20250193542A1 (en) Selective reading in colour filter arrays having smallest repeating units with different sub-units
JP2013157729A (ja) 画像処理装置、撮像装置及び画像処理方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19951624

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2021554529

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19951624

Country of ref document: EP

Kind code of ref document: A1