WO2024257318A1 - 機械学習用訓練データ生成方法、機械学習方法、機械学習プログラム、及び画像処理装置 - Google Patents

機械学習用訓練データ生成方法、機械学習方法、機械学習プログラム、及び画像処理装置 Download PDF

Info

Publication number
WO2024257318A1
WO2024257318A1 PCT/JP2023/022328 JP2023022328W WO2024257318A1 WO 2024257318 A1 WO2024257318 A1 WO 2024257318A1 JP 2023022328 W JP2023022328 W JP 2023022328W WO 2024257318 A1 WO2024257318 A1 WO 2024257318A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
captured
captured image
subject distance
machine learning
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2023/022328
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
健 井岡
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Olympus Medical Systems Corp
Original Assignee
Olympus Medical Systems Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Olympus Medical Systems Corp filed Critical Olympus Medical Systems Corp
Priority to PCT/JP2023/022328 priority Critical patent/WO2024257318A1/ja
Priority to JP2025527170A priority patent/JPWO2024257318A1/ja
Priority to CN202380099255.5A priority patent/CN121311921A/zh
Publication of WO2024257318A1 publication Critical patent/WO2024257318A1/ja
Priority to US19/270,836 priority patent/US20250342570A1/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/60—Image enhancement or restoration using machine learning, e.g. neural networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10068—Endoscopic image
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]

Definitions

  • the present invention relates to a method for generating training data for machine learning, a machine learning method, a machine learning program, and an image processing device.
  • the teacher data is a captured image generated by a first imaging device (hereinafter, referred to as a first captured image), whereas the training data is a simulation image obtained by adding blur to the first captured image.
  • Patent Literature 1 when generating a simulation image serving as training data from a first captured image serving as teacher data, the difference between the subject distance of the first captured image captured by the first imaging device and the subject distance of an image captured by the second imaging device is not taken into consideration.
  • the subject distance is assumed to be a specific subject distance, and a blur is added to the first captured image to generate a simulation image.
  • the simulation image cannot be generated correctly. As a result, a correct trained model cannot be generated, and a high-quality inference image cannot be generated with high accuracy.
  • the present invention has been made in consideration of the above, and aims to provide a method for generating training data for machine learning, a machine learning method, a machine learning program, and an image processing device that enable the generation of high-quality inference images with high accuracy.
  • the method for generating training data for machine learning acquires a first captured image generated by a first imaging device and a first subject distance of the first captured image, and generates a simulation image simulating a second captured image captured at a second subject distance by a second imaging device that generates an captured image of lower image quality than the first captured image as training data for machine learning corresponding to the first captured image serving as teacher data, by correcting the image quality of the first captured image based on a conversion table that defines the amount of correction based on the correlation between the first subject distance and the first imaging device, and the second subject distance and the second imaging device to generate the simulation image.
  • the machine learning method acquires teacher data and training data, and generates a learned model by machine learning based on the teacher data and the training data, where the teacher data is a first captured image captured by a first imaging device at a first subject distance, and the training data is a simulation image simulating a second captured image captured at a second subject distance by a second imaging device that generates an captured image of lower quality than the first captured image, and the simulation image is one in which the image quality of the first captured image has been corrected based on a conversion table that determines the amount of correction based on the correlation between the first subject distance and the first imaging device, and the second subject distance and the second imaging device.
  • the machine learning program of the present invention is a machine learning program that uses as teacher data a first captured image captured by a first imaging device at a first subject distance, and as training data a simulation image that imitates a second captured image captured at a second subject distance by a second imaging device that generates an image of lower quality than the first captured image, and the training data is an image in which the image quality of the first captured image has been corrected based on a conversion table that determines the amount of correction based on the correlation between the first subject distance and the first imaging device, and the second subject distance and the second imaging device.
  • the image processing device includes a distance information acquisition unit that acquires a subject distance indicating the subject distance of a processing target image for which image quality improvement processing is to be performed, a trained model selection unit that selects a trained model corresponding to the subject distance acquired by the distance information acquisition unit from among a plurality of trained models, and an image processing unit that acquires the processing target image and performs the image quality improvement processing on the processing target image using the trained model selected by the trained model selection unit to generate a high image quality inference image.
  • the machine learning training data generation method, machine learning method, machine learning program, and image processing device make it possible to generate high-quality inference images with high accuracy.
  • FIG. 1 is a block diagram showing a configuration of a machine learning training data generating device according to an embodiment.
  • FIG. 2 is a flowchart illustrating a method for generating training data for machine learning.
  • FIG. 3 is a diagram illustrating the generation process (step S1C).
  • FIG. 4 is a diagram illustrating the generation process (step S1C).
  • FIG. 5 is a block diagram showing the configuration of a machine learning device.
  • FIG. 6 is a flow chart illustrating a machine learning method.
  • FIG. 7 is a diagram illustrating the learning process (step S2B).
  • FIG. 8 is a block diagram showing the configuration of the second endoscope system.
  • FIG. 9 is a flowchart showing an image processing method.
  • FIG. 10 is a diagram illustrating a first modified example of the embodiment.
  • FIG. 10 is a diagram illustrating a first modified example of the embodiment.
  • FIG. 11 is a diagram illustrating the second modification of the embodiment.
  • FIG. 12 is a diagram illustrating the second modification of the embodiment.
  • FIG. 13 is a diagram illustrating the second modification of the embodiment.
  • FIG. 14 is a diagram illustrating the second modification of the embodiment.
  • FIG. 15 is a diagram illustrating a third modified example of the embodiment.
  • FIG. 16 is a diagram illustrating a fourth modified example of the embodiment.
  • FIG. 17 is a diagram illustrating a fourth modified example of the embodiment.
  • the machine learning training data generation device 100 and its method for generating training data for machine learning the machine learning device 200 and its method for performing machine learning using the training data to generate a trained model
  • the second endoscope system 300 and its method for performing image quality improvement processing using the trained model to generate a high-quality inference image we will explain in order the machine learning training data generation device 100 and its method for generating training data for machine learning, the machine learning device 200 and its method for performing machine learning using the training data to generate a trained model, and the second endoscope system 300 and its method for performing image quality improvement processing using the trained model to generate a high-quality inference image.
  • FIG. 1 is a block diagram showing a configuration of an apparatus 100 for generating machine learning training data according to an embodiment.
  • the machine learning training data generation device 100 is an information processing device such as a PC (Personal Computer) or a server, and generates simulation images that serve as training data required for generating a trained model used in image quality improvement processing (super-resolution).
  • the machine learning training data generation device 100 generates the training data from a first captured image generated by a first endoscope system 400.
  • the first endoscope system 400 is a system used in the medical field for observing the inside of a subject (inside a living body). As shown in FIG. 1, the first endoscope system 400 includes a first endoscope 410 and a first image processing device 420.
  • the first endoscope 410 corresponds to a first imaging device according to the present invention.
  • the first endoscope 410 is configured, for example, by a flexible endoscope having an imaging unit 411 ( FIG. 1 ) that is partially inserted into a living body and captures an image of a subject inside the living body.
  • the imaging unit 411 has an imaging element such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) that receives a subject image and converts it into an electrical signal.
  • CCD charge coupled device
  • CMOS complementary metal oxide semiconductor
  • the first image processing device 420 is configured to include a controller such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit), or an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array), and controls the operation of the entire first endoscope system 400. As shown in FIG. 1, the first image processing device 420 includes an image processing unit 421 and an external interface 422.
  • a controller such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit), or an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array)
  • ASIC Application Specific Integrated Circuit
  • FPGA Field-Programmable Gate Array
  • the image processing unit 421 performs a predetermined image processing on the first captured image.
  • the first captured image after the image processing is performed is then displayed on a display device (not shown).
  • the first captured image is also output to the outside via the external interface 422.
  • the machine learning training data generation device 100 includes an external interface 110, a storage unit 120, and a generation processing unit 130.
  • the storage unit 120 stores the first captured image acquired via the external interface 110.
  • Fig. 1 illustrates the machine learning training data generation device 100 configured to acquire the first captured image directly from the first endoscope system 400, the present invention is not limited to this.
  • the machine learning training data generation device 100 may be configured to acquire the first captured image output from the first endoscope system 400 and stored in a server or the like from the server via the external interface 110.
  • the storage unit 120 also stores various programs executed by the generation processing unit 130, information necessary for the processing of the generation processing unit 130, and the like.
  • the generation processing unit 130 includes a controller such as a CPU or an MPU, or an integrated circuit such as an ASIC or an FPGA, and generates a simulation image by executing a generation process described below.
  • a controller such as a CPU or an MPU, or an integrated circuit such as an ASIC or an FPGA, and generates a simulation image by executing a generation process described below.
  • the detailed functions of the generation processing unit 130 will be described later in the section entitled "Method of generating training data for machine learning.”
  • FIG. 2 is a flowchart illustrating a method for generating training data for machine learning.
  • the generation processing unit 130 acquires a first captured image stored in the memory unit 120 (step S1A), and acquires first subject distance information indicating the first subject distance of the first captured image when captured by the first endoscope 410 (step S1B). Furthermore, information on the imaging device that captured the first captured image (hereinafter, also referred to as endoscope information) may be acquired.
  • the imaging device information include at least one of the model and model number of the imaging device, and the optimal subject distance at which the endoscope that captured the image is in focus.
  • Examples of the imaging device include an endoscope, an eye catheter, and a digital camera.
  • the imaging device information may be obtained from at least one of an imaging device, such as an endoscope, an endoscope processor, and an image.
  • the method for generating training data for machine learning can be executed again using the same image, or when executing the method for generating training data for machine learning on an image or series of images extracted from the same video, the acquisition of the imaging device information can be omitted by using the imaging device information that has already been acquired.
  • the imaging unit 411 is configured with a stereo camera.
  • the generation processing unit 130 acquires the first object distance information or the first object distance information and the endoscope information as shown below. Specifically, the generation processing unit 130 calculates (acquires) first subject distance information indicating a first subject distance based on the principle of triangulation by using a relative shift amount in the image of the same subject in each captured image (first captured image) captured simultaneously from different viewpoints by the stereo camera.
  • the generation processing unit 130 calculates the first subject distance information indicating the first subject distance to a subject captured in the center of the first captured image or the first subject distance to a predetermined subject captured in the first captured image.
  • the generation processing unit 130 acquires the first subject distance information or the first subject distance information and endoscope information as shown below. Specifically, the generation processing unit 130 calculates (acquires) first subject distance information indicating a first subject distance based on pixel information corresponding to phase difference pixels in the first captured image. Here, the generation processing unit 130 calculates the first subject distance information indicating a first subject distance to a subject appearing in the center of the first captured image or a first subject distance to a predetermined subject appearing in the first captured image.
  • step S1B the generation processing unit 130 executes a generation process as described below to generate a simulation image (step S1C).
  • step S1C 3 and 4 are diagrams for explaining the generation process (step S1C). Specifically, FIG. 4 is a diagram showing a corrected PSF (Point Spread Function) stored in the storage unit 120.
  • PSF Point Spread Function
  • the generation processing unit 130 generates a simulation image corresponding to an image captured at a predetermined object distance by a second endoscope having a predetermined optimum object distance different from the first endoscope, based on the first object distance information and the endoscope information acquired in step S1B.
  • the simulation image serves as training data for machine learning.
  • the first object distance is, for example, 3 mm.
  • the above-mentioned value indicating the first object distance is merely an example, and other values may be used. For convenience of explanation, the above-mentioned value is used below.
  • the generation processing unit 130 generates an optical image of the first endoscope 410 by projecting a first captured image CI1 captured by the first endoscope 410 at an optimal subject distance for the first endoscope 410 without blurring onto an image plane (step S1C1). Specifically, the generation processing unit 130 generates an optical image of the first endoscope 410 by enlarging the length and width of the first captured image CI1 by a predetermined magnification.
  • the generation processing unit 130 reads a corrected PSF (corrected PSF(1) (FIG. 4)) corresponding to the first subject distance of the first captured image CI1 from the storage unit 120, as shown in FIG. 3 (step S1C2).
  • the memory unit 120 stores image quality correction information (corrected PSF(1)) as a corrected PSF corresponding to the first subject distance (3 mm) and the second subject distance (2 mm) of the second endoscope 310 (see FIG. 8).
  • a conversion table is used in which the degree of blur of the first imaging device and the degree of blur of the second imaging device are correlated in advance to calculate the necessary correction PSF. For example, looking at the conversion table shown in FIG. 4, it can be seen that in order to obtain a blurred image with a subject distance of 2 mm with the second imaging device, the image with the optimal subject distance captured by the first imaging device must be blurred to a subject distance of 3 mm. In other words, it can be seen that the image must be blurred to 3 mm, not 2 mm.
  • the corresponding conversion table may be identified from the imaging device information described above.
  • the second endoscope 310 generates an image of lower quality than the first captured image.
  • the simulation image is training data corresponding to the teacher data, in which the first captured image captured by the first endoscope 410 at the first subject distance (3 mm) is used as teacher data (correct image), and is a simulation image that imitates the second captured image captured by the second endoscope 310 at the second subject distance (2 mm).
  • the above-mentioned values indicating the second subject distance are merely examples, and other values may be used. In the following, the above-mentioned values are used for convenience of explanation.
  • the corrected PSF(1) is calculated as shown below.
  • MTF1 which is the amount of blur (MTF (Modulation Transfer Function)) on the image plane of the optical system constituting the first endoscope 410 at a first subject distance (3 mm)
  • MTF2 which is the amount of blur (MTF) on the image plane of the optical system constituting the second endoscope 310 at a second subject distance (2 mm)
  • Corrected PSF(1) is image quality correction information for correcting the amount of blur so that MTF1 becomes MTF2, and is configured by, for example, a two-dimensional filter.
  • step S1C2 the generation processing unit 130, as shown in FIG. 3, convolves the correction PSF(1) with the optical image generated in step S1C2 to correct the blur (image quality) of the optical image (first captured image) (step S1C3). As a result, an optical image of the second endoscope 310 is generated (step S1C4).
  • step S1C4 the generation processing unit 130 samples (thins out pixels) the optical image generated in step S1C4, as shown in FIG. 3, to generate a simulation image SI1.
  • the generation processing unit 130 stores the first captured image CI1 (teacher data) used to generate the simulation image SI1 and the simulation image SI1 (training data) as a pair in the storage unit 120.
  • FIG. 5 is a block diagram showing the configuration of the machine learning device 200.
  • the machine learning device 200 is an information processing device such as a PC or a server, and performs machine learning using teacher data and training data to generate a learned model.
  • the machine learning device 200 includes an external interface 210, a storage unit 220, and a learning processing unit 230.
  • the storage unit 220 stores a plurality of sets of teacher data (first captured image CI1) and training data (simulation image SI1) acquired via the external interface 210.
  • the machine learning device 200 is illustrated as having a configuration in which the machine learning training data generation device 100 directly acquires a plurality of sets of teacher data (first captured image CI1) and training data (simulation image SI1), but is not limited thereto.
  • the machine learning device 200 may be configured to acquire a plurality of sets of teacher data (first captured image CI1) and training data (simulation image SI1) output from the machine learning training data generation device 100 and stored in a server or the like from the server via the external interface 210.
  • the storage unit 220 also stores various programs executed by the learning processing unit 230, information necessary for the processing of the learning processing unit 230, and the like.
  • the learning processing unit 230 is configured to include a controller such as a CPU or MPU, or an integrated circuit such as an ASIC or FPGA, and generates a trained model by executing a learning process described below.
  • a controller such as a CPU or MPU
  • an integrated circuit such as an ASIC or FPGA
  • FIG. 6 is a flow chart illustrating a machine learning method.
  • the learning processing unit 230 acquires a plurality of sets of teacher data (first captured images CI1) and training data (simulation images SI1) stored in the storage unit 220 (step S2A).
  • step S2A the learning processing unit 230 generates a trained model by executing a learning process as described below (step S2B).
  • FIG. 7 is a diagram illustrating the learning process (step S2B).
  • the learning processing unit 230 repeatedly executes a learning process for the learning model using a plurality of sets of teacher data (first captured images CI1) and training data (simulation images SI1), and generates the learning model after the learning as a trained model MD.
  • the learning model used in the learning process is, for example, a CNN (Convolutional Neural Network).
  • the learning processing unit 230 calculates the weight values and bias values of each layer of the CNN, generates these values as a trained model MD, and stores the trained model MD in the storage unit 220.
  • the neural network used in the learning process is not limited to CNN, and other neural networks may be used.
  • various well-known learning algorithms may be used as the machine learning algorithm in the neural network. For example, a supervised learning algorithm using backpropagation may be used.
  • FIG. 8 is a block diagram showing the configuration of the second endoscope system 300.
  • the second endoscope system 300 is a system used in the medical field for observing the inside of a subject (inside a living body).
  • the second endoscope system 300 includes a second endoscope 310 and a second image processing device 320.
  • the second endoscope 310 corresponds to a second imaging device according to the present invention.
  • the second endoscope 310 is configured, for example, by a flexible endoscope having an imaging unit 311 ( FIG. 8 ) that is partially inserted into a living body and captures an image of a subject inside the living body.
  • the imaging section 311 differs from the imaging section 411 only in that the imaging section 311 generates a captured image of lower image quality than the imaging section 411.
  • the captured image generated by the imaging section 311 will be referred to as a second captured image.
  • the second image processing device 320 includes a controller such as a CPU or MPU, or an integrated circuit such as an ASIC or FPGA, and controls the operation of the entire second endoscope system 300. As shown in FIG. 8, the second image processing device 320 includes an external interface 321, a storage unit 322, and an image processing unit 323.
  • the storage unit 322 stores the trained model MD acquired via the external interface 321.
  • Fig. 8 illustrates a configuration in which the second endoscope system 300 acquires the trained model MD directly from the machine learning device 200, this is not limiting.
  • the second endoscope system 300 may be configured to acquire the trained model MD output from the machine learning device 200 and stored in a server or the like from the server via the external interface 321.
  • the storage unit 322 also stores various programs executed by the image processing unit 323, information necessary for the processing of the image processing unit 323, and the like.
  • the image processor 323 performs a predetermined image processing on the second captured image.
  • the second captured image after the image processing is then displayed on a display device (not shown).
  • the detailed functions of the image processing unit 323 will be described later in the "Image Processing Method" section.
  • FIG. 9 is a flowchart showing an image processing method.
  • the image processing unit 323 acquires a second captured image (step S3A) and reads the trained model MD from the storage unit 322 (step S3B).
  • the image processing unit 323 executes image quality improvement processing on the second captured image (processing target image) acquired in step S3A using the trained model MD to generate a high-image quality inference image (step S3C).
  • a high-image quality inference image such as the first captured image CI1 captured at a first subject distance (3 mm) by the first endoscope 410 that generates a captured image with higher image quality than the second endoscope 310 is generated.
  • step S1C when a simulation image simulating a second captured image captured at a second object distance by a second endoscope 310 that generates a captured image of lower image quality than the first captured image CI1 is generated as training data for machine learning corresponding to the first captured image CI1 (teacher data), the image quality of the first captured image CI1 is corrected based on the first object distance to generate the simulation image SI1. That is, the simulation image SI1 is generated taking into consideration the first object distance of the first captured image CI1 when captured by the first endoscope 410. Therefore, according to this embodiment, a simulation image SI1 can be correctly generated, and as a result, a correct trained model MD can be generated, and a high-quality inference image can be generated with high accuracy.
  • Fig. 10 is a diagram for explaining the first modification of the embodiment. Specifically, Fig. 10 corresponds to Fig. 3 and explains the generation process (step S1C) according to the first modification.
  • the corrected PSF (corrected PSF(1)) used in the blur correction process (step S1C3) is stored in the storage unit 120, but this is not limiting.
  • the corrected PSF (corrected PSF(1)) may be calculated before the blur correction process (step S1C3).
  • step S1C5 is employed instead of step S1C2.
  • the generation processing unit 130 calculates MTF1, which is the amount of blur (MTF) on the imaging plane of the optical system constituting the first endoscope 410 at the first object distance (3 mm) D1, by optical simulation from the first object distance (approximately 3 mm) D1 and the first lens configuration information I1 related to the optical system constituting the first endoscope 410, as shown in Fig. 10.
  • the generation processing unit 130 also calculates MTF2, which is the amount of blur (MTF) on the imaging plane of the optical system constituting the second endoscope 310 at the second object distance (2 mm) D2, by optical simulation from the second object distance (2 mm) D2 and the second lens configuration information I2 related to the optical system constituting the second endoscope 310. Then, the generation processing unit 130 calculates a correction PSF (correction PSF(1)), which is image quality correction information for correcting the amount of blur so that MTF1 becomes MTF2. The calculated corrected PSF (corrected PSF(1)) is used in the blur correction process (step S1C3).
  • FIG. 11 is a diagram corresponding to Fig. 4, and is a diagram showing a corrected PSF stored in the storage unit 120 according to the modified example 2.
  • Fig. 12 is a diagram corresponding to Fig. 7, and is a diagram for explaining the learning process (step S1B) according to the modified example 2.
  • Fig. 13 is a diagram corresponding to Fig. 8, and is a block diagram showing the configuration of a second endoscope system 300A according to the modified example 2.
  • Fig. 14 is a diagram corresponding to Fig. 9, and is a flowchart showing an image processing method according to the modified example 2.
  • the machine learning training data generation device 100 generates only the simulation image SI1 as training data corresponding to the first captured image CI1 (teacher data) in which the first subject distance is 3 mm, but this is not limited thereto. For example, it may generate a simulation image as training data corresponding to a first captured image in which the first subject distance is a value other than 3 mm.
  • the generation processing unit 130 in this second modified example generates a simulation image SI1 serving as training data corresponding to a first captured image CI1 in which the first subject distance based on the first subject distance information is a first specific distance, a simulation image SI2 serving as training data corresponding to a first captured image CI2 in which the first subject distance is a second specific distance, and a simulation image SI3 serving as training data corresponding to a first captured image CI3 in which the first subject distance is a third specific distance.
  • the first specific distance is 3 mm, as in the embodiment described above.
  • the second specific distance is, for example, 7.5 mm.
  • the third specific distance is, for example, 12.5 mm. Note that the above values indicating the first to third specific distances are merely examples, and other values may also be used. For the sake of convenience in the following description, the above values will be used.
  • step S1A the generation processing unit 130 in this modified example 2 obtains a first captured image CI1 captured at a first subject distance (3 mm), a first captured image CI2 captured at a first subject distance (7.5 mm), and a first captured image CI3 captured at a first subject distance (12.5 mm).
  • step S1B the generation processing unit 130 in this variation 2 obtains first subject distance information indicating the first subject distance (3 mm), first subject distance information indicating the first subject distance (7.5 mm), and first subject distance information indicating the first subject distance (12.5 mm).
  • the storage unit 120 stores image quality correction information (corrected PSF (1)) as a corrected PSF corresponding to the first subject distance (3 mm) of the first endoscope 410 and the second subject distance (2 mm) of the second endoscope 310.
  • the storage unit 120 also stores image quality correction information (corrected PSF (2)) as a corrected PSF corresponding to the first subject distance (7.5 mm) of the first endoscope 410 and the second subject distance (5 mm) of the second endoscope 310.
  • the storage unit 120 also stores image quality correction information (corrected PSF (3)) as a corrected PSF corresponding to the first subject distance (12.5 mm) of the first endoscope 410 and the second subject distance (9 mm) of the second endoscope 310.
  • the generation processing unit 130 executes step S1C using the first captured image CI1 at the first subject distance (3 mm) and the corrected PSF (1) to generate a simulation image SI1.
  • This simulation image SI1 uses the first captured image CI1 captured by the first endoscope 410 at the first subject distance (3 mm) as teacher data, and is training data corresponding to the teacher data, and is a simulation image that imitates the second captured image captured by the second endoscope 310 at the second subject distance (2 mm).
  • the generation processing unit 130 executes step S1C using the first captured image CI2 at the first object distance (7.5 mm) and the corrected PSF (2) to generate a simulation image SI2.
  • This simulation image SI2 is training data corresponding to the teacher data, which is the first captured image CI2 captured by the first endoscope 410 at the first object distance (7.5 mm), and is a simulation image simulating a second captured image captured by the second endoscope 310 at the second object distance (5 mm).
  • the above-mentioned values indicating the second object distance are merely examples, and other values may be used. In the following, the above-mentioned values are used for convenience of explanation.
  • the generation processing unit 130 executes step S1C using the first captured image CI3 at the first subject distance (12.5 mm) and the corrected PSF (3) to generate a simulation image SI3.
  • This simulation image SI3 is a simulation image that uses the first captured image CI3 captured by the first endoscope 410 at the first subject distance (12.5 mm) as teacher data, is training data corresponding to the teacher data, and imitates the second captured image captured by the second endoscope 310 at the second subject distance (9 mm).
  • the above-mentioned values indicating the second subject distance are merely examples, and other values may be used. In the following, the above-mentioned values are used for convenience of explanation.
  • the generation processing unit 130 stores the first captured image CI1 (teacher data) used to generate the simulation image SI1 and the simulation image SI1 (training data) as a pair in the storage unit 120.
  • the generation processing unit 130 also stores the first captured image CI2 (teacher data) used to generate the simulation image SI2 and the simulation image SI2 (training data) as a pair in the storage unit 120.
  • the generation processing unit 130 also stores the first captured image CI3 (teacher data) used to generate the simulation image SI3 and the simulation image SI3 (training data) as a pair in the storage unit 120.
  • the processing of the machine learning device 200 also differs from that in the above-described embodiment. Specifically, as shown in FIG. 12, the learning processing unit 230 in this variant example 2 repeatedly executes a learning process (step S2B (step S2B1)) for the learning model using multiple sets of teacher data (first captured image CI1) and training data (simulation image SI1), and generates the learning model after learning as a learned model MD1.
  • a learning process step S2B (step S2B1)
  • teacher data first captured image CI1
  • training data stimulation image SI1
  • the learning processing unit 230 repeatedly executes a learning process (step S2B (step S2B2)) for the learning model using multiple sets of teacher data (first captured images CI2) and training data (simulation images SI2), and generates the learning model after the learning as a learned model MD2.
  • step S2A the learning processing unit 230 in this modified example 2 acquires multiple sets of teacher data (first captured image CI1) and training data (simulation image SI1), multiple sets of teacher data (first captured image CI2) and training data (simulation image SI2), and multiple sets of teacher data (first captured image CI3) and training data (simulation image SI3).
  • the learning processing unit 230 stores the generated trained model MD1 in the storage unit 220 in association with the second subject distance (2 mm).
  • the learning processing unit 230 also stores the generated trained model MD2 in association with the second subject distance (5 mm) in the storage unit 220.
  • the learning processing unit 230 also stores the generated trained model MD3 in association with the second subject distance (9 mm) in the storage unit 220.
  • the configuration of the second endoscope system 300A is also different from the second endoscope system 300 described in the above-mentioned embodiment.
  • the second endoscope system 300A according to the present modification 2 has a second image processing device 320 whose configuration is changed with respect to the second endoscope system 300.
  • the second image processing device 320 according to the present modification 2 will be referred to as a second image processing device 320A.
  • the second image processing device 320A corresponds to the image processing device according to the present invention. As shown in FIG. 13, the second image processing device 320A has the functions of a distance information acquisition unit 324 and a trained model selection unit 325 added to the second image processing device 320.
  • the image processing method according to the present modified example 2 will be described, while also explaining the functions of the distance information acquisition unit 324 and the trained model selection unit 325.
  • the image processing method according to the second modification adds steps S3D and S3E instead of step S3B to the image processing method described in the above embodiment.
  • Step S3D is executed after step S3A. Specifically, in step S3D, the distance information acquisition unit 324 acquires second subject distance information indicating the second subject distance of the second captured image acquired in step S3A.
  • the distance information acquisition unit 324 acquires second subject distance information in step S3D as shown below. Specifically, the distance information acquisition unit 324 calculates (acquires) first subject distance information indicating the second subject distance based on pixel information corresponding to the phase difference pixels in the second captured image. Here, the distance information acquisition unit 324 calculates second subject distance information indicating the second subject distance to a subject appearing in the center of the second captured image or the second subject distance to a predetermined subject appearing in the second captured image.
  • the trained model selection unit 325 selects a trained model corresponding to the second object distance based on the second object distance information acquired in step S3D from among the trained models MD1 to MD3 stored in the storage unit 322 (step S3E). Then, in the high image quality processing in step S3C, the trained model selected in step S3E is used. That is, by performing the high image quality processing on the second captured image captured by the second endoscope 310, a high image quality inference image such as the first captured image CI1 captured at the first object distance and the optimal object distance by the first endoscope 410 that generates a captured image of higher image quality than the second endoscope 310 is generated.
  • the image to be subjected to high image quality application to the learning model trained so that an image with an object distance of 2 mm becomes the optimal object distance does not necessarily have to have an object distance of 2 mm.
  • the image has an object distance of 1 mm to 3 mm, for example, it is possible to convert the image quality so that the image becomes the optimal object distance.
  • the image quality can be improved using the same machine learning model.
  • a high-image-quality inference image is generated, such as the first captured image CI2 captured at the optimum subject distance by the first endoscope 410 that generates a captured image of higher image quality than the second endoscope 310.
  • a high-image-quality inference image is generated, such as the first captured image CI3 captured at the optimum subject distance by the first endoscope 410 that generates a captured image of higher image quality than the second endoscope 310.
  • Fig. 15 is a diagram for explaining the third modification of the embodiment. Specifically, Fig. 15 corresponds to Fig. 12 and explains the learning process (step S1B) according to the third modification.
  • the learned models MD1 to MD3 are generated by machine learning from multiple sets of teacher data (first captured image CI1) and training data (simulation image SI1), multiple sets of teacher data (first captured image CI2) and training data (simulation image SI2), and multiple sets of teacher data (first captured image CI3) and training data (simulation image SI3), but this is not limited to this.
  • step S2B the learning processing unit 230 of this modified example 3 repeatedly executes learning processing for the same learning model using multiple sets of teacher data (first captured image CI1) and training data (simulation image SI1), multiple sets of teacher data (first captured image CI2) and training data (simulation image SI2), and multiple sets of teacher data (first captured image CI3) and training data (simulation image SI3), and generates the learning model after learning as a learned model MD4.
  • the learned model MD4 corresponds to the third learned model of the present invention.
  • the image processing method according to the third modification is the same as the embodiment described above, except that the trained model MD is changed to the trained model MD4. That is, in the third modification, as in the second modification, a high-quality inference image is generated, such as the first captured image CI1 captured at the first subject distance and the optimal subject distance by the first endoscope 410 that generates a captured image of higher quality than the second endoscope 310, by performing image quality improvement processing on the second captured image captured by the second endoscope 310.
  • the image to be improved in quality to be applied to the learning model trained to have an image with a subject distance of 2 mm as the optimal subject distance during machine learning does not necessarily have to have a subject distance of 2 mm.
  • the image quality can be improved using the same machine learning model.
  • a high-image-quality inference image is generated, such as the first captured image CI2 captured at the optimum subject distance by the first endoscope 410 that generates a captured image of higher image quality than the second endoscope 310.
  • a high-image-quality inference image is generated, such as the first captured image CI3 captured at the optimum subject distance by the first endoscope 410 that generates a captured image of higher image quality than the second endoscope 310.
  • a single learned model MD4 is generated by machine learning using multiple sets of teacher data (first captured image CI1) and training data (simulation image SI1), multiple sets of teacher data (first captured image CI2) and training data (simulation image SI2), and multiple sets of teacher data (first captured image CI3) and training data (simulation image SI3). Therefore, when the second endoscope 310 generates a second captured image at various second subject distances, a high-quality inference image can be generated with high accuracy from the second captured image using the trained model MD4, even if the second subject distance is not obtained.
  • FIG. 16 and 17 are diagrams for explaining the fourth modification of the embodiment.
  • Fig. 16 corresponds to Fig. 3 and is a diagram for explaining the generation process (step S1C) according to the fourth modification.
  • the generation processing unit 130 according to the present modified example 4 generates the following simulation image SI4 in addition to generating the simulation image SI1.
  • the simulation image SI4 is teacher data corresponding to the training data, using the simulation image SI1 as training data, and is a simulation image simulating an image captured at a predetermined subject distance by a third endoscope (not shown) that generates an image having a lower image quality than the first captured image CI1 but a higher image quality than the second captured image generated by the second endoscope 310.
  • step S1C the generation processing unit 130 according to the present modified example 4 generates a simulation image SI1 in the same manner as in the above-described embodiment, and also generates a simulation image SI4 as described below.
  • the generation processing unit 130 according to the present modification 4 generates an optical image of the first endoscope 410 by projecting a first captured image CI1 captured by the first endoscope 410 at a first subject distance (3 mm) onto an image plane (step S1C6).
  • the generation processing unit 130 generates an optical image of the first endoscope 410 by enlarging the length and width of the first captured image CI1 by a predetermined magnification.
  • the corrected PSF(1) is calculated as shown below.
  • MTF1 which is the amount of blur (MTF) on the image plane of the optical system constituting the first endoscope 410 at a first subject distance (3 mm)
  • MTF3 which is the amount of blur (MTF) on the image plane of the optical system constituting the third endoscope at a specified subject distance
  • Correction PSF(4) is image quality correction information for correcting the amount of blur so that MTF1 becomes MTF3, and is configured by, for example, a two-dimensional filter.
  • step S1C7 the generation processing unit 130 according to this fourth modification convolves the optical image generated in step S1C6 with the correction PSF(4) as shown in FIG. 16, and corrects the blur (image quality) of the optical image (first captured image) (step S1C7). As a result, an optical image of the third endoscope is generated (step S1C8).
  • step S1C8 the generation processing unit 130 according to this fourth modification samples (thins out pixels) the optical image generated in step S1C8, as shown in FIG. 16, to generate a simulation image SI4.
  • the generation processing unit 130 stores the simulation image SI4 (teacher data) and the simulation image SI1 (training data) as a pair in the storage unit 120.
  • step S2A the learning processing unit 230 in this modification 4 acquires multiple sets of teacher data (simulation image SI4) and training data (simulation image SI1).
  • the image processing method according to this modification 4 is the same as the embodiment described above, except that the trained model MD is changed to trained model MD5. That is, in this modification 4, by performing image quality improvement processing on a second captured image captured by a second endoscope 310 at a second subject distance (approximately 2 mm), a high-quality inference image is generated, which is similar to an image captured at a predetermined subject distance by a third endoscope that generates captured images of higher quality than the second endoscope 310.
  • the simulation image SI1 is used as training data, and a simulation image SI4 is generated which is teacher data corresponding to the training data and imitates an image captured at a predetermined subject distance by a third endoscope (not shown) which generates a captured image having a lower image quality than the first captured image CI1 and a higher image quality than the second captured image generated by the second endoscope 310.
  • a learned model MD5 is generated by machine learning using multiple sets of teacher data (simulation images SI4) and training data (simulation images SI1).
  • the first to third imaging devices according to the present invention are constituted by endoscopes, but this is not limited thereto, and any other imaging device may be adopted as long as it is an imaging device that generates an image by capturing an image.
  • the PSF of the optical image (two-dimensional image) is corrected to generate the simulation image SI1, but this is not limiting.
  • the simulation image SI1 may be generated by converting the first captured image CI1 into a frequency space and correcting an OTF (Optical Transfer Function) in the frequency space.
  • OTF Optical Transfer Function
  • Machine learning training data generating device 110 External interface 120 Memory unit 130 Generation processing unit 200 Machine learning device 210 External interface 220 Memory unit 230 Learning processing unit 300, 300A Second endoscope system 310 Second endoscope 311 Imaging unit 320, 320A Second image processing device 321 External interface 322 Memory unit 323 Image processing unit 324 Distance information acquisition unit 325 Trained model selection unit 400 First endoscope system 410 First endoscope 411 Imaging unit 420 First image processing device 421 Image processing unit 422 External interface CI1 to CI3 First captured image D1 First subject distance D2 Second subject distance I1 First lens configuration information I2 Second lens configuration information MD, MD1 to MD5 Trained model SI1 to SI4 Simulation images

Landscapes

  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Image Analysis (AREA)
PCT/JP2023/022328 2023-06-15 2023-06-15 機械学習用訓練データ生成方法、機械学習方法、機械学習プログラム、及び画像処理装置 Ceased WO2024257318A1 (ja)

Priority Applications (4)

Application Number Priority Date Filing Date Title
PCT/JP2023/022328 WO2024257318A1 (ja) 2023-06-15 2023-06-15 機械学習用訓練データ生成方法、機械学習方法、機械学習プログラム、及び画像処理装置
JP2025527170A JPWO2024257318A1 (enExample) 2023-06-15 2023-06-15
CN202380099255.5A CN121311921A (zh) 2023-06-15 2023-06-15 机器学习用训练数据生成方法、机器学习方法、机器学习程序以及图像处理装置
US19/270,836 US20250342570A1 (en) 2023-06-15 2025-07-16 Machine learning training data generation method, machine learning method, and computer-readable recording medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2023/022328 WO2024257318A1 (ja) 2023-06-15 2023-06-15 機械学習用訓練データ生成方法、機械学習方法、機械学習プログラム、及び画像処理装置

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US19/270,836 Continuation US20250342570A1 (en) 2023-06-15 2025-07-16 Machine learning training data generation method, machine learning method, and computer-readable recording medium

Publications (1)

Publication Number Publication Date
WO2024257318A1 true WO2024257318A1 (ja) 2024-12-19

Family

ID=93851719

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2023/022328 Ceased WO2024257318A1 (ja) 2023-06-15 2023-06-15 機械学習用訓練データ生成方法、機械学習方法、機械学習プログラム、及び画像処理装置

Country Status (4)

Country Link
US (1) US20250342570A1 (enExample)
JP (1) JPWO2024257318A1 (enExample)
CN (1) CN121311921A (enExample)
WO (1) WO2024257318A1 (enExample)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2021174070A (ja) * 2020-04-20 2021-11-01 キヤノン株式会社 画像処理方法、学習済みモデルの製造方法、プログラム、および、画像処理装置
JP2021189929A (ja) * 2020-06-03 2021-12-13 キヤノン株式会社 画像処理方法、プログラム、画像処理装置、および、画像処理システム
JP2021190814A (ja) * 2020-05-29 2021-12-13 キヤノン株式会社 学習方法、画像処理方法、およびプログラム

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2021174070A (ja) * 2020-04-20 2021-11-01 キヤノン株式会社 画像処理方法、学習済みモデルの製造方法、プログラム、および、画像処理装置
JP2021190814A (ja) * 2020-05-29 2021-12-13 キヤノン株式会社 学習方法、画像処理方法、およびプログラム
JP2021189929A (ja) * 2020-06-03 2021-12-13 キヤノン株式会社 画像処理方法、プログラム、画像処理装置、および、画像処理システム

Also Published As

Publication number Publication date
CN121311921A (zh) 2026-01-09
JPWO2024257318A1 (enExample) 2024-12-19
US20250342570A1 (en) 2025-11-06

Similar Documents

Publication Publication Date Title
JP7258604B2 (ja) 画像処理方法、画像処理装置、プログラム、および学習済みモデルの製造方法
JP7362284B2 (ja) 画像処理方法、画像処理装置、プログラム、画像処理システム、および、学習済みモデルの製造方法
JP7721626B2 (ja) 学習データの生成方法、学習方法、学習データ製造装置、学習装置、およびプログラム
JP5237978B2 (ja) 撮像装置および撮像方法、ならびに前記撮像装置のための画像処理方法
JP2020201540A (ja) 画像処理方法、画像処理装置、画像処理システム、学習済みウエイトの製造方法、および、プログラム
JP7234057B2 (ja) 画像処理方法、画像処理装置、撮像装置、レンズ装置、プログラム、記憶媒体、および、画像処理システム
JP2026083367A (ja) 画像処理方法、画像処理装置、画像処理プログラム、学習方法、学習装置、学習プログラム
JP2022056769A (ja) 画像処理方法、プログラム、画像処理装置、学習済みモデルの製造方法、学習方法、学習装置、および、画像処理システム
JP2009111596A (ja) 撮像装置
JP6487008B1 (ja) 高解像度撮像装置
JP2021189929A (ja) 画像処理方法、プログラム、画像処理装置、および、画像処理システム
JP2020187611A (ja) 画像処理方法、画像処理装置、画像処理システム、撮像装置、および、プログラム
JP2012065113A (ja) 撮像装置、撮像方法及びプログラム
JP2015115733A (ja) 画像処理方法、画像処理装置、撮像装置および画像処理プログラム
JP2018148512A (ja) 撮像装置と撮像装置の制御方法、及びプログラム
JP2020030569A (ja) 画像処理方法、画像処理装置、撮像装置、レンズ装置、プログラム、および、記憶媒体
JP5543280B2 (ja) 画像処理装置、画像処理方法
US20240144536A1 (en) Image processing device, image processing method, and storage medium
JP2021190814A (ja) 学習方法、画像処理方法、およびプログラム
JP2020154669A (ja) 学習装置、学習方法、学習プログラム、データの製造方法、画像処理装置、画像処理方法、および、画像処理プログラム
JP2024038523A (ja) 画像処理方法、学習済みモデルの製造方法、プログラム、および、画像処理装置
US20250342570A1 (en) Machine learning training data generation method, machine learning method, and computer-readable recording medium
JP2023069527A (ja) 画像処理装置および画像処理方法
JP2021086272A (ja) 画像処理方法、プログラム、学習済みモデルの製造方法、画像処理装置、および、画像処理システム
JP2021103404A (ja) 画像処理方法、プログラム、学習済みモデルの製造方法、画像処理装置、および、画像処理システム

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23941620

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2025527170

Country of ref document: JP

Kind code of ref document: A

WWE Wipo information: entry into national phase

Ref document number: 2025527170

Country of ref document: JP

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 23941620

Country of ref document: EP

Kind code of ref document: A1