JP2021069818A - Imaging apparatus, imaging apparatus control method, and program - Google Patents

Abstract

To acquire a medical image having reduced failures by re-imaging a subject.SOLUTION: An imaging apparatus comprises: first drive means for driving an imaging optical unit for imaging a subject; second drive means for driving an optical member included in the imaging optical unit; control means for controlling the first drive means and the second drive means; and output means for outputting information about a failure in a medical image obtained by imaging the subject with the imaging optical unit, by using a learned model, from the medical image obtained by imaging.SELECTED DRAWING: Figure 5

Description

The present invention relates to a photographing device for photographing a subject, a control method of the photographing device, and a program.

Conventionally, for example, in the field of ophthalmology, an image of an eye to be inspected has been taken by various imaging devices for the purpose of acquiring a medical image for diagnosing the eye to be inspected. For example, a fundus camera is used as an ophthalmologic device, which is one of the imaging devices for diagnosing the fundus of the eye to be inspected. Further, for example, an apparatus using an optical coherence tomography (OCT) using interference by low coherent light as an ophthalmic apparatus for diagnosing a tomography of an eye to be inspected (hereinafter, this is referred to as an OCT apparatus). used.

Generally, in the design of an ophthalmic apparatus, various eye models to be inspected are assumed, and the design is performed with a sufficient margin. However, in the actual eye to be inspected, there may be more variation than expected with respect to the modeled eye due to the influence of diseases and the like. In this case, there is a possibility that an eye image suitable for diagnosis cannot be obtained by a normal imaging method.

For example, in shooting with a fundus camera, there is a time lag between the time when the operator presses the shooting switch and the time when the shooting light source generates a flash as shooting light. If the subject blinks in the meantime, the blinking may block the irradiation of the fundus with the photographing light, and excessive reflected light from the anterior segment of the photographing light may be obtained. Further, if the eye to be inspected moves from an appropriate alignment position at the moment of photographing and the so-called misalignment occurs, flare occurs in the peripheral portion of the fundus image. As described above, the fundus image acquired in an unsuitable state is regarded as an image having photo loss, and is not suitable for diagnosis. In this way, for example, a fundus image in which a state such as poor alignment, poor focus, or underexposure appears is regarded as an image having photo loss, and such poor alignment, poor focus, or underexposure causes photo loss.

At this time, a method of having the examiner judge the quality of the image and the cause of the image loss, and automatically re-shooting the eye image when it is determined that the obtained eye image has image loss due to blinking or flare, is It is disclosed in Patent Document 1. Further, in the method disclosed in Patent Document 1, when it is determined that the copying loss is caused by flare, the position of the fundus camera at the time of re-imaging is automatically calculated for readjustment of the alignment.

Japanese Unexamined Patent Publication No. 2016-7737

In the method disclosed in Patent Document 1 in which the examiner determines the cause of the copying loss, if the examiner makes a mistake in the cause of the copying loss, the image is re-photographed under inappropriate shooting conditions. Similar image loss may occur on the image. In addition to the above-mentioned misalignment, poor focus, and underexposure, there are various causes of this image loss, such as unclear fundus due to disease. For this reason, it is desirable that a discerning examiner makes a judgment and perform re-imaging under appropriate imaging conditions according to the cause of the image loss. However, there may be cases where a knowledgeable examiner cannot operate the ophthalmologic device, for example, during screening by group medical examination. In this case, even if re-imaging is performed, the same image loss may occur on the obtained image.

In view of the above-mentioned problems, one of the objects of the present invention is to make it possible to obtain a medical image in which copying loss is reduced by re-imaging a subject.

In order to solve the above problems, the photographing apparatus according to one embodiment of the present invention is provided.
The first driving means for driving the photographing optical unit for photographing the subject,
A second driving means for driving the optical member included in the photographing optical unit, and
A control means for controlling the first drive means and the second drive means,
An output means for outputting information on copying loss in the medical image obtained by photographing the subject using the trained model from the medical image obtained by photographing the subject by the photographing optical unit.
It is characterized by having.

According to one of the present inventions, it is possible to obtain a medical image with reduced copying loss by re-imaging the subject.

It is a figure which shows the schematic structure of the fundus camera which is an example of the ophthalmic apparatus which concerns on Example 1 of this invention. It is a schematic diagram which shows the optical composition of the fundus camera illustrated in FIG. It is a block diagram which shows typically the structure of the control part used in the fundus camera shown in FIG. It is a schematic diagram explaining the alignment operation with respect to the anterior eye part performed in the fundus camera illustrated in FIG. It is a schematic diagram explaining the focus operation with respect to the fundus performed by the fundus camera illustrated in FIG. It is a flowchart explaining the process of fundus photography which concerns on Example 1 of this invention. It is a schematic diagram explaining CNN which is an example of a machine learning model. It is a figure which shows typically the optical composition of the imaging system which can replace the fundus camera in Example 1. FIG. It is a figure which shows typically the optical composition of the other imaging system which can replace the fundus camera in Example 1. FIG. It is a figure which shows the schematic structure of the OCT apparatus which is an example of the ophthalmic apparatus which concerns on Example 2 of this invention. It is a block diagram which shows typically the structure of the control part of the ophthalmic apparatus shown in FIG. It is a flowchart explaining the process of the tomographic image taking which concerns on Example 2 of this invention. It is the schematic explaining one aspect of the preview displayed on the display monitor in Example 2. FIG.

As an exemplary embodiment of the present invention, an ophthalmic apparatus is illustrated as an aspect of the imaging apparatus, and will be described in detail below with reference to the drawings. However, the dimensions, materials, shapes, relative positions of the components, etc. described in the following examples are arbitrary and can be changed according to the configuration of the device to which the present invention is applied or various conditions. Also, in the drawings, the same reference numerals are used between the drawings to indicate elements that are the same or functionally similar.

In this embodiment, a trained model related to a machine learning model is used. A machine learning model is a learning model based on a machine learning algorithm such as deep learning. In deep learning, a neural network is used to generate features and coupling weighting coefficients for learning. Examples of machine learning algorithms other than deep learning include support vector machines, nearest neighbor methods, linear regression, averaging methods, neural networks, and the like. Of these algorithms, any available algorithm can be applied to the present invention as appropriate. The trained model is a model obtained (trained) by training a machine learning model by an arbitrary machine learning algorithm using appropriate learning data in advance. However, although the trained model is obtained by using appropriate training data in advance, it does not mean that no further training is performed, and additional training can be performed. Additional learning can be performed even after the ophthalmic device has been installed at the site of use. In the following, the teacher data refers to learning data and is composed of a pair of input data and output data. The output data of the learning data (teacher data) is also referred to as correct answer data.

In the following examples, images that reflect inappropriate phenomena and conditions for imaging such as poor alignment, poor focus, underexposure, indistinct fundus due to illness, blinking of the eye to be inspected, etc. Is an image having photo loss. For example, poor alignment causes so-called flare in the image, and poor focus causes so-called out-of-focus. That is, an image having photo loss refers to an image in a state inappropriate for diagnosis, such as flare or out-of-focus. In addition, the causes of image loss may be, for example, the above-mentioned misalignment, poor focus, underexposure, indistinctness of the fundus due to disease, blinking of the eye to be inspected, and the like. In the following, an example will be described in which the ophthalmic apparatus is controlled so that an image with reduced photo loss can be acquired at the time of re-imaging when re-imaging is performed because an image having photo loss is obtained.

[Example 1]
In this embodiment, a fundus camera is illustrated as an ophthalmic apparatus, and examples in which the present invention is applied to a fundus camera will be described with reference to FIGS. 1 to 6. In this embodiment, the photographed fundus image and the learned model described later are used to output the correction value under the imaging conditions for executing reimaging under the imaging conditions that reduce the imaging loss. Then, the output correction value is used to perform reimaging with an ophthalmic apparatus. As a result, even if the examiner does not have sufficient insight into the copying loss, it is possible to reduce the possibility of the copying loss occurring at the time of re-shooting. Hereinafter, this embodiment will be described in detail.

<Outline configuration of the device>
The schematic configuration of an example of the fundus camera used in this embodiment will be described with reference to FIG. The fundus camera C includes a base C1, a photographing optical unit C2, and a joystick C3. The photographing optical unit C2 houses an optical system that can move in the left-right direction (X direction), the up-down direction (Y direction), and the front-back direction (Z direction) with respect to the base C1. The photographing optical unit C2 is movable in the three-dimensional (XYZ) direction with respect to the eye to be inspected by a driving unit C4 (see FIG. 2B) composed of a pulse motor or the like provided on the base C1. Further, the drive unit C4 can move the photographing optical unit C2 in the three-dimensional (XYZ) direction by operating the joystick C3.

<Optical configuration of the device>
The optical configuration of the fundus camera C according to this embodiment will be described with reference to FIG. 2 (a). The optical system corresponds to a photographing optical system for observing, imaging, or photographing an eye to be inspected. The configuration of the optical system is divided into a plurality of optical paths L1 to L5 in which various optical elements are arranged, and each of them will be described in order.

The optical path L1 is arranged with a light source that emits illumination light for observing or photographing the fundus and related configurations, and is arranged along the optical path of the illumination light emitted from the light source. On the optical path L1, in order from the observation light source 1 that emits constant light such as a halogen lamp, the condenser lens 2, the filter 3 that transmits infrared light and blocks visible light, the photographing light source 4 such as a strobe, and the lens 5 , And the mirror 6 is arranged. The optical path L2 is arranged in the reflection direction of the mirror 6. On the optical path L2, a ring diaphragm 7 having a ring-shaped opening, a crystalline lens baffle 31, a relay lens 8, a corneal baffle 32, and a perforated mirror 9 having a central opening are arranged in order from the mirror 6.

The optical path L3 is arranged in the reflection direction of the perforated mirror 9. On the optical path L3, the dichroic mirror 24 and the objective lens 10 facing the eye E to be inspected are arranged in order from the perforated mirror. The dichroic mirror 24 is removable with respect to the optical path L3. The photographing diaphragm 11 is arranged in the hole portion of the perforated mirror 9, and the focus lens 12, the photographing lens 13, and the half mirror 100 are arranged in this order behind the photographing aperture 11. The focus lens 12 is driven in the direction of the optical path L3 indicated by the arrow in the figure to adjust the focusing state with respect to the fundus Er. Further, an image sensor 14 for both moving image observation and still image shooting is arranged at the tip of the half mirror 100, and an internal fixation lamp 101 is arranged at the tip of the optical path L4 which is the reflection direction of the half mirror 100. The internal fixation lamp 101 promotes fixation of the eye E to be inspected, thereby enabling imaging of a desired position on the fundus Er.

Here, the configuration of the optical system used for alignment (hereinafter, this is referred to as alignment) and adjustment of the in-focus state (hereinafter, this is referred to as focus) will be described. First, the configuration of the anterior segment observation optical system for alignment will be described.

A lens 61, an aperture 62, a prism 63, a lens 64, and a two-dimensional image sensor 65 having sensitivity in the infrared region are arranged in the optical path L5 in the reflection direction of the dichroic mirror 24. With these configurations, an anterior segment observation optical system for observing the anterior segment is formed. Here, the light incident on the prism 63 is refracted and separated in the left and right directions opposite to each other in the upper half and the lower half of the prism 63. Therefore, when the distance between the eye to be inspected E and the photographing optical unit C2 is longer than the appropriate operating distance, the observation image of the anterior eye portion is formed by the lens 61 on the side closer to the lens 61 than the prism 63, and the observation image of the observation image is formed. The upper half is shifted to the right and the lower half is shifted to the left. When the distance between the eye to be inspected E and the photographing optical unit C2 is shorter than the proper operating distance, the observed image is imaged with the observation image shifted upside down.

A light source 105 for observing the anterior segment of the eye is provided on the outside of the objective lens 10 as seen from the optical path L3. The anterior segment of the eye E to be inspected is illuminated by the anterior segment observation light source 105 that emits light in an infrared region different from the wavelength transmitted through the filter 3 that blocks visible light. The above-mentioned anterior segment observation optical system makes it possible to detect the alignment state of the eye E to be inspected with the anterior segment. The configuration and the driving unit arranged on the base C1 to drive the photographing optical unit C2 in the triaxial direction constitute an alignment means for controlling the alignment of the photographing optical system with respect to the eye to be inspected in this embodiment. The arrangement of each optical path and the arrangement of various configurations provided in the fundus camera C described above are examples, and can be replaced with various known arrangements as needed.

Next, the configuration of the focus optical system will be described.
A focus index projection unit 22 is arranged between the ring diaphragm 7 on the optical path L2 and the relay lens 8. The focus index projection unit 22 is for projecting the divided split index on the pupil Ep of the eye E to be inspected, and is removable with respect to the optical path L2. Then, the focus index projection unit 22 and the focus lens 12 move in conjunction with the directions of the optical path L2 and the optical path L3 indicated by the arrows in the figure, respectively, based on the control from the control unit 18 (drive control unit 201). It has become. At this time, the focus index projection unit 22 and the image sensor 14 are optically in a conjugated relationship. With these focus optical systems, the focus state of the fundus Er of the eye E to be inspected can be detected.

In this embodiment, the configuration constitutes a focusing means for obtaining the focus state of the imaging optical system with respect to the fundus Er of the eye E to be inspected. The focus unit (lens drive unit (not shown) that drives the focus lens 12) in the focus means is driven by the drive control unit 201. Next, the operation of these focusing means and the like will be described in detail with reference to FIGS. 3 and 4. The manual control described below can be executed by the user controlling the drive unit or the like described above according to the tilt angle or tilt direction of the joystick C3, for example, as the operation input unit 21. Further, the manual shooting can be performed by pressing the shooting button provided on the joystick C3, taking the above-mentioned joystick C3 as an example.

Next, the details of the control unit 18 in the fundus camera C used in this embodiment will be described with reference to FIG. 2 (b). FIG. 2B is a block diagram showing a schematic configuration of the control unit 18. As shown in FIG. 2B, the control unit 18 includes a drive control unit 201, a display control unit 202, an acquisition unit 210, and a processing unit 220. Further, the control unit 18 is connected to a focus lens 12, an image sensor 14, an image processing unit 17, an operation input unit 21, a focus index projection unit 22, a storage memory 41, a two-dimensional image sensor 65, a drive unit C4, and the like. To. The operation input unit 21 includes the joystick C3 described above.

The drive control unit 201 executes drive control of the focus lens 12, the image sensor 14, the image processing unit 17, the focus index projection unit 22, the two-dimensional image sensor 65, the drive unit C4, and the like. The display control unit 202 is connected to the display monitor 15 via the image processing unit 17, and an image to be displayed on the display monitor 15 or a user instruction input via, for example, an instruction screen displayed on the display monitor 15 or the like. Accept. The display monitor 15 is an arbitrary display such as an LCD display, and various information such as a GUI for operating the photographing optical unit C2 and the control unit 18, a generated image, an image subjected to arbitrary processing, and patient information. Can be displayed. The output image from the image pickup device 14 is combined with the character image prepared in advance through the image processing unit 17 and displayed on the display monitor 15.

Further, when displaying the GUI described above, the operation input unit 21 is used to operate the control unit 18 by operating the GUI or inputting information. The operation input unit 21 includes, for example, a mouse, a touch pad, a trackball, a touch panel display, a pointing device such as a stylus pen, a keyboard, and the like. When a touch panel display is used, the display monitor 15 and the operation input unit 21 can be integrally configured. In this embodiment, the photographing optical unit C2, the control unit 18, the operation input unit 21, and the display monitor 15 are separate elements, but some or all of them are integrally configured. May be good.

The acquisition unit 210 includes an image acquisition unit 211, an information acquisition unit 212, and a storage unit 213. The image acquisition unit 211 is connected to the image sensor 14 and the two-dimensional image sensor 65, and acquires an image signal output from these or an image obtained from the image signal. The information acquisition unit 212 acquires the information input via the joystick or the display monitor 15 described above, or the eye-tested information stored in the storage memory 41 in advance along with the eye-tested eye E. The storage unit 213 stores various images and information acquired by the acquisition unit 210, as well as output information such as a copying loss cause and a correction value, which will be described later, output from the processing unit 220. The processing unit 220 has an output unit 221 and a selection unit 222. The output unit 221 outputs the cause of copying loss, the correction value, and the like by using, for example, a learned model described later, and outputs these. The selection unit 222 is used by the output unit 221 from a learned model in which a plurality of types are stored in the storage unit 213 according to, for example, the cause of photo loss and shooting conditions input by the user acquired by the information acquisition unit 212. Select a trained model.

Each component of the control unit 18 other than the storage unit 213 may be composed of a software module executed by a processor such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). The processor may be, for example, a GPU (Graphical Processing Unit), an FPGA (Field-Programmable Gate Array), or the like. In addition, each component may be configured by a circuit or the like that performs a specific function such as an ASIC. The storage unit 213 may be configured by any storage medium such as an optical disk or a memory.

<Outline of anterior segment alignment>
Next, the alignment of the photographing optical unit C2 and the like with respect to the eye to be inspected E performed at the time of actual photographing will be described. FIG. 3 is an observation image obtained on the two-dimensional image sensor 65 of the anterior segment observation optical system described with reference to FIG. 2A, and is acquired by the image acquisition unit 211 and displayed by the display control unit 202 on the display monitor 15. The image to be displayed in is schematically shown. The anterior segment of the eye E to be inspected illuminated by the anterior segment observation light source 105 in FIG. 2A is divided into upper and lower parts by a prism 63 and shown in FIG. 3A on the two-dimensional imaging element 65. Observed as an image like. As shown in the figure, the portion other than the pupil portion appears white because a large amount of reflected light from the anterior segment observation light source 105 is reflected and enters. On the other hand, the pupil portion P looks black because the reflected light does not enter.

The image processing unit 17 can extract the pupil portion P from this contrast difference and can determine the pupil position. In FIG. 3A, the pupil center PO is detected from the lower pupil portion P of the upper and lower pupil portions P. The control unit 18 moves the drive unit C4 so that the pupil center PO detected by the image processing unit 17 is located at the image center O of the two-dimensional image sensor 65 shown in FIG. 3 (b). By this operation, the anterior segment alignment, which is the alignment between the anterior segment of the eye E to be inspected and the imaging optical unit C2, can be performed. The observed image of the two-dimensional image sensor 65 can be displayed on the display monitor 15 as it is via the control unit 18.

<Outline of fundus focus>
Next, FIG. 4 is an observation image obtained on the image pickup device 14 that combines the moving image observation and the still image shooting described in FIG. 2A, and is acquired by the image acquisition unit 211 and displayed by the display control unit 202. The image to be displayed on the display monitor 15 is schematically shown. In the figure, the split indexes 22a and 22b indicate the indexes projected on the pupil of the eye E to be inspected by the focus index projection unit 22 of the focus optical system.
When the photographing optical unit C2 is aligned with the anterior eye portion of the eye E to be inspected so as to be in the state of FIG. 3B, the image shown in FIG. 4A is observed on the image sensor 14. .. As described above, the focus index projection unit 22 and the focus lens 12 are provided with an optical path L2 and an optical path L3 by means of a drive unit (not shown) including a motor or the like provided corresponding to each of the focus index projection unit 22 and the focus lens 12 based on the control from the control unit 18. It is driven in conjunction with the direction. Further, the image sensor 14 is optically conjugated with the focus index projection unit 22. Therefore, by moving the focus index projection unit 22 in the optical path L2 direction, the split indexes 22a and 22b move as observation images on the image sensor 14, and the focus lens 12 moves in conjunction with the optical path L3 direction.

The image processing unit 17 obtains the focus state, the moving direction and the moving amount of the focus lens 12 from the split indexes 22a and 22b. Based on the information regarding the movement of the focus lens 12 obtained from the image processing unit 17, the drive control unit 201 changes from the state of FIG. 4 (a) to the state of FIG. 4 (b) (straight line) on the image sensor 14. The drive unit is controlled so as to. This makes it possible to focus the fundus Er of the eye E to be inspected.

As described above, in the fundus camera according to this embodiment, alignment and focusing can be performed. Further, for example, the above-mentioned alignment operation and focus operation are automatically performed, and after detecting that the alignment operation and the focus operation are completed, the light source 4 for photographing is emitted to emit light, and the image sensor 14 performs the photographing operation of the fundus of the eye. Can be executed. By automatically performing the above operations, the user can automatically take a picture of the fundus by the fundus camera C simply by pressing the shooting start button of the operation input unit 21 (joystick C3). Here, the criterion for determining the end of the alignment operation and the focus operation is set based on the alignment position and the focus accuracy using a plurality of models of the eye to be inspected. That is, the operation is set to end when the index or the like deviates within a predetermined allowable range from the state in which the optimum alignment position and focus accuracy are secured. The captured still image of the fundus is combined with the character image by the image processing unit 17 and displayed on the display monitor 15.

<Fundus photography sequence>
Next, a basic imaging sequence for performing automatic imaging of the eye to be inspected will be described with reference to the flowchart of FIG. The user gives an instruction to the fundus camera C to roughly align the subject and the device and to start the photographing operation by using various input means configured in the operation input unit 21. At that time, the subject is placed on the jaw support (not shown), and the position of the eye to be inspected in the Y-axis direction is adjusted by the jaw support drive mechanism (not shown) so as to have a predetermined height. The user drives the photographing optical unit C2 by the joystick C3 to a position where the pupil of the eye to be inspected E displayed on the display monitor 15 is displayed, and presses the photographing start button after the driving is completed. When the control unit 18 detects that the shooting start button is pressed, the control unit 18 shifts the flow to step S501.

In step S501, the drive control unit 201 executes the alignment operation described with reference to FIG. 3, and aligns the anterior eye portion of the eye E to be inspected with the photographing optical unit C2. Specifically, the control unit 18 determines the alignment state in the front-rear direction (Z direction) of the photographing optical unit C2 from the front eye portion observation image divided by the prism 63. When the upper half and the lower half of the observed image are displaced from each other as shown in FIG. 3A, it is determined that alignment is necessary. In this case, the drive control unit 201 controls the drive unit so that the photographing optical unit C2 moves in a direction in which the observation image does not shift as shown in FIG. 3 (b). Further, in the vertical and horizontal alignment, the drive control unit 201 detects the pupil center P0 from the front eye observation image, and the drive control unit 201 detects the pupil center P0 and positions it at the image center O of the two-dimensional image pickup element 65. To control. When the control unit 18 determines that the vertical and center positions of the anterior segment observation image coincide with the center of the image, or a deviation state within a predetermined range is obtained from the matching state, and the alignment is completed, the control unit 18 performs a flow. The process proceeds to step S502. Since the two-dimensional image sensor 65 and the image sensor 14 can perform image analysis independently, the automatic alignment operation described here is continued even after step S502.

In step S502, the focus operation described with reference to FIG. 4 is executed to perform focusing. After the alignment is completed in step S501, the image processing unit 17 starts analyzing the output signal from the image sensor 14. Since the focusing is not optimal when only the normal alignment is completed, the image sensor 14 captures a fundus image in which the split indexes 22a and 22b do not match as shown in FIG. 4A. Therefore, as shown in FIG. 4B, the drive control unit 201 drives and controls the focus lens 12 so that the split indexes 22a and 22b imaged by the image sensor 14 are aligned. When the control unit 18 determines that the alignment and focusing are in the finished state, the control unit 18 shifts the flow to S503.

In step S503, the drive control unit 201 retracts the focus index projection unit 22 from the optical path L2 by a control unit motor (not shown). After the evacuation is completed, the drive control unit 201 issues the photographing light source 4. As a result, the fundus Er of the eye E to be inspected is photographed. When the control unit 18 determines that the shooting is completed, the control unit 18 shifts the flow to S504. In step S504, the image of the fundus Er taken by the display control unit 202 is displayed on the display monitor 15, and the control unit 18 shifts the flow to step S505.

In step S505, it is possible to determine the copying loss of the captured image by the processing unit 220 described later. However, in this embodiment, the user who observes the captured image displayed on the display monitor 15 determines whether or not the image is lost. The user further inputs the determination result to the control unit 18 via the operation input unit 21 or the like, and the determination result is acquired by the acquisition unit 210 (information acquisition unit 212). Specifically, the captured image and a plurality of causes of photo loss are displayed on the display monitor 15, and the user performs an operation such as moving the cursor or the like to an item that seems to be the cause of the photo loss and clicking the mouse. The judgment result can be input. When the control unit 18 detects an input when it is determined that there is no copying loss by the user, the control unit 18 shifts the flow to step S506. In step S506, the control unit 18 saves the captured image in the storage memory 41 and ends the photographing process.

Further, when the control unit 18 detects the input when it is determined that the photo loss is caused by the user, the control unit 18 shifts the flow to step S507. As described above, in the present embodiment, the user also inputs the possible cause of the copying loss when determining the copying loss via the operation input unit 21. Here, it is assumed that, for example, a plurality of causes of image loss are displayed on the display monitor 15, and the user selects the cause of image loss by operating a cursor or the like displayed on the display monitor 15. However, the method of selecting the cause of photo loss is not limited to the one illustrated here, and if the user can select an arbitrary cause of photo loss from among a plurality of causes of photo loss, other known methods may be applied. it can.

The user changes the shooting conditions according to the cause of the shooting loss and reshoots. In this embodiment, at that time, whether to manually change the shooting conditions using the operation input unit 21 or the like to perform re-shooting, or to automatically obtain a correction value and perform re-shooting using the correction value. Can be selected by the user. In step S507, when the user inputs a manual operation selection, the control unit 18 shifts the flow to step S508 so that the setting of the shooting conditions at the time of re-shooting can be manually executed from automatic. In this case, the user drives the photographing optical unit C2 by using various input means including, for example, the operation input unit 21 and the joystick C3, and presses the photographing start button after the driving is completed to start re-imaging. When the control unit 18 detects that the shooting start button has been pressed by the user, the control unit 18 shifts the flow to step S509. In step S509, the drive control unit 201 causes the photographing light source 4 to emit light to photograph the fundus Er of the eye E to be inspected. Then, the control unit 18 shifts the flow to step S504 according to the completion of shooting.

In step S507, when the selection of the automatic operation is input, the control unit 18 shifts the flow to step S510. In step S510, the control unit 18 outputs the correction value using the trained model with the image determined to be a copy loss (hereinafter, referred to as a copy loss image) as input data. More specifically, the selection unit 222 selects a trained model to be used from a plurality of trained models stored in the storage unit 213 according to the photo loss cause input by the user. A copy loss image is input to the trained model. The output unit 221 outputs a correction value from the captured image using the selected trained model. After the correction value is obtained, the control unit 18 shifts the flow to S511.

For example, when the user determines that the cause of the image loss is so-called flare, it is considered that the WD (operating distance: the distance between the imaging optical unit C2 and the eye E to be inspected) is too close or too far as the imaging condition. Be done. Therefore, if the same eye to be inspected is re-photographed under the same alignment conditions, the obtained image will have the same flare. Therefore, when performing automatic alignment, it is necessary to correct the alignment state. As shown in the description of FIG. 3, it is a condition for determining the end of alignment that the front eye images divided vertically by the prism 63 match. Therefore, when the cause of the copying loss is flare, the output unit 221 outputs a correction value for this determination condition, so that automatic alignment that does not cause the copying loss can be enabled.

Further, for example, when the user determines that the cause of the image loss is so-called out-of-focus, it is conceivable that the focus lens 12 is too close to or too far from the fundus Er. Therefore, if the same eye to be inspected is re-photographed under the same focus condition, the obtained image will be the same out-of-focus image. Therefore, it is necessary to correct the focus state when performing automatic focusing. As shown in the explanation of FIG. 4, it is a condition for determining the end of the focus operation that the split indexes 22a and 22b projected on the pupil of the eye to be inspected by the focus index projection unit 22 are aligned. Therefore, when the cause of the shooting loss is out of focus, the output unit 221 outputs a correction value for this determination condition, so that automatic focusing that does not cause the shooting loss can be enabled.

In step S511, the control unit 18 automatically drives the photographing optical unit C2 or the focus lens 12 according to the output correction value. When the control unit 18 determines that the drive is completed, the control unit 18 causes the light source 4 for photographing to emit light to photograph the fundus Er of the eye E to be inspected. Then, the flow is shifted to step S504. After that, the user determines in step S505 whether or not the image obtained by re-shooting has a copying loss.

As described above, in the present embodiment, the user determines whether or not the captured image has a copying loss and further, the cause of the copying loss. Then, using the trained model using the captured image as input data, the correction value of the photographing condition in the photographing optical unit C2 corresponding to the cause of the photographed loss is output. As a result, if the user can determine the cause of the photo loss to a certain extent, it is possible to derive an appropriate correction value according to the photo loss situation even if the user is not an expert.

In this embodiment, the copying loss determination including the cause of the copying loss in step S505 is determined by input by the user. However, the apparatus may automatically perform the copying loss determination by evaluating the image quality such as the contrast of the obtained image. Further, it may be allowed to select in advance whether or not to perform the manual selection in step S507.
Further, the correction value output from the trained model in step S510 may be stored in the storage memory 41 for each subject and used at the next shooting of the subject.

<Generation of trained model>
Next, the generation of the trained model according to this embodiment will be described. As described above, the trained model in this embodiment is a model in which a learning model according to an arbitrary machine learning algorithm such as deep learning is trained in advance using appropriate learning data. Specific algorithms for machine learning include the nearest neighbor method, the naive Bayes method, a decision tree, and a support vector machine. In addition, deep learning (deep learning) in which features and coupling weighting coefficients for learning are generated by themselves using a neural network can also be mentioned. As appropriate, any of the above algorithms that can be used can be applied to the learning model according to the embodiment. The trained model is a model in which training (learning) is performed in advance using appropriate teacher data (learning data) for a machine learning model according to an arbitrary machine learning algorithm. However, the trained model does not require further learning, and additional learning can be performed. The teacher data is composed of one or more pairs of input data and output data. The format and combination of the input data and output data of the pair group that composes the teacher data may be one of which is an image and the other of which is a numerical value, or one of which is composed of multiple image groups and the other of which is a character string. May be an image or the like, which is suitable for a desired configuration.

Hereinafter, the learning data used for training when obtaining the trained model according to this embodiment will be described. The training data consists of a pair of input data corresponding to the data actually input to the trained model and output data (correct answer data or teacher data) corresponding to the data generated and output by the trained model. The training data may be referred to as teacher data, and the teacher data may be referred to as being composed of a pair of input data and output data.
In this embodiment, the captured image is used as the input data of the learning data. Then, the correction value is used as the output data of the learning data.

For example, when the cause of photo loss is so-called flare, the reason why flare occurs may be that the WD is too close or too far. More specifically, when the WD is approached, the cornea enters the region where the illumination light flux overlaps with the photographing light beam, a part of the illumination light is reflected by the cornea, and this reflected light enters the photographing light beam to generate corneal flare. This corneal flare turns red because the long wavelength side of the illumination luminous flux overlaps with the imaging luminous flux. On the other hand, when the WD is far away, the rear surface of the crystalline lens enters the region where the illumination light flux overlaps with the shooting light beam, and a part of the illumination light is reflected by the rear surface of the crystal body, and this reflected light enters the shooting light beam to cause lens flare. .. This crystalline lens flare becomes blue because the short wavelength side of the illumination luminous flux overlaps with the photographing luminous flux. Therefore, based on the color of the flare, it is possible to determine whether the WD is too close to cause a photo loss or the WD is too far to cause a photo loss.

Therefore, the WD is photographed by moving the WD closer to or further away from an appropriate position (the position where the anterior eye images divided vertically by the prism 63 coincide with each other) by a certain amount, and under these shooting conditions, flare, which is a photo loss, is taken. Acquire a fundus image with. The image of the fundus with flare captured by this is used as the input data of the training data, and the difference between the positions where the photographing optical unit C2 is moved in the optical path direction by a certain amount from the appropriate position is used as the correction value, and the output data of the training data is used. can do. Further, by further changing the position of the photographing optical unit C2 by a certain amount to obtain a plurality of missed images, a pair of input data of a plurality of learning data and output data of the learning data (hereinafter, this is referred to as a learning data pair). (Note) can be created. Then, by training using the created plurality of training data pairs, it is possible to generate a trained model for the case where the cause of photo loss is flare.

For example, when the cause of the image loss is so-called out-of-focus, it is conceivable that the focus lens 12 is too close to or too far from the fundus Er as the reason for the out-of-focus. However, from the fundus image, it is not easy to determine whether the focus lens 12 is out of focus because it is close to the fundus Er and is out of focus because it is far from it. Here, as an example in which the fundus Er is distant, the case where the axial length of the eye E to be inspected is long is illustrated. In this case, the axial length can be estimated by knowing the diopter D of the eye to be inspected E from the position of the focus lens 12.

Therefore, the focus lens 12 takes a picture by moving the focus lens 12 closer to or further away from an appropriate position by a certain amount. As a result, it is possible to obtain a fundus image having out-of-focus, which is a loss of image, and an amount of deviation from an appropriate position on the optical path of the focus lens 12 at that time. In addition, there is a difference in the mode of blurring whether the fundus is out of focus due to being close to the fundus Er or is out of focus due to being far away. More specifically, when the focus lens 12 is close to the fundus Er, the obtained fundus image becomes darker at the focal position, but conversely becomes slightly brighter at the periphery, and the visible region expands by that amount. Further, when the focus lens 12 approaches the fundus Er, it becomes darker from the position where the focal point was, and finally, only the periphery remains as a visible region. Further, when the focus lens 12 is far from the fundus Er, the obtained fundus image becomes dark as a whole. However, since the periphery is also darkened, the visible region does not expand. Further, when the focus lens 12 moves away from the fundus Er, it becomes dark from the surroundings, and finally, only the focal position remains as a visible region. Therefore, from the distribution of light and darkness of the fundus image Er, it is possible to know whether the focus lens 12 is out of focus because it is close to the fundus Er and is out of focus because it is far from it.

From the above, it is possible to use the fundus image as the input data of the learning data and the correction value which is the difference between the positions where the focus lens 12 is moved from the appropriate position by a certain amount as the output data of the learning data. Further, by changing the position of the focus lens 12 on the optical path by a certain amount, a plurality of learning data pairs can be created. Then, by training using the created plurality of training data pairs, it is possible to generate a trained model for the case where the cause of copying loss is out of focus. In the case of a fundus camera, the diopter D of the eye E to be inspected can be known from the position of the focus lens 12 at the time of the first fundus photography. By referring to this diopter D, it is possible to confirm whether the fundus is out of focus due to being close to the fundus Er or being out of focus due to being far away.

In the above-mentioned example, the correction value is calculated only when the photographing optical unit C2 or the focus lens 12 is moved by a certain amount. However, an image taken without moving from an appropriate position and its correction value (difference = 0) can also be used as a learning data pair. By training the trained model using the training data pair, it is possible to improve the training accuracy. In addition, learning data pairs for diseased eyes such as fundus indistinct due to cataract cannot be created by the above method. Therefore, in order to generate a trained model for corresponding to an image having such image loss, a correction value calculated from an eye image determined to be image loss by photographing a diseased eye and a re-imaging position at that time is used. Is used as a training data pair.

In the above-described embodiment, the user inputs the cause of the copy loss, and only the copy loss image is used as the input data. However, the number of input data may be one or more and can be appropriately selected. For example, further data related to the eye to be inspected such as an image of the anterior segment of the eye, an intraocular pressure value, and an axial length can be added as input data of training data. However, the number and type of input data (input data of training data) at the time of creating the trained model and the number and type of input data at the time of using the trained model must be the same. Further, for example, in the case of the above-mentioned diseased eye, it is assumed that an appropriate image cannot be obtained by the ophthalmic apparatus used. In the case of such an image, as the output of the trained model, for example, an output such as no correction or no correction may be set.

As the trained model, the one trained in advance using the fundus camera C may be used as it is, or the trained model may be updated (additional learning) while using it. Specifically, when the captured image is determined to be a copy loss and the position is manually adjusted and re-photographed, the captured image determined to be a copy loss is used as the input data of the learning data. Further, for example, the correction value calculated from the position on the optical path of the focus lens 12 at the time of shooting loss and the position on the optical path at the time of re-shooting is used as the output data of the learning data. Then, additional learning is performed using the learning data pair. Further, additional learning may be performed for each subject, in which case a trained model for the subject is generated.

Here, a CNN (Convolutional Neural Network), which is a machine learning model in this embodiment, will be described. FIG. 6 is a schematic view showing the structure of CNN. CNN is composed of a plurality of layers that process and output input data. Types of the plurality of groups include Convolution layer 601, Activation layer 602, Pooling layer 603, Fully connected layer 604, and Output layer 605.

The Convolution layer 601 is a layer that performs convolution processing on the input group according to parameters such as the kernel size of the set filter, the number of filters, the stride value, and the dilation value. The kernel size of the filter may be changed according to the number of dimensions of the input image. The Activation layer 602 determines the activation of the total sum of the input signals, and is composed of a step function, a sigmoid function, a ReLU (Rectifier Line Unit), and the like. The Pooling layer 603 is a layer that performs a process such as Max Pooling process in which the number of output value groups is reduced from the number of input value groups by thinning or synthesizing the input value groups. The fully connected layer 604 is a layer that combines the image data from which the feature portion has been extracted through the processing so far into one and outputs the value converted by the activation function.

When generating the trained model, training data consisting of an image as input data and a correction value as output data is used. Then, when an image is actually input to the trained model, the output layer 605 outputs the probability that the input image will be the value of each of the plurality of correction values used in the previous output data. More specifically, for each of the plurality of correction values used as output data in the training data, the possibility of becoming the correction value corresponding to the image input to the trained model is output as, for example, the probability of each correction value. .. The output unit 221 determines the correction value to be finally output with reference to these output probabilities. For example, the output correction value is determined as the correction value to be finally output based on the relative magnitude of the probabilities of the individual correction values, and is output from the output unit 221. Will be done. Alternatively, for example, each correction value output from the output layer 605 is weighted according to its probability, and the correction value obtained by performing an calculation such as addition averaging is output from the output unit 221 as the final correction value. You can also do it. In the case of this embodiment, by executing such a process in the output unit 221, it is possible to obtain the output data as a numerical value even when the image is used as the input data. The correction value finally output from the output layer 605 may be configured such that the fully coupled layer 604 performs the optimum value calculation for the correction value output.

When data is input to the trained model, the data according to the design of the trained model is output. For example, data that is likely to correspond to the input data is output according to the tendency trained using the training data. In this embodiment, when an image at the time of copying loss is input to the trained model trained by the training data described above, a correction value having a high possibility of reducing copying loss is output.

The trained model used in the output unit 221 in this embodiment may be used in the processing unit 220 or the control unit 18. The trained model may be composed of, for example, a CPU, a software module executed by a processor such as an MPU, GPU, or FPGA, or a circuit or the like that performs a specific function such as an ASIC. Further, these trained models may be used in a device of another server connected to the processing unit 220 or the control unit 18. In this case, the processing unit 220 or the control unit 18 can use the trained model by connecting to a server or the like provided with the trained model via an arbitrary network such as the Internet. Here, the server provided with the trained model may be, for example, a cloud server, a fog server, an edge server, or the like.

In this embodiment, the processing unit 220 has an output unit 221 that uses a trained model obtained in advance. However, the processing unit 220 may be provided with a learning unit for generating a trained model. The learning unit can generate a trained model to be used in the processing unit by inputting the above-mentioned input data and output data. Further, in this case, the learning unit may include an error detecting unit and an updating unit. The error detection unit obtains an error between the output data output from the output layer of the neural network and the teacher data according to the input data input to the input layer. The error detection unit may use the loss function to calculate the error between the output data from the neural network and the teacher data. Based on the error obtained by the error detection unit, the update unit updates the coupling weighting coefficient and the like between the nodes of the neural network so that the error becomes small. This updating unit updates the coupling weighting coefficient and the like by using, for example, the backpropagation method. The error backpropagation method is a method of adjusting the coupling weighting coefficient between the nodes of each neural network so that the above error becomes small.

Further, in this embodiment, the control unit 18 is described as the control unit 18 included in the fundus camera C. However, these configurations and various configurations included in the control unit 18 may be configured by two or more devices capable of communicating with each other, or may be configured by a single device. Further, each component of the control unit 18 may be composed of a software module executed by a processor such as a CPU, MPU, GPU, or FPGA. In addition, each component may be configured by a circuit or the like that performs a specific function such as an ASIC. Further, it may be configured by a combination of any other hardware and any software. Since the GPU can perform efficient calculations by processing more data in parallel, it is effective to use this when learning is performed a plurality of times using learning data such as deep learning. ..

As described above, the ophthalmic apparatus according to the first embodiment includes a photographing optical unit C2, a driving means (driving unit C4), a control means (control unit 18), an input means (operation input unit 21), and the like. It includes an output means (output unit 221) in the control unit 18. The photographing optical unit C2 has the above-mentioned various optical members as a configuration for capturing a fundus image of the eye E to be inspected. The drive control unit 201 controls a second drive unit that drives various drive-controlled optical members such as the focus lens 12 included in the photographing optical unit C2. The drive unit C4 (first drive unit) is composed of, for example, a pulse motor or the like, and drives the position of the photographing optical unit C2 with respect to the eye E to be inspected in the three axes of XYZ. The input means can be composed of, for example, a GUI displayed on the display monitor 15, and accepts input by the user regarding the presence or absence of photo loss and the cause of photo loss in the captured image. The control unit 18 is connected to an optical member such as a focus lens 12 and a drive unit C4 included in the photographing optical unit C2 to control them. The output unit 221 learns corresponding to the input information on the photo loss from among a plurality of trained models based on the cause of the input photo loss (or information on the photo loss including the presence or absence of the photo loss). Select a completed model. Then, the output unit 221 includes a focus lens 12 and the like and a drive unit C4 for re-imaging the fundus image with reduced photo loss using the learned model selected from the photographed fundus image of the eye E to be inspected. At least one of the control values (correction values) can be output. The step performed by the output unit 221 can also constitute, for example, one step of the control method of the ophthalmic apparatus (a step of inputting the cause of copying loss and a step of outputting the control value) as a step of outputting the control value. ..

In the above-mentioned ophthalmic apparatus, as shown in the flowchart of FIG. 5, at least one of the control of the focus lens 12 and the like and the drive unit C4 at the time of re-imaging is automatically controlled or is controlled by the user. You can choose to execute according to the instructions. This selection can be executed via an operation input unit 21 that constitutes the selection means, for example, a GUI displayed on the display monitor 15. Further, when re-shooting is selected according to the instruction from the user, a new correction value between the focus lens 12 or the like and the drive unit C4 input via the GUI or the like can be obtained. For the trained model obtained by using the training data in which the image before re-shooting and the cause of the copying loss determined at that time are used as input data and the obtained correction value is used as output data. Additional learning may be performed. This additional learning can be executed by, for example, further providing a configuration (GPU or the like) as a means for generating a learning model in the output unit 221.

Further, as a cause of copying loss, out-of-focus was described as one aspect of the above-described embodiment. When it is determined that the cause of the image loss is out-of-focus, the out-of-focus is caused by, for example, the arrangement of the focus lens 12 exemplified as the focusing member on the optical path (on the optical path L4). For this reason, the arrangement of the focus lens 12 on the optical path L4 is adjusted at the time of re-imaging. Specifically, the output unit 221 sets the focus lens 12 as a correction value (control value) from the position on the optical path L4 at the time of the first shooting (first position) to the position at the time of re-shooting (second position). Outputs the amount of movement to move. As described above, the direction on the optical path L4 for moving the focus lens 12 can be determined depending on the mode of out-of-focus. At that time, by referring to the shape information of the eye E to be inspected and the past diagnostic information, for example, the diopter, it can be determined whether the focus lens 12 is out of focus because it is too close to the fundus Er and is out of focus because it is too far away. You can also check.

When the cause of the image loss is determined to be flare, the flare is caused by, for example, WD, which is the distance between the photographing optical unit C2 and the eye E to be inspected. Therefore, at the time of re-imaging, the distance between the photographing optical unit C2 and the eye E to be examined on the optical path L3 is adjusted. Specifically, the output unit 221 sets the correction value (control value) from the position on the optical path L3 at the time of the first shooting (first position) to the position at the time of re-shooting (second position). The amount of movement that moves C2 is output. Further, as described above, whether the position of the focus lens 12 is too close to or too far from the eye E to be inspected can be determined by whether the flare is reddish (included) or bluish (included). Therefore, the output unit 221 can determine the positive or negative of the movement amount when moving from the first position to the second position based on this color of the input fundus image.

In the above-described embodiment, flare and out-of-focus are described as examples of the causes of the image loss. However, the causes of the image loss include, for example, blinking of the eye to be inspected at the time of photographing, or a disease such as cataract. , Underexposure, etc. are also included. When the photographed image is lost due to blinking, it is appropriate as an imaging condition, but it is preferable to photograph under the same conditions because there is only a problem in the timing of photographing. Therefore, when it is determined that the image loss is caused by blinking, or when this determination result is output, the control value for re-imaging is zero (the correction value for the control state when the eye E to be inspected is photographed is zero). It should be output. In addition, when the captured image is lost due to a disease such as cataract, it may be difficult to capture an appropriate image even if the imaging conditions are changed due to cloudiness of the eye to be inspected or the like. As described above, the cause of the image loss is not due to the control state of the ophthalmic apparatus for the eye to be inspected, but may be due to the state of the eye to be inspected such as blinking or cataract. In this way, when the image loss is caused by the condition of the eye to be inspected, the trained model does not output the control value, but outputs the information that an appropriate control value does not exist, or shoots under the original shooting conditions. Is appropriate, and the control value of zero should be output. Further, when the state of the eye to be inspected is input as the cause of image loss, the control unit 18 may directly output zero as a control value without using the trained model. Further, in the case of insufficient exposure or the like, the trained model can output control values related to the exposure time and the amount of light of the photographing light source 4.

In order to deal with the above-mentioned causes of copying loss, it is advisable to generate a trained model corresponding to each of these causes of copying loss in advance. When generating this trained model, it is advisable to use an image having a copying loss corresponding to the cause of the copying loss as input data, and add a result such as shooting under the same conditions or not outputting a control value to the output data. The control unit 18 can re-photograph the photographed eye E using the control value output by the output unit 221. Further, at the time of re-shooting, the user may manually change the shooting conditions. From this, when the display control unit 202 (display control means) controls the display form so that the control value output by the output unit 221 is displayed on the display monitor 15 (display means) in correspondence with the cause of the copying loss. Good. Further, for example, when the subject's eye E to be examined is photographed at a later date, it is assumed that it is preferable to use this control value again. Therefore, it is desirable that the output control value is stored or stored in a storage means for storing the output control value, such as a storage unit 213 or an external storage device arranged in the control unit 18. Then, when the fundus camera C re-photographs the eye E to be inspected, the control unit 18 controls the optical member and the drive unit C4 included in the photographing optical unit C2 by using the stored control values. ..

With the above configuration, re-shooting can be performed under shooting conditions according to the situation of shooting loss. As a result, even if the user is immature and the cause of the image loss can be determined to some extent, but appropriate re-imaging conditions cannot be set, automatic re-imaging is performed to obtain a fundus image suitable for diagnosis. be able to. It should be noted that the above-mentioned causes of image loss are examples, and for other known causes of image loss such as underexposure and their correction values, images having image loss are similarly generated and used as input data. Therefore, a trained model corresponding to the cause of this copying loss can be obtained.

[Modification 1]
In the first embodiment described above, the user selects and inputs the cause of the copying loss. Then, the case where the selection unit 222 selects the trained model according to the input cause of the photo loss in the processing unit 220 and inputs the photo loss image to the trained model has been described. However, for example, in mass examinations, the skill level of users is low, and it is possible that the cause of photo loss cannot be determined. The purpose of this modification is to deal with such a case. In this modified example, the configuration of the fundus camera C is the same as the configuration described in the first embodiment except that the selection unit 222 is removed in the processing unit 220. Omit. Hereinafter, the processing executed by the processing unit 220 and the trained model used at that time will be described.

The trained model used in this embodiment is a pair in which the captured fundus image is used as input data, and whether or not the fundus image has photo loss and, if so, the cause of the photo loss is used as output data. It is trained by learning data consisting of. Further, the processing unit described in the first embodiment does not have the selection unit 222 in this modification, but has only the output unit 221. At the time of actual fundus photography, in this modified example, whether or not the photographed fundus image is input to the trained model and has a photo loss from the trained model, and if there is a photo loss, the photo loss. The cause is output. As a result, even if the user is inexperienced in capturing the fundus image, it is possible to know whether or not there is a copying loss in the captured fundus image and the cause thereof.

In this modified example, in a mass examination or the like, for example, in the flow of the photographing process shown in FIG. 5, the processing unit 220 can execute the determination and input of the copying loss performed by the user in step S505. At the time of taking an actual fundus image, for example, the fundus image displayed in step S504 shown in FIG. 5 can be input to the trained model in the processing unit 220. The output unit 221 outputs information on the presence or absence of image loss and the cause of the image loss of the photographed fundus image from this trained model. In this case, the output layer 605 (see FIG. 6) described above outputs a value such as a ratio that causes image loss corresponding to the image, and the output unit 221 determines using the value output from the output layer 605. Output the cause of the lost image. Further, in the present embodiment, the trained model can output that the captured fundus image does not have a copying loss. This eliminates the need for an inexperienced user to judge whether or not there is a photo loss by himself, and allows the ophthalmologic device to know whether or not an image suitable for diagnosis is obtained, without unnecessary re-imaging. Images can be saved. The determination of copying loss by the processing unit 220 described above may be executed after the determination by the user in step S505 instead of step S505. As a result, for example, even if the user determines that there is a copying loss, if the processing unit 220 determines that there is no copying loss, unnecessary re-shooting can be avoided. Further, for example, even if the user determines that there is no copying loss, if the processing unit 220 determines that there is a copying loss, re-shooting can be efficiently executed. At this time, when the judgment result of the user and the judgment result of the processing unit 220 are different, the judgment result of the processing unit 220 is displayed, so that the shooting process can be efficiently advanced, for example. Further, when it is not possible to determine whether the fundus image has a photo loss, the user may determine in step S505 whether or not the fundus image has the above-mentioned photo loss by assuming that the image has a photo loss. Alternatively, the flow can be shifted to step S510 to output a determination (correction value = 0) that there is no correction. In these cases, by further shifting the flow to step S506, even if the user is uncertain about the presence or absence of copying loss, the presence or absence of the copying loss can be accurately determined, which is unnecessary. It is possible to avoid re-imaging and reduce the burden on the subject.

More specifically, the output data in the training data includes information such as the cause of the copying loss and the presence / absence of the copying loss as information on a plurality of copying losses. The output data includes a value such as the ratio at which the fundus image used as the input data will correspond to any of the information regarding the copying loss, or the probability that the image is allocated to any of them. Therefore, from the trained model, values such as a ratio corresponding to the input image are output from the input image, for example, for each of the causes of copying loss. The output unit 221 refers to these output values to determine the cause of the image loss that is finally output. For example, as the output cause of copying loss, the cause of copying loss having the largest proportion is determined from the relative magnitude of the proportions that are individual values, and is output from the output unit 221. For example, when the subsequent processing in the first embodiment described above is to be performed and the information that the copying is lost is output, it is conceivable that the control unit 18 shifts the flow to step S507. As a result, the processes of steps S508 to S511 described above can be executed.

The ophthalmic apparatus according to this modification includes a photographing optical unit C2, a driving unit C4, a control unit 18, and an output means (output unit 221) in the control unit 18. The photographing optical unit C2 has the above-mentioned various optical members as a configuration for capturing a fundus image of the eye E to be inspected. Among these various optical members, for example, the focus lens 12, which is driven and controlled at the time of photographing, is driven by a second driving means including a drive control unit 201 and a motor (not shown) for driving the focus lens 12. .. As the first driving means, the driving unit C4 is composed of, for example, a pulse motor or the like, and drives the position of the photographing optical unit C2 with respect to the eye E to be inspected in the three axial directions of XYZ. The control unit 18 is connected to the photographing optical unit C2 and the drive unit C4 to control them. The output unit 221 can output the cause of the image loss from the photographed fundus image of the eye E to be examined by using the trained model. Further, the output unit 221 can determine that the fundus image does not have a copying loss by the trained model, and can output the determination result. That is, the output unit 221 can output information on the cause of the copying loss (cause of the copying loss, presence / absence of the copying loss, etc.) from the image of the eye E to be inspected taken by the trained model. It should be noted that the output unit 221 may further have a plurality of further trained models capable of outputting control values that may reduce each cause of copying loss in response to each cause of copying loss. That is, the output unit 221 outputs at least one control value (correction value) of the optical member and the drive unit C4 of the photographing optical unit C2 for reducing the photographing loss by using a plurality of trained models. it can. The display control unit 202 can display the cause of image loss output by the output unit 221 on the display monitor 15. At that time, the output control value can also be displayed on the display monitor 15 in correspondence with the cause of the copying loss. The step performed by the output unit 221 can also constitute one step of the control method of the ophthalmic apparatus, for example, as a step of outputting a control value.

With the above configuration, it is possible to know whether or not the photographed fundus image has a copying loss, and if there is a copying loss, the cause of the copying loss. As a result, even if the user is immature, it is possible to appropriately discriminate the captured fundus image and accurately know the cause in the case of loss of image. Further, by combining with the configuration of the first embodiment, it is possible to perform re-imaging under the imaging conditions according to the situation of image loss regardless of the user's proficiency in observing the fundus image. As a result, even if the user is immature, automatic re-imaging can be performed under appropriate imaging conditions, and a fundus image suitable for diagnosis can be obtained. The trained model described above can be obtained by using training data consisting of a pair in which the image of the eye E to be inspected is used as input data and information on the cause of copying loss corresponding to the image is used as output data. Further, in this trained model, a value corresponding to a plurality of information regarding copying loss used as output data when the input image is generated from the trained model is output from the output layer 605. The output unit 221 outputs information on the image loss corresponding to the input image, more specifically, information on the cause of the image loss, based on the value corresponding to any of the information on the plurality of image loss. It should be noted that the eye to be inspected used for generating the learning data may be the same as or different from the eye to be inspected in which an image actually taken and considered to have photo loss was obtained. By using images related to different eyes to be inspected and information related to photo loss as training data, it is possible to obtain a trained model with high accuracy by using more training data. In addition, the training data for obtaining a trained model that can output control values also includes an image that is actually photographed and is considered to have photo loss, and a control value of an ophthalmic apparatus used to obtain the image. Other than training data can be used. That is, from the image having the image loss obtained by a different ophthalmic device instead of the ophthalmic device actually used for photographing, and the control value of the driving means of the ophthalmic device that reduces the image loss of the image in the ophthalmic device. Data can also be used in the learning.

[Modification 2]
In Example 1 described above, a case where a fundus camera is used as an example of a device for photographing the fundus of the eye has been described. However, the configuration for photographing the fundus is not limited to the illustrated fundus camera, and a so-called color scanning laser ophthalmoscope (hereinafter referred to as color SLO) capable of color photographing the fundus may be used. The configuration of the optical system of the color SLO used in this modification will be described below with reference to FIG. 7, which schematically shows the optical configuration. The other configurations attached to the optical system such as the display monitor, the control unit, the operation input unit, the image processing unit, and the storage memory are the same as the configurations arranged along with the fundus camera described in the first embodiment. Therefore, the explanation here is omitted.

The color SLO 701 shown as this modification includes an illumination optical system 710 and a light receiving optical system 720. The illumination optical system 710 is composed of an optical path L1 and an optical member arranged in the optical path L3. Specifically, the illumination optical system 710 includes a lens 712, a perforated mirror 713, a focus lens 714, a lens 715, a scanning unit 716, and an objective lens 717, which are arranged in order from the laser light source 711 toward the eye E to be inspected. Have. The laser light source 711 is capable of emitting light in four wavelength regions of blue, green, red, and infrared (hereinafter referred to as illumination light), and is capable of emitting light from the fundus Er of blue, green, and red during color photography. The reflected light of is used. Further, these illumination lights can also be used for, for example, visible fluorescence photography and infrared fluorescence photography. The focus lens 714 can be driven by the drive system 714a along the optical path L1 in the direction of the arrow in the figure, and the emitted light can be focused at an arbitrary position on the fundus Er.

The illumination light emitted from the laser light source 711 passes through the opening 713a of the perforated mirror 713, reaches the scanning unit 716 via the focus lens 714 and the lens 715, and is further irradiated to the fundus Er through the objective lens 717. The illumination light reflected by the fundus Er follows this path in the reverse direction, and is guided to the optical path L2 of the light receiving optical system 720 by the perforated mirror 713. The scanning unit 716 can be composed of, for example, a pair of galvanometer mirrors 716a and 716b. At this time, one mirror scans the illumination light emitted from the laser light source 711 in the X direction, for example, and the other mirror scans the illumination light in the Y direction, so that the illumination light rasterizes on the fundus Er. Can be scanned. The laser light source 711, the drive system 714a, and the scanning unit 716 are controlled by the control unit 18 (see FIGS. 1 and 2).

The light receiving optical system 720 has a lens 721, a pinhole plate 722, and an optical separation unit 730, which are arranged in order from the perforated mirror 713 in the optical path L2. The light separation unit 730 separates the reflected light from the fundus Er into the light in the blue, green, and red (and infrared) wavelength bands described above. Therefore, the light separation unit 730 has dichroic mirrors 731 and 732 that can separate the illumination light according to the wavelength. Light receiving elements 724,726,728 corresponding to light in each wavelength band are arranged in the transmission direction and the reflection direction of these dichroic mirrors 731 and 732. With these configurations, the reflected light of blue, green, and red light from the fundus Er can be separated and received, and a color image of the fundus Er can be obtained by synthesizing the images obtained from each of these light receiving elements. Be done.

Between the dichroic mirror and the light receiving element, there are a lens 723,725,727 for condensing light in each wavelength range on the light receiving element, and a filter 733 for removing the light reflected from the fundus during fluorescence imaging. 734,735 are arranged. Further, the light receiving optical system 720 has a pinhole plate 722 and a filter insertion / removal portion 740. The pinhole plate 722 is arranged in the inland conjugate with the fundus Er and is used as a confocal diaphragm. Further, the light receiving optical system 720 described above shares each member arranged from the objective lens 717 to the perforated mirror 713 with the illumination optical system 710. The fundus image obtained by the ophthalmologic apparatus using the color SLO701 described above is also affected by the processing by the control unit 18 described in the first embodiment, regardless of the user's proficiency in observing the fundus image. Re-shooting can be performed under the shooting conditions according to the shooting conditions. As a result, even if the user is immature, automatic re-imaging can be performed under appropriate imaging conditions, and a fundus image suitable for diagnosis can be obtained.

[Modification 3]
It is also possible to configure the objective optical system in the color SLO 701 in the ophthalmic apparatus described as the above-mentioned modification 2 by the reflection optical system, and make the color SLO an ultra-wide angle of view SLO. The configuration of the optical system of the ultra-wide angle of view SLO used in this modification will be described below with reference to FIG. 8 which schematically shows the optical configuration. The other configurations attached to the optical system such as the display monitor, the control unit, the operation input unit, the image processing unit, and the storage memory are the same as the configurations arranged along with the fundus camera described in the first embodiment. Therefore, the explanation here is omitted.

The ultra-wide angle of view SLO 800 shown as a modified example includes a light source unit 801, an illumination optical system, and a light receiving optical system. The light source unit 801 includes an infrared light laser light source 801a, a red laser light source 820a, a green laser light source 820b, and a blue laser light source 820c. Further, the light source unit 801 has a dichroic mirror 823, a dichroic mirror 821a, and a dichroic mirror 821b for making the light from these light sources coaxial in the optical path L1. The illumination optical system includes a perforated mirror 802, a lens 803, a diopter correction optical system (804,805), a concave mirror (806,808,810), and scanning, which are arranged in order from the light source unit 801 toward the eye E to be inspected. It has an optical system (807,809).

The diopter correction optical system has mirrors 804 and 805, and the optical path length can be changed by moving these mirrors in the direction of the arrow in the figure by a drive system (not shown), and the diopter correction of the focus can be performed. The scanning optical system includes a polygon mirror 807 and a galvano mirror 809. The polygon mirror 807 scans the laser beam horizontally, for example, on the fundus of the eye, and the galvano mirror 809 scans the laser beam vertically. The light receiving optical system includes a lens 812, a pinhole plate 813, a condenser lens 814, and a light receiving element 815, which are arranged in order from the perforated mirror 802 in the reflection direction of the perforated mirror 802. The reflected light from the fundus is reflected by the peripheral portion of the hole of the perforated mirror 802 and guided to the light receiving element 815 via these optical members.

The scanning laser light emitted by the light source unit 801 passes through the hole of the perforated mirror 802 after passing through the dichroic mirror 823, and is guided to the polygon mirror 807 via the diopter correction optical system and the concave mirror 806. The laser beam scanned horizontally on the fundus by the polygon mirror 807 is further vertically scanned by the galvano mirror 809 via the concave mirror 808. The laser beam passing through the galvano mirror 809 is applied to the eye E to be inspected reflected by the concave mirror 810. The laser beam reflected from the fundus of the eye passes through these optical members in reverse, is reflected by the perforated mirror 802, and is received by the light receiving element 815. By constructing the objective optical system with such a reflective optical system, it is possible to capture an image on the fundus with a wide angle of view. The fundus image obtained by the ophthalmologic apparatus using the ultra-wide angle of view SLO800 described above is also lost due to the processing by the control unit 18 described in the first embodiment regardless of the user's proficiency in observing the fundus image. It is possible to re-shoot under the shooting conditions according to the situation. As a result, even if the user is immature, automatic re-imaging can be performed under appropriate imaging conditions, and a fundus image suitable for diagnosis can be obtained.

[Example 2]
In Example 1, an ophthalmic apparatus that outputs imaging conditions at the time of re-imaging using a trained model has been described. On the other hand, in the second embodiment, as another example of using the trained model, an OCT device (optical coherence tomography device) is used for an ophthalmic device that outputs the cause of image loss using the trained model from a captured image. As an example, it will be described with reference to FIGS. 9 to 11.

<Optical configuration of the device>
The optical configuration of the OCT apparatus used in this embodiment will be described with reference to FIG. The OCT apparatus used in this embodiment includes a photographing optical system 900, a spectroscope 960, a control unit 980, and a display monitor 990 which is an image display unit. The photographing optical system 900 corresponds to the photographing optical unit C2 shown in FIG. 1 in the first embodiment, and has a left-right direction (X direction), a vertical direction (Y direction), and a front-back direction (Z) with respect to a base (not shown). It is said that it can be moved in the direction).

The photographing optical system 900 is composed of a measuring optical system for capturing a two-dimensional image and a tomographic image of the anterior eye Ea of the eye to be inspected E and the fundus Er of the eye to be inspected. An objective lens 901-1 is installed facing the eye E to be inspected, and the optical path is branched by a first dichroic mirror 902 and a second dichroic mirror 903 that function as optical path branching portions on the optical axis thereof. .. That is, the optical path from the eye E to be inspected is divided into the anterior segment observation optical path L91, the optical path L92 between the fundus observation optical path and the fixation light optical path, and the optical path L93 (OCT optical system measurement optical path) for each wavelength band by these dichroic mirrors. Be separated. The mode of branching of these optical paths and the configuration arranged in the reflection direction and the transmission direction of the dichroic mirror are not limited to the modes exemplified here, and may be various known modes.

The first dichroic mirror lens 941 and an image sensor 942 for front eye observation are arranged on the front eye observation optical path L91 in the transmission direction of the first dichroic mirror 902. The optical path L92 in the reflection direction of the second dichroic mirror 903 is further branched by the third dichroic mirror 918 into an optical path to an APD (avalanche photodiode) 915 and a fixation lamp 916 for observing the fundus of the eye for each wavelength band. On the optical path L92, a lens 901-2, a scanning unit, a focus lens 911, a lens 912, and a third dichroic mirror 918 are arranged in this order from the second dichroic mirror 903. The focus lens 911 is driven in the direction of the optical path indicated by the arrow in the figure by a drive motor (not shown) for adjusting the focus for observing the fixation lamp and the fundus. The APD 915 arranged in the transmission direction of the third dichroic mirror 918 has a sensitivity in the wavelength of the illumination light for fundus observation (not shown), specifically in the vicinity of 780 nm. On the other hand, the fixation lamp 916 arranged in the reflection direction of the third dichroic mirror 918 generates visible light to promote fixation of the subject.

Further, in the optical path L92, an X scanner 917-1 and a Y scanner 917-2 are arranged as scanning units for scanning the light emitted from a fundus observation illumination light source (not shown) on the fundus Er of the eye E to be inspected. Will be done. The X scanner 917-1 is used for scanning the main scanning direction of the measurement light, and the Y scanner 917-2 is used for scanning in the sub scanning direction intersecting the main scanning direction of the measurement light. The lens 901-2 is arranged with the vicinity of the center position of the X scanner 917-1 and the Y scanner 917-2 as the focal position. The APD915 is a single detector and detects the light scattered / reflected from the fundus Er and returned. The third dichroic mirror 918 is a prism in which a perforated mirror or a hollow mirror is vapor-deposited, and separates the illumination light and the return light from the fundus Er.

The optical member arranged in the optical path L93 forms an OCT optical system and is used for capturing a tomographic image of the fundus Er of the eye to be inspected, specifically, an interference signal for forming a tomographic image. Used to obtain. A lens 901-3, a mirror 921, an OCT scanning unit, an OCT focus lens 923, and a lens 924 are arranged on the optical path L93 in order from the second dichroic mirror 903. The scanning unit that scans the light on the fundus Er of the eye E to be inspected has an OCTX scanner 922-1 and an OCTY scanner 922-2. The OCTX scanner 922-1 and the OCTY scanner 922-2 are arranged so that the vicinity of the center on the optical path L93 is the focal position of the lens 901-3. Further, the vicinity of the center and the position of the pupil of the eye E to be inspected have an optical conjugate relationship. In order to focus the measurement light on the fundus Er, the OCT focus lens 923 is movable in the direction of the optical path indicated by the arrow in the figure by a drive motor (not shown).

Next, the optical path from the measurement light source 930, the reference optical system, and the spectroscope 960 will be described. The measurement light source 930 is a low coherent light source for obtaining the measurement light incident on the measurement optical path (optical path L93). The OCT optical system illustrated here uses a Michelson interferometer, which includes a measurement light source 930, an optical coupler 925, an optical fiber 925 to 1-4, a lens 951, a dispersion compensation glass 952, and a reference mirror 953. And a spectroscope 960. Optical fibers 925-1 to 925-1 to 4 are single-mode optical fibers connected to and integrated with an optical coupler 925.

The light emitted from the measurement light source 930 is divided into measurement light guided to the optical fiber 925-2 side via the optical coupler 925 and reference light guided to the optical fiber 925-3 side through the optical fiber 925-1. The measurement light is applied to the fundus Er of the eye E to be observed through the optical path L93 of the OCT optical system. The measured light reflected / scattered by the retina follows the same optical path L93 in reverse as return light and reaches the optical coupler 925. On the other hand, the reference light reaches the reference mirror 953 through the optical fiber 925-3, the lens 951, and the dispersion compensation glass 952 and is reflected. Then, it follows the same optical path in the opposite direction to reach the optical coupler 925. The return light and the reference light that have reached the optical coupler 925 are combined by the optical coupler 925 to become interference light. Here, interference occurs when the optical path length of the measurement light and the optical path length of the reference light are almost the same. The position of the reference mirror 953 can be adjusted in the optical axis direction indicated by the arrow in the drawing by a drive motor or the like (not shown), and the optical path length can be adjusted by moving the reference mirror 953 in the optical axis direction. The obtained interference light is guided to the spectroscope 960 via the optical fiber 925-4.

The spectroscope 960 includes a lens 961, a diffraction grating 962, a lens 963, and a line sensor 964. The interference light emitted from the optical fiber 925-4 becomes substantially parallel light through the lens 961, is separated by the diffraction grating 962, and is imaged on the line sensor 964 by the lens 963. The line sensor 964 has a configuration in which a plurality of pixels, that is, light receiving elements are arranged in a row, and all the pixels can be read out at once by a predetermined clock. Although the Michelson interferometer is used as the interferometer in this embodiment, a Mach-Zehnder interferometer may be used.

Next, the details of the control unit 980 in the OCT apparatus used in this embodiment will be described with reference to FIG. FIG. 10 is a block diagram showing a schematic configuration of the control unit 980. As shown in FIG. 10, the control unit 980 includes a drive control unit 1001, a display control unit 1002, an acquisition unit 1010, and a processing unit 1020. Further, the control unit 980 includes an OCT focus lens 923, an APD 915, a fixed eye lamp 916, a display monitor 990, an image sensor 942, a scanning unit (917), an OCT scanning unit (922), a focus lens 911, a reference mirror 953, and a line. It is connected to the sensor 964 and the like.

The drive control unit 1001 drives the OCT focus lens 923, the APD 915, the fixation lamp 916, the image sensor 942, the scanning unit (917), the OCT scanning unit (922), the focus lens 911, the reference mirror 953, the line sensor 964, and the like. Take control. The display control unit 1002 is connected to the display monitor 990, and receives an image to be displayed on the display monitor 990, a user instruction input via an instruction screen displayed on the display monitor 990, or the like. The display monitor 990 is an arbitrary display such as an LCD display, and is a GUI for operating each drive unit and control unit 980 of the OCT device, a generated image, an image subjected to arbitrary processing, various types of patient information, and the like. Information can be displayed.

Further, when displaying the GUI described above, an operation input unit is further connected to the control unit 980, and the GUI is operated or information is input via the operation input unit. You can also operate. For the operation input unit, for example, a mouse, a touch pad, a trackball, a touch panel display, a pointing device such as a stylus pen, a keyboard, or the like can be used. When a touch panel display is used, the display monitor 990 and the operation input unit can be integrally configured. In this embodiment, the photographing optical system 900, the spectroscope 960, the control unit 980, and the display monitor 990 are separate elements, but even if some or all of them are integrally configured. Good.

The acquisition unit 1010 includes an image acquisition unit 1011, an information acquisition unit 1012, and a storage unit 1013. The image acquisition unit 1011 is connected to the APD 915 and the image generation unit 1023 described later, and acquires an image signal output from these or an image generated by the image generation unit 1023. The information acquisition unit 212 acquires information input via the display monitor 990 described above, or eye examination information previously stored in the storage unit 1013 in association with the eye to be inspected E. The storage unit 1013 stores various images and information acquired by the acquisition unit 1010, as well as output information such as a copying loss cause and a correction value, which will be described later, output from the processing unit 1020. The processing unit 1020 has an output unit 1021 and an image generation unit 1023. The output unit 1021 outputs, for example, a learning model described later to output the cause of the copying loss, and outputs a correction value or the like for reducing the cause of the copying loss. The image generation unit 1023 generates a fundus image or a tomographic image using, for example, a signal acquired from the APD 915 or the line sensor 964 acquired by the image acquisition unit 1011. The generated image is stored in the storage unit 1013 by the image acquisition unit 1011.

Each component of the control unit 980 other than the storage unit 1013 may be composed of a software module executed by a processor such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). The processor may be, for example, a GPU (Graphical Processing Unit), an FPGA (Field-Programmable Gate Array), or the like. In addition, each component may be configured by a circuit or the like that performs a specific function such as an ASIC. The storage unit 1013 may be composed of any storage medium such as an optical disk or a memory.

<Overview of SLO Focus>
In the OCT apparatus, the focus operation is executed by the SLO optical system, and the focus in the OCT is performed using the obtained conditions. Here, an outline of the focus operation performed in the SLO optical system will be described. The control unit 980 controls the optical members of the SLO optical system so that the optimum imaging conditions can be obtained in the SLO optical system of the photographing optical system 900. Specifically, the drive control unit 1001 controls the focus of the focus lens 911 in the SLO optical system.

The drive control unit 1001 first moves the focus lens 911 to a position on the optical axis corresponding to the lowest position of the diopter by a drive motor or the like (not shown). Next, in the control unit 980, the image generation unit generates an SLO image (fundus image) from the signal obtained via the APD 915 in response to the completion of the movement of the focus lens 911. The output unit 1021 analyzes the generated SLO image and calculates the focus evaluation value of the SLO image. The focus evaluation value is obtained, for example, based on the contrast and sharpness of the generated image. Here, it is judged that the higher the calculated focus evaluation, the better the focus state of the SLO optical system with respect to the fundus Er of the eye E to be inspected.

Subsequently, the drive control unit 1001 moves the focus lens 911 again, and adds an additional 0.125 diopter to the current diopter in the SLO optical system. The control unit 980 generates a new SLO image according to the completion of the movement of the focus lens 911, and calculates the focus evaluation value of the SLO image. When the current focus evaluation value is obtained, the output unit 1021 compares the current focus evaluation value with the front focus evaluation value which is the focus evaluation value acquired immediately before. If the current focus evaluation value is higher than the prefocus evaluation value, the current focus evaluation value is updated as a new prefocus evaluation value. After that, the drive control unit 1001 further adds a predetermined amount (0.125 in this case) of the diopter to the current diopter, generates a further SLO image, and acquires the focus evaluation value thereof, and newly obtained the focus. Compare the evaluation value and the front focus evaluation value.

In the comparison between the current focus evaluation value and the prefocus evaluation value, if it is determined that the prefocal evaluation value is higher and the focus evaluation value does not increase, the output unit 1021 determines that the diopter that obtained the prefocus evaluation value Judge that it is close to the optimum focus state. In this way, the optimum focus state is repeated by changing the diopter, generating an SLO image, calculating the focus evaluation value, comparing the obtained focus evaluation value and the prefocus evaluation value, and changing the diopter based on the comparison result. Look for. When the focus evaluation value of the SLO image becomes the highest through such control, it can be determined that the focus state of the SLO optical system, particularly the position of the focus lens 911, is in the optimum focus position.
In this embodiment, the predetermined amount for moving the focus lens 911 is 0.125 diopter, but the value is not limited to this value, and other movement amounts may be used. Further, the movement unit is not limited to the diopter, and may be, for example, the movement distance of the lens, the rotation speed of the motor for moving the lens, or the like.

<Overview of OCT reference mirror position adjustment>
Next, the outline of the position adjustment of the reference mirror 953 in the OCT apparatus will be described. The display mode of the tomographic image generated by using the interference light obtained through the OCT optical system changes depending on the optical path length difference between the measurement light and the reference light by the reference mirror 953. That is, if the reference mirror 953 is not adjusted to an appropriate position, a tomographic image such as a retina, which is an inspection target site (predetermined site), may not be displayed at an appropriate position in the generated tomographic image. Therefore, the drive control unit 1001 controls the photographing optical system 900 so that the inspection target portion is displayed at an appropriate position in the tomographic image of the eye E to be inspected obtained in the optimum focus state. Specifically, the position of the reference mirror 953 in the OCT optical system is adjusted in the optical axis direction, and the display position of the inspection target portion in the tomographic image is finely adjusted.

The drive control unit 1001 first drives a drive motor or the like (not shown) so that the reference optical path length of the OCT optical system becomes the longest, and moves the reference mirror 953. Next, the image generation unit 1023 generates a tomographic image (OCT image) using the interference light obtained in this state in response to the completion of the movement of the reference mirror 953. The output unit 1021 analyzes the generated tomographic image and outputs whether or not the tomographic image includes a site to be inspected of the eye E to be inspected, such as the retina. When the tomographic image includes the inspection target portion, the drive control unit 1001 outputs a determination result that further adjustment of the reference mirror 953 is unnecessary.

When the tomographic image does not include the inspection target site, the drive control unit 1001 moves the reference mirror 953 so that the reference optical path length is 0.125 mm shorter than the current state. The image generation unit 1023 generates a new tomographic image from the interference signal obtained in that state in response to the completion of the movement of the reference mirror 953. The output unit 1021 outputs whether or not the tomographic image to be inspected is included in the generated tomographic image, and if it is not included, a series of processing from the movement of the reference mirror 953 by the drive control unit 1001 or the like is necessary. And repeat. As a result, the inspection target site is displayed at an appropriate position in the generated tomographic image.
In this embodiment, the movement amount of the reference mirror 953 is 0.125 mm, but the movement amount is not limited to this value, and other movement amounts may be used. Further, the movement unit is not limited to mm, and may be, for example, the rotation speed of the motor for moving the reference mirror.

<Sequence of tomographic image imaging>
Next, a basic imaging sequence for performing automatic imaging of the eye to be inspected will be described with reference to the flowchart of FIG. The user gives an instruction to the OCT apparatus to roughly align the subject and the photographing optical system 900 and to start the preview operation, for example, via the GUI of the display monitor 990. At that time, the subject is placed on the jaw support (not shown), and the position of the eye to be inspected in the Y-axis direction is adjusted by the jaw support drive mechanism (not shown) so as to have a predetermined height. When the rough position adjustment is completed, the user presses the preview start button displayed on the GUI, for example. When the control unit 980 detects that the preview start button has been pressed, the control unit 980 shifts the flow to step S1101.

In step S1101, an alignment operation is executed to align the anterior segment of the eye E to be inspected with the photographing optical system 900. Since the details of the alignment operation for the anterior segment of the eye are the same as the operation described with reference to FIG. 3 executed in the first embodiment, the details thereof will be omitted here. When the control unit 980 determines that the alignment is completed, the control unit 980 shifts the flow to step S1102.

In step S1102, the focus operation of the SLO optical system described above is executed to perform focusing. The drive control unit 1001 moves the focus lens 911 to change the diopter. The image generation unit 1023 generates an SLO image (fundus image) using the signal obtained by the APD915. The output unit 1021 calculates the focus evaluation value of the generated fundus image and executes the operation for obtaining the optimum focus state described above. When the control unit 980 determines that the focus evaluation value of the fundus image becomes the highest and the focus state of the SLO optical system, particularly the position on the optical axis of the focus lens 911 is in the optimum focus position, the flow shifts to step S1103. Let me.

In step S1103, the drive control unit 1001 controls the optical member of the OCT optical system so that the optimum imaging conditions can be obtained in the OCT optical system in the photographing optical system 900. Specifically, the focus of the OCT focus lens 923 in the OCT optical system is controlled.
The OCT optical system and the SLO optical system have a correlation with the position of each focusing lens when a focused state is obtained. Therefore, once the position on the optical axis of the focus lens 911 from which the optimum focus state has been obtained in the SLO optical system is grasped, the position of the OCT focus lens 923 of the OCT optical system can be adjusted in association with this. As a result, an optimum focus state can be obtained even in the OCT optical system. In step S1103, the drive control unit 1001 uses the position on the optical axis of the focus lens 911 and the diopter value obtained by the focus processing of the SLO optical system in step S1102 to position the OCT focus lens 923 at the corresponding position on the optical axis. Move to. When the control unit 980 determines that the movement of the OCT focus lens 923 is completed, the control unit 980 shifts the flow to step S1104.

In step S1104, the operation of adjusting the position of the OCT reference mirror described above is executed to adjust the display position of the inspection target portion in the tomographic image. The drive control unit 1001 moves the reference mirror 953 in the optical axis direction. Then, the acquisition unit 1010 acquires an interference signal from the interference light received by the line sensor 964, and the image generation unit 1023 generates a tomographic image from the interference signal. The output unit 1021 outputs a determination result as to whether or not the tomographic image to be inspected is included in the generated tomographic image. The control unit 980 shifts the flow to step S1105 when the tomographic image includes the inspection target site and the determination result that further adjustment is unnecessary is output.

In step S1105, the image illustrated in FIG. 12 is displayed as a preview on the display monitor 990 under the control of the display control unit 1002. The anterior eye image 1201, the fundus image 1202, and the tomographic image 1203 are displayed in the preview. The anterior eye image 1201 is an image obtained from the output of the image sensor 942, the fundus image 1202 is an image generated from the output of the APD 915, and the tomographic image 1203 is an image generated from the output of the line sensor 964. In this embodiment, the preview display is performed after steps S1101 to S1104 are completed. However, since the image sensor 942, the APD 915, and the line sensor 964 can be independently driven for image acquisition, even if there is an image generated during steps S1101 to S1104, these preview displays can be performed. Good.

In step S1106, the output unit 1021 uses the fundus image and the tomographic image generated by the image generation unit 1023 as input data to output the cause of the image loss and the imaging conditions corresponding to the cause of the image loss using the trained model. .. If there is no cause of copying loss output by the output unit 1021, the control unit 980 shifts the flow to step S1107. In step S1107, the drive control unit 1001 performs so-called main imaging, which is the imaging of the tomographic image provided for the actual diagnosis under the imaging conditions at the time of preview, and the control unit 980 shifts the flow to step S1108. In the main shooting, for example, the conditions related to the focus lens, the reference mirror, and the like are not changed, but the conditions such as the number of signal acquisitions in this one scanning line are changed. In step S1108, the control unit 980 saves the tomographic image or the like acquired by photographing in the storage unit 1013 and ends the process. Further, when the cause of copying loss or the like is output in step S1106, the control unit 980 shifts the flow to step S1109.

For example, in the case of the focus operation of the SLO optical system, the control unit 980 moves the focus lens 911 to change the diopter, acquires a fundus image, and calculates the focus evaluation value thereof. Then, the optimum focus state is searched for by repeatedly comparing the obtained focus evaluation value and the prefocus evaluation value and changing the diopter based on the comparison result. However, when a clear fundus image cannot be obtained due to unclear fundus due to cataract, for example, a constant focus evaluation value may not be obtained even if the position of the focus lens 911 is adjusted. If a constant focus evaluation value is not obtained, the focus operation of the SLO optical system will not be completed, and it will be interrupted after a certain period of time, and a blurred fundus image and a tomographic image will be acquired.

Further, in the focus operation of the OCT optical system, the drive control unit 1001 moves the OCT focus lens 923 to a corresponding position on the optical axis by using the position of the focus lens 911 and the diopter value obtained in step S1102. Therefore, if the focus lens 911 cannot be moved to an appropriate position by the focus operation of the SLO optical system, the OCT focus lens 923 cannot be moved to an appropriate position either.

For example, in the case of adjusting the position of the OCT reference mirror, the control unit 980 moves the reference mirror 953, acquires a tomographic image, and determines whether or not the inspection target site is arranged at an appropriate position in the tomographic image. To do. However, if the optical path length adjustment of the reference light by the reference mirror 953 is insufficient, or if the curvature of the fundus is large and the optical path length of the measurement light varies greatly depending on the location, the tomographic image may be folded. If the tomographic image is folded back, the structure of the retina becomes unclear and the inspection accuracy decreases.

In this way, if the position of the focus lens cannot be determined or the position of the reference mirror cannot be adjusted with normal control, appropriate shooting conditions can be obtained even if the lost image is input to the trained model. I can't. In such a case, information on the structure of the eye to be inspected, such as the fact that the focus operation of the SLO optical system has not been achieved or the curvature of the fundus is large, can be added to the data input to the trained model. As a response to such a case, it is preferable that the trained model to be used also refers to these data and outputs that the trained model cannot output appropriate shooting conditions. Therefore, when the cause of the image loss is output in step S1106, the display control unit 1002 causes the display monitor 990 to display the output cause of the image loss in addition to the fact that the cause of the image loss cannot be dealt with. Good. Further, at that time, by displaying the shooting conditions controlled at the time of re-shooting or the correction value of the shooting conditions, it is possible to increase the information to be referred to when the user selects in the subsequent step S1109.

In step S1109, when the control unit 980 determines that the manual operation has been selected, the flow shifts to step S1110, and the control of the shooting condition is changed from the automatic control to the manual control. The user drives each configuration of the photographing optical system 900 by using the GUI of the display monitor 990 described above and various other input means (not shown). Then, the control unit 980 shifts the flow to step S1105 in order to preview the tomographic image obtained under the conditions after the manual control.

In step S1109, when the control unit 980 determines that the automatic operation is selected, the control unit 980 shifts the flow to step S1111. In step S1111, the control unit 980 automatically drives the photographing optical unit 700 and the like in order to re-photograph the fundus image under the imaging conditions corresponding to the output imaging loss conditions output by the output unit 1021. Then, the control unit 980 shifts the flow to step S1105 in order to preview the tomographic image obtained under the conditions after the automatic control.

Appropriate determination of the cause of image loss as described above for the tomographic image and the fundus image used to obtain the focus condition needs to be made by an knowledgeable user. However, a knowledgeable user does not always operate the ophthalmic device. According to this embodiment, it is possible to appropriately output the cause of the copying loss and the correction value for dealing with the cause of the copying loss from the preview image using the trained model. Therefore, it is possible to automatically control for obtaining appropriate shooting conditions so that the problematic shooting loss can be reduced and re-shooting can be performed.
In this embodiment, the control unit 980 makes a copying loss determination using the image displayed in the preview after the preview is displayed in step S1105. However, after the so-called main imaging in step S1107, a copying loss determination may be made using the obtained tomographic image of the main imaging. In addition, whether or not to perform the manual selection in step S1109 may be selected in advance.

<Generation of trained model>
Next, the generation of the trained model according to this embodiment will be described. As described in Example 1, the trained model in this example is a machine learning model that follows an arbitrary machine learning algorithm such as deep learning, and is trained using appropriate learning data in advance. It is a model that was done.

Hereinafter, the learning data used for training when obtaining the trained model according to this embodiment will be described.
In this embodiment, a fundus image and a tomographic image are used as input data for the learning data. Further, the input data may include the diopter described above. Then, as the output data of the learning data, the cause of the photo loss and the shooting conditions corresponding to the cause of the photo loss or the correction value of the shooting conditions are used. The method of creating the learning data is basically the same as the method of creating the learning data in the above-described first embodiment.

For example, the fundus image is taken by moving the position of the focus lens 911 on the optical axis closer to or further from the fundus Er by a certain amount. As a result, as the input data of the training data, a fundus image having out-of-focus, which is one aspect of copying loss, can be obtained. Depending on the aspect of the fundus image, it is possible to determine whether the focus lens 911 is due to being too close to or too far from the fundus Er. More specifically, when the focus lens 911 is close to the fundus Er, the obtained fundus image becomes darker at the focal position, but conversely the periphery becomes slightly brighter, and the visible region expands accordingly. Further, when the focus lens 911 approaches the fundus Er, it becomes darker from the position where the focal point was, and finally, only the periphery remains as a visible region. Further, when the focus lens 911 is far from the fundus Er, the obtained fundus image becomes dark as a whole. However, since the periphery is also darkened, the visible region does not expand. Further, when the focus lens 911 moves away from the fundus Er, it becomes dark from the surroundings, and finally, only the focal position remains as a visible region. Therefore, from the distribution of light and darkness of the fundus image Er, it is possible to know whether the focus lens 911 is out of focus because it is close to the fundus Er and is out of focus because it is far from it.

From the above, by using the amount of deviation (correction value) from the focus position of the focus lens corresponding to these fundus images as the output data, it is possible to obtain a learned model that outputs the cause of this copying loss and the correction value. Further, as a loss image of the same aspect, for example, a tomographic image in which the focus position in the depth direction of the retina is not at a predetermined position can be used as input data. For such a tomographic image, a trained model that can deal with out-of-focus can be obtained by using the same training data as the fundus image.

Further, for example, a tomographic image is taken by shortening or lengthening the optical path length of the reference mirror 953 from a desired position where the optical path length of the reference light becomes a predetermined optical path length. As a result, a tomographic image in which the inspection target site, which is one aspect of photo loss, is deviated from a desired position in the tomographic image can be obtained. Since the vertical position of the inspection target part in the tomographic image can be associated with the length of the optical path length of the reference light, how should the optical path length of the reference light be changed from the captured tomographic image? Can be identified. Further, by using the amount of deviation (correction value) of the position on the optical axis of the reference mirror with respect to these tomographic images from a predetermined position as output data, a learned model that outputs the cause of this copying loss and the correction value can be obtained. ..

In addition, the trained model in this embodiment can output an output that there is no copying loss. Therefore, the image taken without moving from an appropriate position and the cause of the copying loss (no copying loss) are also used as the learning data pair. By training using the learning data pair, the learning accuracy can be improved. Further, as described in Example 1, for example, learning data in a diseased eye cannot be created in the above-described manner. Therefore, it is advisable to use the eye image obtained by photographing the diseased eye and the cause of the image loss at that time as a learning data pair. It should be noted that the above-mentioned causes of image loss are examples, and for other known causes of image loss such as underexposure and their correction values, images having image loss are similarly generated and used as input data. Therefore, a trained model corresponding to the cause of this copying loss can be obtained.

The number of input data may be one or more, and can be appropriately selected. For example, a focus evaluation value or the like can be added as input data for learning data. Further, a trained model applicable to each image, for example, an anterior eye image, a fundus image, and a tomographic image, or a trained model corresponding to additional data such as other diopters may be prepared in advance. In this case, it is preferable that an appropriate trained model is selected according to the data to be input to the trained model. Further, as described in the above-described modification 1, first, the cause of the photo loss is determined using a learned model that outputs only the cause of the photo loss from the acquired image, and the image is photographed from the cause of the photo loss and the acquired image. It is also possible to use a trained model that outputs a condition or a correction value.

Further, as the trained model, the one trained in advance using the OCT device may be used as it is, or the trained model may be updated (additional learning) while using it. Specifically, the captured image determined to be a copying loss is used as the input data of the learning data, and the cause of the copying loss and the shooting conditions at that time are used as the output data of the learning data. Then, additional learning is performed using the learning data pair. Further, additional learning may be performed for each subject, in which case a trained model for the subject is generated. The CNN, which is a machine learning model in this embodiment, is the same as the contents described in Examples 1 and 6, and thus the description thereof is omitted here. In this embodiment, when a captured image is input to the trained model trained by the training data described above, the cause of the copying loss and the shooting conditions for reducing the cause of the shooting loss and performing re-shooting are output.

In Example 2 described above, the misalignment of the inspection target site in the tomographic image was described as the cause of the copying loss. In the second embodiment, the photographing optical unit is obtained by dividing the return light from the eye E to be irradiated and the light from the measurement light source 930, which is obtained by dividing the light from the measurement light source 930. It has an interference optical system that interferes with the reference light. The reference light in this case corresponds to the measurement light. When it is determined that the cause of the image loss is the deviation of the arrangement of the inspection target portion, the deviation of the arrangement is caused by, for example, the optical path length difference between the optical path length of the reference light and the optical path length of the measurement light. Therefore, this optical path length difference is adjusted at the time of re-shooting. Specifically, the output unit 1021 sets the optical path length difference as a correction value (control value) from the optical path length difference (first difference) at the time of the first shooting to the optical path length difference (second difference) at the time of re-shooting. ) Is output. More specifically, the amount of change in the position of the reference mirror 953 on the optical path is output. In the second embodiment, the reference mirror 953 is moved to change the optical path length of the reference light, but the optical path length difference may be changed by changing the optical path length of the measurement light. When changing the optical path length difference, whether the optical path length difference should be increased or decreased depends on the arrangement of the inspection target part (depending on the upper side or the lower side in the image) in the tomographic image of the eye E to be inspected. ) Can be determined. As a result, the output unit 1021 can output the positive and negative of the change amount of the difference.

[Modification example 4]
In Example 2 described above, the case where the SLO optical system is used as the configuration for photographing the fundus of the eye has been described. However, the configuration for photographing the fundus is not limited to the illustrated embodiment, and the fundus can be photographed in color. For example, the color SLO described in the above-described modification 2 may be used instead of the SLO optical system described in the second embodiment. In this case, in the optical path L92 of FIG. 9, for example, the color SLO701 shown in FIG. 7 can be arranged instead of the various configurations of the illustrated SLO optical system. Further, the objective optical system in the color SLO 701 may be configured by the reflection optical system as described in Example 3 described above, and the color SLO may be, for example, the ultra-wide angle of view SLO 800 shown in FIG. Further, instead of the SLO optical system in the above-mentioned Example 2, it is also possible to adopt the optical configuration of the fundus camera C of FIG. 2A as described in the above-mentioned Example 1.

According to the above-described embodiment, it is possible to output an appropriate correction value at the time of copying loss using the trained model. Further, according to the above-described embodiment, it is possible to output an appropriate cause of copying loss by using the trained model. Therefore, it is possible to reduce the occurrence rate of photo loss at the time of re-shooting, and it is also possible to reduce the burden on the subject and the user.

The present invention is not limited to the ophthalmic apparatus exemplified in the above-described examples. For example, the present invention can be applied to devices having arbitrary functions that can be used in the field of ophthalmology, such as an intraocular pressure measuring device, an ocular refraction measuring device, a fundus camera, and an OCT device.

[Modification 5]
In the various examples and modifications described above, the spectral domain OCT (SD-OCT) device using the SLD as the light source has been described as the OCT device, but the configuration of the OCT device according to the present invention is not limited to this. For example, the present invention can be applied to any other type of OCT device such as a wavelength sweep type OCT (SS-OCT) device using a wavelength sweep light source capable of sweeping the wavelength of emitted light. The present invention can also be applied to a Line-OCT device (or SS-Line-OCT device) using line light. The present invention can also be applied to a Full Field-OCT device (or SS-Full Field-OCT device) using area light.

In the various examples and modifications described above, an optical fiber optical system using a coupler is used as the dividing means, but a spatial optical system using a collimator and a beam splitter may be used. Further, the configuration of the photographing optical system 900 is not limited to the above configuration, and a part of the configuration included in the photographing optical system 900 may be a configuration separate from the photographing optical system 900.

Further, in the various examples and modifications described above, the acquisition unit 1010 acquired the interference signal acquired by the photographing optical system 900, the tomographic image generated by the image generation unit 1023, and the like. However, the configuration in which the acquisition unit 1010 acquires these signals and images is not limited to this. For example, the acquisition unit 1010 may acquire these signals from a server or an imaging device connected to the control unit 980 via a LAN, WAN, the Internet, or the like.

Further, the training data of various trained models is not limited to the data obtained by using the ophthalmologic device itself that actually performs the imaging, and the data obtained by using the same type of ophthalmic device or the same type according to the desired configuration. It may be data obtained by using an ophthalmic apparatus or the like.

In the various examples and modifications described above, the tomographic image relating to the fundus portion of the eye to be inspected has been described, but the above image processing may be performed on the tomographic image relating to the anterior segment of the eye to be inspected. In this case, the regions to be subjected to different image processing in the tomographic image include regions such as the crystalline lens, cornea, iris, and anterior chamber of eye. The region may include another region of the anterior segment of the eye. Further, the region of the tomographic image relating to the fundus portion is not limited to the vitreous portion, the retinal portion, and the choroidal portion, and may include other regions relating to the fundus portion. Here, since the tomographic image relating to the fundus portion has a wider gradation than the tomographic image relating to the anterior segment portion, it is possible to more effectively improve the image quality by the image processing according to the above-described embodiment and the modified example. ..

Further, in the various examples and modifications described above, the subject to be examined has been described as an example, but the subject is not limited thereto. For example, the subject may be skin, other organs, or the like. In this case, the OCT device according to the above embodiment and the modified example can be applied to a medical device such as an endoscope in addition to the ophthalmic device.

[Modification 6]
Further, the image processed by the image processing apparatus or the image processing method according to the various examples and modifications described above includes a medical image acquired by using an arbitrary modality (imaging apparatus, imaging method). The medical image to be processed may include a medical image acquired by an arbitrary imaging device or the like, an image processing device according to the above examples and modifications, or an image such as a fundus image or a tomographic image created by an image processing method. it can.

Further, the medical image to be processed is an image of a predetermined part of the subject (subject), and the image of the predetermined part includes at least a part of the predetermined part of the subject. In addition, the medical image may include other parts of the subject. Further, the medical image may be a still image or a moving image, and may be a black-and-white image or a color image. Further, the medical image may be an image showing the structure (morphology) of a predetermined part or an image showing the function thereof. The image showing the function includes, for example, an OCTA image, a Doppler OCT image, an fMRI image, and an image showing blood flow dynamics (blood flow volume, blood flow velocity, etc.) such as an ultrasonic Doppler image. The predetermined part of the subject may be determined according to the subject to be imaged, and the human eye (eye to be examined), brain, lung, intestine, heart, pancreas, kidney, liver and other organs, head, chest, etc. Includes any part such as legs and arms.

Further, the medical image may be a tomographic image of the subject or a frontal image. The frontal image is, for example, a frontal image of the fundus, a frontal image of the anterior segment of the eye, a fundus image photographed by fluorescence, and data acquired by OCT (three-dimensional OCT data) in at least a part range in the depth direction of the imaged object. Includes En-Face images generated using the data from. The En-Face image is an OCTA En-Face image (motion contrast front image) generated by using at least a part of the data in the depth direction of the shooting target for the three-dimensional OCTA data (three-dimensional motion contrast data). ) May be. Further, three-dimensional OCT data and three-dimensional motion contrast data are examples of three-dimensional medical image data.

Here, the motion contrast data is data indicating a change between a plurality of volume data obtained by controlling the measurement light to be scanned a plurality of times in the same region (same position) of the eye to be inspected. At this time, the volume data is composed of a plurality of tomographic images obtained at different positions. Then, motion contrast data can be obtained as volume data by obtaining data showing changes between a plurality of tomographic images obtained at substantially the same position at different positions. The motion contrast frontal image is also referred to as an OCTA frontal image (OCTA En-Face image) relating to OCTA angiography (OCTA) for measuring the movement of blood flow, and the motion contrast data is also referred to as OCTA data. The motion contrast data can be obtained, for example, as a decorrelation value, a variance value, or a maximum value divided by a minimum value (maximum value / minimum value) between two tomographic images or corresponding interference signals. , It may be obtained by any known method. At this time, the two tomographic images can be obtained, for example, by controlling the measurement light to be scanned a plurality of times in the same region (same position) of the eye to be inspected.

The En-Face image is, for example, a front image generated by projecting data in the range between two layer boundaries in the XY directions. At this time, the front image is at least a part of the depth range of the volume data (three-dimensional tomographic image) obtained by using optical interference, and is the data corresponding to the depth range determined based on the two reference planes. Is projected or integrated on a two-dimensional plane. The En-Face image is a frontal image generated by projecting the data corresponding to the depth range determined based on the detected retinal layer of the volume data onto a two-dimensional plane. As a method of projecting data corresponding to a depth range determined based on two reference planes on a two-dimensional plane, for example, a representative value of data within the depth range is set as a pixel value on the two-dimensional plane. Techniques can be used. Here, the representative value can include a value such as an average value, a median value, or a maximum value of pixel values within a range in the depth direction of a region surrounded by two reference planes. Further, the depth range related to the En-Face image is, for example, a range including a predetermined number of pixels in a deeper direction or a shallower direction with respect to one of the two layer boundaries relating to the detected retinal layer. May be good. Further, the depth range related to the En-Face image may be, for example, a range changed (offset) according to an operator's instruction from a range between two layer boundaries related to the detected retinal layer. Good.

The imaging device is a device for capturing an image used for diagnosis. The photographing device detects, for example, a device that obtains an image of a predetermined part by irradiating a predetermined part of the subject with radiation such as light or X-rays, electromagnetic waves, ultrasonic waves, or the like, or radiation emitted from the subject. This includes a device for obtaining an image of a predetermined part. More specifically, the imaging devices according to the various examples and modifications described above include at least an X-ray imaging device, a CT device, an MRI device, a PET device, a SPECT device, an SLO device, an OCT device, an OCTA device, and a fundus. Includes cameras, endoscopes, etc.

The OCT apparatus may include a time domain OCT (TD-OCT) apparatus and a Fourier domain OCT (FD-OCT) apparatus. Further, the Fourier domain OCT apparatus may include a spectral domain OCT (SD-OCT) apparatus and a wavelength sweep type OCT (SS-OCT) apparatus. Further, the SLO device and the OCT device may include a wave surface compensation SLO (AO-SLO) device using a wave surface compensation optical system, a wave surface compensation OCT (AO-OCT) device, and the like. Further, the SLO device and the OCT device may include a polarized SLO (PS-SLO) device, a polarized OCT (PS-OCT) device, and the like for visualizing information on polarization phase difference and polarization elimination.

(Other Examples)
The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiment to a system or device via a network or a storage medium, and a process in which a computer of the system or device reads and executes the program. is there. The computer may have one or more processors or circuits and may include a plurality of separate computers or a network of separate processors or circuits for the computers to read and execute executable instructions. The processor or circuit may include a central processing unit (CPU), a microprocessing unit (MPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a field programmable gateway (FPGA). Also, the processor or circuit may include a digital signal processor (DSP), a data flow processor (DFP), or a neural processing unit (NPU). It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

C2,900: Imaging optical system, 18,980: Control unit, 201,1001: Drive control unit, 202,1002: Display control unit, 210,1010: Acquisition unit, 220,1020: Control unit, 221,1021: Output Unit, 222: Selection unit, 1023: Image generation unit

Claims (24)

The first driving means for driving the photographing optical unit for photographing the subject,
A second driving means for driving the optical member included in the photographing optical unit, and
A control means for controlling the first drive means and the second drive means,
An output means for outputting information on copying loss in the medical image obtained by photographing the subject using the trained model from the medical image obtained by photographing the subject by the photographing optical unit.
A photographing device characterized by being provided with. Using the trained model, the output means uses the trained model as information regarding the copying loss, and when the medical image obtained by taking the picture has a copying loss, the cause of the copying loss in the medical image obtained by taking the picture is determined. The first aspect of claim 1 is to output and output that the medical image obtained by taking a picture does not have a picture loss when the medical image obtained by taking the picture does not have a picture loss. The imaging device described. The imaging according to claim 1 or 2, wherein the trained model is obtained by using training data in which a medical image of a subject is used as input data and information on copying loss in the medical image is used as output data. apparatus. The output data is any one of a plurality of information regarding copying loss, and is
The third aspect of claim 3, wherein the output means outputs information on the copying loss by using a plurality of values corresponding to the information on the plurality of copying loss obtained by using the trained model. Shooting device. The output means further obtains the first medical image in which the image loss in the photographed medical image is reduced when the photographed medical image has a copy loss. The imaging device according to claim 1 or 2, wherein a control value of at least one of the driving means and the second driving means is output. The first driving means for driving the photographing optical unit for photographing the subject,
A second driving means for driving the optical member included in the photographing optical unit, and
A control means for controlling the first drive means and the second drive means,
An input means for inputting information on copying loss in a medical image obtained by photographing a subject by the photographing optical unit, and an input means.
From the medical image obtained by taking the picture, the image was taken using the learned model corresponding to the input information about the picture loss among a plurality of learned models corresponding to the causes of the picture loss. An output means that outputs a control value of at least one of the first driving means and the second driving means for acquiring a medical image with reduced copying loss in the medical image, and an output means.
A photographing device characterized by being provided with. In the plurality of trained models, the medical image of the subject is used as input data, and the first driving means for driving the photographing optical unit for photographing the subject and the second driving means for driving the optical member included in the photographing optical unit are driven. The photographing apparatus according to claim 6, wherein the photographing apparatus is obtained by using learning data having at least one control value of the driving means as output data. The output data is one of a plurality of control values, and is
The output means uses a plurality of values corresponding to the plurality of control values obtained by using the trained model corresponding to the input information regarding the copying loss, and the copying loss in the medical image obtained by photographing the image. The imaging apparatus according to claim 7, wherein a control value of at least one of the first driving means and the second driving means for acquiring a medical image in which is reduced is output. When the copying loss in the medical image obtained by the photographing is the copying loss caused by the state of the subject photographed by the photographing optical unit, the output means is taken from the medical image obtained by the photographing. The method according to any one of claims 5 to 8, wherein zero can be output as a control value of at least one of the first driving means and the second driving means without using the trained model. The imaging device according to item 1. A selection means for selecting whether to control at least one of the first driving means and the second driving means by the control means based on the control value or according to an instruction from the user. The photographing apparatus according to any one of claims 5 to 9, further comprising. A medical image obtained by taking a picture when it is selected to control at least one of the first driving means and the second driving means by the control means in response to an instruction from a user. The claim is further provided with a means for performing additional learning of the trained model by using at least one of the control values of the first driving means and the second driving means after the control. Item 10. The photographing apparatus according to item 10. The optical member has a focusing member arranged on the optical path in the photographing optical unit.
When the cause of the copying loss in the medical image obtained by the imaging is related to the position of the focusing member on the optical path, the output means has the control value, and the second driving means has the focusing member. The photographing apparatus according to any one of claims 5 to 11, wherein the moving amount for moving the image from the first position to the second position on the optical path is output. When the information regarding the copying loss is related to the distance between the photographing optical unit and the subject, the output means sets the distance between the photographing optical unit and the subject as the control value, and the first driving means uses the first driving means. The imaging according to any one of claims 5 to 12, characterized in that the amount of movement of the photographing optical unit by the first driving means when changing from the first distance to the second distance is output. apparatus. 13. The output means according to claim 13, wherein the output means outputs positive and negative directions along the optical path of the movement amount based on whether the captured image of the subject contains redness or bluishness. Ophthalmic equipment. The photographing optical unit is the return light from the subject irradiated with the measurement light obtained by dividing the light from the light source and the reference light obtained by dividing the light from the light source. Has an interfering optical system that interferes with the reference light corresponding to
When the cause of the image loss in the medical image obtained by the imaging is related to the optical path length difference between the optical path length of the measurement light and the optical path length of the reference light, the output means uses the second control value as the control value. The method according to any one of claims 5 to 14, wherein when the driving means of the above changes the optical path length difference from the first difference to the front second difference, the change amount of the optical path length difference is output. The imaging device described. The imaging device according to claim 15, wherein the output means outputs positive and negative of the amount of change in the optical path length difference according to the position of a predetermined portion in the tomographic image of the photographed subject. Further provided with a storage means for storing the output control value,
When the subject is re-photographed by the photographing optical unit, the control means controls the first driving means and the second driving means by using the stored control value. The photographing apparatus according to any one of claims 5 to 16. The control means is characterized in that after controlling at least one of the first driving means and the second driving means by using the output control value, the photographed subject is re-photographed. The photographing apparatus according to any one of claims 5 to 17. The imaging apparatus according to any one of claims 1 to 18, further comprising a display control means for displaying information on image loss in the medical image obtained by photographing the image on the display means. The display control means controls at least one of the first drive means and the second drive means for acquiring a medical image in which the copying loss in the photographed medical image is reduced. The photographing apparatus according to claim 19, wherein the display means displays the image in a state corresponding to the information regarding the copying loss. The subject is an eye to be inspected and
The information regarding the image loss includes at least one of misalignment, poor focus, underexposure, ambiguity associated with a disease, and blinking of the eye to be inspected. The photographing apparatus according to any one item. The first driving means for driving the photographing optical unit for photographing the subject,
A second driving means for driving the optical member included in the photographing optical unit, and
A method for controlling an imaging device including the first driving means and the controlling means for controlling the second driving means.
A photographing apparatus including a step of outputting information on copying loss in the photographed medical image by using a learned model from a medical image obtained by photographing the subject by the photographing optical unit. Control method. The first driving means for driving the photographing optical unit for photographing the subject,
A second driving means for driving the optical member included in the photographing optical unit, and
A method for controlling an imaging device including the first driving means and the controlling means for controlling the second driving means.
A step of inputting information on copying loss in a medical image obtained by photographing a subject by the photographing optical unit, and
From the medical image of the photographed subject, the image was taken using the trained model corresponding to the input information on the image loss among the plurality of learned models corresponding to the causes of the plurality of image loss. It is characterized by including a step of outputting at least one control value of the first driving means and the second driving means for acquiring a medical image with reduced copying loss in the obtained medical image. How to control the imaging device. A program that, when executed by a processor, causes the processor to perform each step of the imaging device control method according to claim 22 or 23.

Images