WO2022163362A1

WO2022163362A1 - Image processing device, image processing method, and surgical microscope system

Info

Publication number: WO2022163362A1
Application number: PCT/JP2022/000855
Authority: WO
Inventors: 知之大月
Original assignee: ソニーグループ株式会社
Priority date: 2021-01-29
Filing date: 2022-01-13
Publication date: 2022-08-04
Also published as: JP2022116559A

Abstract

An image processing device (13) according to an embodiment of the present disclosure comprises an image input unit (13c) that receives an operating field image of a patient eye, a three-dimensional model reception unit (13a) that receives a three-dimensional model of some or all of the lens of the eye, a stereo image generating unit (13b) that generates a stereo image of the lens from the three-dimensional model, and a display image generating unit (13d) that generates a display image including the operating field image and the stereo image on the basis of the operating field image and the stereo image.

Description

Image processing device, image processing method and surgical microscope system

The present disclosure relates to an image processing device, an image processing method, and a surgical microscope system.

As a method of refractive correction in ophthalmology, by inserting an artificial lens called an intraocular lens (IOL) into the eye, it is widely used to eliminate the refractive error of the crystalline lens and improve visual functions such as visual acuity. It is done. The most widely used intraocular lens is an intraocular lens that is inserted into the lens capsule as a replacement for the lens removed by cataract surgery. In addition to the lens capsule, there are various intraocular lenses such as those that are fixed (dwelled) in the ciliary sulcus (Phakic IOL).

When an operator performs an ophthalmic surgery to place an intraocular lens in the eye, it is difficult for the operator to grasp the positional relationship between the surgical tool and the target area of the eye while referring to the image of the surgical field. Surgical tools may cause unintended injury to the eye. For example, when an operator is performing cataract surgery by grooving the nucleus of the lens, the posterior capsule may be damaged by the surgical instrument because the operator does not know the depth relationship between the surgical instrument and the posterior capsule of the lens. Therefore, in Patent Literature 1, a tomographic image of the eye is used to facilitate understanding of the positional relationship between the surgical tool and the treatment target portion of the eye.

WO2017/065018

However, in order to grasp the positional relationship between the surgical tool and the part of the eye to be treated, when using tomographic images, an optical coherence tomography (OCT) for taking tomographic images should be attached to the surgical microscope. It becomes necessary to provide such a device, which complicates the device and increases the size of the lens barrel.

Therefore, in the present disclosure, an image processing apparatus and an image processing apparatus that are capable of simplifying the apparatus configuration and reducing the size of the apparatus while making it easier for the operator to understand the positional relationship between the surgical tool and the region to be treated of the eye. A method and an operating microscope system are proposed.

An image processing apparatus according to an embodiment of the present disclosure includes an image input unit that receives an operative field image of a patient's eye, a three-dimensional model reception unit that receives a three-dimensional model of part or all of the lens of the eye, A stereo image generator that generates a stereo image of the crystalline lens from the three-dimensional model, and a display image generator that generates a display image including the surgical field image and the stereo image based on the surgical field image and the stereo image. And prepare.

In an image processing method according to an embodiment of the present disclosure, an image processing device receives an operative field image of a patient's eye, receives a three-dimensional model of part or all of the lens of the eye, and from the three-dimensional model generating a stereo image of the lens; and generating a display image including the surgical field image and the stereo image based on the surgical field image and the stereo image.

A surgical microscope system according to an embodiment of the present disclosure includes a surgical microscope that obtains an surgical field image of a patient's eye, an image processing device that generates a display image, and a display device that displays the display image, wherein the image is The processing device includes an image input unit that receives the operative field image, a three-dimensional model reception unit that receives a three-dimensional model of part or all of the lens of the eye, and a stereo image of the lens from the three-dimensional model. and a display image generator for generating the display image including the surgical field image and the stereo image based on the surgical field image and the stereo image.

1 is a diagram showing an example of a schematic configuration of a surgical microscope system according to an embodiment of the present disclosure; FIG. 1 is a diagram showing an example of a schematic configuration of a surgical microscope according to an embodiment of the present disclosure; FIG. 1 is a diagram illustrating an example of a schematic configuration of an image processing device according to an embodiment of the present disclosure; FIG. FIG. 4 is a diagram showing an example of a tomographic image for 3D model generation according to an embodiment of the present disclosure; FIG. 3 is a diagram illustrating an example of a stereoscopic image for 3D model generation according to an embodiment of the present disclosure; FIG. 3 illustrates an example stereo image corresponding to a 3D model according to an embodiment of the present disclosure; FIG. 3 is a diagram showing Example 1 of a display image according to the embodiment of the present disclosure; FIG. FIG. 10 is a diagram showing Example 2 of a display image according to the embodiment of the present disclosure; FIG. 10 is a diagram showing Example 3 of a display image according to the embodiment of the present disclosure; FIG. 10 is a diagram showing Example 4 of a display image according to the embodiment of the present disclosure; FIG. 11 is a diagram showing Example 5 of a display image according to an embodiment of the present disclosure; 1 is a diagram illustrating an example of a schematic configuration of a computer according to an embodiment of the present disclosure; FIG.

Below, embodiments of the present disclosure will be described in detail based on the drawings. Note that the apparatus, method, system, etc. according to the present disclosure are not limited by this embodiment. Further, in each of the following embodiments, basically the same parts are denoted by the same reference numerals, thereby omitting duplicate descriptions.

Each of one or more embodiments (including examples and modifications) described below can be implemented independently. On the other hand, at least some of the embodiments described below may be implemented in combination with at least some of the other embodiments as appropriate. These multiple embodiments may include novel features that differ from each other. Therefore, these multiple embodiments can contribute to solving different purposes or problems, and can produce different effects.

The present disclosure will be described according to the order of items shown below.
1. Embodiment 1-1. Example of schematic configuration of surgical microscope system 1-2. Example of schematic configuration of surgical microscope 1-3. Schematic Configuration of Image Processing Apparatus and Example of Image Processing 1-4. Action and effect 2. An example of a schematic configuration of a computer3. Supplementary note

<1. embodiment>
<1-1. Example of schematic configuration of surgical microscope system>
An example of a schematic configuration of a surgical microscope system 1 according to this embodiment will be described with reference to FIG. FIG. 1 is a diagram showing an example of a schematic configuration of a surgical microscope system 1 according to this embodiment.

As shown in FIG. 1, the surgical microscope system 1 has a surgical microscope 10 and a patient bed 20. This surgical microscope system 1 is a system used for eye surgery. The patient undergoes eye surgery while lying on the patient bed 20 . An operator, who is a doctor, performs surgery while observing the patient's eye through the surgical microscope 10 .

The surgical microscope 10 has an objective lens 11, an eyepiece lens 12, an image processing device 13, and a monitor 14.

The objective lens 11 and the eyepiece lens 12 are lenses for magnifying and observing the eye of the patient to be operated.

The image processing device 13 outputs various images, various information, etc. by performing predetermined image processing on the image captured through the objective lens 11 .

The monitor 14 displays an image captured through the objective lens 11, various images generated by the image processing device 13, various information, and the like. This monitor 14 may be provided separately from the surgical microscope 10 .

In this surgical microscope system 1, for example, the operator looks into the eyepiece 12 and performs surgery while observing the patient's eye through the objective lens 11. Further, the operator performs surgery while confirming various images (for example, an image before image processing, an image after image processing, etc.) and various information displayed on the monitor 14 .

<1-2. Example of schematic configuration of surgical microscope>
An example of the schematic configuration of the surgical microscope 10 according to this embodiment will be described with reference to FIG. FIG. 2 is a diagram showing an example of a schematic configuration of the surgical microscope 10 according to this embodiment.

As shown in FIG. 2, the surgical microscope 10 includes, in addition to the objective lens 11, the eyepiece lens 12, the image processing device 13, and the monitor 14, a light source 51, an observation optical system 52, a front image photographing section 53, and a display It has a section 54 , an interface section 55 and a speaker 56 . Note that the monitor 14 and the presentation unit 54 correspond to a display device.

The light source 51 emits illumination light under the control of the control unit 13A included in the image processing device 13 to illuminate the eyes of the patient.

The observation optical system 52 is composed of optical elements such as the objective lens 11, a half mirror 52a, and lenses (not shown). The observation optical system 52 guides the light (observation light) reflected from the patient's eye to the eyepiece 12 and the front image capturing section 53 .

Specifically, the light reflected from the patient's eye enters the half mirror 52a as observation light via the objective lens 11, a lens (not shown), or the like. Approximately half of the observation light incident on the half mirror 52 a passes through the half mirror 52 a as it is, and enters the eyepiece 12 via the transmissive presentation unit 54 . On the other hand, the other half of the observation light incident on the half mirror 52 a is reflected by the half mirror 52 a and enters the front image capturing section 53 .

The front image capturing unit 53 is composed of, for example, a video camera. The front image photographing unit 53 receives the observation light incident from the observation optical system 52 and photoelectrically converts it to obtain an image of the patient's eye observed from the front, that is, an image of the patient's eye photographed approximately in the eye axis direction. A front image is taken. The front image capturing unit 53 captures (captures) a front image under the control of the image processing device 13 and supplies the obtained front image to the image processing device 13 .

The eyepiece 12 collects the observation light incident from the observation optical system 52 via the presentation unit 54 and forms an optical image of the patient's eye. An optical image of the patient's eye is thereby observed by the operator looking through the eyepiece 12 .

The presentation unit 54 is composed of a transmissive display device or the like, and is arranged between the eyepiece 12 and the observation optical system 52 . The presentation unit 54 transmits observation light incident from the observation optical system 52 and makes it enter the eyepiece 12, and also displays various images (for example, a front image, a stereo image, etc.) and various information supplied from the image processing device 13. are also presented (displayed) as necessary. Various images, various information, and the like may be presented, for example, superimposed on the optical image of the patient's eye, or may be presented in the periphery of the optical image so as not to interfere with the optical image.

The image processing device 13 has a control section 13A that controls the operation of the surgical microscope 10 as a whole. For example, the control section 13A changes the illumination conditions of the light source 51 or changes the zoom magnification of the observation optical system 52 . Further, the control unit 13A controls image acquisition by the front image photographing unit 53 based on the operation information of the operator or the like supplied from the interface unit 55 and the like.

The interface unit 55 is composed of, for example, a communication unit and the like. The communication unit receives commands from an operation unit such as a touch panel superimposed on the monitor 14, a controller, a remote controller (not shown), or the like, and communicates with external devices. The interface unit 55 supplies the image processing apparatus 13 with information and the like according to the operation of the operator. The interface unit 55 also outputs device control information for controlling the external device, which is supplied from the image processing apparatus 13, to the external device.

The monitor 14 displays various images such as front images and stereo images and various information according to control by the control unit 13A of the image processing device 13 .

For example, when a dangerous situation is detected during surgery, the speaker 56 emits a buzzer sound, a melody sound, or the like in order to notify the operator of the dangerous situation in accordance with the control by the control unit 13A of the image processing device 13. Outputs sound, message (voice), and the like. Note that the surgical microscope 10 may be provided with a rotating light or indicator light (lamp) for informing the operator or the like of a dangerous situation.

In the surgical microscope system 1 configured as described above, a display image is generated by the image processing device 13 based on various images such as a front image and a stereo image, and the generated display image is displayed on the monitor 14, the presentation unit 54, or the like. Displaying by the device allows the operator to grasp the positional relationship between the surgical instrument and the surgical target site of the eye.

<1-3. Schematic Configuration of Image Processing Apparatus and Example of Image Processing>
A schematic configuration of the image processing apparatus 13 according to the present embodiment and an example of image processing will be described with reference to FIG. FIG. 3 is a diagram showing an example of a schematic configuration (configuration and processing flow) of the image processing apparatus 13 according to this embodiment.

In the following, an example of the case where the treatment target site is the lens capsule will be explained. In particular, an inexperienced operator does not know the relationship between the surgical tool and the depth of the posterior capsule, and the surgical tool may damage the posterior capsule. In order to eliminate such difficulty in understanding the depth relationship, stereo image presentation processing based on a 3D model obtained by preoperatively measuring the lens (which may include the lens capsule) will be described.

As shown in FIG. 3, the image processing device 13 includes a 3D model reception unit (three-dimensional model reception unit) 13a, a stereo image generation unit 13b, an image input unit 13c, and a display image generation unit 13d. there is

The 3D model receiving unit 13a receives from an external device a 3D model of the treatment target site measured preoperatively in the preoperative planning. The 3D model may include only the site to be treated, or may include other sites for size adjustment and posture adjustment, which will be described later. In addition, the 3D model may include the corneal limbus for size matching based on the cornea in the surgical field image, or may include the iris for posture matching using the iris pattern. The 3D model may also include a preoperative frontal image of the target eye whose positional relationship with the 3D model is known for use in alignment. For example, blood vessels around the cornea in the preoperative front image can be used for posture adjustment.

FIG. 4 is a diagram showing an example of a tomographic image for 3D model generation according to this embodiment. FIG. 5 is a diagram showing an example of a stereoscopic image (a stereoscopic image viewed from a specific direction) for generating a 3D model according to this embodiment. One or both of a tomogram (multiple tomograms) and a stereoscopic image are used to construct a 3D model. This 3D model is supplied to the stereo image generator 13b.

In the examples of FIGS. 4 and 5, the 3D model is obtained by OCT (Optical Coherence Tomography), so the 3D model of the lens behind the iris is missing. For example, if an ultrasonic diagnostic apparatus is used, it is possible to obtain a 3D model of the lens without chipping. It is also possible to prepare a 3D reference model, compensate for missing portions from the reference model, and obtain a 3D model of the crystalline lens with no defects. For the sake of simplicity, the following description is based on the premise that a 3D model of the lens without chipping has been obtained.

Here, OCT irradiates the eye to be treated with light such as near-infrared light, reconstructs the reflected waves from each tissue of the eye, and creates an image (a tomographic image that is a cross-sectional image in the depth direction of the eye). It is a technology to generate. In addition to images captured by OCT, images (tomographic images) captured by a Scheimpflug camera may be used for the 3D model.

Returning to FIG. 3, the stereo image generating unit 13b generates two images of the crystalline lens viewed from two specific viewpoints (corresponding to the left eye and the right eye) based on the 3D model received by the 3D model receiving unit 13a. Generate a stereo image (stereogram) corresponding to the model. The stereo image generator 13b supplies the stereo image of the treatment target site to the display image generator 13d.

FIG. 6 is a diagram showing an example of a stereo image corresponding to the 3D model according to this embodiment. As shown in FIG. 6, the stereo image shows, for example, the lens (which may include the lens capsule) by contour lines. A crystalline lens corresponds to a crystalline lens alone or a crystalline lens having a lens capsule.

In the example of FIG. 6, the stereo image is an image with contour lines to improve visibility, but is not limited to this, and may be an image that gives the operator a 3D feel. In the example of FIG. 6 and the examples of FIGS. 7 to 11 below, only one side of the two images forming the stereo image is shown for the sake of simplification of explanation. shown.

Returning to FIG. 3, the image input unit 13c receives an operating field image (front image) from the front image capturing unit 53 (see FIG. 2), and receives the operating field image (for example, an operating field image at the start of surgery or an operating field image). real-time operating field image, etc.) is supplied to the display image generation unit 13d. In addition, the image input unit 13c supplies the operative field image to the stereo image generation unit 13b as necessary.

The display image generation unit 13d generates a display image by adding the stereo image of the treatment target site supplied from the stereo image generation unit 13b to the surgical field image supplied from the image input unit 13c. For example, the display image generation unit 13d superimposes and synthesizes the stereo image on the surgical field image to generate a display image including the surgical field image and the stereo image.

(Display image example 1)
FIG. 7 is a diagram showing Example 1 of a display image according to the present embodiment. In the example of FIG. 7, the stereo image is positioned on the upper right side of the operative field image (in FIG. 7) and superimposed on the operative field image to generate the display image. This display image is displayed by both or one of the monitor 14 and the presentation unit 54 .

Here, although there are individual differences in the shape of the lens, the operator cannot grasp the depth of the transparent posterior lens capsule from only the front image of the surgical field image received by the image input unit 13c. The operator can grasp the three-dimensional shape of the crystalline lens by visually recognizing the stereo image showing the outline of the crystalline lens. This makes it easier for the operator to grasp the positional relationship between the surgical tool and the lens capsule.

In addition, in example 1 of the display image described above, the size, three-dimensional posture, and position of the stereo image of the treatment target site presented in the display image are not aligned with the surgical field image, but they may all be aligned. In this case, it is possible to superimpose the outline of the stereoscopic image of the crystalline lens on the surgical field image, so that the positional relationship between the surgical instrument and the posterior capsule becomes very easy to understand. Even when the sizes, postures, and positions are aligned independently, the ease of grasping the positional relationship is improved, although it is inferior to the case where all are aligned. Regarding the posture among these, when the three-dimensional posture is aligned, it is particularly easy to grasp the positional relationship. It helps to accurately grasp the shape of the lens, which has anisotropic shape around the axis, and as a result, the grasp of the positional relationship is improved.

(Display image example 2)
FIG. 8 is a diagram showing example 2 of a display image according to the present embodiment. In the example of FIG. 8, the stereo image is positioned on the upper right side of the operative field image (in FIG. 8) and superimposed on the operative field image to generate the display image. A stereo image is generated so that its size and orientation are the same as those of the surgical target site (lens) in the operative field image.

Regarding the size, for example, the size of the stereo image of the treatment target area is aligned with the size of the treatment target area in the surgical field image. In order to make these sizes uniform, for example, the display image generator 13d may adjust the size of the cornea in the operating field image and the size of the cornea in the 3D model to be the same.

In this way, the size of the treatment target area matches the surgical field image, making it easier for the operator to guess the positional relationship between the treatment target area and the surgical tool. For example, when grooving the lens nucleus, the depth of the lens is known, so the operator can easily predict the depth relationship between the surgical tool and the posterior capsule.

Regarding the posture, for example, the posture of the stereo image of the treatment target site is adjusted so that it is three-dimensionally aligned with the posture of the eye in the surgical field image. As a reference for aligning the postures, the iris pattern and the blood vessel pattern in the corneal periphery in both the surgical field image and the 3D model may be used. For example, the display image generation unit 13d matches the patterns of the iris or blood vessels in both the operative field image and the 3D model, and aligns the posture of the stereo image of the treatment target region three-dimensionally with the posture of the eye in the operative field image. good too.

In this way, the stereo image of the treatment target area and the posture of the treatment target area in the surgical field image are aligned, so the operator can easily guess the positional relationship between the treatment target area and the surgical tool.

In addition, based on the relative relationship around the eye axis between the stereo image of the treatment target area and the image of the surgical field, the orientation around the eye axis (the position in the rotational direction) may be adjusted. For example, the orientation around the eye axis may be adjusted so that the upper eye region (portion) in the displayed image in the stereo image of the treatment target site is the same as the surgical field image. As a reference for aligning the directions, the iris pattern and the corneal peripheral blood vessel pattern in both the surgical field image and the 3D model may be used. For example, the display image generation unit 13d matches the patterns of the iris or blood vessels in both the operative field image and the 3D model, and changes the orientation of the stereo image of the treatment target region around the eye axis to the direction around the eye axis of the eye in the operative field image. You can align it. The direction around the eye axis is defined, for example, by the angle in the direction of rotation around the eye axis with respect to a reference line orthogonal to the eye axis.

In this way, by aligning the directions around the eye axis in the stereo image of the treatment target area and the treatment target area in the surgical field image, it becomes easier to guess the shape of the surgical field, and the operator can use the surgical tool and the treatment target. It becomes easier to guess the positional relationship of parts. For example, the treatment target site does not necessarily have the same shape around the eye axis. Therefore, it is effective to align the directions around the eye axis.

(Display image example 3)
FIG. 9 is a diagram showing Example 3 of a display image according to this embodiment. In the example of FIG. 9, the stereo image is positioned on the treatment target region of the surgical field image and superimposed on the surgical field image to generate the display image. A stereo image is generated so that the size, posture, and position are the same as those of the treatment target region (lens) in the surgical field image.

Regarding the position, position alignment is performed so that the position of the stereo image of the treatment target area matches the position of the treatment target area in the surgical field image. As a reference for aligning the positions, the iris pattern and the corneal peripheral blood vessel pattern in both the surgical field image and the 3D model may be used. For example, the display image generator 13d may match the iris or blood vessel patterns in both the surgical field image and the 3D model, and align the position of the stereo image of the treatment target site with the position of the eye in the surgical field image.

In this way, the stereo image is displayed superimposed on the treatment target site in the surgical field, making it easier for the operator to guess the positional relationship between the surgical tool and the treatment target site. In particular, when the sizes and three-dimensional postures of the stereo images and the regions to be treated in the surgical field are the same, the operator can observe the actual positional relationship instead of guessing.

(Display image example 4)
FIG. 10 is a diagram showing Example 4 of a display image according to this embodiment. In the example of FIG. 10, a part of the corresponding stereo image and a part of the treatment target part of the operative field image are enlarged and positioned at the upper left of the operative field image (in FIG. 10). A display image is generated by being superimposed on the operative field image.

It should be noted that FIG. 10 shows an example of enlarged presentation of the surgical field image and the stereo image of the treatment target region, which are of the same size only. In the example of FIG. 10, two stereo images of the treatment target site are presented, but only the enlarged stereo image may be presented.

In this way, by presenting the area to be treated to the operator in an enlarged manner, the operator can observe the area to be treated in detail, making it easier to understand the positional relationship between the surgical tool and the area to be treated. In addition, if the sizes, three-dimensional postures, and positions of the surgical field image and the stereo image of the treatment target area are aligned in the enlarged image, the positional relationship between the surgical tool and the treatment target area will be particularly easy to understand. Clarity is improved without it. In addition, even if only one side of the operative field image and the stereo image of the treatment target area is enlarged and presented, it is inferior to the case where both are presented, but the clarity is improved. You may make it enlarge a site|part.

In addition, when generating a stereo image of the treatment target site described in the above example, the same observation optical system as that for acquiring the surgical field image is used to capture the same stereo inward angle. Adjustments may be made to obtain results.

Furthermore, the three-dimensional position of the surgical tool estimated from the parallax information in the surgical field image, and the shape information of the 3D model of the treatment target site whose size, three-dimensional posture and position are aligned with the surgical field image, are used to determine the surgical tool. and the treatment target site may be calculated, and information about the calculated distance and warning information about warning when the distance is short may be presented. In this case, since the information is presented based on the measurement results rather than subjectively by the operator, the separation distance between the surgical tool and the treatment target site becomes easy to understand. Note that only the distance information may be presented without presenting the stereo image of the treatment target site. There are various methods of presenting information about this distance.

(Display image example 5)
FIG. 11 is a diagram showing example 5 of a display image according to the present embodiment. In the example of FIG. 11, the separation distance is presented by a bar-shaped indicator. This enables the operator to grasp the separation distance between the surgical tool and the treatment target site.

For example, the display image generation unit 13d aligns the size, orientation, and position of the lens of the stereo image with the size, orientation, and position of the lens of the surgical field image, and calculates the separation distance between the lens of the stereo image and the surgical instrument. Then, the display image generation unit 13d generates a bar-shaped indicator as separation distance information regarding the calculated separation distance and adds it to the display image.

Note that the display image generation unit 13d may change the display mode, such as the color of the vicinity of the distal end of the surgical instrument and the color of the image frame attached to the periphery of the image, according to the separation distance. For example, the color may be green when the surgical tool is far from the site to be treated, and may change to orange or red as the surgical tool and the site to be treated are closer. Also, the sound may be presented by the speaker 56 so that the frequency of the presented sound increases as the surgical tool and the site to be treated approach.

Also, instead of presenting the distance information, a warning image or voice may be presented when the distance becomes less than a certain value. For example, the control unit 13A of the image processing device 13 determines whether or not the separation distance is equal to or less than a predetermined value, and if it is determined that the separation distance is equal to or less than the predetermined value, the controller instructs the display device to present a warning image. and control to instruct the speaker 56 to present audio.

For example, the display image generation unit 13d aligns the size, orientation, and position of the lens in the stereo image with the size, orientation, and position of the lens in the surgical field image in accordance with the control by the control unit 13A, and displays the lens in the stereo image and the surgical tool. Calculate the distance between Then, according to the calculated separation distance, when the separation distance becomes equal to or less than a predetermined value, the display image generation unit 13d generates and displays warning information (for example, sentences, symbols, graphics, etc.) indicating that the separation distance is close. Add to image.

Furthermore, when the separation distance falls below a certain value, complications such as posterior capsule damage occur in the form of stopping the ultrasonic vibration of the tip of the surgical instrument by controlling the associated ultrasonic emulsification suction device. may be prevented.

For example, the control unit 13A of the image processing device 13 determines whether or not the separation distance is equal to or less than a predetermined value. Performs control to instruct operation stop such as stopping ultrasonic vibration.

Various display images as described above are used, and these display images may be selectable by the operator, assistant, or the like. The selection of the display image is realized by an input operation on the operation unit by the operator or the assistant. For example, an operator, an assistant, or the like operates an operation unit to select a display mode for displaying a desired display image. According to this selection, the display image generator 13d generates a display image based on the selected display mode. Similarly, regarding various images, the size, position, etc. of the images may be changed by the operator, the assistant, or the like. The display image generation unit 13d generates a display image by changing the size, position, etc. of the image according to the input operation to the operation unit by the operator or the assistant.

<1-4. Action/Effect>
As described above, according to the present embodiment, the image input unit 13c receives the operative field image, and the 3D model receiving unit 13a receives a three-dimensional model of part or all of the crystalline lens of the patient's eye. , the stereo image generation unit 13b generates a stereo image of the crystalline lens from the three-dimensional model, and the display image generation unit 13d generates a display image including the operative field image and the stereo image based on the operative field image and the stereo image. As a result, display images including the operative field image and the stereo image can be presented to the operator. This makes it easier for the operator to understand the positional relationship between the surgical tool and the region of the eye to be treated. Therefore, it is possible for the operator to easily understand the positional relationship between the surgical instrument and the region of the eye to be treated, while simplifying the configuration of the device and reducing the size of the device.

In addition, the display image generation unit 13d generates a display image by aligning the size of the lens of the stereo image with the size of the lens of the surgical field image. This makes it possible to present a display image in which the size of the crystalline lens in the stereo image is the same as the size of the crystalline lens in the surgical field image. It can be easily understood.

In addition, the display image generation unit 13d generates a display image by aligning the orientation of the lens of the stereo image around the eye axis with the orientation of the lens of the surgical field image around the eye axis. This makes it possible to present a display image in which the orientation of the lens in the stereo image around the eye axis is the same as the orientation of the lens in the surgical field image around the eye axis. The positional relationship with the target part can be made easier to understand.

In addition, the display image generation unit 13d generates a display image by aligning the orientation of the lens of the stereo image with the orientation of the lens of the surgical field image. This makes it possible to present a display image in which the orientation of the crystalline lens in the stereo image is the same as the orientation of the crystalline lens in the surgical field image. It can be easily understood.

In addition, the display image generation unit 13d generates a display image by superimposing the stereo image on the operative field image. As a result, it is possible to present a display image in which the crystalline lens of the stereo image and the crystalline lens of the surgical field image are brought closer to each other, so that the operator can more easily understand the positional relationship between the surgical instrument and the surgical target site of the eye.

In addition, the display image generation unit 13d aligns the position of the lens of the stereo image with the position of the lens of the surgical field image to generate the display image. This makes it possible to present a display image in which the position of the crystalline lens in the stereo image is the same as the position of the crystalline lens in the surgical field image. It can be easily understood.

In addition, the display image generation unit 13d expands one or both of the lens of the stereo image and the lens of the surgical field image to generate a display image. As a result, it is possible to present a display image in which one or both of the crystalline lens of the stereo image and the crystalline lens of the surgical field image are magnified, so that the operator can better understand the positional relationship between the surgical tool and the region of the eye to be treated. It can be easily understood.

In addition, the display image generation unit 13d aligns the size, orientation, and position of the lens of the stereo image with the size, orientation, and position of the lens of the surgical field image, calculates the separation distance between the lens of the stereo image and the surgical tool, and calculates Spacing distance information about the spacing distance is generated and added to the display image. As a result, it is possible to present a display image in which the size, orientation, and position of the lens in the stereo image are the same as the size, orientation, and position of the lens in the surgical field image, and which includes separation distance information. It is possible to make the positional relationship between the surgical tool and the eye to be treated easier to understand.

In addition, the display image generation unit 13d aligns the size, orientation, and position of the lens of the stereo image with the size, orientation, and position of the lens of the surgical field image, calculates the separation distance between the lens of the stereo image and the surgical tool, and calculates Warning information is generated according to the separation distance and added to the display image. As a result, it is possible to present warning information based on the measurement results rather than the operator's subjectivity.

<2. Example of schematic configuration of computer>
The series of processes described above can be executed by hardware or by software. When executing a series of processes by software, a program that constitutes the software is installed in the computer. Here, the computer includes, for example, a computer built into dedicated hardware and a general-purpose personal computer capable of executing various functions by installing various programs.

FIG. 12 is a diagram showing an example of a schematic configuration of a computer 500 that executes the series of processes described above by a program.

As shown in FIG. 12, the computer 500 has a CPU (Central Processing Unit) 510, a ROM (Read Only Memory) 520, and a RAM (Random Access Memory) 530.

The CPU 510 , ROM 520 and RAM 530 are interconnected by a bus 540 . An input/output interface 550 is also connected to the bus 540 . An input unit 560 , an output unit 570 , a recording unit 580 , a communication unit 590 and a drive 600 are connected to the input/output interface 550 .

The input unit 560 is composed of a keyboard, mouse, microphone, imaging device, and the like. The output unit 570 is configured with a display, a speaker, and the like. The recording unit 580 is composed of a hard disk, a nonvolatile memory, or the like. The communication unit 590 is configured by a network interface or the like. A drive 600 drives a removable recording medium 610 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory.

In the computer 500 configured as described above, the CPU 510 loads, for example, the program recorded in the recording unit 580 into the RAM 530 via the input/output interface 550 and the bus 540, and executes it. A series of processes are performed.

A program executed by the computer 500, that is, the CPU 510 can be provided by being recorded on a removable recording medium 610 such as a package medium, for example. Also, the program can be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

In the computer 500 , the program can be installed in the recording unit 580 via the input/output interface 550 by loading the removable recording medium 610 into the drive 600 . Also, the program can be received by the communication unit 590 and installed in the recording unit 580 via a wired or wireless transmission medium. In addition, the program can be installed in the ROM 520 or the recording unit 580 in advance.

It should be noted that the program executed by the computer 500 may be a program in which processing is performed in chronological order according to the order described in this specification, or in parallel or at a necessary timing such as when a call is made. It may be a program in which processing is performed in

Also, in this specification, a system means a set of multiple components (devices, modules (parts), etc.), and it does not matter whether all the components are in the same housing. Therefore, a plurality of devices housed in separate housings and connected via a network, and a single device housing a plurality of modules in one housing, are both systems. .

Further, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the present technology.

For example, this technology can take the configuration of cloud computing in which one function is shared by multiple devices via a network and processed jointly.

In addition, each step described in the above process flow can be executed by a single device or shared by a plurality of devices.

Furthermore, when one step includes multiple processes, the multiple processes included in the one step can be executed by one device or shared by multiple devices.

Also, the effects described in this specification are merely examples and are not limited, and there may be effects other than those described in this specification.

<3. Note>
Note that the present technology can also take the following configuration.
(1)
an image input for receiving an operative field image for the patient's eye;
a three-dimensional model receiving unit that receives a three-dimensional model of part or all of the lens of the eye;
a stereo image generator that generates a stereo image of the crystalline lens from the three-dimensional model;
a display image generation unit that generates a display image including the surgical field image and the stereo image based on the surgical field image and the stereo image;
An image processing device comprising:
(2)
The display image generation unit is
aligning the size of the lens of the stereo image with the size of the lens of the surgical field image to generate the display image;
The image processing apparatus according to (1) above.
(3)
The display image generation unit is
generating the display image by aligning the orientation of the lens of the stereo image around the eye axis with the orientation of the lens of the surgical field image around the eye axis;
The image processing apparatus according to (1) or (2) above.
(4)
The display image generation unit is
generating the display image by aligning the orientation of the lens in the stereo image with the orientation of the lens in the surgical field image;
The image processing apparatus according to any one of (1) to (3) above.
(5)
The display image generation unit is
generating the display image by superimposing the stereo image on the operative field image;
The image processing apparatus according to any one of (1) to (4) above.
(6)
The display image generation unit is
generating the display image by aligning the position of the lens in the stereo image with the position of the lens in the surgical field image;
The image processing apparatus according to (5) above.
(7)
The display image generation unit is
magnifying one or both of the lens of the stereo image and the lens of the surgical field image to generate the display image;
The image processing apparatus according to any one of (1) to (6) above.
(8)
The display image generation unit is
Aligning the size, orientation, and position of the lens in the stereo image with the size, orientation, and position of the lens in the surgical field image, calculating the separation distance between the lens in the stereo image and the surgical instrument, and calculating the calculated separation generating separation distance information about the distance and adding it to the displayed image;
The image processing apparatus according to any one of (1) to (7) above.
(9)
The display image generation unit is
Aligning the size, orientation, and position of the lens in the stereo image with the size, orientation, and position of the lens in the surgical field image, calculating the separation distance between the lens in the stereo image and the surgical instrument, and calculating the calculated separation generating warning information according to the distance and adding it to the display image;
The image processing apparatus according to any one of (1) to (8) above.
(10)
The image processing device
receiving surgical field images for the patient's eye;
receiving a three-dimensional model of part or all of the lens of the eye;
generating a stereo image of the lens from the three-dimensional model;
generating a display image including the operative field image and the stereo image based on the operative field image and the stereo image;
An image processing method comprising:
(11)
a surgical microscope for obtaining an operative field image of the patient's eye;
an image processing device that generates a display image;
a display device for displaying the display image;
with
The image processing device is
an image input unit that receives the operative field image;
a three-dimensional model receiving unit that receives a three-dimensional model of part or all of the lens of the eye;
a stereo image generator that generates a stereo image of the crystalline lens from the three-dimensional model;
a display image generation unit that generates the display image including the surgical field image and the stereo image based on the surgical field image and the stereo image;
A surgical microscope system having
(12)
An image processing method using the image processing apparatus according to any one of (1) to (9) above.
(13)
A surgical microscope system comprising the image processing device according to any one of (1) to (9) above.

1 Operating Microscope System 10 Operating Microscope 11 Objective Lens 12 Eyepiece Lens 13 Image Processing Device 13a 3D Model Receiving Section (3D Model Receiving Section)
13b stereo image generation unit 13c image input unit 13d display image generation unit 14 monitor 20 patient bed 51 light source 52 observation optical system 52a half mirror 53 front image capturing unit 54 presentation unit 55 interface unit 56 speaker 500 computer 510 CPU
520 ROMs
530 RAM
540 bus 550 input/output interface 560 input section 570 output section 580 recording section 590 communication section 600 drive 610 removable recording medium

Claims

an image input for receiving an operative field image for the patient's eye;
a three-dimensional model receiving unit that receives a three-dimensional model of part or all of the lens of the eye;
a stereo image generator that generates a stereo image of the crystalline lens from the three-dimensional model;
a display image generation unit that generates a display image including the surgical field image and the stereo image based on the surgical field image and the stereo image;
An image processing device comprising:
The display image generation unit is
aligning the size of the lens of the stereo image with the size of the lens of the surgical field image to generate the display image;
The image processing apparatus according to claim 1.
The display image generation unit is
generating the display image by aligning the orientation of the lens of the stereo image around the eye axis with the orientation of the lens of the surgical field image around the eye axis;
The image processing apparatus according to claim 1.
The display image generation unit is
generating the display image by aligning the orientation of the lens in the stereo image with the orientation of the lens in the surgical field image;
The image processing apparatus according to claim 1.
The display image generation unit is
generating the display image by superimposing the stereo image on the operative field image;
The image processing apparatus according to claim 1.
The display image generation unit is
generating the display image by aligning the position of the lens in the stereo image with the position of the lens in the surgical field image;
The image processing apparatus according to claim 5.
The display image generation unit is
magnifying one or both of the lens of the stereo image and the lens of the surgical field image to generate the display image;
The image processing apparatus according to claim 1.
The display image generation unit is
Aligning the size, orientation, and position of the lens in the stereo image with the size, orientation, and position of the lens in the surgical field image, calculating the separation distance between the lens in the stereo image and the surgical instrument, and calculating the calculated separation generating separation distance information about the distance and adding it to the displayed image;
The image processing apparatus according to claim 1.
The display image generation unit is
Aligning the size, orientation, and position of the lens in the stereo image with the size, orientation, and position of the lens in the surgical field image, calculating the separation distance between the lens in the stereo image and the surgical instrument, and calculating the calculated separation generating warning information according to the distance and adding it to the display image;
The image processing apparatus according to claim 1.
The image processing device
receiving surgical field images for the patient's eye;
receiving a three-dimensional model of part or all of the lens of the eye;
generating a stereo image of the lens from the three-dimensional model;
generating a display image including the operative field image and the stereo image based on the operative field image and the stereo image;
An image processing method comprising:
a surgical microscope for obtaining an operative field image of the patient's eye;
an image processing device that generates a display image;
a display device for displaying the display image;
with
The image processing device is
an image input unit that receives the operative field image;
a three-dimensional model receiving unit that receives a three-dimensional model of part or all of the lens of the eye;
a stereo image generator that generates a stereo image of the crystalline lens from the three-dimensional model;
a display image generation unit that generates the display image including the surgical field image and the stereo image based on the surgical field image and the stereo image;
A surgical microscope system having