JP2020137103A - Information processing system and information processing program - Google Patents

Abstract

To provide an information processing system and an information processing program capable of identifying an object that is hard to be discriminated.SOLUTION: According to an embodiment of the present information, an information processing system includes an acquisition part, and a display part. The acquisition part acquires position information in a picked-up image of an identification object or an area including the identification object by performing object detection in a segmentation image obtained from the picked-up image including the identification object. The display part displays a video obtained by overlapping a marker generated on the basis of the position information acquired by the acquisition part on the picked-up image.SELECTED DRAWING: Figure 3

Description

The present invention relates to an information processing system and an information processing program for identifying an object that is difficult to identify.

Eye surgery is commonly used to treat eye disorders such as cataracts. When performing eye surgery, the healthcare professional covers the patient's face with a drape to expose the eye to be operated on through the opening of the drape (see Non-Patent Document 1).

https: // www. tsukazaki-hp. jp / care / ophthalmology / ai

If the patient's face is covered with drapes, it can be difficult to tell whether the eyes exposed from the drapes are the left or right eye. Therefore, the medical staff may mistakenly think that the eye of the person performing the surgery is exposed when the eye of the person who does not perform the surgery is mistakenly exposed. The problem of such misidentification is a problem that can occur in a field other than the medical field. Therefore, it is desirable to provide an information processing system and an information processing program capable of identifying an object that is difficult to identify.

The first information processing system according to the embodiment of the present invention includes an acquisition unit and a display unit. The acquisition unit acquires the position information in the captured image of the identification target or the region including the identification target by detecting the object on the Segmentation image obtained from the captured image including the identification target. The display unit displays an image in which a marker generated based on the position information acquired by the acquisition unit is superimposed on the captured image.

The second information processing system according to the embodiment of the present invention includes an acquisition unit and a synthesis unit. The acquisition unit acquires the position information in the captured image of the identification target or the region including the identification target by detecting the object on the Segmentation image obtained from the captured image including the identification target. The synthesizing unit superimposes the marker generated based on the position information acquired by the acquiring unit on the image of the surgical field including the identification target formed by the optical system.

A third information processing system according to an embodiment of the present invention includes an acquisition unit and a transmission unit. The acquisition unit acquires the position information in the captured image of the identification target or the region including the identification target by detecting the object on the Segmentation image obtained from the captured image including the identification target. The transmission unit transmits the position information acquired by the acquisition unit to the information processing device provided with the display unit.

The first information processing program according to an embodiment of the present invention causes a computer to perform the following two steps.
(1) Acquisition step of acquiring the position information in the captured image of the identification target or the region including the identification target by detecting the object on the Segmentation image obtained from the captured image including the identification target (2) Acquisition step Generation step of generating a video signal for displaying an image in which the marker generated based on the position information acquired in is superimposed on the captured image.

The second information processing program according to one embodiment of the present invention causes a computer to perform the following two steps.
(1) Acquisition step (2) Acquisition step of acquiring the position information in the captured image of the identification target or the region including the identification target by detecting the object on the Segmentation image obtained from the captured image including the identification target. Generation step of generating a video signal for generating projected light for superimposing the marker generated based on the position information acquired in the above on the image of the surgical field including the identification target formed by the optical system.

A third information processing program according to an embodiment of the present invention causes a computer to perform the following two steps.
(1) Acquisition step of acquiring the position information in the captured image of the identification target or the region including the identification target by detecting the object on the Segmentation image obtained from the captured image including the identification target (2) Acquisition step Transmission step of transmitting the position information acquired in step 2 to an information processing device equipped with a display unit.

In the first information processing system, the second information processing system, the third information processing system, the first information processing program, the second information processing program, and the third information processing program according to the embodiment of the present invention. By performing object detection on the Information Processing image obtained from the captured image including the identification target, the position information in the captured image of the identification target or the region including the identification target is acquired. As a result, for example, the marker generated based on the acquired position information is displayed as an image superimposed on the captured image, or superimposed on the image of the surgical field including the identification target formed by the optical system. It becomes possible. As a result, the elements necessary for identifying the identification target can be reliably included in the captured image, so that the object that is difficult to identify can be identified.

It is a figure which shows an example of the operative field image in which the marker for the position adjustment of an imaging region is superposed. It is a figure which shows an example of the operative field image in which the marker for the position adjustment of an imaging region is superposed. It is a figure which shows an example of the schematic structure of the left-right eye discrimination system which concerns on 1st Embodiment of this invention. It is a figure which shows an example of the functional block of the left-right eye identification system of FIG. It is a figure which shows an example of the image data stored in the memory of a server apparatus. It is a figure which shows an example of the Segmentation image. It is a flow chart which shows an example of the right-and-left eye identification procedure in the left-right eye identification system of FIG. It is a figure which shows an example of the Detection image obtained from the Segmentation image of FIG. It is a figure which shows an example of the image displayed on the display surface of FIG. It is a figure which shows an example of the image displayed on the display surface of FIG. It is a figure which shows an example of the image displayed on the display surface of FIG. It is a figure which shows an example of the image displayed on the display surface of FIG. It is a figure which shows an example of the image displayed on the display surface of FIG. It is a figure which shows an example of the image displayed on the display surface of FIG. It is a figure which shows one modification of the functional block of the left-right eye identification system of FIG. It is a flow chart which shows an example of the right-and-left eye identification procedure in the left-right eye identification system of FIG. It is a figure which shows one modification of the schematic structure of the left-right eye identification system of FIG. It is a figure which shows an example of the functional block of the left-right eye identification system of FIG. It is a flow chart which shows an example of the right-and-left eye identification procedure in the left-right eye identification system of FIG. It is a figure which shows one modification of the schematic structure of the left-right eye identification system of FIG. It is a figure which shows an example of the functional block of the left-right eye identification system of FIG. It is a figure which shows an example of the functional block of the left-right eye identification system of FIG. It is a figure which shows one modification of the schematic structure of the left-right eye identification system of FIG. It is a figure which shows an example of the functional block of the left-right eye identification system of FIG. It is a figure which shows an example of the schematic structure of the left-right eye identification system which concerns on the 2nd Embodiment of this invention. It is a figure which shows an example of the functional block of the left-right eye identification system of FIG. It is a figure which shows an example of the image emitted from the projection part of FIG. It is a figure which shows an example of the image emitted from the projection part of FIG. It is a figure which shows an example of the image emitted from the projection part of FIG. It is a figure which shows an example of the image emitted from the projection part of FIG. It is a figure which shows an example of the image emitted from the projection part of FIG. It is a figure which shows an example of the image which is visually recognized by the eyepiece part of FIG. It is a flow chart which shows an example of the right-and-left eye identification procedure in the left-right eye identification system of FIG. It is a figure which shows one modification of the functional block of the left-right eye identification system of FIG. FIG. 3 is a flow chart showing an example of a left-right eye identification procedure in the left-right eye identification system of FIG. 34. It is a figure which shows one modification of the schematic structure of the left-right eye identification system of FIG. It is a figure which shows an example of the functional block of the left-right eye identification system of FIG. It is a flow chart which shows an example of the right-and-left eye identification procedure in the left-right eye identification system of FIG.

Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the drawings. The following description is a specific example of the present invention, and the present invention is not limited to the following aspects. Further, the present invention is not limited to the arrangement, dimensions, dimensional ratio, etc. of each component shown in each figure. Further, in the following description, the directions such as up / down, left / right, front / back, etc. are the directions seen from the medical staff side unless otherwise specified. Regarding the vertical direction, the direction from the objective lens described later toward the surgical field is downward, and the opposite direction is upward. Generally, the patient is operated on his back, so the vertical and vertical directions are the same. The description will be given in the following order.
1. 1. Background of the invention 2. First Embodiment (FIGS. 1 to 14)
The server device generates segmentation images and identifies the left and right eyes.
Example of performing brightness judgment, focus judgment, generation image generation and position judgment with an information processing device. Modification example of the first embodiment Modification A (FIGS. 15 and 16)
An example in which brightness judgment, focus judgment, generation image generation and position judgment are also performed by a server device (an example in which an information processing device functions as an interface).
Modification B (FIGS. 17 to 19)
Example of omitting the server device Modification C (FIGS. 20 to 22)
In the first embodiment and the modified example A,
Example of transmitting an captured image to the information processing device by a communication different from the communication between the information processing device and the server device Modification D (FIGS. 23 and 24)
Example of omitting the server device in the modified example C 4. Second Embodiment (FIGS. 25-33)
A server device is used to generate Segmentation images and identify the left and right eyes.
An example of performing brightness judgment, focus judgment, generation image generation and position judgment with an information processing device with a camera. Modification example of the second embodiment Modification E (FIGS. 34 and 35)
An example in which brightness judgment, focus judgment, generation image generation and position judgment are also performed by a server device (an example in which an information processing device with a camera functions as an interface)
Modification F (FIGS. 36 to 38)
Example of omitting the server device

<1. Background of the invention>
1 and 2 show an example of a surgical field image Ia (captured image) in which markers Mc for adjusting the position of an imaging region are superimposed. In the surgical field image Ia shown in FIGS. 1 and 2, the eye 10 which is the identification target in the left and right eye identification system (hereinafter, simply referred to as “left and right eye identification system”) according to the embodiment of the present invention is included. include. The eye 10 includes an eyeball 11 and a caruncle 12. The caruncle 12 exists at the end (inner corner of the eye) of the eye 10 near the nose. The surgical field images Ia shown in FIGS. 1 and 2 correspond to, for example, the state of the surgical field when a medical worker stands beside the patient's head and looks into a microscope installed above the patient's head. .. Therefore, as shown in FIG. 1, when the caruncle 12 and the inner corner of the eye are included in the image (Detection image Ib) obtained by performing object detection (Detection) on the surgical field image Ia. For example, by superimposing the frame of the Detection image Ib on the surgical field image Ia as a marker and displaying it, the medical staff can intuitively grasp whether or not the entire eye 10 is within the imaging region. be able to. In this case, the medical staff adjusts the imaging region so that the marker is located within the predetermined region in the surgical field image Ia, so that the entire eye 10 falls within the predetermined region in the surgical field image Ia. Whether the eye 10 shown in the field image Ia is the left eye or the right eye can be correctly identified by the left and right eye identification system.

However, when an object is actually detected on the surgical field image Ia, for example, as shown in FIG. 2, the Detection image Ib includes the eyeball 11, but does not include the caruncle 12 or the inner corner of the eye. .. Therefore, as shown in FIG. 2, when the frame of the Detection image Ib is superimposed on the surgical field image Ia as a marker and displayed, the medical staff can determine whether or not the entire eye 10 is within the imaging region. It can be difficult to grasp intuitively. In this case, even if the medical staff adjusts the imaging region so that the marker is located within the predetermined region in the surgical field image Ia, a part of the eye 10 is out of the predetermined region in the surgical field image Ia. There is a possibility that the left and right eye identification system cannot correctly distinguish whether the eye 10 shown in the surgical field image Ia is the left eye or the right eye. Therefore, the present invention proposes a method for ensuring that the Deception image Ib contains an element that facilitates identification of the left and right eyes (for example, the caruncle 12 and the inner corner of the eye).

<2. First Embodiment>
[Constitution]
FIG. 3 shows an example of a schematic configuration of the left and right eye identification system 100 according to the first embodiment of the present invention. The left and right eye identification system 100 corresponds to a specific example of the "information processing system" of the present invention. FIG. 4 shows an example of the functional blocks of the left and right eye identification system 100. In FIG. 3, a microscope system 200 for ophthalmic surgery for a medical professional to magnify and visually recognize a surgical field is illustrated together with a left-right eye discrimination system 100. In the present embodiment, the surgical field is an area of the face of the patient 400 that is exposed within the opening 610 of the drape 600 that covers the face of the patient 400. The surgical field includes the eye on which surgery is performed. The patient 400 is lying on his back on the operating table 300, the body of the patient 400 is covered with sheets 500, and the face of the patient 400 is covered with a drape 600.

(Microscope system for ophthalmic surgery 200)
The microscope system 200 for ophthalmic surgery includes, for example, a support 210, a first arm 220, a second arm 230, a drive device 240, a medical professional microscope 250, and a foot switch 260.

The stanchions 210 support the entire ophthalmologic surgical microscope system 200. One end of the first arm 220 is connected to the upper end of the support column 210. One end of the second arm 230 is connected to the other end of the first arm 220. A drive device 240 is connected to the other end of the second arm 230. The medical professional microscope 250 is suspended by a drive device 240. The camera 120 used in the left and right eye identification system 100 is attached to the microscope 250 for medical staff. The foot switch 260 is used to perform various operations with the feet of a medical worker or the like. The drive device 240 moves the microscope 250 for medical workers and the camera 120 three-dimensionally in the vertical direction and the horizontal direction in response to an operation by a medical worker or the like.

The microscope 250 for medical professionals has a lens barrel portion 251 for accommodating various optical systems, a drive system, and the like, and a pair of left and right eyepiece portions 252. The medical worker looks into the eyepiece 252 and observes the surgical field with both eyes through the lens barrel 251. The lens barrel portion 251 includes, for example, an objective lens, an illumination optical system, an observation optical system, and a photographing optical system. The observation optical system is an optical system connected to the objective lens, and the photographing optical system is an optical system connected to the camera 120.

The illumination optical system illuminates the surgical field through the objective lens. The observation optical system is used for observing the surgical field illuminated by the illumination optical system with a microscope 250 for medical staff through an objective lens. A pair of left and right observation optical systems are provided, the right observation optical system is connected to the right eyepiece 252, and the left observation optical system is connected to the left eyepiece 252. The observation optical system on the right side is provided with, for example, a beam splitter, and a part of the light guiding the observation optical system on the right side from the surgical field is guided to the imaging optical system by the beam splitter.

(Left and right eye identification system 100)
The left and right eye identification system 100 includes, for example, an information processing device 110, a camera 120, and a server device 130. The information processing device 110 and the server device 130 are connected to each other via the network 140. The information processing device 110 is configured to be able to communicate with the server device 130 via the network 140. The server device 130 is configured to be able to communicate with the information processing device 110 via the network 140. The information processing device 110 and the camera 120 are connected via the network 140. The information processing device 110 is configured to be able to communicate with the camera 120 via the network 140. The camera 120 is configured to be able to communicate with the information processing device 110 via the network 140.

The network 140 is, for example, a network that communicates using a communication protocol (TCP / IP) that is standardly used on the Internet. The network 140 may be, for example, a secure network that communicates using a communication protocol unique to the network. The network 140 is, for example, the Internet, an intranet, or a local area network. The connection between the network 140 and the information processing device 110, the server device 130, or the camera 120 may be, for example, a wired LAN (Local Area Network) such as Ethernet, a wireless LAN such as Wi-Fi, or a mobile phone. It may be a line or the like.

The information processing device 110 includes, for example, a control unit 111, a memory 112, a display unit 113, an input unit 114, and a network IF (Interface) 115. In the present embodiment, the control unit 111 corresponds to a specific example of the "acquisition unit" of the present invention. The display unit 113 corresponds to a specific example of the “display unit” of the present invention. The control unit 111 is configured to include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and the like, and executes, for example, a web browser program stored in the memory 112, an operating system, and the like (not shown). The control unit 111 further executes, for example, the program 112A stored in the memory 112. The program 112A corresponds to a specific example of the "information processing program" of the present invention. Program 112A causes control unit 111 to perform a series of procedures for identifying the type of eye (left eye or right eye) included in the surgical field. A series of procedures for the program 112A to execute the control unit 111 will be described in detail later.

The memory 112 stores a program (for example, a web browser program or an operating system) executed by the control unit 111. The memory 112 is composed of, for example, a RAM (Random Access Memory), a ROM (Read Only Memory), an auxiliary storage device (hard disk, etc.), and the like. The memory 112 further stores the program 112A.

The display unit 113 includes, for example, a display device such as a liquid crystal panel or an organic EL (Electroluminescence) panel. The display unit 113 displays an image based on the video signal from the control unit 111 on the display surface 110A. The input unit 114 receives an instruction from the outside (for example, a user) and outputs the received instruction to the control unit 111. The input unit 114 may be, for example, a mechanical input interface including buttons, dials, or the like, or a voice input interface including a microphone or the like. The input unit 114 may be, for example, a touch panel provided on the display surface 110A of the information processing device 110. The network IF 115 is a communication interface for communicating with the server device 130 via the network 140.

The camera 120 is configured to include, for example, a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary metal-oxide-semiconductor) image sensor. For example, the camera 120 takes an image under the control of the information processing device 110, and outputs the image (operative field image Ia) obtained thereby to the information processing device 110 via the network 140.

The server device 130 includes, for example, a control unit 131, a segmentation learning model 132, a left / right eye discrimination learning model 133, a memory 134, a network IF 135, and the like. The left-right eye discrimination learning model 133 corresponds to a specific example of the "discrimination unit" of the present invention. The control unit 131 is configured to include a CPU, a GPU, and the like, and executes, for example, a web server program stored in the memory 134, an operating system, and the like. The control unit 131 further transmits, for example, the identification result obtained by the left and right eye identification learning model 133 to the information processing device 110 via the network IF135.

The network IF 135 is a communication interface for communicating with the information processing device 110 via the network 140. The memory 134 stores a program (for example, a web server program or an operating system) executed by the control unit 131. The memory 134 is composed of, for example, a RAM, a ROM, an auxiliary storage device (hard disk, etc.) and the like. The memory 134 further stores a plurality of image data 134A, a plurality of image data 134B, and a plurality of surgical data 134C.

Each image data 134A is one of the two types of learning data input to the Segmentation learning model 132, and is one of the two types of learning data input to the left / right eye discrimination learning model 133. There is one. Each image data 134A is, for example, a surgical field image Ia obtained by the camera 120 as shown in FIGS. 1 and 2. Each image data 134B is one of two types of training data input to the Segmentation learning model 132. For example, as shown in FIG. 5, the meaning indicated by the pixels of each image data 134A is labeled. It is a attached Segmentation image. The image data 134B shown in FIG. 5 is composed of pixels labeled as the eye region Ra and pixels labeled as the peripheral region Rb. Each surgical data 134C is surgical data of a patient, and includes, for example, a patient name, a patient ID, a disease name, a surgical date and time, a surgical eye (left eye, right eye), a surgical procedure, a doctor in charge, and the like.

In the learning process, the Segmentation learning model 132 performs machine learning using a set of image data 134A and 134B as learning data. The Segmentation learning model 132 generates and holds information on the correspondence between the image data 134A and the image data 134B as a learning result as a result of machine learning. In the segmentation learning model 132, when the surgical field image Ia obtained by the camera 120 is input in the conversion process, the surgical field image Ia is converted into the segmentation image Ic by using the learning result (see FIG. 6).

In the learning process, the left-right eye discrimination learning model 133 performs machine learning using the image data 134A and the type of eye (left eye or right eye) included in the image data 134A as learning data. The left-right eye discrimination learning model 133 generates and holds information about the correspondence between the image data 134A and the eye type as a learning result as a result of machine learning. In the discrimination process, the left-right eye discrimination learning model 133 identifies the type of eye (left eye or right eye) included in the surgical field image Ia by using the learning result when the surgical field image Ia is acquired.

[motion]
Next, an example of the operation of the left and right eye identification system 100 according to the present embodiment will be described.

(Machine learning)
The user instructs the Segmentation learning model 132 to start learning via the input unit 114. Then, the Segmentation learning model 132 executes the learning process. Specifically, the Segmentation learning model 132 performs machine learning using a plurality of sets of image data 134A and image data 134B stored in the memory 134 as training data. The Segmentation learning model 132 generates and holds information on the correspondence between the image data 134A and the image data 134B as a learning result as a result of machine learning. In this way, machine learning by the Segmentation learning model 132 is completed.

Next, the user instructs the left and right eye discrimination learning model 133 to start learning via the input unit 114. Then, the left and right eye discrimination learning model 133 executes the learning process. Specifically, the left-right eye discrimination learning model 133 informs the input request of the eye type (left eye, right eye) for each image data 134A together with the thumbnails and identifiers of the plurality of image data 134A stored in the memory 134. It is transmitted to the processing device 110. If there is no problem such as communication speed, the left and right eye identification learning model 133 may transmit the original image data 134A instead of the thumbnail to the information processing device 110. When the information processing apparatus 110 receives an input request for an eye type (left eye, right eye) for each image data 134A together with thumbnails and identifiers of a plurality of image data 134A from the left and right eye discrimination learning model 133, the plurality of received information devices 110 The thumbnail of the image data 134A of the above is displayed on the display unit 113, and a display prompting the user to input the type of eye (left eye, right eye) is displayed on the display unit 113. The user inputs the eye type (left eye, right eye) to the input unit 114 for each image data 134A according to the display of the display unit 113. The input unit 114 transmits the eye type (left eye, right eye) for each image data 134A input by the user to the left / right eye discrimination learning model 133 together with the identifier of the image data 134A. When the left and right eye discrimination learning model 133 receives the eye type (left eye, right eye) for each image data 134A from the information processing device 110 together with the identifier of the image data 134A, the eye type (left eye, right eye) And the correspondence relationship with the identifier of the image data 134A is generated and held. In this way, machine learning by the left and right eye discrimination learning model 133 is completed.

(Left and right eye identification)
FIG. 7 is a flow chart showing an example of the left / right eye identification procedure in the left / right eye identification system 100. First, the user instructs the start of left / right eye identification via the input unit 114. The user instructs, for example, to start the program 112A via the input unit 114. Then, the information processing device 110 (control unit 111) reads the program 112A from the memory 112 and loads it into the control unit 111. The control unit 111 loaded with the program 112A executes a series of steps for distinguishing the left and right eyes. Specifically, first, the control unit 111 outputs an imaging instruction to the camera 120. The camera 120 images the inside of the opening of the drape 600 according to the imaging instruction, and transmits the surgical field image Ia obtained thereby to the information processing device 110.

The information processing device 110 (control unit 111) determines the brightness of the received surgical field image Ia. The information processing device 110 (control unit 111) converts, for example, the surgical field image Ia into HSV, and when the V value of each pixel obtained by the conversion is out of a predetermined range, the surgical field image Ia is bright. The display unit 113 indicates that the image is insufficient or too bright. The information processing device 110 (control unit 111) performs focus determination, for example, when the V value of each obtained pixel is within a predetermined range. For example, the information processing device 110 (control unit 111) performs Laplacian differentiation on the surgical field image Ia, and when the score obtained by the Laplacian differentiation is lower than a predetermined threshold value, the surgical field image Ia is out of focus. The fact that the information is displayed is displayed on the display unit 113. The information processing device 110 (control unit 111) transmits, for example, the surgical field image Ia to the server device 130 when the derived score is equal to or higher than a predetermined threshold value.

When the server device 130 receives the surgical field image Ia from the information processing device 110, the segmentation learning model 132 executes the conversion process. Specifically, the Segmentation learning model 132 converts the surgical field image Ia into the Segmentation image Ic by using the learning result (see FIG. 6). The segmentation learning model 132 transmits the segmentation image Ic obtained by the conversion to the information processing device 110. The information processing device 110 (control unit 111) detects an object on the received Segmentation image Ic. As shown in FIG. 8, the information processing device 110 (control unit 111) cuts out a rectangular region including the eye region Ra (for example, a rectangular region in contact with the eye region Ra) from the segmentation image Ic, for example. .. The image obtained in this way is referred to as a Detection image Id.

The information processing device 110 (control unit 111) uses the position information of the obtained Detection image Id in the surgical field image Ia as the position information in the surgical field image Ia of the region including the eye to be identified. The information processing device 110 (control unit 111) uses, for example, the position information of the outer edge portion of the obtained Detection image Id in the surgical field image Ia as the position information of the marker Mc. That is, the information processing device 110 (control unit 111) acquires the position information in the surgical field image Ia of the region including the eye to be identified by detecting the object on the Segmentation image Ic.

The information processing device 110 (control unit 111) may use the position information of the obtained Detection image Id in the surgical field image Ia as the position information of the eye to be identified in the surgical field image Ia. The information processing device 110 (control unit 111) may use, for example, the position information in the surgical field image Ia of the outer edge portion of the eye region Ra of the obtained Detection image Id as the position information of the marker Mc described later. In this case, the information processing device 110 (control unit 111) acquires the position information of the eye to be identified in the surgical field image Ia by detecting the object on the segmentation image Ic.

The information processing device 110 (control unit 111) generates a marker Mc based on the acquired position information, and then displays an image (marker image Ie) in which the generated marker Mc is superimposed on the surgical field image Ia. A signal is generated and output to the display unit 113. For example, as shown in FIG. 9, the display unit 113 displays an image (marker image Ie) in which the marker Mc is superimposed on the surgical field image Ia based on the video signal from the control unit 111 on the display surface 110A. ..

The left and right eye identification system 100 periodically performs a series of procedures from the above-mentioned imaging instruction to the marker Mc display. At this time, the medical staff fine-tunes the imaging area of the camera 120 while referring to the image displayed on the display surface 110A. Whether or not the position of the marker Mc in the surgical field image Ia protrudes from the predetermined region each time the information processing device 110 (control unit 111) executes a series of procedures from the above-mentioned imaging instruction to the marker Mc display. Is determined. As a result, when the position of the marker Mc in the surgical field image Ia protrudes from the predetermined area, the information processing apparatus 110 (control unit 111) may, for example, display the position of the marker Mc in the surgical field image Ia in the predetermined area. A video signal for displaying an image including information meaning that it protrudes from the image is generated and output to the display unit 113. The display unit 113 displays on the display surface 110A an image including information indicating that the position of the marker Mc in the surgical field image Ia protrudes from a predetermined region and the surgical field image Ia. For example, as shown in FIG. 10, the display unit 113 displays an image including the status ST such as “microscope position is deviated”, the marker Mc, and the surgical field image Ia on the display surface 110A. .. The information processing device 110 (control unit 111) uses a method other than displaying the image on the display surface 110A (for example, sound, lighting of a patrol lamp, etc.) to deviate the position of the marker Mc from the predetermined region in the surgical field image Ia. You may inform your healthcare professional that you are doing so.

On the other hand, when the position of the marker Mc in the surgical field image Ia is within a predetermined area, the information processing apparatus 110 (control unit 111) causes the position of the marker Mc in the surgical field image Ia to be within the predetermined area. A video signal for displaying an image indicating the existence is generated and output to the display unit 113. The display unit 113 includes, for example, information indicating that the position of the marker Mc in the surgical field image Ia is within a predetermined region based on the video signal from the control unit 111, and the surgical field image Ia. The image is displayed on the display surface 110A. For example, as shown in FIG. 11, the display unit 113 displays an image including the status ST such as “microscope position OK”, the marker Mc, and the surgical field image Ia on the display surface 110A. The information processing device 110 (control unit 111) uses a method other than displaying an image on the display surface 110A (for example, sound, lighting of a patrol lamp, etc.) to position the marker Mc in a predetermined area in the surgical field image Ia. You may inform your healthcare professional that there is.

When the position of the marker Mc in the surgical field image Ia is within a predetermined region, the information processing device 110 (control unit 111) refers to the left and right eyes with respect to the left and right eye discrimination learning model 133 in the server device 130. Output the identification instruction. When the left / right eye discrimination learning model 133 receives the left / right eye discrimination instruction, the left / right eye discrimination learning model 133 executes the discrimination process. Specifically, the left-right eye discrimination learning model 133 uses the learning result to discriminate the eyes included in the Detection image Ib. More specifically, the left-right eye discrimination learning model 133 uses the learning result to identify the type of eye (left eye, right eye) included in the Detection image Ib. The left and right eye discrimination learning model 133 transmits the discrimination result DR to the information processing device 110. The control unit 111 generates a video signal for displaying an image including the status ST, the identification result DR, the marker Mc, and the surgical field image Ia on the display surface 110A, and outputs the video signal to the display unit 113. For example, as shown in FIG. 12, the display unit 113 displays an image including the status ST, the identification result DR, the marker Mc, and the surgical field image Ia based on the video signal from the control unit 111. Display on 110A. For example, the display unit 113 may superimpose the identification result DR on the surgical field image Ia as shown in FIG. 12, or perform the identification result DR as shown in FIG. 13, for example. It may be displayed in an area different from the field image Ia.

By looking at the identification result DR displayed on the display surface 110A, the medical staff can determine whether the eye exposed in the opening 610 of the drape 600 is the right eye or the left eye, and by extension, the eye to be operated on. Can be confirmed. In this way, left and right eye discrimination is performed.

The information processing device 110 (control unit 111) further reads, reads out, and includes the surgical data 134C corresponding to the patient 400 from the plurality of surgical data 134C stored in the memory 134, for example. The surgical eye is compared with the identification result DR. The information processing device 110 (control unit 111) displays a video signal for displaying an image including the comparison result CR, the status ST, the identification result DR, the marker Mc, and the surgical field image Ia on the display surface 110A. Generate and output to the display unit 113. For example, as shown in FIG. 14, the display unit 113 displays the comparison result CR, the status ST, the identification result DR, the marker Mc, and the surgical field image Ia based on the video signal from the control unit 111. The including image is displayed on the display surface 110A. Note that FIG. 14 illustrates, as the comparison result CR, the display when the surgical eye read and included in the surgical data 134C and the identification result DR match.

[effect]
Next, the effect of the left and right eye identification system 100 according to the present embodiment will be described.

Eye surgery is commonly used to treat eye disorders such as cataracts. When performing eye surgery, the healthcare professional covers the patient's face with a drape, exposing the eye to be operated on through the opening of the drape. If the patient's face is covered with drapes, it can be difficult to tell whether the eyes exposed from the drapes are the left or right eye. Therefore, the medical staff may mistakenly think that the eye of the person performing the surgery is exposed when the eye of the person who does not perform the surgery is mistakenly exposed.

On the other hand, in the present embodiment, the eye to be identified or the eye to be identified is included by detecting the object on the Segmentation image Ic obtained from the surgical field image Ia including the eye to be identified. The position information of the region in the surgical field image Ia is acquired. Further, in the present embodiment, an image (marker image Ie) in which the marker Mc generated based on the acquired position information is superimposed on the surgical field image Ia is displayed. As a result, the marker Mc can surely include an element (for example, the caruncle 12 and the inner corner of the eye) that facilitates the identification of the left and right eyes. As a result, the medical worker adjusts the imaging region of the camera 120 so that the marker Mc is located within a predetermined region in the surgical field image Ia, so that the eye reflected in the surgical field image Ia is the left eye. The left and right eye identification system 100 can correctly identify whether the eye is the right eye or the right eye. Therefore, it is possible to identify the type of eye that is difficult to identify.

Further, in the present embodiment, the discrimination result DR by the left and right eye discrimination learning model 133 is displayed on the display surface 110A of the display unit 113. Further, in the present embodiment, the identification result DR by the left and right eye identification learning model 133 is displayed on the display surface 110A of the display unit 113 together with the surgical field image Ia. As a result, it is possible to perform reliable confirmation work without oversight.

Further, in the present embodiment, by performing object detection on the Segmentation image Ic, after the Detection image Id is cut out from the Segmentation image Ic, the position information of the cut out Detection image Id in the surgical field image Ia is obtained. , The position information in the surgical field image Ia of the eye to be identified or the region including the eye to be identified. As a result, the object is simply detected for the surgical field image Ia, and the position information of the eye to be identified in the surgical field image Ia is derived based on the detection image obtained thereby. , The marker Mc can surely include an element (for example, the caruncle 12 and the inner corner of the eye) that facilitates the identification of the left and right eyes. As a result, the medical worker adjusts the imaging region of the camera 120 so that the marker Mc is located within a predetermined region in the surgical field image Ia, so that the eye reflected in the surgical field image Ia is the left eye. The left and right eye identification system 100 can correctly identify whether the eye is the right eye or the right eye. Therefore, it is possible to identify the type of eye that is difficult to identify.

Further, in the present embodiment, when the marker Mc protrudes from the predetermined region in the surgical field image Ia, it includes information (for example, status ST) meaning that the marker Mc protrudes from the predetermined region. The image is displayed on the display surface 110A of the display unit 113. This makes it possible to reliably inform the medical staff that the imaging area of the camera 120 needs to be fine-tuned.

Further, in the present embodiment, after the surgical field image Ia obtained when the marker Mc is within a predetermined area in the surgical field image Ia is transmitted to the server device 130, the identification result of the left and right eyes is transmitted to the server device. The information processing apparatus 110 is provided with a network IF 115 that receives from the 130. As a result, a device (server device 130) different from the information processing device 110 can handle the heavy-load processing performed by the segmentation learning model 132, the left-right eye discrimination learning model 133, and the like. As a result, for example, since each information processing device 110 installed in each operating room can share one server device 130, the left and right eye identification system 100 can be identified as compared with the case where the left and right eye identification system 100 is installed in each operating room. The installation cost of the system 100 can be reduced.

Further, in the present embodiment, the surgical field image Ia transmitted from the camera 120 is received in the network IF 115. As a result, it is not necessary to provide a network IF separate from the network IF 115 for communication with the camera 120, so that an inexpensive information processing device 110 can be used. As a result, the cost of the left and right eye identification system 100 can be reduced.

<3. Modification example of the first embodiment>
Next, a modified example of the left and right eye identification system 100 according to the first embodiment will be described.

[[Modification A]]
FIG. 15 shows a modified example of the functional block of the left and right eye identification system 100 according to the above embodiment. The information processing device 110 according to this modification stores the program 112B in the memory 112 instead of the program 112A. The server device 130 according to this modification further stores the program 134D in the memory 134. Program 134D corresponds to a specific example of the "identification program" of the present invention.

The program 112B causes the control unit 111 to execute a procedure for causing the information processing device 110 to function as a user interface in the left and right eye identification system 100. The program 134D causes the control unit 131 to execute a series of procedures for identifying the type of eye (left eye or right eye) included in the surgical field.

(Left and right eye identification)
FIG. 16 is a flow chart showing an example of the left / right eye identification procedure in the left / right eye identification system 100 according to this modified example. First, the user instructs the start of left / right eye identification via the input unit 114. The user instructs, for example, to start the programs 112B and 134D via the input unit 114. Then, the information processing device 110 (control unit 111) reads the program 112B from the memory 112 and loads it into the control unit 111. The control unit 111 loaded with the program 112B executes a procedure for causing the information processing device 110 to function as a user interface in the left and right eye identification system 100. The server device 130 (control unit 131) reads the program 134D from the memory 134 and loads it into the control unit 131. The control unit 131 loaded with the program 134D executes a series of procedures for distinguishing the left and right eyes.

Specifically, first, the control unit 111 outputs an imaging instruction to the camera 120. The camera 120 images the inside of the opening of the drape 600 according to the imaging instruction, and transmits the surgical field image Ia obtained thereby to the information processing device 110.

The information processing device 110 (control unit 111) transmits the received surgical field image Ia to the server device 130. The server device 130 (control unit 131) determines the brightness of the received surgical field image Ia. The server device 130 (control unit 131) converts, for example, the surgical field image Ia into HSV, and when the V value of each pixel obtained by the conversion is out of a predetermined range, the surgical field image Ia is bright. A signal for displaying on the display unit 113 that the image is insufficient or too bright is transmitted to the information processing device 110. The information processing device 110 displays on the display unit 113 that the surgical field image Ia is insufficient or too bright based on the signal from the server device 130.

The server device 130 (control unit 131) performs a focus determination, for example, when the V value of each obtained pixel is within a predetermined range. For example, the server device 130 (control unit 131) performs Laplacian differentiation on the surgical field image Ia, and when the score obtained by the Laplacian differentiation is below a predetermined threshold value, the surgical field image Ia is out of focus. A signal for displaying this on the display unit 113 is transmitted to the information processing device 110. The information processing device 110 displays on the display unit 113 that the surgical field image Ia is out of focus based on the signal from the server device 130. For example, when the derived score is equal to or higher than a predetermined threshold value, the server device 130 (control unit 131) inputs the surgical field image Ia into the segmentation learning model 132.

The Segmentation learning model 132 executes a conversion process when the surgical field image Ia is input. Specifically, the Segmentation learning model 132 converts the surgical field image Ia into the Segmentation image Ic by using the learning result (see FIG. 6). The Segmentation learning model 132 inputs the Segmentation image Ic obtained by the conversion to the control unit 131. The control unit 131 detects an object with respect to the input Segmentation image Ic. For example, as shown in FIG. 8, the control unit 131 cuts out a rectangular region including the eye region Ra (for example, a rectangular region in contact with the eye region Ra) from the Segmentation image Ic. The image obtained in this way is referred to as a Detection image Id.

The control unit 131 uses the position information of the obtained Detection image Id in the surgical field image Ia as the position information in the surgical field image Ia of the region including the eye to be identified. For example, the control unit 131 uses the position information of the outer edge portion of the obtained Detection image Id in the surgical field image Ia as the position information of the marker Mc. That is, the control unit 131 acquires the position information in the surgical field image Ia of the region including the eye to be identified by detecting the object on the Segmentation image Ic.

The control unit 131 may use the position information of the obtained Detection image Id in the surgical field image Ia as the position information of the eye to be identified in the surgical field image Ia. For example, the control unit 131 may use the position information in the surgical field image Ia of the outer edge portion of the eye region Ra of the obtained Deception image Id as the position information of the marker Mc. In this case, the control unit 131 acquires the position information of the eye to be identified in the surgical field image Ia by detecting the object on the Segmentation image Ic.

The control unit 131 generates a marker Mc based on the acquired position information, and then transmits an image (marker image Ie) in which the generated marker Mc is superimposed on the surgical field image Ia to the information processing device 110. The information processing device 110 (control unit 111) generates a video signal for displaying an image including the received marker image IE, and outputs the video signal to the display unit 113. For example, as shown in FIG. 9, the display unit 113 displays an image in which the marker Mc is superimposed on the surgical field image Ia based on the video signal from the control unit 111 on the display surface 110A.

The left and right eye identification system 100 periodically performs a series of procedures from the above-mentioned imaging instruction to the marker Mc display. At this time, the medical staff fine-tunes the imaging area of the camera 120 while referring to the image displayed on the display surface 110A. Whether or not the position of the marker Mc in the surgical field image Ia protrudes from the predetermined region each time the server device 130 (control unit 131) executes a series of procedures from the above-mentioned imaging instruction to the marker Mc display. To judge. As a result, when the position of the marker Mc in the surgical field image Ia protrudes from the predetermined area, the server device 130 (control unit 131) may, for example, move the position of the marker Mc in the surgical field image Ia from the predetermined area. Information indicating that it is protruding is transmitted to the information processing apparatus 110. The information processing device 110 (control unit 111) generates a video signal for displaying an image including the received information and outputs the video signal to the display unit 113. The display unit 113 includes, for example, information indicating that the position of the marker Mc in the surgical field image Ia protrudes from a predetermined region based on the video signal from the control unit 111, the marker Mc, and the surgical field image Ia. An image including and is displayed on the display surface 110A. For example, as shown in FIG. 10, the display unit 113 displays an image including the status ST such as “microscope position is deviated”, the marker Mc, and the surgical field image Ia on the display surface 110A. .. The information processing device 110 (control unit 111) uses a method other than displaying the image on the display surface 110A (for example, sound, lighting of a patrol lamp, etc.) to deviate the position of the marker Mc from the predetermined region in the surgical field image Ia. You may inform your healthcare professional that you are doing so.

On the other hand, when the position of the marker Mc in the surgical field image Ia is within the predetermined region, the server device 130 (control unit 131) has the position of the marker Mc in the surgical field image Ia within the predetermined region. Information indicating that is transmitted to the information processing device 110. The information processing device 110 (control unit 111) generates a video signal for displaying an image including the received information and outputs the video signal to the display unit 113. The display unit 113 includes, for example, information indicating that the position of the marker Mc in the surgical field image Ia is within a predetermined region, the marker Mc, and the surgical field image based on the video signal from the control unit 111. An image including Ia is displayed on the display surface 110A. The information processing device 110 (control unit 111) uses a method other than displaying an image on the display surface 110A (for example, sound, lighting of a patrol lamp, etc.) to position the marker Mc in a predetermined area in the surgical field image Ia. You may inform your healthcare professional that there is.

The server device 130 (control unit 131) outputs a left / right eye identification instruction to the left / right eye identification learning model 133 when the position of the marker Mc is within a predetermined region in the surgical field image Ia. When the left / right eye discrimination learning model 133 receives the left / right eye discrimination instruction, the left / right eye discrimination learning model 133 executes the discrimination process. Specifically, the left-right eye discrimination learning model 133 uses the learning result to discriminate the eyes included in the Detection image Ib. More specifically, the left-right eye discrimination learning model 133 uses the learning result to identify the type of eye (left eye, right eye) included in the Detection image Ib. The left and right eye discrimination learning model 133 transmits the discrimination result to the control unit 131. The control unit 131 transmits the identification result DR to the information processing device 110. The control unit 111 generates a video signal for displaying an image including the status ST, the identification result DR, the marker Mc, and the surgical field image Ia on the display surface 110A, and outputs the video signal to the display unit 113. For example, as shown in FIG. 12, the display unit 113 displays an image including the status ST, the identification result DR, the marker Mc, and the surgical field image Ia based on the video signal from the control unit 111. Display on 110A. For example, the display unit 113 may superimpose the identification result DR on the surgical field image Ia as shown in FIG. 12, or perform the identification result DR as shown in FIG. 13, for example. It may be displayed in an area different from the field image Ia.

By looking at the identification result DR displayed on the display surface 110A, the medical staff can determine whether the eye exposed in the opening 610 of the drape 600 is the right eye or the left eye, and by extension, the eye to be operated on. Can be confirmed. In this way, left and right eye discrimination is performed.

The information processing device 110 (control unit 111) further reads, reads out, and includes the surgical data 134C corresponding to the patient 400 from the plurality of surgical data 134C stored in the memory 134, for example. The surgical eye is compared with the identification result DR, and a video signal for displaying an image including the comparison result CR on the display surface 110A is generated and output to the display unit 113. For example, as shown in FIG. 14, the display unit 113 displays an image including the comparison result CR, the identification result DR, the status ST, and the surgical field image Ia based on the video signal from the control unit 111. Displayed on surface 110A. Note that FIG. 14 illustrates, as the comparison result CR, the display when the surgical eye read and included in the surgical data 134C and the identification result DR match.

In this modification, the server device 130 detects the segmentation image Ic obtained from the surgical field image Ia including the eye to be identified, thereby detecting the eye to be identified or the eye to be identified. The position information in the surgical field image Ia of the region including the above is acquired. Further, in this modification, an image (marker image Ie) in which the marker Mc generated based on the acquired position information is superimposed on the surgical field image Ia is transmitted to the information processing apparatus 110. As a result, the information processing device 110 can display the marker image Ie, so that the marker Mc can surely include an element (for example, the caruncle 12 or the inner corner of the eye) that facilitates the identification of the left and right eyes. it can. As a result, the medical worker adjusts the imaging region of the camera 120 so that the marker Mc is located within a predetermined region in the surgical field image Ia, so that the eye reflected in the surgical field image Ia is the left eye. The left and right eye identification system 100 can correctly identify whether the eye is the right eye or the right eye. Therefore, it is possible to identify the type of eye that is difficult to identify.

Further, in this modified example, by detecting the object on the Segmentation image Ic, after the Detection image Id is cut out from the Segmentation image Ic, the position information of the cut out Detection image Id in the surgical field image Ia is obtained. It is the position information in the surgical field image Ia of the eye to be identified or the region including the eye to be identified. As a result, the object is simply detected for the surgical field image Ia, and the position information of the eye to be identified in the surgical field image Ia is derived based on the detection image obtained thereby. , The marker Mc can surely include an element (for example, the caruncle 12 and the inner corner of the eye) that facilitates the identification of the left and right eyes. As a result, the medical worker adjusts the imaging region of the camera 120 so that the marker Mc is located within a predetermined region in the surgical field image Ia, so that the eye reflected in the surgical field image Ia is the left eye. The left and right eye identification system 100 can correctly identify whether the eye is the right eye or the right eye. Therefore, it is possible to identify the type of eye that is difficult to identify.

[[Modification B]]
In the above embodiment, for example, as shown in FIG. 17, the server device 130 may be omitted, and the information processing device 110 may have the function of the server device 130. At this time, for example, as shown in FIG. 18, the information processing apparatus 110 includes a Segmentation learning model 132 and a left / right eye discrimination learning model 133, as well as a plurality of image data 134A, a plurality of image data 134B, and a plurality of surgical data. 134C is stored in the memory 112.

(Left and right eye identification)
FIG. 19 is a flow chart showing an example of the left / right eye identification procedure in the left / right eye identification system 100 according to this modified example. First, the user instructs the start of left / right eye identification via the input unit 114. The user instructs, for example, to start the program 112A via the input unit 114. Then, the information processing device 110 (control unit 111) reads the program 112A from the memory 112 and loads it into the control unit 111. The control unit 111 loaded with the program 112A executes a series of steps for distinguishing the left and right eyes.

Specifically, first, the control unit 111 outputs an imaging instruction to the camera 120. The camera 120 images the inside of the opening of the drape 600 according to the imaging instruction, and transmits the surgical field image Ia obtained thereby to the information processing device 110.

The control unit 111 determines the brightness of the received surgical field image Ia. For example, the control unit 111 converts the surgical field image Ia into HSV, and when the V value of each pixel obtained by the conversion is out of the predetermined range, the surgical field image Ia is insufficiently bright or too bright. The presence is displayed on the display unit 113.

For example, when the V value of each obtained pixel is within a predetermined range, the control unit 111 performs a focus determination. For example, the control unit 111 performs Laplacian differentiation on the surgical field image Ia, and when the score obtained by the Laplacian differentiation is below a predetermined threshold value, the display unit 113 indicates that the surgical field image Ia is out of focus. Display on. For example, when the derived score is equal to or higher than a predetermined threshold value, the control unit 111 inputs the surgical field image Ia into the segmentation learning model 132.

The Segmentation learning model 132 executes a conversion process when the surgical field image Ia is input. Specifically, the Segmentation learning model 132 converts the surgical field image Ia into the Segmentation image Ic by using the learning result (see FIG. 6). The Segmentation learning model 132 inputs the Segmentation image Ic obtained by the conversion to the control unit 111. The control unit 111 detects an object with respect to the input Segmentation image Ic. For example, as shown in FIG. 8, the control unit 111 cuts out a rectangular region including the eye region Ra (for example, a rectangular region in contact with the eye region Ra) from the Segmentation image Ic. The image obtained in this way is referred to as a Detection image Id.

The control unit 111 uses the position information of the obtained Detection image Id in the surgical field image Ia as the position information in the surgical field image Ia of the region including the eye to be identified. For example, the control unit 111 uses the position information of the outer edge portion of the obtained Detection image Id in the surgical field image Ia as the position information of the marker Mc. That is, the control unit 111 acquires the position information in the surgical field image Ia of the region including the eye to be identified by detecting the object on the Segmentation image Ic.

The control unit 111 may use the position information of the obtained Detection image Id in the surgical field image Ia as the position information of the eye to be identified in the surgical field image Ia. For example, the control unit 111 may use the position information of the outer edge portion of the eye region Ra of the obtained Deception image Id in the surgical field image Ia as the position information of the marker Mc. In this case, the control unit 111 acquires the position information of the eye to be identified in the surgical field image Ia by detecting the object on the Segmentation image Ic.

The control unit 111 generates a marker Mc based on the acquired position information, then generates a video signal for displaying an image in which the generated marker Mc is superposed on the surgical field image Ia, and outputs the video signal to the display unit 113. .. For example, as shown in FIG. 9, the display unit 113 displays an image in which the marker Mc is superimposed on the surgical field image Ia based on the video signal from the control unit 111 on the display surface 110A.

The left and right eye identification system 100 periodically performs a series of procedures from the above-mentioned imaging instruction to the marker Mc display. At this time, the medical staff fine-tunes the imaging area of the camera 120 while referring to the image displayed on the display surface 110A. For example, each time the control unit 111 executes a series of procedures from the above-mentioned imaging instruction to the marker Mc display, the control unit 111 determines whether or not the position of the marker Mc in the surgical field image Ia protrudes from a predetermined region. As a result, when the position of the marker Mc in the surgical field image Ia protrudes from the predetermined region, the control unit 111 indicates that, for example, the position of the marker Mc in the surgical field image Ia protrudes from the predetermined region. A video signal for displaying an image including meaning information is generated and output to the display unit 113. The display unit 113 is, for example, an image including information indicating that the position of the marker Mc in the surgical field image Ia protrudes from a predetermined region based on the video signal from the control unit 111, and the surgical field image Ia. Is displayed on the display surface 110A. For example, as shown in FIG. 10, the display unit 113 displays an image including the status ST such as “microscope position is deviated”, the marker Mc, and the surgical field image Ia on the display surface 110A. .. The control unit 111 is engaged in medical treatment that the position of the marker Mc in the surgical field image Ia is out of a predetermined region by a method other than displaying the image on the display surface 110A (for example, sound or lighting of a patrol lamp). You may inform the person.

On the other hand, when the position of the marker Mc in the surgical field image Ia is within a predetermined region, the control unit 111 displays an image indicating that the position of the marker Mc in the surgical field image Ia is within the predetermined region. A video signal for display is generated and output to the display unit 113. The display unit 113 includes, for example, information indicating that the position of the marker Mc in the surgical field image Ia is within a predetermined region, the marker Mc, and the surgical field image based on the video signal from the control unit 111. An image including Ia is displayed on the display surface 110A. For example, as shown in FIG. 11, the display unit 113 displays an image including the status ST such as “microscope position OK”, the marker Mc, and the surgical field image Ia on the display surface 110A. The control unit 111 indicates that the position of the marker Mc in the surgical field image Ia is within a predetermined region by a method other than displaying the image on the display surface 110A (for example, sound or lighting of a patrol lamp). You may inform.

When the position of the marker Mc is within a predetermined region in the surgical field image Ia, the control unit 111 outputs a left / right eye identification instruction to the left / right eye identification learning model 133. When the left / right eye discrimination learning model 133 receives the left / right eye discrimination instruction, the left / right eye discrimination learning model 133 executes the discrimination process. Specifically, the left-right eye discrimination learning model 133 uses the learning result to discriminate the eyes included in the Detection image Ib. More specifically, the left-right eye discrimination learning model 133 uses the learning result to identify the type of eye (left eye, right eye) included in the Detection image Ib. The left and right eye discrimination learning model 133 inputs the discrimination result DR to the control unit 111. The control unit 111 generates a video signal for displaying an image including the status ST, the identification result DR, the marker Mc, and the surgical field image Ia on the display surface 110A, and outputs the video signal to the display unit 113. For example, as shown in FIG. 12, the display unit 113 displays an image including the status ST, the identification result DR, the marker Mc, and the surgical field image Ia based on the video signal from the control unit 111. Display on 110A. For example, the display unit 113 may superimpose the identification result DR on the surgical field image Ia as shown in FIG. 12, or perform the identification result DR as shown in FIG. 13, for example. It may be displayed in an area different from the field image Ia.

By looking at the identification result DR displayed on the display surface 110A, the medical staff can determine whether the eye exposed in the opening 610 of the drape 600 is the right eye or the left eye, and by extension, the eye to be operated on. Can be confirmed. In this way, left and right eye discrimination is performed.

The control unit 111 further reads, for example, from the plurality of surgical data 134C stored in the memory 112, the surgical data 134C corresponding to the patient 400, and the surgical eye included in the read and surgical data 134C and the identification result. The DR is compared, and a video signal for displaying the comparison result CR on the display surface 110A is generated and output to the display unit 113. For example, as shown in FIG. 14, the display unit 113 displays an image including the comparison result CR, the identification result DR, the status ST, and the surgical field image Ia based on the video signal from the control unit 111. Displayed on surface 110A. Note that FIG. 14 illustrates, as the comparison result CR, the display when the surgical eye read and included in the surgical data 134C and the identification result DR match.

In this modification, the information processing device 110 has the function of the server device 130 according to the above embodiment. As a result, the left and right eyes can be identified simply by preparing the information processing device 110 and the camera 120 and connecting them to the network 140. Therefore, it is possible to reduce the time and effort required to install the left and right eye identification system 100 because the server device 130 is unnecessary.

[[Modification C]]
In the above embodiment, for example, as shown in FIGS. 20 and 21, communication between the information processing device 110 and the camera 120 may be performed via a network 150 different from the network 140. Further, in the above-described modification A, for example, as shown in FIGS. 20 and 22, communication between the information processing device 110 and the camera 120 may be performed via a network 150 different from the network 140. At this time, the network 150 may be short-range wireless communication such as Bluetooth (registered trademark). Even in the case of such a configuration, it has the same effect as that of the above-described embodiment.

[[Modification D]]
In the above embodiment, for example, as shown in FIGS. 23 and 24, the information processing device 110 has the function of the server device 130, and the communication between the information processing device 110 and the camera 120 provides the network 150. It may be done through. At this time, the information processing apparatus 110 includes a Segmentation learning model 132 and a left / right eye discrimination learning model 133, and stores a plurality of image data 134A, a plurality of image data 134B, and a plurality of surgical data 134C in the memory 112. Further, the information processing device 110 has a network IF 116 that communicates with the camera 120 via the network 150.

In this modification, when the network 150 is short-range wireless communication such as Bluetooth (registered trademark), it becomes easier to establish communication between the information processing device 110 and the camera 120 as compared with the above embodiment.

<4. Second Embodiment>
[Constitution]
FIG. 25 shows an example of a schematic configuration of the left and right eye identification system 700 according to the second embodiment of the present invention. The left and right eye identification system 700 corresponds to a specific example of the "information processing system" of the present invention. FIG. 26 shows an example of the functional blocks of the left and right eye identification system 700. In FIG. 25, a microscope system 200 for ophthalmic surgery for a medical professional to magnify and visually recognize a surgical field is illustrated together with a left-right eye discrimination system 700.

(Left and right eye identification system 700)
The left and right eye identification system 700 includes, for example, an information processing device 160 with a camera and a server device 130. The information processing device 160 with a camera and the server device 130 are connected via a network 140. The information processing device 160 with a camera is configured to be able to communicate with the server device 130 via the network 140. The server device 130 is configured to be able to communicate with the information processing device 160 with a camera via the network 140.

The information processing device 160 with a camera includes, for example, a control unit 111, a memory 112, an input unit 114, a network IF 115, a projection unit 117, and a camera 120. The projection unit 117 is connected to the eyepiece unit 252, and causes the image light generated based on the image signal from the control unit 111 to enter the eyepiece unit 252.

The image light generated by the projection unit 117 may be, for example, a luminous flux that generates an image including the marker Mc and the status ST, as shown in FIGS. 27 and 28. The image light generated by the projection unit 117 may be, for example, a luminous flux that generates an image including the marker Mc, the status ST, and the identification result DR, as shown in FIGS. 29 and 30. At this time, in the image light generated by the projection unit 117, for example, the identification result DR may be superimposed on the marker Mc as shown in FIG. 29, or as shown in FIG. 30, for example, identification. The result DR may be arranged at a place different from the marker Mc. The image light generated by the projection unit 117 may be, for example, a luminous flux that generates an image including the marker Mc, the status ST, the identification result DR, and the comparison result CR, as shown in FIG. 31. ..

The image light incident on the eyepiece 252 and the light guiding the observation optical system from the surgical field are combined in the eyepiece 252 and incident on the eyes of the medical staff. That is, the eyepiece 252 displays the status ST, the identification result DR or the comparison result CR, and the marker Mc together with the image of the surgical field formed by the observation optical system. As a result, the medical staff visually recognizes an image as shown in FIG. 32, for example. In FIG. 32, the marker Mc is superposed on the image of the surgical field, and the status ST, the identification result DR, and the comparison result CR are displayed separately from the image of the surgical field (that is, the image of the surgical field is displayed). The case where it is displayed (without superposition) is illustrated. The status ST, the identification result DR, and the comparison result CR may be superposed on the image of the surgical field.

(Left and right eye identification)
FIG. 33 is a flow chart showing an example of the left and right eye identification procedure in the left and right eye identification system 700. First, the user instructs the start of left / right eye identification via the input unit 114. The user instructs, for example, to start the program 112A via the input unit 114. Then, the information processing device 160 with a camera (control unit 111) reads the program 112A from the memory 112 and loads it into the control unit 111. The control unit 111 loaded with the program 112A executes a series of steps for distinguishing the left and right eyes. Specifically, first, the control unit 111 outputs an imaging instruction to the camera 120. The camera 120 images the inside of the opening of the drape 600 according to the imaging instruction, and outputs the surgical field image Ia obtained thereby to the control unit 111.

The control unit 111 determines the brightness of the surgical field image Ia input from the camera 120. For example, the control unit 111 converts the surgical field image Ia into HSV, and when the V value of each pixel obtained by the conversion is out of the predetermined range, the surgical field image Ia is insufficiently bright or too bright. The presence is displayed on the display unit 113. For example, when the V value of each obtained pixel is within a predetermined range, the control unit 111 performs a focus determination. For example, the control unit 111 performs Laplacian differentiation on the surgical field image Ia, and when the score obtained by the Laplacian differentiation is below a predetermined threshold value, the display unit 113 indicates that the surgical field image Ia is out of focus. Display on. The information processing device 110 (control unit 111) transmits, for example, the surgical field image Ia to the server device 130 when the derived score is equal to or higher than a predetermined threshold value.

When the server device 130 receives the surgical field image Ia from the information processing device 160 with a camera, the Segmentation learning model 132 executes the conversion process. Specifically, the Segmentation learning model 132 converts the surgical field image Ia into the Segmentation image Ic by using the learning result (see FIG. 6). The segmentation learning model 132 transmits the segmentation image Ic obtained by the conversion to the information processing device 160 with a camera. The information processing device 160 with a camera (control unit 111) detects an object for the received Segmentation image Ic. As shown in FIG. 8, for example, the information processing device 160 with a camera (control unit 111) has a rectangular region including the eye region Ra (for example, a rectangular region in contact with the eye region Ra) from the segmentation image Ic. Cut out. The image obtained in this way is referred to as a Detection image Id.

The information processing device 160 with a camera (control unit 111) uses the position information of the obtained Detection image Id in the surgical field image Ia as the position information in the surgical field image Ia of the region including the eye to be identified. The information processing device 160 with a camera (control unit 111) uses, for example, the position information of the outer edge portion of the obtained Detection image Id in the surgical field image Ia as the position information of the marker Mc. That is, the information processing device 160 with a camera (control unit 111) acquires the position information in the surgical field image Ia of the region including the eye to be identified by detecting the object on the Segmentation image Ic.

The information processing device 160 with a camera (control unit 111) may use the position information of the obtained Detection image Id in the surgical field image Ia as the position information of the eye to be identified in the surgical field image Ia. The information processing apparatus 160 with a camera (control unit 111) may use, for example, the position information in the surgical field image Ia of the outer edge portion of the eye region Ra of the obtained Deception image Id as the position information of the marker Mc. In this case, the information processing apparatus 160 with a camera (control unit 111) acquires the position information of the eye to be identified in the surgical field image Ia by detecting the object on the Sedication image Ic.

The information processing device 160 with a camera (control unit 111) generates a marker Mc based on the acquired position information, then generates a video signal for generating video light including the generated marker Mc, and causes the projection unit 117 to generate a video signal. Output. The projection unit 117 generates image light including the marker Mc based on the image signal from the control unit 111, and emits the image light to the eyepiece unit 252. As a result, the image light incident on the eyepiece 252 and the light guiding the observation optical system from the surgical field are combined in the eyepiece 252 and incident on the eyes of the medical staff. As a result, the medical staff visually recognizes the image in which the marker Mc is superimposed on the image of the surgical field. That is, the eyepiece 252 superimposes and displays the marker Mc on the image of the surgical field formed by the observation optical system.

The left and right eye identification system 700 periodically performs a series of procedures from the above-mentioned imaging instruction to the marker Mc display. At this time, the medical staff fine-tunes the imaging region of the camera 120 while referring to the image of the visual field (surgical field) seen through the eyepiece 252. For example, each time the information processing apparatus 160 with a camera (control unit 111) executes a series of procedures from the above-mentioned imaging instruction to the marker Mc display, the position of the marker Mc in the surgical field image Ia protrudes from a predetermined area. Judge whether or not. As a result, when the position of the marker Mc in the surgical field image Ia protrudes from the predetermined region, the information processing apparatus 160 with a camera (control unit 111) determines, for example, the position of the marker Mc in the surgical field image Ia. A video signal for generating video light including information meaning that it is out of the region of is generated and output to the projection unit 117. Based on the video signal from the control unit 111, the projection unit 117 generates information indicating that the position of the marker Mc in the surgical field image Ia protrudes from a predetermined region, and video light including the marker Mc. , Exits to the eyepiece 252. As a result, the image light incident on the eyepiece 252 and the light guiding the observation optical system from the surgical field are combined in the eyepiece 252 and incident on the eyes of the medical staff. As a result, the medical staff visually recognizes, for example, an image in which the image shown in FIG. 27 is superimposed on the image of the surgical field. The information processing device 160 with a camera (control unit 111) can be operated by a method other than displaying the marker Mc in the field of view (operative field) seen through the eyepiece 252 (for example, sound or lighting of a patrol lamp). The medical staff may be informed that the position of the marker Mc is out of the predetermined region in the field image Ia.

On the other hand, when the position of the marker Mc in the surgical field image Ia is within a predetermined area, the information processing apparatus 160 with a camera (control unit 111) has the position of the marker Mc in the surgical field image Ia in a predetermined area. A video signal for generating video light including information indicating that it is inside is generated and output to the projection unit 117. Based on the video signal from the control unit 111, the projection unit 117 generates video light including information indicating that the position of the marker Mc in the surgical field image Ia is within a predetermined region and the marker Mc. Then, the light is emitted to the eyepiece 252. As a result, the image light incident on the eyepiece 252 and the light guiding the observation optical system from the surgical field are combined in the eyepiece 252 and incident on the eyes of the medical staff. As a result, the medical staff visually recognizes, for example, an image in which the image shown in FIG. 28 is superimposed on the image of the surgical field. The information processing device 160 with a camera (control unit 111) is operated by a method other than displaying the marker Mc in the field of view (operative field) seen through the eyepiece 252 (for example, sound or lighting of a patrol lamp). The medical staff may be informed that the position of the marker Mc in the field image Ia is within a predetermined region.

When the position of the marker Mc in the surgical field image Ia is within a predetermined region, the information processing device 160 with a camera (control unit 111) refers to the left / right eye discrimination learning model 133 in the server device 130. Outputs left and right eye identification instructions. When the left / right eye discrimination learning model 133 receives the left / right eye discrimination instruction, the left / right eye discrimination learning model 133 executes the discrimination process. Specifically, the left-right eye discrimination learning model 133 uses the learning result to discriminate the eyes included in the Detection image Ib. More specifically, the left-right eye discrimination learning model 133 uses the learning result to identify the type of eye (left eye, right eye) included in the Detection image Ib. The left-right eye discrimination learning model 133 transmits the discrimination result DR to the information processing device 160 with a camera. The control unit 111 generates a video signal for generating video light including the status ST, the identification result DR, and the marker Mc, and outputs the video signal to the projection unit 117. The projection unit 117 generates image light including the status ST, the identification result DR, and the marker Mc based on the image signal from the control unit 111, and emits the image light to the eyepiece unit 252. As a result, the image light incident on the eyepiece 252 and the light guiding the observation optical system from the surgical field are combined in the eyepiece 252 and incident on the eyes of the medical staff. As a result, the medical staff visually recognizes, for example, an image in which the image shown in FIG. 29 or FIG. 30 is superimposed on the image of the surgical field.

By looking at the identification result DR in the image of the visual field (surgical field) seen through the eyepiece 252, the medical staff can determine whether the eye exposed in the opening 610 of the drape 600 is the right eye or the left eye. As a result, it is possible to confirm whether or not the eye is the target of surgery. In this way, left and right eye discrimination is performed.

The information processing device 160 with a camera (control unit 111) further reads, reads out the surgical data 134C corresponding to the patient 400 from the plurality of surgical data 134C stored in the memory 134, and reads the surgical data 134C into the surgical data 134C. The included surgical eye is compared with the identification result DR. The information processing device 160 with a camera (control unit 111) generates a video signal for generating video light including the comparison result CR, the status ST, the identification result DR, and the marker Mc, and causes the projection unit 117 to generate a video signal. Output. The projection unit 117 generates image light including the comparison result CR, the status ST, the identification result DR, and the marker Mc based on the image signal from the control unit 111, and emits the image light to the eyepiece unit 252. As a result, the image light incident on the eyepiece 252 and the light guiding the observation optical system from the surgical field are combined in the eyepiece 252 and incident on the eyes of the medical staff. As a result, for example, as shown in FIG. 32, the medical staff can see the comparison result CR, the identification result DR, the status ST, and the marker Mc through the eyepiece 252 (operative field). Visually visible in the image of.

[effect]
Next, the effect of the left and right eye identification system 700 according to the present embodiment will be described.

In the present embodiment, the object is detected on the Segumentation image Ic obtained from the surgical field image Ia including the eye to be identified, so that the eye to be identified or the region including the eye to be identified is included. , The position information in the surgical field image Ia is acquired. Further, in the present embodiment, the marker Mc generated based on the acquired position information is superposed on the image of the surgical field formed by the observation optical system and displayed. As a result, the marker Mc can surely include an element (for example, the caruncle 12 and the inner corner of the eye) that facilitates the identification of the left and right eyes. As a result, the medical staff adjusts the imaging region of the camera 120 so that the marker Mc is located within a predetermined region in the image of the surgical field formed by the observation optical system, so that the marker Mc is reflected in the surgical field image Ia. The left and right eye identification system 100 can correctly discriminate whether the eye is the left eye or the right eye. Therefore, it is possible to identify the type of eye that is difficult to identify.

Further, in the present embodiment, the identification result DR is displayed on the microscope 250 for medical staff by superimposing it on the image of the surgical field formed by the observation optical system. Further, in the present embodiment, in the microscope 250 for medical professionals, the identification result DR is displayed superposed on the image of the surgical field formed by the observation optical system together with the marker Mc. As a result, it is possible to perform reliable confirmation work without oversight.

Further, in the present embodiment, by performing object detection on the Segmentation image Ic, after the Detection image Id is cut out from the Segmentation image Ic, the position information of the cut out Detection image Id in the surgical field image Ia is obtained. , The position information in the surgical field image Ia of the eye to be identified or the region including the eye to be identified. As a result, the object is simply detected for the surgical field image Ia, and the position information of the eye to be identified in the surgical field image Ia is derived based on the detection image obtained thereby. , The marker Mc can surely include an element (for example, the caruncle 12 and the inner corner of the eye) that facilitates the identification of the left and right eyes. As a result, the medical staff adjusts the imaging region of the camera 120 so that the marker Mc is located within a predetermined region in the image of the surgical field formed by the observation optical system, so that the marker Mc is reflected in the surgical field image Ia. The left and right eye identification system 100 can correctly discriminate whether the eye is the left eye or the right eye. Therefore, it is possible to identify the type of eye that is difficult to identify.

Further, in the present embodiment, when the marker Mc protrudes from a predetermined region in the image of the surgical field formed by the observation optical system, the marker Mc and the marker Mc protrude from the predetermined region. An image containing information (eg, status ST) is superimposed on the image of the surgical field formed by the observation optics. This makes it possible to reliably inform the medical staff that the imaging area of the camera 120 needs to be fine-tuned.

Further, in the present embodiment, after the surgical field image Ia obtained when the marker Mc is within a predetermined area in the surgical field image Ia is transmitted to the server device 130, the identification result of the left and right eyes is transmitted to the server device. A network IF 115 that receives from 130 is provided in the information processing apparatus 160 with a camera. As a result, the heavy-load processing performed by the segmentation learning model 132, the left-right eye discrimination learning model 133, and the like can be assigned to a device (server device 130) different from the information processing device 160 with a camera. As a result, for example, since each information processing device 160 with a camera installed in each operating room can share one server device 130, the left and right eye identification systems 700 are installed in each operating room. The installation cost of the eye identification system 700 can be reduced.

<5. Modification example of the second embodiment>
Next, a modified example of the left / right eye identification system 700 according to the second embodiment will be described.

[[Modification E]]
FIG. 34 shows a modified example of the functional block of the left and right eye identification system 700 according to the second embodiment. The information processing apparatus 160 with a camera according to this modification stores the program 112B in the memory 112 instead of the program 112A. The server device 130 according to this modification further stores the program 134D in the memory 134.

The program 112B causes the control unit 111 to execute a procedure for causing the information processing device 160 with a camera to function as a user interface in the left and right eye identification system 100. The program 134D causes the control unit 131 to execute a series of procedures for identifying the type of eye (left eye or right eye) included in the surgical field.

(Left and right eye identification)
FIG. 35 is a flow chart showing an example of the left / right eye identification procedure in the left / right eye identification system 700 according to this modified example. First, the user instructs the start of left / right eye identification via the input unit 114. The user instructs, for example, to start the programs 112B and 134D via the input unit 114. Then, the information processing device 160 with a camera (control unit 111) reads the program 112B from the memory 112 and loads it into the control unit 111. The control unit 111 loaded with the program 112B executes a procedure for causing the information processing device 160 with a camera to function as a user interface in the left-right eye identification system 700. The server device 130 (control unit 131) reads the program 134D from the memory 134 and loads it into the control unit 131. The control unit 131 loaded with the program 134D executes a series of procedures for distinguishing the left and right eyes.

Specifically, first, the information processing device 160 with a camera (control unit 111) outputs an imaging instruction to the camera 120. The camera 120 images the inside of the opening of the drape 600 according to the imaging instruction, and outputs the surgical field image Ia obtained thereby to the control unit 111.

The information processing device 160 with a camera (control unit 111) transmits the surgical field image Ia input from the camera 120 to the server device 130. The server device 130 (control unit 131) determines the brightness of the received surgical field image Ia. The server device 130 (control unit 131) derives, for example, the average value of the brightness of each pixel of the surgical field image Ia, and when the derived average value falls below a predetermined threshold value, the surgical field image Ia has the brightness. A signal for displaying the shortage on the display unit 113 is transmitted to the information processing device 160 with a camera. The information processing device 160 with a camera displays on the display unit 113 that the surgical field image Ia is insufficient in brightness based on the signal from the server device 130.

The server device 130 (control unit 131) performs focus determination, for example, when the derived average value is within a predetermined range. For example, the server device 130 (control unit 131) performs Laplacian differentiation on the surgical field image Ia, and when the score obtained by the Laplacian differentiation is below a predetermined threshold value, the surgical field image Ia is out of focus. A signal for displaying this on the display unit 113 is transmitted to the information processing apparatus 160 with a camera. The information processing device 160 with a camera displays on the display unit 113 that the surgical field image Ia is out of focus based on the signal from the server device 130. For example, when the derived score is equal to or higher than a predetermined threshold value, the server device 130 (control unit 131) inputs the surgical field image Ia into the segmentation learning model 132.

The Segmentation learning model 132 executes a conversion process when the surgical field image Ia is input. Specifically, the Segmentation learning model 132 converts the surgical field image Ia into the Segmentation image Ic by using the learning result (see FIG. 6). The Segmentation learning model 132 inputs the Segmentation image Ic obtained by the conversion to the control unit 131. The control unit 131 detects an object on the input Segmentation image Ic. For example, as shown in FIG. 10, the control unit 131 cuts out a rectangular region including the eye region Ra (for example, a rectangular region in contact with the eye region Ra) from the Segmentation image Ic. The image obtained in this way is referred to as a Detection image Id.

The server device 130 (control unit 131) uses the position information of the obtained Detection image Id in the surgical field image Ia as the position information in the surgical field image Ia of the region including the eye to be identified. The server device 130 (control unit 131) uses, for example, the position information of the outer edge portion of the obtained Detection image Id in the surgical field image Ia as the position information of the marker Mc. That is, the server device 130 (control unit 131) acquires the position information in the surgical field image Ia of the region including the eye to be identified by detecting the object on the Segmentation image Ic.

The server device 130 (control unit 131) may use the position information of the obtained Detection image Id in the surgical field image Ia as the position information of the eye to be identified in the surgical field image Ia. The server device 130 (control unit 131) may use, for example, the position information in the surgical field image Ia of the outer edge portion of the eye region Ra of the obtained Deception image Id as the position information of the marker Mc. In this case, the server device 130 (control unit 131) acquires the position information of the eye to be identified in the surgical field image Ia by detecting the object on the Segmentation image Ic.

The server device 130 (control unit 131) transmits the acquired position information to the information processing device 160 with a camera. The information processing device 160 with a camera generates a marker Mc based on the position information from the server device 130, then generates a video signal for generating video light including the generated marker Mc, and outputs the video signal to the projection unit 117. To do. The projection unit 117 generates image light including the marker Mc based on the image signal from the control unit 111, and emits the image light to the eyepiece unit 252. As a result, the image light incident on the eyepiece 252 and the light guiding the observation optical system from the surgical field are combined in the eyepiece 252 and incident on the eyes of the medical staff. As a result, the medical staff visually recognizes, for example, an image in which the marker Mc is superimposed on the image of the surgical field. At this time, the position and size of the marker Mc are defined by the video signal incident on the projection unit 117 from the control unit 111. That is, the eyepiece 252 superimposes and displays the marker Mc on the image of the surgical field formed by the observation optical system.

The left and right eye identification system 700 periodically performs a series of procedures from the above-mentioned imaging instruction to the marker Mc display. At this time, the medical staff fine-tunes the imaging region of the camera 120 while referring to the image of the visual field (surgical field) seen through the eyepiece 252. Whether or not the position of the marker Mc in the surgical field image Ia protrudes from the predetermined region each time the server device 130 (control unit 131) executes a series of procedures from the above-mentioned imaging instruction to the marker Mc display. To judge. As a result, when the position of the marker Mc in the surgical field image Ia protrudes from the predetermined area, the server device 130 (control unit 131) may, for example, move the position of the marker Mc in the surgical field image Ia from the predetermined area. Information indicating that it is protruding is transmitted to the information processing apparatus 160 with a camera. The information processing device 160 with a camera (control unit 111) generates a video signal for displaying an image including the received information and outputs the video signal to the display unit 113. The display unit 113, for example, based on the video signal from the control unit 111, provides information indicating that the position of the marker Mc in the surgical field image Ia protrudes from a predetermined region, and video light including the marker Mc. It is generated and emitted to the eyepiece 252. As a result, the image light incident on the eyepiece 252 and the light guiding the observation optical system from the surgical field are combined in the eyepiece 252 and incident on the eyes of the medical staff. As a result, the medical staff visually recognizes, for example, an image in which the image shown in FIG. 27 is superimposed on the image of the surgical field. The information processing device 160 with a camera (control unit 111) can be operated by a method other than displaying the marker Mc in the field of view (operative field) seen through the eyepiece 252 (for example, sound or lighting of a patrol lamp). The medical staff may be informed that the position of the marker Mc is out of the predetermined region in the field image Ia.

On the other hand, when the position of the marker Mc in the surgical field image Ia is within the predetermined region, the server device 130 (control unit 131) has the position of the marker Mc in the surgical field image Ia within the predetermined region. Information indicating that is transmitted to the information processing apparatus 160 with a camera. The information processing device 160 with a camera (control unit 111) generates a video signal for generating video light including the received information and outputs the video signal to the projection unit 117. Based on the video signal from the control unit 111, the projection unit 117 generates video light including information indicating that the position of the marker Mc in the surgical field image Ia is within a predetermined region and the marker Mc. Then, the light is emitted to the eyepiece 252. As a result, the image light incident on the eyepiece 252 and the light guiding the observation optical system from the surgical field are combined in the eyepiece 252 and incident on the eyes of the medical staff. As a result, the medical staff visually recognizes, for example, an image in which the image shown in FIG. 28 is superimposed on the image of the surgical field. The information processing device 160 with a camera (control unit 111) is operated by a method other than displaying the marker Mc in the field of view (operative field) seen through the eyepiece 252 (for example, sound or lighting of a patrol lamp). The medical staff may be informed that the position of the marker Mc in the field image Ia is within a predetermined region.

The server device 130 (control unit 131) outputs a left / right eye identification instruction to the left / right eye identification learning model 133 when the position of the marker Mc is within a predetermined region in the surgical field image Ia. When the left / right eye discrimination learning model 133 receives the left / right eye discrimination instruction, the left / right eye discrimination learning model 133 executes the discrimination process. Specifically, the left-right eye discrimination learning model 133 uses the learning result to discriminate the eyes included in the Detection image Ib. More specifically, the left-right eye discrimination learning model 133 uses the learning result to identify the type of eye (left eye, right eye) included in the Detection image Ib. The left and right eye discrimination learning model 133 transmits the discrimination result DR to the control unit 131. The server device 130 (control unit 131) transmits the identification result DR to the information processing device 160 with a camera. The information processing device 160 with a camera (control unit 111) generates a video signal for generating video light including the status ST, the identification result DR, and the marker Mc, and outputs the video signal to the projection unit 117. The projection unit 117 generates image light including the status ST, the identification result DR, and the marker Mc based on the image signal from the control unit 111, and emits the image light to the eyepiece unit 252. As a result, the image light incident on the eyepiece 252 and the light guiding the observation optical system from the surgical field are combined in the eyepiece 252 and incident on the eyes of the medical staff. As a result, the medical staff visually recognizes, for example, an image in which the image shown in FIG. 29 or FIG. 30 is superimposed on the image of the surgical field.

By looking at the identification result DR in the image of the visual field (surgical field) seen through the eyepiece 252, the medical staff can determine whether the eye exposed in the opening 610 of the drape 600 is the right eye or the left eye. As a result, it is possible to confirm whether or not the eye is the target of surgery. In this way, left and right eye discrimination is performed.

The server device 130 (control unit 131) further reads, for example, from the plurality of surgical data 134C stored in the memory 134, the surgical data 134C corresponding to the patient 400, reads out, and includes the surgical data 134C. The eye is compared with the identification result DR, and the comparison result CR is transmitted to the information processing apparatus 160 with a camera. The information processing device 160 with a camera (control unit 111) generates a video signal for generating video light including a status ST, a comparison result CR, an identification result DR, and a marker Mc, and outputs the video signal to the projection unit 117. To do. The projection unit 117 generates image light including the status ST, the comparison result CR, the identification result DR, and the marker Mc based on the image signal from the control unit 111, and emits the image light to the eyepiece unit 252. As a result, the image light incident on the eyepiece 252 and the light guiding the observation optical system from the surgical field are combined in the eyepiece 252 and incident on the eyes of the medical staff. As a result, the medical staff visually recognizes, for example, an image in which the image shown in FIG. 31 is superimposed on the image of the surgical field.

In this modification, the server device 130 detects the segmentation image Ic obtained from the surgical field image Ia including the eye to be identified, thereby detecting the eye to be identified or the eye to be identified. The position information in the surgical field image Ia of the region including the above is acquired. Further, in this modification, the acquired position information is transmitted to the information processing apparatus 160 with a camera. As a result, in the information processing apparatus 160 with a camera, the marker Mc based on the position information can be superposed on the image of the surgical field and displayed, so that the element (in the marker Mc) that facilitates the identification of the left and right eyes ( For example, the caruncle 12 and the inner corner of the eye) can be reliably included. As a result, the medical staff adjusts the imaging region of the camera 120 so that the marker Mc is located within a predetermined region in the image of the surgical field formed by the observation optical system, so that the marker Mc is reflected in the surgical field image Ia. The left and right eye identification system 100 can correctly discriminate whether the eye is the left eye or the right eye. Therefore, it is possible to identify the type of eye that is difficult to identify.

Further, in this modified example, by detecting the object on the Segmentation image Ic, after the Detection image Id is cut out from the Segmentation image Ic, the position information of the cut out Detection image Id in the surgical field image Ia is obtained. It is the position information in the surgical field image Ia of the eye to be identified or the region including the eye to be identified. As a result, the object is simply detected for the surgical field image Ia, and the position information of the eye to be identified in the surgical field image Ia is derived based on the detection image obtained thereby. , The marker Mc can surely include an element (for example, the caruncle 12 and the inner corner of the eye) that facilitates the identification of the left and right eyes. As a result, the medical worker adjusts the imaging region of the camera 120 so that the marker Mc is located within a predetermined region in the surgical field image Ia, so that the eye reflected in the surgical field image Ia is the left eye. The left and right eye identification system 100 can correctly identify whether the eye is the right eye or the right eye. Therefore, it is possible to identify the type of eye that is difficult to identify.

[[Modification F]]
In the second embodiment, for example, as shown in FIG. 36, the server device 130 may be omitted, and the information processing device 160 with a camera may have the function of the server device 130. At this time, the information processing apparatus 160 with a camera includes, for example, a Segmentation learning model 132 and a left / right eye discrimination learning model 133, as shown in FIG. 37, and a plurality of image data 134A, a plurality of image data 134B, and a plurality of image data 134B. The operation data 134C is stored in the memory 112.

(Left and right eye identification)
FIG. 38 is a flow chart showing an example of the left / right eye identification procedure in the left / right eye identification system 700 according to this modified example. First, the user instructs the start of left / right eye identification via the input unit 114. The user instructs, for example, to start the program 112A via the input unit 114. Then, the information processing device 160 with a camera (control unit 111) reads the program 112A from the memory 112 and loads it into the control unit 111. The control unit 111 loaded with the program 112A executes a series of steps for distinguishing the left and right eyes.

Specifically, first, the control unit 111 outputs an imaging instruction to the camera 120. The camera 120 images the inside of the opening of the drape 600 according to the imaging instruction, and transmits the surgical field image Ia obtained thereby to the control unit 111.

The control unit 111 determines the brightness of the received surgical field image Ia. The control unit 111 derives, for example, the average value of the brightness of each pixel of the surgical field image Ia, and when the derived average value is less than a predetermined threshold value, it indicates that the surgical field image Ia is insufficient in brightness. It is displayed on the display unit 113.

The control unit 111 performs a focus determination, for example, when the derived average value is equal to or greater than a predetermined threshold value. For example, the control unit 111 performs Laplacian differentiation on the surgical field image Ia, and when the score obtained by the Laplacian differentiation is below a predetermined threshold value, the display unit 113 indicates that the surgical field image Ia is out of focus. Display on. For example, when the derived score is equal to or higher than a predetermined threshold value, the control unit 111 inputs the surgical field image Ia into the segmentation learning model 132.

The Segmentation learning model 132 executes a conversion process when the surgical field image Ia is input. Specifically, the Segmentation learning model 132 converts the surgical field image Ia into the Segmentation image Ic by using the learning result (see FIG. 6). The Segmentation learning model 132 inputs the Segmentation image Ic obtained by the conversion to the control unit 131. The control unit 111 detects an object with respect to the input Segmentation image Ic. For example, as shown in FIG. 8, the control unit 111 cuts out a rectangular region including the eye region Ra (for example, a rectangular region in contact with the eye region Ra) from the Segmentation image Ic. The image obtained in this way is referred to as a Detection image Id.

The control unit 111 uses the position information of the obtained Detection image Id in the surgical field image Ia as the position information in the surgical field image Ia of the region including the eye to be identified. For example, the control unit 111 uses the position information of the outer edge portion of the obtained Detection image Id in the surgical field image Ia as the position information of the marker Mc. That is, the control unit 111 acquires the position information in the surgical field image Ia of the region including the eye to be identified by detecting the object on the Segmentation image Ic.

The control unit 111 may use the position information of the obtained Detection image Id in the surgical field image Ia as the position information of the eye to be identified in the surgical field image Ia. For example, the control unit 111 may use the position information of the outer edge portion of the eye region Ra of the obtained Deception image Id in the surgical field image Ia as the position information of the marker Mc. In this case, the control unit 111 acquires the position information of the eye to be identified in the surgical field image Ia by detecting the object on the Segmentation image Ic.

The control unit 111 generates the marker Mc based on the acquired position information, then generates a video signal for generating the video light including the generated marker Mc, and outputs the video signal to the projection unit 117. The projection unit 117 generates image light including the marker Mc based on the image signal from the control unit 111, and emits the image light to the eyepiece unit 252. As a result, the image light incident on the eyepiece 252 and the light guiding the observation optical system from the surgical field are combined in the eyepiece 252 and incident on the eyes of the medical staff. As a result, the medical staff visually recognizes, for example, an image in which the marker Mc is superimposed on the image of the surgical field. At this time, the position and size of the marker Mc are defined by the video signal incident on the projection unit 117 from the control unit 111. That is, the eyepiece 252 superimposes and displays the marker Mc on the image of the surgical field formed by the observation optical system.

The left and right eye identification system 100 periodically performs a series of procedures from the above-mentioned imaging instruction to the marker Mc display. At this time, the medical staff fine-tunes the imaging region of the camera 120 while referring to the image of the visual field (surgical field) seen through the eyepiece 252. For example, each time the control unit 111 executes a series of procedures from the above-mentioned imaging instruction to the marker Mc display, the control unit 111 determines whether or not the position of the marker Mc in the surgical field image Ia protrudes from a predetermined region. As a result, when the position of the marker Mc in the surgical field image Ia protrudes from the predetermined region, the control unit 111 indicates that, for example, the position of the marker Mc in the surgical field image Ia protrudes from the predetermined region. An image light containing the meaning information is generated and emitted to the eyepiece 252. As a result, the image light incident on the eyepiece 252 and the light guiding the observation optical system from the surgical field are combined in the eyepiece 252 and incident on the eyes of the medical staff. As a result, the medical staff visually recognizes, for example, an image in which the image shown in FIG. 27 is superposed on the image of the surgical field. The control unit 111 positions the marker Mc in the surgical field image Ia by a method other than displaying the marker Mc in the visual field (operative field) visible through the eyepiece 252 (for example, sound or lighting of a patrol lamp). May inform the healthcare professional that is out of the predetermined area.

On the other hand, when the position of the marker Mc in the surgical field image Ia is within a predetermined region, the control unit 111 provides information indicating that the position of the marker Mc in the surgical field image Ia is within the predetermined region. A video signal for generating the included video light is generated and output to the projection unit 117. Based on the video signal from the control unit 111, the projection unit 117 generates video light including information indicating that the position of the marker Mc in the surgical field image Ia is within a predetermined region and the marker Mc. Then, the light is emitted to the eyepiece 252. As a result, the image light incident on the eyepiece 252 and the light guiding the observation optical system from the surgical field are combined in the eyepiece 252 and incident on the eyes of the medical staff. As a result, the medical staff visually recognizes, for example, an image in which the image shown in FIG. 28 is superimposed on the image of the surgical field. The information processing device 160 with a camera (control unit 111) is operated by a method other than displaying the marker Mc in the field of view (operative field) seen through the eyepiece 252 (for example, sound or lighting of a patrol lamp). The medical staff may be informed that the position of the marker Mc in the field image Ia is within a predetermined region.

When the position of the marker Mc is within a predetermined region in the surgical field image Ia, the control unit 111 outputs a left / right eye identification instruction to the left / right eye identification learning model 133. When the left / right eye discrimination learning model 133 receives the left / right eye discrimination instruction, the left / right eye discrimination learning model 133 executes the discrimination process. Specifically, the left-right eye discrimination learning model 133 uses the learning result to discriminate the eyes included in the Detection image Ib. More specifically, the left-right eye discrimination learning model 133 uses the learning result to identify the type of eye (left eye, right eye) included in the Detection image Ib. The left and right eye discrimination learning model 133 inputs the discrimination result DR to the control unit 111. The control unit 111 generates a video signal for generating the video light including the status ST, the identification result DR, and the marker Mc, and outputs the video signal to the projection unit 117. The projection unit 117 generates image light including the status ST, the identification result DR, and the marker Mc based on the image signal from the control unit 111, and emits the image light to the eyepiece unit 252. As a result, the image light incident on the eyepiece 252 and the light guiding the observation optical system from the surgical field are combined in the eyepiece 252 and incident on the eyes of the medical staff. As a result, the medical staff visually recognizes, for example, an image in which the image shown in FIG. 29 or FIG. 30 is superimposed on the image of the surgical field.

By looking at the identification result DR in the image of the visual field (surgical field) seen through the eyepiece 252, the medical staff can determine whether the eye exposed in the opening 610 of the drape 600 is the right eye or the left eye. As a result, it is possible to confirm whether or not the eye is the target of surgery. In this way, left and right eye discrimination is performed.

The control unit 111 further reads, for example, from the plurality of surgical data 134C stored in the memory 112, the surgical data 134C corresponding to the patient 400, and the surgical eye included in the read and surgical data 134C and the identification result. Compare with DR. The control unit 111 generates a video signal for generating video light including the status ST, the comparison result CR, the identification result DR, and the marker Mc, and outputs the video signal to the projection unit 117. The projection unit 117 generates image light including the status ST, the comparison result CR, the identification result DR, and the marker Mc based on the image signal from the control unit 111, and emits the image light to the eyepiece unit 252. As a result, the image light incident on the eyepiece 252 and the light guiding the observation optical system from the surgical field are combined in the eyepiece 252 and incident on the eyes of the medical staff. As a result, the medical staff visually recognizes, for example, an image in which the image shown in FIG. 31 is superposed on the image of the surgical field.

In this modification, the information processing device 160 with a camera has the function of the server device 130 according to the second embodiment. As a result, the left and right eyes can be identified simply by preparing the information processing device 160 with a camera. Therefore, it is possible to reduce the time and effort required to install the left and right eye identification system 700 because the server device 130 is unnecessary.

Although the present invention has been described above with reference to the embodiments and examples thereof, the present invention is not limited to the embodiments and the like, and various modifications are possible. In embodiments and the like, the present invention has been applied to eye surgery, but is suitably applicable to fields where it is necessary to identify an object that is difficult to identify. The effects described in the present specification are merely examples. The effects of the present invention are not limited to the effects described herein. The present invention may have effects other than those described herein.

10 ... Eye, 11 ... Eyeball, 12 ... Tear Hill, 100 ... Left and right eye identification system, 110 ... Information processing device, 110A ... Display surface, 111 ... Control unit, 112 ... Memory, 112A, 112B ... Program, 113 ... Display unit , 114 ... Input unit, 115, 116 ... Network IF, 117 ... Projection unit, 120 ... Camera, 130 ... Server device, 131 ... Control unit, 132 ... Segmentation learning model, 133 ... Left and right eye discrimination learning model, 134 ... Memory, 134A, 134B ... image data, 134C ... surgical data, 134D ... program, 135 ... network IF, 140, 150 ... network, 160 ... information processing device with camera, 200 ... ophthalmic surgery microscope system, 210 ... support, 220 ... 1 arm, 230 ... 2nd arm, 240 ... drive device, 250 ... medical worker microscope, 251 ... lens barrel, 252 ... eyepiece, 260 ... foot switch, 300 ... operating table, 400 ... patient, 500 ... Sheets, 600 ... Drape, 610 ... Opening, 700 ... Left and right eye identification system, CR ... Comparison result, DR ... Identification result, Ia ... Surgical field image, Ib ... Detection image, Ic ... Segment image, Id ... Detection image, Mc ... Marker, Ra ... eye area, Rb ... peripheral area, ST ... status.

Claims (15)

An acquisition unit that acquires position information in the captured image of the identification target or a region including the identification target by detecting an object on the Segmentation image obtained from the captured image including the identification target.
An information processing system including a display unit that displays an image in which a marker generated based on the position information acquired by the acquisition unit is superimposed on the captured image. The information processing system according to claim 1, wherein the display unit further displays an image including the identification result of the identification target included in the captured image as the image. An acquisition unit that acquires position information in the captured image of the identification target or a region including the identification target by detecting an object on the Segmentation image obtained from the captured image including the identification target.
An information processing system including a compositing unit that superimposes a marker generated based on the position information acquired by the acquisition unit on an image of a visual field including the identification target formed by an optical system. The information processing system according to claim 3, wherein the synthesis unit generates an image including the marker, the identification result of the identification target included in the captured image, and the image of the visual field. An acquisition unit that acquires position information in the captured image of the identification target or a region including the identification target by detecting an object on the Segmentation image obtained from the captured image including the identification target.
An information processing system including a transmission unit that transmits the position information acquired by the acquisition unit to an information processing device including a display unit. The acquisition unit cuts out a Detection image from the Segmentation image by detecting an object on the Segmentation image, and obtains the position information of the cut out Detection image in the captured image as the identification target or the identification. The information processing system according to any one of claims 1 to 5, which is used as position information in the captured image of a region including an object. The display unit displays, when the marker protrudes from a predetermined area, an image including information indicating that the marker protrudes from the predetermined area as the image. Alternatively, the information processing system according to claim 2. When the marker protrudes from a predetermined region, the synthesis unit includes an image including the marker, information meaning that the marker protrudes from the predetermined region, and an image of the visual field. The information processing system according to claim 3 or 4, which is generated. Claims 1 to claim 1 further include a first communication unit that transmits the captured image obtained when the marker is within a predetermined area to an external device and receives the identification result from the external device. The information processing system according to any one of 4. The information processing system according to claim 2 or 4, further comprising an identification unit that identifies the identification target included in the captured image obtained when the marker is within a predetermined area. The information processing system according to claim 9, wherein the first communication unit receives the captured image transmitted from the imaging device. The information processing system according to claim 9, further comprising a second communication unit that receives the captured image transmitted from the image pickup device by wireless communication. An acquisition step of acquiring position information in the captured image of the identification target or a region including the identification target by performing object detection on the Segmentation image obtained from the captured image including the identification target.
An information processing program that causes a computer to execute a generation step of generating a video signal for displaying an image in which a marker generated based on the position information acquired in the acquisition step is superimposed on the captured image. An acquisition step of acquiring position information in the captured image of the identification target or a region including the identification target by performing object detection on the Segmentation image obtained from the captured image including the identification target.
A generation step of generating a video signal for generating projected light for superimposing a marker generated based on the position information acquired in the acquisition step on an image of a visual field including the identification target formed by an optical system. An information processing program that causes a computer to execute and. An acquisition step of acquiring position information in the captured image of the identification target or a region including the identification target by performing object detection on the Segmentation image obtained from the captured image including the identification target.
An information processing program that causes a computer to execute a transmission step of transmitting the position information acquired in the acquisition step to an information processing device provided with a display unit.