JP2017176414A - Control device, method and surgery system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To execute calibration easily and surely.SOLUTION: A control device acquires a detection value of a sight line by an input part, and detects a reference point by a reference point detection part. The control device estimates that the sight line points to the reference point. When it is estimated that the sight line points to the reference point by the estimation part, the control device corrects the detection value of the sight line by a correction part based on a difference between the detection value of the sight line and the reference point. The technique can be applied for, for example, when an operator performs surgery using a surgical tool, correcting a detection value of a sight line of the operator.SELECTED DRAWING: Figure 5

Description

  The present technology relates to a control apparatus and method and a surgical system, and more particularly to a control apparatus and method and a surgical system which can perform calibration easily and reliably.

  It has been proposed to control the operation of a computer by detecting the line of sight of the operator and selecting menus, objects, etc. displayed on the monitor by the line of sight. In order to realize such control, it is necessary to accurately detect the line of sight. For this reason, calibration is required to correct the deviation between the actual line-of-sight position and the detected line-of-sight position.

  For example, Patent Document 1 proposes correcting a shift based on a relationship between a selected object and another displayed object.

  Japanese Patent Application Laid-Open No. 2004-228561 proposes correcting a deviation based on the position of an object when a voice keyword is detected.

JP 2012-65781 A JP 2007-25963 A

  However, according to the above proposal, the calibration becomes simpler than when correction is performed on a calibration-dedicated screen, but the accuracy is not sufficient, and it is difficult to apply to a medical system in particular. there were.

  The present technology has been made in view of such a situation, and makes it possible to execute calibration easily and reliably.

  One aspect of the present technology includes an input unit that acquires a detection value of a line of sight, a reference point detection unit that detects a reference point, an estimation unit that estimates that the line of sight is directed to the reference point, and the estimation A control unit that corrects the detection value of the line of sight based on a difference between the detection value of the line of sight and the reference point when the line of sight estimates that the line of sight is directed to the reference point Device.

  The line of sight is a line of sight during an operation of an operator who operates using a surgical instrument.

  The estimation unit can estimate that the line of sight is directed to the reference point when the surgical instrument is in use.

  The surgical instrument can be used while being energized.

  The reference point detection unit detects a tip of the surgical tool used by being energized as the reference point, and the correction unit corresponds to an error between the detected value of the line of sight and the tip of the surgical tool, The line-of-sight detection value can be corrected.

  The reference point detection unit detects a trajectory of the surgical tool used by being energized as the reference point, and the correction unit corresponds to an error between the detected value of the line of sight and the trajectory of the surgical tool, The line-of-sight detection value can be corrected.

  When the pattern of the object displayed on the monitor is detected and the motion based on the pattern of the object displayed on the monitor is switched, the estimation unit directs the line of sight toward the reference point. Can be estimated.

  The reference point detection unit detects an object displayed on a monitor when receiving a voice command as the reference point, and the estimation unit detects the display of the object that receives the voice command and the line of sight is the reference It can be estimated that the point is oriented.

  One aspect of the present technology includes an input step of acquiring a detection value of a line of sight, a reference point detection step of detecting a reference point, an estimation step of estimating that the line of sight is directed to the reference point, and the estimation A correction step of correcting the detected value of the line of sight based on a difference between the detected value of the line of sight and the reference point when it is estimated that the line of sight is directed to the reference point by the processing of the step; It is a control method including.

  One aspect of the present technology is an imaging unit that images the surgical field of the subject, a display unit that displays the surgical field image captured by the imaging unit, an input unit that acquires a detection value of the line of sight, and a reference point A reference point detection unit that detects the line of sight, an estimation unit that estimates that the line of sight is directed to the reference point, and the estimation unit that estimates that the line of sight is directed to the reference point And a correction unit that corrects the detected value of the line of sight based on a difference between the detected value of the line of sight and the reference point.

  In one aspect of the present technology, a gaze detection value is obtained, a reference point is detected, a gaze is estimated to be directed to the reference point, and a gaze is directed to the reference point Sometimes, the line-of-sight detection value is corrected based on the difference between the line-of-sight detection value and the reference point.

  As described above, according to one aspect of the present technology, calibration can be performed easily and reliably.

  Note that the effects described in this specification are merely examples, and are not limited, and may have additional effects.

It is a perspective view showing typically the composition of the operation system of one embodiment of this art. It is a block diagram which shows the functional structure of the surgery system of one embodiment of this technique. It is a block diagram which shows the functional structure of the surgery system of one embodiment of this technique. It is a flowchart explaining the gaze correction process of one embodiment of this technique. It is a figure explaining line-of-sight correction processing of one embodiment of this art. It is a figure explaining line-of-sight correction processing of one embodiment of this art. It is a flowchart explaining the gaze correction process of one embodiment of this technique. It is a figure explaining line-of-sight correction processing of one embodiment of this art. It is a flowchart explaining the gaze correction process of one embodiment of this technique. It is a figure explaining line-of-sight correction processing of one embodiment of this art.

Hereinafter, an embodiment for carrying out the present technology will be described. The description will be given in the following order.
1. Embodiment: Surgery System (1) Configuration of Surgery System (Fig. 1)
(2) Functional configuration of the surgical system (FIGS. 2 and 3)
(3) Gaze correction processing (FIGS. 4 to 10)
2. Other

<Embodiment>
(1) Configuration of surgical system (Fig. 1)
FIG. 1 is a perspective view schematically illustrating a configuration of a surgical system according to an embodiment of the present technology.

  The surgical system 10 includes a surgical field camera 11, a camera arm 12, a control device 13, a monitor arm 14, a monitor 15, a motion recognition camera 16, a foot switch 17, a microphone 18, and an operating table 19. The surgical system 10 is disposed in an operating room or the like, and enables a procedure such as a surgical operation with reference to an image taken by the surgical field camera 11.

  Specifically, the surgical field camera 11 (surgical imaging device) of the surgical system 10 is a modality device such as a 3D camera held by the camera arm 12. The operative field camera 11 images the operative field or the like of the patient 32 lying on the operating table 19 and transmits the 3D image obtained as a result to the control device 13 as an operative field image. The camera arm 12 holds the operative field camera 11 and controls the position and angle of the operative field camera 11.

  The motion recognition camera 16 is a 2D camera, for example, and is disposed on the monitor 15. The motion recognition camera 16 photographs the operator 31 with the microphone 18 mounted on the head 31A. The motion recognition camera 16 transmits a 2D image obtained as a result of imaging to the control device 13 as an operator image. The monitor arm 14 holds the monitor 15 and controls the positions and angles of the monitor 15 and the motion recognition camera 16.

  The monitor 15 is a 3D monitor having a screen for displaying an image, and is disposed at a position relatively close to the operator 31 (in the example of FIG. 1, a position close to the operator 31 on the operating table 19). The monitor 15 displays an operation field image transmitted from the control device 13. An action recognition camera 16 is disposed on the monitor 15.

  The control device 13 sets the operation mode to the manual mode or the hands-free mode. The manual mode is a mode in which the surgical system 10 is controlled based on an input by the operator 31 (for example, an applied force to the camera arm 12 or an operation of an operation button (not shown) provided in each part). is there. The hands-free mode is based on non-contact input such as voice, line of sight, movement and direction of the head 31A, gestures, etc., and input by contact of the foot 31B to the foot foot switch 17 without depending on the hand of the operator 31. In this mode, the operation system 10 is controlled.

  Further, the control device 13 recognizes the movement and direction of the head 31 </ b> A by detecting the position of the head 31 </ b> A in the surgeon image transmitted from the motion recognition camera 16. The control device 13 detects the direction of the line of sight of the operator 31 from the surgeon image, and recognizes the position of the line of sight on the screen of the monitor 15 based on the direction. Furthermore, the control device 13 recognizes the gesture of the operator 31 from the operator image.

  The control device 13 receives the voice transmitted from the microphone 18 and performs voice recognition on the voice. The control device 13 receives an operation signal representing an operation on the foot switch 17 transmitted from the foot switch 17, and recognizes the content of the operation on the foot switch 17 based on the operation signal.

  Furthermore, when the operation mode is the hands-free mode, the control device 13 uses the following information as input information. That is, the movement and direction of the head 31 </ b> A, the gesture of the surgeon 31, line-of-sight position information indicating the position of the line of sight on the screen of the monitor 15, voice recognition result, volume, and operation information indicating the content of operations on the foot switch 17 Input information. The control device 13 recognizes the command from the operator 31 and the state of the operator 31 based on the input information.

  The control device 13 permits a command from the operator 31 according to the state of the operator 31. The control device 13 controls the imaging of the operative field camera 11, controls the drive of the camera arm 12 and the monitor arm 14, controls the display of the monitor 15, and changes the operation mode according to the permitted command. Or change it.

  The surgeon 31 recognizes the operative field image displayed on the monitor 15 with the naked eye as a 3D image.

  The microphone 18 is attached to the head 31 </ b> A of the operator 31. The microphone 18 acquires surrounding sounds including the voice of the operator 31 and transmits it to the control device 13.

  The foot switch 17 is disposed around the operator 31 and is operated by contact of the leg 31B of the operator 31. The foot switch 17 transmits an operation signal representing the operation of the leg 31 </ b> B from the operator 31 to the control device 13.

  In the surgical operation system 10 configured as described above, the surgeon 31 lays the surgical subject 32 on the operating table 19 and looks at the surgical field image displayed on the monitor 15 while performing surgery and the like. Take action.

  Further, the operator 31 performs non-contact input or input by touching the foot when changing the operation mode, the imaging conditions of the surgical field camera 11, the position and angle of the surgical field camera 11, the display of the monitor 15, and the like. Therefore, the operator 31 can perform input while holding a surgical tool (not shown). Further, the surgeon 31 does not need to perform sterilization each time input is performed.

  It should be noted that any method can be adopted as the line-of-sight detection method, the movement and direction of the head 31A of the operator 31, the gesture detection method, and the voice acquisition method. For example, the line-of-sight detection device or the microphone 18 may not be a wearable device.

  In this specification, the horizontal direction of the monitor 15 is referred to as the x direction, the vertical direction is referred to as the y direction, and the direction perpendicular to the screen of the monitor 15 is referred to as the z direction.

  In the surgical system 10, the line of sight detection is performed using an operator image taken by the motion recognition camera 16, but the operator 31 wears glasses equipped with a line of sight detection device, and the line of sight is detected. The detection device may perform line-of-sight detection.

  In the surgical system 10, since the distance between the motion recognition camera 16 and the operator 31 is short, the movement and direction of the head 31 </ b> A are detected from the operator image, but the operator 31 wears a marker (not shown). Then, the movement and direction of the head 31A may be detected from the position of the marker in the operator image.

  Further, the monitor 15 may be arranged at a position relatively far from the operator 31. The monitor 15 may be a 3D monitor that allows the operator 31 to recognize a 3D image by using the 3D polarized glasses, and the operator 31 may wear the 3D polarized glasses.

(2) Functional Configuration of Surgery System FIG. 2 is a block diagram illustrating a functional configuration of the surgery system according to the embodiment of the present technology. In this embodiment, the control device 13 is configured to control operations of the operative field camera 11, the camera arm 12, the monitor 15, the motion recognition camera 16, and the like, and predetermined information is received from the information input interface 51. It is configured to be entered.

  The information input interface 51 includes an audio detection unit 61, a line-of-sight detection unit 62, a head track unit 63, a gesture detection unit 64, a foot switch 65, and the like. The sound detection unit 61 includes the microphone 18 as a component, and detects sound emitted from the operator 31 and the like. The line-of-sight detection unit 62 detects the line of sight of the operator 31 from the image signal of the motion recognition camera 16. The head track unit 63 detects the movement of the head 31 </ b> A of the operator 31 from the image signal of the motion recognition camera 16. The gesture detection unit 64 detects the gesture of the operator 31 from the image signal of the motion recognition camera 16. Therefore, it can be considered that the line-of-sight detection unit 62, the head track unit 63, and the gesture detection unit 64 are configured in the image signal processing unit 74. The foot switch 65 detects the operation of the foot switch 17 in FIG. 1 and outputs a signal corresponding to the operation.

  The control device 13 includes a sensor input unit 71, a calibration unit 72, an operation control unit 73, and an image signal processing unit 74. The sensor input unit 71 inputs a signal from the information input interface 51 and outputs it to the calibration unit 72 and the operation control unit 73. The calibration unit 72 includes a detection value correction unit 81 and a correction value update unit 82.

  The detection value correction unit 81 and the correction value update unit 82 are input with the detected value of the line of sight of the operator 31 detected by the line of sight detection unit 62 via the sensor input unit 71. The detection value correction unit 81 corrects the input detection value based on the correction value supplied from the correction value update unit 82, and outputs the corrected detection value to the operation control unit 73. The correction value update unit 82 receives the corrected detection value from the detection value correction unit 81 as the current detection value, and is supplied from the line-of-sight detection unit 62 via the sensor input unit 71. Compared with, a new correction value is calculated as necessary. The calculation result is output to the detection value correction unit 81.

  The operation control unit 73 of the control device 13 controls the operation of the image signal processing unit 74 in addition to controlling the operation of the operative field camera 11, the camera arm 12, the monitor 15, and the operation recognition camera 16. The image signal processing unit 74 processes image signals of images taken by the operative field camera 11 and the motion recognition camera 16 and outputs them to the monitor 15 for display.

  The operation performed by the operator 31 is supported by the operation system 10 having the above basic configuration. That is, a predetermined part of the patient 32 is photographed by the operative field camera 11 and the image is displayed on the monitor 15. The surgeon 31 performs an operation while viewing the image.

  FIG. 3 is a block diagram illustrating a functional configuration of the surgical operation system according to the embodiment of the present technology. The surgical operation system 10 according to the present embodiment is configured as shown in FIG. 3 so that calibration can be easily and reliably performed in addition to the basic configuration shown in FIG. That is, the image signal processing unit 74 includes the reference point detection unit 101. The reference point detection unit 101 detects a reference point from the image signal output from the operative field camera 11 and supplies it to the correction value update unit 82. The correction value update unit 82 calculates a correction value using the reference point.

  As the reference point, a display position of a surgical instrument used during surgery, a display position of a predetermined object, or the like can be used.

  The control device 13 includes an information input unit 111. The information input unit 111 inputs information related to the operation of the surgeon 31 during the operation, and performs a determination process for correction timing. For example, when the surgeon 31 uses a surgical instrument (see FIG. 5 described later), a determination process is performed upon receiving an input of a signal indicating that the surgical instrument is being used. Further, the information input unit 111 determines from the output of the operation control unit 73 whether an instruction to switch the operation mode of the camera arm 12 or the monitor arm 14 or an instruction to switch the system operation by line of sight has been issued. Further, the information input unit 111 determines whether a predetermined object (see FIG. 10 described later) for receiving a voice command is displayed. The determination result is output to the correction value update unit 82. Other configurations are the same as those of the surgical system 10 of FIG.

(3) Gaze correction processing (3-1) Gaze correction processing 1
Next, the first line-of-sight correction process of the surgical system 10 will be described. FIG. 4 is a flowchart for describing line-of-sight correction processing according to an embodiment of the present technology.

  In step S <b> 11, the line-of-sight detection unit 62 detects the line of sight of the operator 31. That is, the motion recognition camera 16 captures the head 31A (in this case, the face) of the operator 31 and detects the line of sight from the position of the eyes. The line-of-sight detection process is continued thereafter, and the detection value correction unit 81 and the correction value update unit 82 always receive the current line-of-sight detection value.

  In step S12, the detected value correction unit 81 calibrates the initial line of sight. This calibration process is performed by displaying a calibration-dedicated screen on the monitor 15 so that the operator 31 can visually recognize it and correcting the error. This calibration process is initially performed only once.

  In step S13, the correction value update unit 82 updates the correction value. In this case, the correction value is updated based on the calibration result. The detection value correction unit 81 corrects the detection value of the line of sight that is currently detected based on the correction value.

  In step S14, the reference point detection unit 101 detects a reference point. Reference points will be described with reference to FIG. FIG. 5 is a diagram for describing line-of-sight correction processing according to an embodiment of the present technology. FIG. 5 shows an image taken by the operative field camera 11 and displayed on the monitor 15. In FIG. 5, an electric knife is shown as a surgical instrument 201 used by the surgeon 31 energized. Here, the distal end 202 of the surgical instrument 201 is used as a reference point. The reference point position information detected by the reference point detection unit 101 is supplied to the correction value update unit 82.

  In step S15, the correction value update unit 82 calculates an error. That is, an error between the current gaze detection value (that is, the gaze detection position coordinates) and the reference point coordinates is calculated. In the example of FIG. 5, the line-of-sight detection position 211 is located far from the distal end 202 of the surgical instrument 201 and is located at the upper right thereof, and an error between the position of the line-of-sight detection position 211 and the distal end 202 is calculated.

  In step S16, the correction value update unit 82 determines the error amount. That is, it is determined whether the error amount calculated in step S15 is within the reference range. When the error amount is within the reference range, that is, when the error amount is not less than the small reference value and not more than the large reference value, the correction value update unit 82 determines a correction value in step S17. Specifically, the error value is set as the correction value.

  In step S18, the information input unit 111 (estimation unit) determines the correction timing. That is, it is determined whether it is a correction timing now. For example, when the surgeon 31 uses an electric knife as the surgical instrument 201, a switch for turning on the power is operated in order to turn on the power. Alternatively, the foot switch 17 is operated. When using the electric knife, the operator 31 is surely viewing the tip 202. Therefore, when a signal indicating that the surgical instrument 201 (in this case, an electric knife) is used is input, it is estimated that the operator 31 is surely viewing the tip 202. That is, the error amount at that time is not an accidental error but an error amount that is correctly detected.

  Therefore, when the surgical instrument 201 is being used, it is determined in step S18 that the correction timing is reached, the process returns to step S13, and a process for updating the correction value is executed. That is, the correction value update unit 82 updates the original correction value with the correction value based on the error amount determined in step S17. The detection value correction unit 81 corrects the detection value of the line of sight based on the correction value. That is, in FIG. 5, the current gaze detection position 211 is corrected so as to coincide with the position of the tip 202 as a reference point. As a result, the detected value can be reliably corrected to the correct position value.

  Thereafter, a new reference point is detected in step S14 for the new line-of-sight detection value, and the same processing is repeated.

  When it is determined in step S18 that it is not the timing for correction, that is, when the surgical instrument 201 is not used, it cannot be estimated that the operator 31 is viewing the distal end 202 of the surgical instrument 201. Therefore, in this case, the correction value update unit 82 discards the correction value in step S19. That is, the correction value determined in step S17 is discarded. Then, the process returns to step S14, and the subsequent processes are repeated. That is, the same processing is repeated based on the new detection value and the reference point.

  If the amount of error is smaller than the lower reference value, it is considered that there is almost no error, so there is no need for correction. If the error amount is larger than the large reference value, it is considered that an abnormal value has been detected accidentally. Therefore, if it is determined in step S16 that the error amount is not within the reference range, the process returns from step S16 to step S14. Then, a new line-of-sight detection value and a new reference point are detected, and the same processing is repeated.

  FIG. 6 is a diagram for describing line-of-sight correction processing according to an embodiment of the present technology. In FIG. 6, a line 251 represents the locus of the distal end 202 of the surgical instrument 201. In other words, the trajectory when the surgical instrument 201 is sequentially moved to the surgical instruments 201A, 201B, and 201C is represented as a line 251.

  A line 252 represents the actual line of sight of the operator 31. The actual line-of-sight line 252 of the surgeon 31 is a line that approximates the line 251 of the path of the distal end 202 of the surgical instrument 201. On the other hand, a line 253 represents the locus of the eye gaze detection position of the operator 31. The line 253 representing the locus of the visual line detection position is separated from the line 252 of the actual visual locus of the operator 31. However, the line 253 representing the locus of the line-of-sight detection position can be brought close to the line 252 of the locus of the actual line of sight of the operator 31 by the correction processing described above.

  The locus lines 251 and 253 may be calculated and the correction process may be performed based on the calculated lines.

  As described above, the calibration is easily and reliably performed on the back of the operator 31 while performing an operation necessary for the operation without feeling a special burden.

(3-2) Line-of-sight correction processing 2
Next, the second line-of-sight correction process will be described with reference to FIG. FIG. 7 is a flowchart for describing line-of-sight correction processing according to an embodiment of the present technology.

  In step S31, the line-of-sight detection unit 62 detects the line of sight of the operator 31. That is, the motion recognition camera 16 captures the head 31A (in this case, the face) of the operator 31 and detects the line of sight from the position of the eyes. The line-of-sight detection process is continued thereafter, and the detection value correction unit 81 and the correction value update unit 82 always receive the current line-of-sight detection value.

  In step S32, the detection value correcting unit 81 calibrates the initial line of sight. This calibration process is performed by displaying a calibration-dedicated screen on the monitor 15 so that the operator 31 can visually recognize it and correcting the error. This calibration process is initially performed only once.

  In step S33, the correction value update unit 82 updates the correction value. In this case, the correction value is updated based on the calibration result. The detection value correction unit 81 corrects the detection value based on the correction value. The processing so far is the same as the processing in steps S11 to S13 in FIG.

  In step S34, the line-of-sight detection unit 62 extracts a pattern from the line-of-sight detection frequency. Here, the pattern will be described with reference to FIG.

  FIG. 8 is a diagram for describing line-of-sight correction processing according to an embodiment of the present technology. As shown in FIG. 8, five objects 311 to 315 are displayed in an area 301 in the lower center of the monitor 15. These are, for example, objects related to the operation mode switching operation of the camera arm 12 and the monitor arm 14 during the operation of the surgical system 10 or the system operation switching operation based on the line of sight.

  The operator 31 selects and sets a predetermined operation mode by visually recognizing these objects. That is, this visual recognition is not performed for calibration but is performed as part of an operation for surgery. In this case, the operator 31 visually recognizes the objects 311 to 315 in the region 301. As a result, the time for which the line of sight is directed to each of the objects 311 to 315 becomes longer, and the frequency of detection of the line of sight at that position is higher than in other areas. In the example of FIG. 8, the line-of-sight detection positions 341 to 345 in the region 331 are detected as positions where the line-of-sight detection frequency is high. That is, in this example, detection is performed based on the accumulated value of the line of sight, not the instantaneous line of sight.

  In this way, the line-of-sight detection unit 62 detects, as a pattern, an area where the line-of-sight detection frequency is greater than other areas. In this example, a state in which five frequently detected points are arranged at a predetermined interval is detected as a pattern so that the points are arranged almost on a straight line.

  In step S35, the reference point detection unit 101 detects a reference point. In the example of FIG. 8, a pattern displayed at a predetermined interval so that the objects 311 to 315 are arranged in a straight line in the region 301 is detected as a reference point. The reference point position information detected by the reference point detection unit 101 is supplied to the correction value update unit 82.

  In step S36, the correction value update unit 82 determines whether the patterns match. That is, the pattern extracted in step S34 is compared with the pattern detected as the reference point in step S35.

  If it is determined that the pattern extracted in step S34 matches the pattern detected as the reference point in step S35, the correction value update unit 82 calculates an error in step S37. That is, an error between the current gaze detection value (that is, the gaze detection position coordinates) and the reference point coordinates is calculated. In the example of FIG. 8, the line-of-sight detection positions 341 to 345 in the area 331 are located in the upper right direction so that they are slightly rotated clockwise from the objects 311 to 315 in the area 301. Here, the position error between the region 331 where the line-of-sight detection positions 341 to 345 exist and the region 301 where the objects 311 to 315 exist is calculated.

  In step S38, the correction value update unit 82 determines the amount of error. That is, it is determined whether the error amount calculated in step S37 is within the reference range. When the error amount is within the reference range, that is, when the error amount is not less than the small reference value and not more than the large reference value, the correction value update unit 82 determines the correction value in step S39. Specifically, the error value is set as the correction value.

  In step S40, the information input unit 111 determines the correction timing. That is, it is determined whether it is a correction timing now. The operation control unit 73 controls the operative field camera 11, the camera arm 12, the monitor arm 14, the image signal processing unit 74, the monitor 15, and the like, and has control information thereof. The information input unit 111 monitors the control information of the operation control unit 73 and detects when an instruction to switch the operation mode of the camera arm 12 or the monitor arm 14 or an instruction to switch the system operation by line of sight is issued. To do. When the detection is performed by the information input unit 111, it can be estimated that the operator 31 is viewing the objects 311 to 315 for the switching. That is, the error amount at that time is not an accidental error but an error amount that is correctly detected.

  Therefore, when the information input unit 111 determines that an operation mode switching instruction or a system operation switching instruction based on the line of sight has been input, it is determined in step S40 that the correction timing is reached, and the process returns to step S33. . And the process which updates a correction value is performed. That is, the correction value update unit 82 updates the original correction value with the correction value based on the error amount determined in step S39. The detection value correction unit 81 corrects the detection value of the line of sight based on the correction value. That is, in FIG. 8, the correction is performed so that the position of the area 331 matches the position of the area 301. That is, the current gaze detection positions 341 to 345 are corrected so as to coincide with the positions of the objects 311 to 315 as reference points. As a result, the detected value can be reliably corrected to the correct position value.

  Thereafter, a new line-of-sight detection value is detected, a new pattern is extracted in step S34, and the same processing is repeated.

  When an instruction to switch the operation mode or an instruction to switch the system operation by line of sight has not been detected by the information input unit 111, it is determined in step S40 that it is not the timing for correction. That is, at this time, it cannot always be estimated that the operator 31 is viewing the region 301. Therefore, at this time, the correction value update unit 82 discards the correction value in step S41. That is, the correction value determined in step S39 is discarded. Then, the process returns to step S34, and the subsequent processes are repeated. That is, the same processing is repeated based on the new pattern.

  If it is determined in step S36 that the patterns do not match, the surgeon 31 has visually recognized a pattern different from the extracted pattern. If the correction is made in this case, the error becomes larger, and the correction should not be executed. Therefore, in this case, the process returns from step S36 to step S34. Then, a new pattern is extracted and the same process is repeated.

  When the amount of error is smaller than the small reference value, it is considered that there is almost no error, so there is no need for correction. If the error amount is larger than the large reference value, it is considered that an abnormal value has been detected accidentally. Therefore, if it is determined in step S38 that the error amount is not within the reference range, the process returns from step S38 to step S34. Then, a new pattern is extracted and the same process is repeated.

  As described above, the calibration is easily and reliably performed on the back of the operator 31 while performing an operation necessary for the operation without feeling a special burden.

(3-3) Line-of-sight correction processing 3
Next, the third line-of-sight correction process will be described with reference to FIG. FIG. 9 is a flowchart for describing line-of-sight correction processing according to an embodiment of the present technology.

  In step S <b> 51, the line-of-sight detection unit 62 detects the line of sight of the operator 31. That is, the motion recognition camera 16 captures the head 31A (in this case, the face) of the operator 31 and detects the line of sight from the position of the eyes. The line-of-sight detection process is continued thereafter, and the detection value correction unit 81 and the correction value update unit 82 always receive the current line-of-sight detection value.

  In step S52, the detection value correction unit 81 calibrates the initial line of sight. This calibration process is performed by displaying a calibration-dedicated screen on the monitor 15 so that the operator 31 can visually recognize it and correcting the error. This calibration process is initially performed only once.

  In step S53, the correction value update unit 82 updates the correction value. In this case, the correction value is updated based on the calibration result. The detection value correction unit 81 corrects the detection value based on the correction value.

  In step S54, the reference point detection unit 101 detects a reference point. The process so far is the same as the process of step S11 thru | or step S14 of FIG. Here, reference points in this example will be described.

  FIG. 10 is a diagram for describing line-of-sight correction processing according to an embodiment of the present technology. In FIG. 10, an object 401 is displayed on the upper left of the monitor 15. This object 401 is displayed at a timing when an input of a voice command from the surgeon 31 is accepted, and only an input of a voice command issued by the surgeon 31 while viewing the object 401 is accepted.

  That is, in order for the surgeon 31 to input a voice command, the object 401 must be visually recognized. Therefore, the operator 31 needs to visually recognize the object 401 at the timing when the voice command is input. Therefore, in this example, the object 401 is detected as a reference point. The reference point position information detected by the reference point detection unit 101 is supplied to the correction value update unit 82.

  In step S55, the voice detection unit 61 determines whether a voice command has been recognized. The voice command recognized here is not for calibration, but is issued as part of an operation for surgery. In this case, the operator 31 visually recognizes the object 401. When it is determined that the voice command is recognized, the correction value update unit 82 calculates an error in step S56. That is, an error between the current gaze detection value (that is, the gaze detection position coordinates) and the reference point coordinates is calculated. In the example of FIG. 10, an error between the line-of-sight detection position 402 and the position of the object 401 is calculated.

  In step S57, the correction value update unit 82 determines the amount of error. That is, it is determined whether the error amount calculated in step S56 is within the reference range. When the error amount is within the reference range, that is, when the error amount is not less than the small reference value and not more than the large reference value, the correction value update unit 82 determines the correction value in step S58. Specifically, the error value is used as the correction value.

  In step S59, the information input unit 111 determines the correction timing. That is, it is determined whether it is a correction timing now. The reference point detection unit 101 detects the object 401 as a reference point. This object 401 is displayed at a timing when an input of a voice command from the surgeon 31 is accepted, and only an input of a voice command issued by the surgeon 31 while viewing the object 401 is accepted. Accordingly, when a voice command is recognized at the timing when the object 401 is displayed, it can be estimated that the surgeon 31 is viewing the object 401. That is, the error amount at that time is not an accidental error but an error amount that is correctly detected.

  Since the voice command has already been detected in step S55, it can be estimated that the surgeon 31 is viewing the object 401 if the object 401 is displayed that the voice command cannot be accepted unless it is visually recognized. . Therefore, when the object 401 is displayed that the voice command cannot be received unless it is visually recognized, it is determined in step S59 that the correction timing is reached, the process returns to step S53, and the process of updating the correction value is executed. Is done. That is, the correction value update unit 82 updates the original correction value with the correction value based on the error amount determined in step S58. The detection value correction unit 81 corrects the detection value of the line of sight based on the correction value. That is, in FIG. 10, the current gaze detection position 402 is corrected so as to coincide with the position of the object 401 as a reference point. As a result, the detected value can be reliably corrected to the correct position value.

  Thereafter, a new reference point is detected in step S54, and the same processing is repeated.

  When the amount of error is smaller than the small reference value, it is considered that there is almost no error, so there is no need for correction. If the error amount is larger than the large reference value, it is considered that an abnormal value has been detected accidentally. Therefore, when it is determined in step S57 that the error amount is not within the reference range, the correction value update unit 82 discards the correction value in step S60. That is, the correction value determined in step S56 is discarded. Then, the process returns to step S54, and the subsequent processes are repeated. That is, new line-of-sight detection points and reference points are detected, and the same processing is repeated.

  Even when it is determined in step S55 that the voice command is not recognized, the process returns to step S54, and the subsequent processes are repeated. That is, new line-of-sight detection points and reference points are detected, and the same processing is repeated.

  As described above, the calibration is easily and reliably performed on the back of the operator 31 while performing an operation necessary for the operation without feeling a special burden.

  In the above description, the line-of-sight correction processes in FIGS. 4, 7, and 9 have been individually described. However, they may be performed in parallel at the same time, or may be repeatedly performed in series in a predetermined order. You may make it do.

  In this specification, the system means a set of a plurality of components (devices, modules (parts), etc.), and it does not matter whether all the components are in the same housing. Accordingly, a plurality of devices housed in separate housings and connected via a network and a single device housing a plurality of modules in one housing are all systems. .

  In addition, the effect described in this specification is an illustration to the last, and is not limited, There may exist another effect.

  The embodiments of the present technology are not limited to the above-described embodiments, and various modifications can be made without departing from the gist of the present technology.

  For example, in the first embodiment, the control device 13 performs both control based on a combination of a plurality of types of non-contact inputs and restricts control according to the state of the operator 31. Thus, although the safety of the operation is improved, the safety of the operation may be improved by only one of them.

  Further, the control target by the control device 13 may be anything as long as it is a surgical device. For example, the control device 13 can also control a surgical imaging device such as an endoscope or a video microscope.

  Furthermore, the zoom control may be performed by processing the surgical field image in the image signal processing unit 74 instead of being performed by the imaging control of the surgical field camera 11.

  In this case, the image signal processing unit 74 enlarges the surgical field image transmitted from the surgical field camera 11 in response to the zoom-in photographing command, thereby zooming in on the subject corresponding to the position of the line of sight. An electronic zoom is performed to generate an image from an operative field image. Similarly, the image signal processing unit 74 performs zoom-out shooting around the subject corresponding to the position of the line of sight by reducing the surgical field image transmitted from the surgical field camera 11 in response to the zoom-out shooting command. A zoomed-out image is generated from the operative field image. At this time, the image signal processing unit 74 may superimpose a marker at a position corresponding to the line of sight in the zoom-in image or the zoom-out image based on the line-of-sight position information.

  Further, annotation display may always be performed while the surgical field image is displayed on the monitor 15. The non-contact input is not limited to the voice of the operator 31, the line of sight, the movement and direction of the head 31A, and the gesture. For example, the movement or posture of the operator 31 other than the head 31A may be used.

  The means for accepting non-contact input may be wearable like the microphone 18 or may not be wearable.

  Even when the operation mode is the manual mode, the control device 13 estimates the state of the operator 31 and controls the operative field camera 11, the camera arm 12, and the image signal processing unit 74 according to the state. You may make it restrict | limit.

<Others>
In addition, this technique can also take the following structures.
(1)
An input unit for obtaining a detection value of the line of sight;
A reference point detector for detecting a reference point;
An estimation unit that estimates that the line of sight is directed to the reference point;
A correction unit that corrects the detection value of the line of sight based on a difference between the detection value of the line of sight and the reference point when the estimation unit estimates that the line of sight is directed to the reference point; A control device provided.
(2)
The control device according to (1), wherein the line of sight is a line of sight during an operation of an operator who operates using a surgical instrument.
(3)
The control unit according to (1) or (2), wherein the estimation unit estimates that the line of sight is directed toward the reference point when the surgical instrument is in use.
(4)
The control device according to (1), (2), or (3), wherein the surgical instrument is used by being energized.
(5)
The reference point detection unit detects the distal end of the surgical instrument used by being energized as the reference point,
The control device according to any one of (1) to (4), wherein the correction unit corrects the detection value of the line of sight according to an error between the detection value of the line of sight and a distal end of the surgical instrument.
(6)
The reference point detection unit detects a trajectory of the surgical tool used by being energized as the reference point,
The control device according to any one of (1) to (5), wherein the correction unit corrects the detection value of the line of sight according to an error between the detection value of the line of sight and a trajectory of the surgical instrument.
(7)
When the pattern of the object displayed on the monitor is detected and the motion based on the pattern of the object displayed on the monitor is switched, the estimation unit directs the line of sight toward the reference point. The control device according to any one of (1) to (6).
(8)
The reference point detection unit detects an object displayed on a monitor when receiving a voice command as the reference point,
The control device according to any one of (1) to (7), wherein when the display of the object that receives the voice command is detected, the estimation unit estimates that the line of sight is directed to the reference point.
(9)
An input step for obtaining a detected value of the line of sight;
A reference point detecting step for detecting a reference point;
An estimation step for estimating that the line of sight is directed to the reference point;
A correction step of correcting the detected value of the line of sight based on the difference between the detected value of the line of sight and the reference point when it is estimated that the line of sight is directed to the reference point by the process of the estimating step. A control method including and.
(10)
An imaging unit for imaging the surgical field of the subject;
A display unit for displaying an operative field image photographed by the photographing unit;
An input unit for obtaining a detection value of the line of sight;
A reference point detector for detecting a reference point;
An estimation unit that estimates that the line of sight is directed to the reference point;
A correction unit that corrects the detection value of the line of sight based on a difference between the detection value of the line of sight and the reference point when the estimation unit estimates that the line of sight is directed to the reference point; Surgery system provided.

  DESCRIPTION OF SYMBOLS 10 Surgical system, 11 Surgical field camera, 12 Camera arm, 13 Control apparatus, 15 Monitor, 51 Information input interface, 61 Voice detection part, 62 Eye gaze detection part, 72 Calibration part, 73 Operation control part, 74 Image signal processing part , 81 Detection value correction unit, 82 Correction value update unit, 101 Reference point detection unit, 111 Information input unit

Claims (10)

An input unit for obtaining a detection value of the line of sight;
A reference point detector for detecting a reference point;
An estimation unit that estimates that the line of sight is directed to the reference point;
A correction unit that corrects the detection value of the line of sight based on a difference between the detection value of the line of sight and the reference point when the estimation unit estimates that the line of sight is directed to the reference point; A control device provided. The control device according to claim 1, wherein the line of sight is a line of sight during an operation of an operator who operates using a surgical instrument. The control device according to claim 2, wherein the estimation unit estimates that the line of sight is directed toward the reference point when the surgical instrument is in use. The control device according to claim 3, wherein the surgical instrument is used while being energized. The reference point detection unit detects the distal end of the surgical instrument used by being energized as the reference point,
The control device according to claim 4, wherein the correction unit corrects the detection value of the line of sight according to an error between the detection value of the line of sight and a distal end of the surgical instrument. The reference point detection unit detects a trajectory of the surgical tool used by being energized as the reference point,
The control device according to claim 4, wherein the correction unit corrects the detection value of the line of sight according to an error between the detection value of the line of sight and a trajectory of the surgical instrument. When the pattern of the object displayed on the monitor is detected and the motion based on the pattern of the object displayed on the monitor is switched, the estimation unit directs the line of sight toward the reference point. The control device according to claim 2. The reference point detection unit detects an object displayed on a monitor when receiving a voice command as the reference point,
The control device according to claim 2, wherein when the display of the object that receives the voice command is detected, the estimation unit estimates that the line of sight is directed to the reference point. An input step for obtaining a detected value of the line of sight;
A reference point detecting step for detecting a reference point;
An estimation step for estimating that the line of sight is directed to the reference point;
A correction step of correcting the detected value of the line of sight based on the difference between the detected value of the line of sight and the reference point when it is estimated that the line of sight is directed to the reference point by the process of the estimating step. A control method including and. An imaging unit for imaging the surgical field of the subject;
A display unit for displaying an operative field image photographed by the photographing unit;
An input unit for obtaining a detection value of the line of sight;
A reference point detector for detecting a reference point;
An estimation unit that estimates that the line of sight is directed to the reference point;
A correction unit that corrects the detection value of the line of sight based on a difference between the detection value of the line of sight and the reference point when the estimation unit estimates that the line of sight is directed to the reference point; Surgery system provided.

Images