KR101114234B1 - Surgical robot system and laparoscope handling method thereof - Google Patents

Abstract

PURPOSE: A surgical robot system and a laparoscope handling method thereof are provided to increase the accuracy of an operation by automatically controlling the position of a celioscope and image input angle through recognizing a surgical part. CONSTITUTION: A telescopic display unit(20) visually recognizes an image of a target area to be operated. A movement sensor(310) detect the direction and size of a face. A manipulation command generator(320) generates a manipulation command to control the position of a celioscope and an image input angle. A transmitting unit(330) transmits the manipulation command to a slave robot.

Description

Surgical robot system and laparoscope handling method

The present invention relates to surgery, and more particularly to a surgical robot system and a laparoscopy manipulation method thereof.

A surgical robot refers to a robot having a function that can replace a surgical operation performed by a doctor. Such a surgical robot has the advantage of being capable of accurate and precise operation and remote surgery compared to humans.

Surgical robots currently being developed worldwide include bone surgery robots, laparoscopic surgery robots, and stereotactic surgery robots. The laparoscopic surgical robot is a robot that performs minimally invasive surgery using a laparoscope and a small surgical tool.

Laparoscopic surgery is an advanced surgical technique that involves surgery after inserting a laparoscope, which is an endoscope for looking into the belly with a hole about 1 cm in the navel area, and is expected to be developed in the future.

Recent laparoscopic is equipped with a computer chip to obtain a clearer and enlarged image than the naked eye, and has been developed so that any operation can be performed using specially designed laparoscopic surgical instruments while viewing the screen through the monitor.

Moreover, laparoscopic surgery has the same range of surgery as laparotomy, but has fewer complications than laparotomy, and can start treatment much faster after the procedure, and it has the advantage of maintaining the stamina or immune function of the patient. have. As a result, laparoscopic surgery is increasingly recognized as a standard surgery for treating colorectal cancer in the US and Europe.

The surgical robot system generally consists of a master robot and a slave robot. When the operator manipulates a manipulator (for example, a handle) provided in the master robot, a surgical tool coupled to the robot arm of the slave robot or held by the robot arm is operated to perform surgery.

However, the conventional surgical robot system requires a separate user operation for the operator to move the laparoscope to a desired position or to adjust the image input angle in order to obtain an image of the surgical site. That is, the operator must input a manipulation for the control of the laparoscope separately using the hands or feet during the surgical procedure.

However, this may be a cause of weakening the operator's concentration in the course of the surgery to maintain a high level of concentration, and incomplete surgery due to the weakened concentration has a problem that can cause serious sequelae to the surgical patient.

The present invention is to provide a surgical robot system and a laparoscopy manipulation method that allows the operator to control the position and image input angle of the laparoscope only by looking at the desired surgical site.

In another aspect, the present invention is to provide a surgical robot system and a laparoscopy manipulation method that allows the operator to concentrate only on the surgical operation is not necessary because the operator does not need a separate operation for the operation of the laparoscope.

In addition, the present invention is to provide a surgical robot system and a laparoscopic operation method that does not require learning about the device operation method using face recognition, and intuitively understand the operation method.

In addition, the present invention is to provide a surgical robot system and a laparoscopy manipulation method capable of controlling the endoscope device in a variety of ways by using only the movement of the face in three-dimensional space without using the arm.

The technical problems other than the present invention can be easily understood from the following description.

According to an aspect of the present invention, a surgical robot for controlling at least one of the position and the image input angle of the vision unit using the operation signal, the flow in the direction and size corresponding to the movement direction and size of the contacted operator's face ) And a motion detector for outputting sensing information corresponding to the direction and size of the contact of the contact part, and generating and outputting an operation command for at least one of the position and the image input angle of the vision part using the sensing information. There is provided a surgical robot including an operation command generation unit.

If the operation command relates to one or more of straight line operation and rotational operation of the vision unit, the operation handle direction of the surgical robot can be changed and operated accordingly.

The contact portion may be formed as part of a console panel of the surgical robot.

The support may be formed to protrude at one or more points of the contact portion to fix the position of the operator's face.

The eyepiece may be perforated in the contacting portion such that the image obtained by the vision portion is shown as visual information.

The contact portion may be formed of a material of light transmitting material so that the image obtained by the vision portion is shown as visual information.

The surgical robot includes a touch sensing unit for detecting whether the operator's face is in contact with the contact portion or the support, and a reference state in which the contact portion is designated as a default position and state when the contact release is recognized by the sensing of the contact sensing unit. It may further include an original state recovery unit for processing to return to.

The original state restoration unit may process the contact portion to return to the reference state by reverse manipulation of the contact portion flow direction and size according to the sensing information.

The surgical robot may further include an eye tracker unit configured to compare the sequentially generated image data in order of time to generate analysis information for analyzing one or more of a change in the position of the eye, a change in the shape of the eye, and a gaze direction. The camera unit may further include a camera unit for generating image data by capturing toward the contact portion inside the surgical robot, and a storage unit for storing the generated image data.

The operation command generation unit may determine whether the analysis information satisfies a preset change as an arbitrary operation command, and output a corresponding operation command when the operation information is satisfied.

The vision portion may be any one of a microscope and an endoscope, and the endoscope may be one or more of laparoscopic, thoracoscopic, arthroscopic, parenteral, bladder, rectal, duodenum, mediastinal and cardiac.

The contact portion is formed on the front surface of the console panel to be supported by the elastic body, the elastic body may provide a restoring force so that the contact portion is returned to its original position when the external force for the movement of the contact portion is removed.

According to another aspect of the present invention, a surgical robot for controlling at least one of the position and the image input angle of the vision unit using the operation signal, the eyepiece for providing the image obtained by the vision unit as visual information; The eye tracker unit for generating analysis information analyzing one or more of a change in the position of the pupil, a change in the shape of the eye, and a viewing direction seen through the eyepiece; and whether the analysis information satisfies a preset change as an arbitrary operation command. There is provided a surgical robot including an operation command generation unit that determines and outputs an operation command for operating the vision unit when satisfied.

The eye tracker unit includes a camera unit for imaging the eyepiece from the inside of the surgical robot to generate image data, a storage unit for storing the generated image data, and sequentially comparing the image data generated sequentially to determine the position of the pupil. It may include an eye tracker unit for generating analysis information that interprets one or more of a change, an eye shape change, and a viewing direction.

The eyepiece may be perforated in a contact portion formed as part of a console panel of the surgical robot.

According to still another aspect of the present invention, there is provided a surgical robot for controlling a vision unit by using an operation signal, comprising: an imaging unit for imaging an object to generate image data, and a center point of both eyes in a face included in the image data Analyze the size and rotation direction of the cabinet formed by the straight line extending from the screen and the center line of the screen, and compare the size and rotation direction of the analyzed cabinet with respect to previously photographed image data to obtain displacement information on the rotation direction and rotation angle of the face. There is provided a surgical robot including an angle and distance calculator to generate and an operation command generator for generating and outputting an operation command for operating the vision unit according to the displacement information.

The angle and distance calculator further calculates distance information between the reference point in the face interpreted from the previously photographed image data and the reference point in the face interpreted from the image data, and the calculated distance information is applied to the displacement amount information for parallel movement manipulation of the vision unit. May be included.

The angle and distance calculator further calculates the amount of change between the distance between the two eyes in the face analyzed in the previously photographed image data and the distance between the eyes in the face interpreted in the image data. It may be included in the displacement information.

The vision part may be one or more of laparoscopic, thoracoscopic, arthroscopic, parenteral, bladder, rectal, duodenum, mediastinal, cardiac, or cardiac.

The vision unit may be a device for acquiring 3D images, and the degree of overlapping the left and right images for 3D image processing may be adjusted according to positions of the face and eyes.

The surgical robot calculates a similarity between a storage unit storing a photographic image of an authenticated user, a feature of a face included in the image data and a feature of a face included in the photo image, and the calculated similarity is equal to or greater than a predetermined value. The apparatus may further include a determining unit configured to control the manipulation command generator to generate and output the manipulation command only. Here, the feature elements of the face may be one or more of the position and shape of the constituent parts, the color of the eyes, the shape of the face, the skin color, the form of skin wrinkles, the iris, the constituent parts may be one or more of the eyes, eyebrows, nose and mouth have.

The surgical robot includes a storage unit for storing effective user information that is one or more of area information on which a face shape is to be located and reference value of a face size included in the image data, and whether the face included in the image data satisfies the effective user information. The apparatus may further include a determiner configured to control the manipulation command generation unit to generate and output the manipulation command only when it is determined and satisfied.

The surgical robot determines whether the facial motion state included in the image data is maintained for a predetermined time or more, and further determines a control unit for controlling the operation command generation unit to generate and output an operation command only when the facial motion state is maintained for a predetermined time or more. It may include.

The surgical robot may further include a determination unit configured to determine whether the degree of facial movement included in the image data exceeds a preset range, and to control the manipulation command generation unit to generate and output the manipulation command only when the degree of movement of the face exceeds the preset range. Can be.

The surgical robot includes a storage unit which stores operation command information to be generated for at least one change of the head movement and the facial expression, and generates a change information corresponding to at least one of the facial expression and the head movement by interpreting the plurality of image data. The apparatus may further include a determining unit configured to control the operation command generator to generate and output an operation command according to the operation command information.

According to another aspect of the invention, the surgical robot for controlling one or more of the position of the vision portion and the image input angle, the step of outputting the sensing information corresponding to the direction and the size of the contact portion flows; And generating and outputting an operation command for at least one of the position and the image input angle of the vision unit using the sensing information, wherein the contact portion is formed as part of the console panel of the surgical robot, Provided is a method for operating a vision unit, which is configured to flow in a direction and a size corresponding to the direction and size of movement.

The vision unit manipulation method may further include determining whether the operator's face is in contact with the contact portion, and if the contact is in contact with the operator, controlling to start output of sensing information.

The operation method of the vision unit includes determining whether the contact portion exists as a reference state which is a position and state designated as default when the contact is released, and when the contact state is not present, returning to the reference state. It may further comprise the step of processing.

The return to the reference state may be performed by inverse manipulation of the contact portion flow direction and size according to the sensing information.

Vision operating method includes the steps of generating and storing the image data taken from the inner side of the surgical robot toward the contact portion, and comparing the stored image data in chronological order to determine the position of the pupil, the change of the eye shape and the direction of attention The method may further include generating interpretation information of one or more interpretations.

The vision unit manipulation method may further include determining whether the analysis information satisfies a preset change as an arbitrary manipulation command, and outputting a corresponding manipulation command that is preset accordingly if satisfied.

The contact portion is formed on the front surface of the console panel to be supported by the elastic body, the elastic body may provide a restoring force so that the contact portion is returned to its original position when the external force for the movement of the contact portion is removed.

According to another aspect of the present invention, a surgical robot for controlling at least one of the position and the image input angle of the vision unit using the operation signal, a contact for providing the image obtained by the vision unit as visual information An analysis processor for generating analysis information for analyzing the face part, the movement of the face seen through the contact part, and determining whether the analysis information satisfies a preset change as an arbitrary operation command, and if so, to operate the vision part. There is provided a surgical robot including an operation command generation unit for outputting an operation command.

The analysis processing unit includes a camera unit for generating image data by imaging toward the contact portion inside the surgical robot, a storage unit for storing the generated image data, and a change in position of a predetermined feature point in the sequentially generated image data. It may include an analysis unit for generating the analysis information on the movement of the face by comparing and determining in chronological order.

The contact portion may be formed as part of a console panel of the surgical robot, and the contact portion may be formed of a material of light transmitting material so that the image obtained by the vision part is viewed as visual information. Can be.

According to another aspect of the invention, the surgical robot is a control method for controlling one or more of the position and the image input angle of the vision portion, one of the position change, eye shape change and the direction of eye viewing through the eyepiece Generating analysis information that interprets the above, determining whether the analysis information satisfies a preset change as an arbitrary operation command, and if satisfactory, manipulating at least one of the position of the vision unit and the image input angle. A vision operation method is provided that includes generating and outputting a command.

The generating of the analysis information includes generating and storing image data photographed toward the eyepiece from the inside of the surgical robot, and comparing the stored image data in chronological order to determine the position of the pupil, the change of the eye shape, and the gaze direction. It may include generating analysis information that interprets one or more of the.

According to still another aspect of the present invention, a method for operating a vision unit by a surgical robot includes imaging an object to generate image data, and extending a center point of both eyes from a face included in the image data. Analyzing the size and rotation direction of the cabinet formed by the straight line and the center line of the screen, and generating displacement information on the rotation direction and the rotation angle of the face by comparing the size and rotation direction of the analyzed cabinet with respect to previously photographed image data. And generating and outputting an operation command for causing the vision unit to be operated in correspondence with the displacement amount information.

The generating of the displacement information may include calculating distance information between a reference point in the face interpreted from previously photographed image data and a reference point in the face interpreted in the image data, and converting the calculated distance information into a vision unit for parallel movement operation. It may include the step of including in the displacement information.

The generating of the displacement information may include calculating a change amount between the distance between the both eyes in the face analyzed from the previously photographed image data and the distance between the both eyes in the face interpreted from the image data; It may include the step of including in the displacement information for scaling.

The vision part may be one or more of laparoscopic, thoracoscopic, arthroscopic, parenteral, bladder, rectal, duodenum, mediastinal, cardiac, or cardiac.

The vision unit may be a device for acquiring 3D images, and the degree of overlapping the left and right images for 3D image processing may be adjusted according to positions of the face and eyes.

The operation method of the vision unit includes calculating a similarity between a feature element of a face included in the image data and a photographic image of an authenticated user previously stored in the storage unit, and generating an operation command only when the calculated similarity is equal to or greater than a predetermined value. The method may further include controlling the outputting to be performed. Here, the feature elements of the face may be one or more of the position and shape of the constituent parts, the color of the eyes, the shape of the face, the skin color, the form of skin wrinkles, the iris, the constituent parts may be one or more of the eyes, eyebrows, nose and mouth have.

The vision unit manipulation method may further include determining whether a face included in the image data satisfies pre-stored effective user information, and controlling to generate and output an operation command only when the face is satisfied. Can be. The storage unit may store in advance effective user information that is one or more of area information on which a face shape is to be located in the image data and reference values of a face size included in the image data.

The vision unit manipulation method includes controlling whether the face motion state included in the image data is maintained for a predetermined time or more, and generating and outputting an operation command only when the face motion state is maintained for a predetermined time or more. It may further comprise the step.

The operation method of the vision unit includes determining whether the degree of facial movement included in the image data exceeds a preset range, and controlling to perform a step of generating and outputting an operation command only when the preset range is exceeded. It may further include.

The vision unit manipulation method includes: generating a change information of at least one of facial expression and head movement by analyzing a plurality of image data, and performing an operation command according to operation command information corresponding to at least one change information of facial expression and head movement. The method may further include generating and outputting. The storage unit may store in advance manipulation command information to be generated for changes in head movement and facial expression.

Other aspects, features, and advantages will become apparent from the following drawings, claims, and detailed description of the invention.

According to an embodiment of the present invention, the operator can control the position of the laparoscope and the image input angle only by the act of seeing the desired surgical site.

In addition, the operator does not need a separate operation for the operation of the laparoscopic has the effect of allowing the operator to focus on the operation only.

In addition, there is an effect that the operation method can be intuitively understood without learning about a device operation method using face recognition.

In addition, there is an effect that can control the endoscope device in a variety of ways by using only the movement of the face in three-dimensional space without using the arm.

1 is a plan view showing the overall structure of a surgical robot according to an embodiment of the present invention.
Figure 2a is a conceptual diagram showing a master interface of the surgical robot according to an embodiment of the present invention.
2b to 2e are diagrams illustrating the movement form of the contact portion according to an embodiment of the present invention.
Figure 3 is a block diagram schematically showing the configuration of a telescopic display unit for generating a laparoscopic operation command according to an embodiment of the present invention.
4 is a flowchart illustrating a method of transmitting a laparoscope manipulation command according to an embodiment of the present invention.
5 is a block diagram schematically illustrating a configuration of a telescopic display unit for generating a laparoscopic manipulation command according to another embodiment of the present invention.
6 is a flowchart illustrating a method of transmitting a laparoscope manipulation command according to another embodiment of the present invention.
FIG. 7 is a block diagram schematically illustrating a configuration of a telescopic display unit for generating a laparoscopic manipulation command according to another embodiment of the present invention. FIG.
8 is a flowchart illustrating a method of transmitting a laparoscopic manipulation command according to another embodiment of the present invention.
9 is a flowchart illustrating a method of transmitting a laparoscopic manipulation command according to another embodiment of the present invention.
10 is a view illustrating an image display form by a telescopic display unit according to an embodiment of the present invention.
11 is a flowchart illustrating a method of transmitting a laparoscopic manipulation command according to another embodiment of the present invention.
12 is a conceptual diagram showing a master interface of a surgical robot according to another embodiment of the present invention.
13A is a block diagram schematically showing the configuration of a laparoscopic manipulation unit according to another embodiment of the present invention.
13B is a view showing an operation concept of a laparoscopic manipulation unit according to another embodiment of the present invention.
14 and 15 illustrate facial movements for laparoscopic manipulation, respectively, according to another embodiment of the present invention.
Figure 16 is a flow chart showing the operation of the laparoscopic operation unit according to another embodiment of the present invention.
FIG. 17 is a flow diagram specifically illustrating step 1610 of FIG. 16 in accordance with another embodiment of the present invention. FIG.

The present invention is capable of various modifications and various embodiments, and specific embodiments are illustrated and described in the drawings. However, this is not intended to limit the present invention to specific embodiments, it should be understood to include all transformations, equivalents, and substitutes included in the spirit and scope of the present invention. In the following description of the present invention, if it is determined that the detailed description of the related known technology may obscure the gist of the present invention, the detailed description thereof will be omitted.

The terms first, second, etc. may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another.

The terminology used herein is for the purpose of describing particular example embodiments only and is not intended to be limiting of the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, the terms "comprise" or "have" are intended to indicate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, and one or more other features. It is to be understood that the present invention does not exclude the possibility of the presence or the addition of numbers, steps, operations, components, components, or a combination thereof.

In addition, the terms "... unit", "... group", "module" and the like that can be described in the specification means a unit for processing at least one function or operation, which means hardware or software or hardware and It can be implemented in a combination of software.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, and in the following description with reference to the accompanying drawings, the same or corresponding components are given the same reference numerals and redundant description thereof will be omitted. Shall be.

In addition, in describing various embodiments of the present invention, each embodiment should not be interpreted or implemented independently, and the feature elements and / or technical ideas described in each embodiment may be combined with other embodiments separately described. It is to be understood that it may be interpreted or practiced.

In addition, through the following description it will be apparent that the present invention is a technical idea that can be used universally for surgery or experiments in which vision parts such as endoscopes and microscopes are used. In addition, the endoscope may be diversified into laparoscopic, thoracoscopic, arthroscopic, parenteral, bladder, rectal, duodenum, mediastinal, cardiac, and the like. However, hereinafter, the case of the vision unit is a kind of endoscope, that is, laparoscope for the convenience of explanation and understanding will be described as an example.

1 is a plan view showing the overall structure of a surgical robot according to an embodiment of the present invention, Figure 2a is a conceptual diagram showing a master interface of the surgical robot according to an embodiment of the present invention, Figures 2b to 2e FIG. Is a view illustrating a movement form of a contacting unit according to an exemplary embodiment of the present invention.

1 and 2A, a surgical robot system includes a slave robot 2 performing surgery on a patient lying on an operating table and a master robot 1 remotely controlling a slave robot 2. The master robot 1 and the slave robot 2 are not necessarily separated into separate devices that are physically independent, but may be integrated into one and integrally formed, in which case the master interface 4 may be, for example, of an integrated robot. May correspond to an interface portion.

The master interface 4 of the master robot 1 comprises a monitor 6, a telescopic display 20 and a master manipulator, and the slave robot 2 comprises a robot arm 3 and a laparoscope 5.

The monitor unit 6 of the master interface 4 may be composed of one or more monitors, and each monitor may be individually displayed information necessary for the operation. 1 and 2 illustrate a case in which the monitor unit 6 is included on each side of the telescopic display unit 20 one by one, but the quantity of the monitor may be variously determined according to the type or type of information requiring display. .

The monitor unit 6 may output one or more biometric information about the patient, for example. In this case, at least one of indicators indicating the patient's condition, for example, biometric information such as body temperature, pulse rate, respiration and blood pressure may be output through at least one monitor of the monitor unit 6, and a plurality of pieces of information may be output. In this case, each piece of information may be divided and output by area. In order to provide such biometric information to the master robot 1, the slave robot 2 includes a biometric information measuring unit including at least one of a body temperature measuring module, a pulse measuring module, a respiratory measuring module, a blood pressure measuring module, an electrocardiogram measuring module, and the like. It may include. The biometric information measured by each module may be transmitted from the slave robot 2 to the master robot 1 in the form of an analog signal or a digital signal, and the master robot 1 monitors the received biometric information. Can be displayed via

The telescopic display unit 20 of the master interface 4 provides the operator with an image input through the laparoscope 5 as a surgical site. The operator views the image through the eyepiece 220 formed on the contact portion 210 of the telescopic display portion 20, and manipulates the robot arm 3 and the end effector by manipulating the master controller to operate on the surgical site. Proceed. 2A illustrates a case in which a panel portion 210 is implemented as an example of the folding portion 210, the folding portion 210 may be recessed to face the inside of the master interface 4. In addition, FIG. 2A illustrates a case in which the eyepiece 220 for the operator to view the image acquired by the laparoscope 5 is formed on the contacting part 210, but the contacting part 210 is shown through the image of the back side. When formed of a losing material, the formation of the eyepiece 220 may be omitted. For example, the folding unit 210 may be formed of a transparent material, coated with a polarizing film, or used to watch a 3D IMAX film so that an image of the rear surface of the folding unit 210 may be transmitted to the operator. It may be formed by a light transmissive material.

The telescopic display unit 20 functions not only as a display device for the operator to check the image of the laparoscope 5 through the eyepiece 220, but also as a control command input unit for controlling the position and image input angle of the laparoscope 5. It is configured to have together.

The operator's face is in contact with or close to the contact portion 210 of the telescopic display unit 20, and a plurality of supports 230 and 240 are formed to protrude so that the operator's face movement can be recognized. For example, the support 230 formed at the top may be used to contact the operator's forehead to fix the forehead position, and the support 240 formed at the side may be located at an area under the operator's eye (for example, the cheekbone area). It can be used to contact and fix the face position. Position and quantity of the support illustrated in FIG. 2A are exemplary, and the position or the shape of the support may be varied, for example, a jaw fixing support, a left or right support 290, or the like. . In the case of the left and right face support, for example, when the left or right side of the face moves, the contact portion 210 may be formed in the form of a rod or a wall to support the movement in the corresponding direction.

The position of the operator's face is fixed by the support parts 230 and 240 formed as described above, and when the operator turns the face in an arbitrary direction while viewing the image by the laparoscope 5 through the eyepiece 220, the facial movement accordingly This can be detected and used as input information for adjusting the position of the laparoscope 5 and / or the image input angle. For example, if the operator wants to check the area on the left side (i.e., the area on the left side of the display screen) rather than the surgical area displayed on the current image, the operator can turn his head so that his face is relatively left. The laparoscope 5 may be manipulated so that the image of the corresponding area is output.

In other words, the contact portion 210 of the telescopic display unit 20 is coupled to the master interface 4 so that the position and / or the angle is changed in accordance with the operator's face movement. To this end, the master interface 4 and the contact portion 210 of the telescopic display portion 20 may be coupled to each other by the flow portion 240. The flow unit 250 may be formed of, for example, an elastic body so as to easily change the position and / or angle of the telescopic display unit 20 and restore the original state when the external force caused by the operator's face movement is removed. In addition, even when the flow unit 250 is formed of an inelastic material, the telescopic display unit 20 may control the original state restoration unit (see FIG. 5) to restore the telescopic display unit 20 to the original state.

The contact part 210 is moved by the flow part 250 based on a virtual center point and coordinates in a three-dimensional space formed on the XYZ axis, or is operated in any direction (eg, clockwise, counterclockwise, etc.). Rotational movement). Here, the virtual center point may be any one point or axis in the contact portion 210, for example, the center point of the contact portion 210.

2b to 2e illustrate the movement of the contact portion 210.

When the operator's face movement direction is parallel to the X, Y or Z axis, the contact portion 210 is moved in a direction in which a force due to the face movement is applied as illustrated in FIG. 2B.

When the operator's face movement direction rotates on the X-Y plane, the contact portion 210 is rotated in a direction in which a force caused by the face movement is applied as illustrated in FIG. 2C. At this time, the contact portion 210 may be rotated in a clockwise or counterclockwise direction depending on the direction in which the force is applied.

When the operator's face movement direction is rotated about the X, Y or Z axis, the contact portion 210 is rotated in a direction in which a force by the face movement is applied about the reference axis as illustrated in FIG. 2D. . In this case, the contact portion 210 may be rotated in a clockwise or counterclockwise direction according to the direction in which the force is applied.

When the force according to the operator's face movement is applied to two of the X, Y, and Z axes, the contact portion 210 is based on the virtual center point and the two axes to which the force is applied, as illustrated in FIG. 2E. Rotation is moved.

As described above, the vertical and horizontal movements of the contact portion 210 and the rotational movement are determined by the direction of the force applied by the face movement, and one or more types of movements described above may be combined.

A method and a configuration in which the telescopic display unit 20 detects an operator's face movement and generates an operation command according to the present invention will be described in detail with reference to the accompanying drawings.

As illustrated in FIGS. 1 and 2A, the master interface 4 is provided with a master manipulator so that the operator can grip and manipulate each hand. The master controller may be implemented by two handles 10 or more handles 10, and an operation signal according to the operator's manipulation of the handle 10 is transmitted to the slave robot 2 so that the robot arm 3 may be moved. Controlled. By operating the handle 10 of the operator, a surgical operation such as a position movement, rotation, and cutting operation of the robot arm 3 may be performed.

For example, the handle 10 may be configured to include a main handle and a sub handle. The operator may operate the slave robot arm 3 or the laparoscope 5 or the like only by the main handle, or may operate the sub handles to simultaneously operate a plurality of surgical equipments in real time. The main handle and the sub handle may have various mechanical configurations depending on the operation method thereof. For example, the robot arm 3 and / or other surgery of the slave robot 2, such as a joystick type, a keypad, a trackball, and a touch screen, may be used. Various input means for operating the equipment can be used.

The master manipulator is not limited to the shape of the handle 10 and may be applied without any limitation as long as it can control the operation of the robot arm 3 through a network.

The master robot 1 and the slave robot 2 may be coupled to each other through a wired communication network or a wireless communication network so that an operation signal and a laparoscope image input through the laparoscope 5 may be transmitted to the counterpart. If a plurality of operation signals by the plurality of handles 10 provided in the master interface 4 and / or an operation signal for adjusting the laparoscope 5 need to be transmitted at the same time and / or at a similar time point, each The operation signal may be transmitted to the slave robot 2 independently of each other. Herein, when each operation signal is 'independently' transmitted, it means that the operation signals do not interfere with each other and one operation signal does not affect the other signal. As described above, in order to transmit the plurality of operation signals independently of each other, in the generation step of each operation signal, header information for each operation signal is added and transmitted, or each operation signal is transmitted in the generation order thereof, or Various methods may be used such as prioritizing each operation signal in advance and transmitting the operation signal accordingly. In this case, the transmission path through which each operation signal is transmitted may be provided independently so that interference between each operation signal may be fundamentally prevented.

The robot arm 3 of the slave robot 2 can be implemented to be driven with multiple degrees of freedom. The robot arm 3 includes, for example, a surgical instrument inserted into a surgical site of a patient, a rocking drive unit for rotating the surgical instrument in the yaw direction according to the surgical position, and a pitch direction perpendicular to the rotational drive of the rocking drive unit. It comprises a pitch drive unit for rotating the surgical instruments, a transfer drive for moving the surgical instruments in the longitudinal direction, a rotation drive for rotating the surgical instruments, a surgical instrument drive unit installed on the end of the surgical instruments to cut or cut the surgical lesion Can be. However, the configuration of the robot arm 3 is not limited thereto, and it should be understood that this example does not limit the scope of the present invention. In addition, the actual control process, such as the operator rotates the robot arm 3 in the corresponding direction by operating the handle 10 is somewhat distanced from the subject matter of the present invention, so a detailed description thereof will be omitted.

One or more slave robots 2 may be used to operate the patient, and the laparoscope 5 for allowing the surgical site to be displayed as an image (that is, an image image) that can be seen through the eyepiece 220 is an independent slave. It may be implemented by the robot (2). In addition, as described above, embodiments of the present invention may be used universally in operations in which various surgical endoscopes (eg, thoracoscopic, arthroscopy, parenteral, etc.) other than laparoscopic are used.

3 is a block diagram schematically illustrating a configuration of a telescopic display unit for generating a laparoscopic manipulation command according to an embodiment of the present invention, and FIG. 4 is a flowchart illustrating a method of transmitting a laparoscopic manipulation command according to an embodiment of the present invention. to be.

Referring to FIG. 3, the telescopic display unit 20 includes a motion detector 310, an operation command generator 320, and a transmitter 330. In addition, the telescopic display unit 20 may further include a component that allows the operator to visually recognize the image of the surgical site input through the laparoscope 5 through the eyepiece 220, but this is the gist of the present invention. Since there is a little distance from the description thereof will be omitted.

The motion detector 310 detects in which direction the operator moves the face in contact with the support 230 and / or 240 of the contact unit 210 and outputs sensing information. The motion detector 310 may include sensing means for detecting a direction and a size (eg, a distance) of the movement of the face. The sensing means is sufficient as sensing means capable of detecting how much the contact portion 210 has moved in which direction, for example, in which direction the flow portion 250 having an elastic force supporting the contact portion 210 is directed. It may be a sensor that detects to what extent the tension (引 張), or a sensor provided to the inside of the master robot 1 to detect how close or / and rotated feature points formed on the inner surface of the contact portion 210, and the like. .

The manipulation command generator 320 analyzes the operator's face movement direction and size using the sensing information received from the motion detector 310, and controls the position and image input angle of the laparoscope 5 according to the analyzed result. Create an operation command to

The transmission unit 330 transmits the operation command generated by the operation command generation unit 320 to the slave robot 2 so that the position and image input angle of the laparoscope 5 are manipulated, and an image is provided accordingly. . The transmission unit 330 may be a transmission unit already provided in the master robot 1 to transmit an operation command for the operation of the robot arm 3.

Figure 4 shows a method for transmitting a laparoscope manipulation command according to an embodiment of the present invention.

Referring to FIG. 4, the telescopic display unit 20 detects the operator's face movement in step 410, and proceeds to step 420 to manipulate the laparoscopic 5 using the sensing information generated by the detection of the face movement. Create a command.

Subsequently, in step 430, the manipulation command generated by step 420 is transmitted to the slave robot 2 for manipulation of the laparoscope 5.

Here, the operation command generated for the operation of the laparoscope 5 may be functioned so that a specific operation is performed also on the master robot 1. For example, when the face is rotated to detect the rotation of the laparoscope 5, a manipulation command for the rotation is transmitted to the slave robot 2 and the direction of the manipulation handle of the master robot 1 is correspondingly changed. By doing so, it is possible to maintain the intuition and ease of operation of the operator. For example, when the rotation signal is detected by the contact unit 210, the laparoscope 5 is rotated by the generated operation signal, and the image displayed on the screen and the position of the surgical tool shown in the image are currently Since it may not coincide with the position of the hand of the manipulation handle, an operation of matching the position of the surgical tool displayed on the screen by moving the position of the manipulation handle may be performed. The control of the operation handle direction is a case where the position / direction of the surgical tool displayed on the screen and the actual operation handle position / direction are not only in the case of the rotary motion of the contact portion 210 but also in the case of the linear motion. The same may apply.

FIG. 5 is a block diagram schematically illustrating a configuration of a telescopic display unit for generating a laparoscopic manipulation command according to another embodiment of the present invention, and FIG. 6 is a flowchart illustrating a method of transmitting a laparoscopic manipulation command according to another embodiment of the present invention. to be.

Referring to FIG. 5, the telescopic display unit 20 may include a motion detector 310, an operation command generator 320, a transmitter 330, a contact detector 510, and an original state restorer 520. have.

The illustrated motion detector 310, the manipulation command generator 320, and the transmitter 330 have been described above with reference to FIG. 3, and thus description thereof will be omitted. However, the motion detector 310 may perform an operation while it is recognized that the operator's face is in contact with the support 230 or / and 240 as the sensing information by the touch detector 510.

The touch detector 510 detects whether the operator's face is in contact with the support 230 or / and 240 and outputs sensing information. To this end, for example, a touch sensor may be provided at the end of the support, and in addition, various sensing schemes may be applied to detect whether a face is in contact.

The original state restoring unit 520 controls the motor driver 530 when the face of the operator is sensed to be in contact with the supporting unit 230 or / and 240 as the sensing information by the contact detecting unit 510. 210 is returned to its original state. The original state restorer 520 may include a motor driver 530 to be described below.

In FIG. 5, the motor driving unit 530 using the motor is illustrated as an operation means for returning the contact portion 210 to its original state, but an operation means for achieving the same purpose is not limited thereto. For example, the contact portion 210 may be treated to return to its original state by various methods such as pneumatic or hydraulic pressure.

The original state restoring unit 520 controls the motor driving unit 530 using, for example, information on the reference state (ie, position and / or angle) of the contacting unit 210, or the operation command generation unit 320. The motor driver 530 may be controlled to be manipulated in the reverse direction and size using the face movement direction and size analyzed by the face movement, so that the contact portion 210 may be returned to its original position.

For example, the operator turns his face in the corresponding direction (by which the contact portion 210 is also moved or rotated) in order to check a region different from the surgical region displayed in the current image or to take an action on the region. ) Detects that the face contact with the contacting unit 210 has ended in a state where it is operated accordingly, the original state restoring unit 520 may return the motor unit to return to the reference state designated by the contacting unit 210 as the default. 530 may be controlled.

The motor driving unit 530 may include a motor that rotates under the control of the original state restoring unit 520, and the motor (eg, position and / or angle) of the contacting unit 210 is adjusted by the rotation of the motor. The driving unit 530 and the contacting unit 210 are coupled to each other. The motor driver 530 may be formed to be received inside the master interface 4. The motor included in the motor driving unit 530 may be, for example, a spherical motor for allowing a degree of freedom movement, and the support structure of the spherical motor may be a spherical bearing in order to remove the limitation of the inclination angle. It may be composed of a circular rotor or a frame structure having three degrees of freedom for supporting the circular rotor.

Even if the contact unit 210 is restored to its original state by the operation of each of the above-described components, the operation command generation unit 320 does not generate and transmit an operation command for the image, and thus is input and output by the laparoscope 5. The image does not change. Therefore, after the operator checks the laparoscopic (5) image through the eyepiece 220 may be consistent in the operation.

Although the case where the original state of the contact portion 210 of the telescopic display unit 20 returns to the original state has been described with reference to FIG. 5, the operation is performed by the flow unit 250 made of an elastic material having elastic force. Naturally, if the external force caused by the movement of the face of the child is removed, it may be returned to its original state. Even when the contact portion 210 is returned to its original state by the elastic force, an operation signal for manipulating the laparoscope 5 will not be generated.

Figure 6 shows a method for transmitting a laparoscope manipulation command according to another embodiment of the present invention.

Referring to FIG. 6, the telescopic display unit 20 detects the operator's face movement in step 410, and proceeds to step 420 to manipulate the laparoscopic 5 using the sensing information generated by the detection of the face movement. Create a command. Thereafter, in step 430, the manipulation command generated by step 420 is transmitted to the slave robot 2 for the manipulation of the laparoscope 5.

Subsequently, the telescopic display unit 20 determines whether the operator releases contact with the contacting unit 210 in step 610. If the contact is maintained, the process proceeds to step 410 again. If the contact is released, the process proceeds to step 620 to control the contact unit 210 to return to its original position.

7 is a block diagram schematically illustrating a configuration of a telescopic display unit for generating a laparoscopic manipulation command according to another embodiment of the present invention.

Referring to FIG. 7, the telescopic display unit 20 includes a contact detector 510, a camera unit 710, a storage unit 720, an eye tracker unit 730, an operation command generation unit 320, and a transmission unit 330. And the controller 740.

The touch detector 510 detects whether the operator's face is in contact with the support 230 or / and 240 formed to protrude from the contact portion 210 and outputs sensing information.

When the camera unit 710 detects that the operator's face is in contact with the contacting unit 210 by sensing information of the contact sensor 510, the camera unit 710 photographs an image of the operator's eyes in real time. The camera unit 710 is arranged to photograph the operator's eye seen through the eyepiece 220 inside the master interface 4. The image of the operator's eye photographed by the camera unit 710 is stored in the storage unit 720 for the eye tracking process of the eye tracker unit 730.

The image photographed by the camera unit 710 is sufficient that the eye tracking process of the eye tracker unit 730 is possible, and after the preprocessing necessary for the processing of the eye tracker unit 730 is performed, the storage unit 720 is stored. May be stored. Since the image generating method and the generated image type for the eye tracking process will be apparent to those skilled in the art, a description thereof will be omitted.

The eye tracker unit 730 analyzes the images stored in the storage unit 720 in real time or at predetermined intervals in chronological order, and analyzes the change of the pupil position of the operator and the gaze direction by the operator and outputs the analysis information. In addition, the eye tracker unit 730 may further analyze the shape of the pupil (for example, blinking eyes, etc.) and output analysis information thereof.

The operation command generation unit 320 refers to the analysis information by the eye tracker unit 730, when the operator's gaze direction is changed, the operation command for controlling the position and / or image input angle of the laparoscope 5 accordingly. Create In addition, the manipulation command generation unit 320 may generate a manipulation command for this if the shape change of the pupil is for inputting a predetermined command. For example, the designation command according to the change in the shape of the pupil is, for example, the laparoscopic (5) approach to the surgical site in the case of two consecutive blinks of the right eye, and the clockwise rotation in the case of two consecutive blinks of the left eye. It can be specified in advance.

The transmission unit 330 transmits the operation command generated by the operation command generation unit 320 to the slave robot 2 so that the position and image input angle of the laparoscope 5 are manipulated, and an image is provided accordingly. . The transmission unit 330 may be a transmission unit already provided in the master robot 1 to transmit an operation command for the operation of the robot arm 3.

The controller 740 controls each of the above components to perform a specified operation.

Up to now, the telescopic display unit 20 that recognizes and processes eye movements using eye tracking technology has been described. However, the present invention is not limited thereto, and the telescopic display unit 20 may be implemented in a manner that detects, recognizes, and processes the movement of the operator's face itself. For example, the camera unit 710 captures a face image, and the analysis processing unit replacing the eye tracker unit 730 captures a feature point (for example, a position of two eyes, a position of a nose, a position of a person, etc.). If the position and change of one or more) is analyzed, the operation command generation unit 320 may generate a corresponding operation command.

8 is a flowchart illustrating a method of transmitting a laparoscopic manipulation command according to another embodiment of the present invention.

Referring to FIG. 8, in operation 810, when the contact of the operator's face is detected by the contact sensor 510, the telescopic display unit 20 activates the camera unit 710 to the eye of the operator visible through the eyepiece 220. The digital image data is generated and stored in the storage unit 720.

In operation 820, the telescopic display unit 20 compares and determines digital image data stored in the storage unit 720 in real time or at predetermined intervals, and generates analysis information about changes in the pupil position and the gaze direction of the operator. In the comparison determination, the telescopic display unit 20 may allow a certain error so that a change in the position information of a certain range may be recognized as not changing the position of the pupil.

In operation 830, the telescopic display unit 20 determines whether the operator's gaze direction changed over a predetermined threshold time is maintained.

If the changed gaze direction is maintained for more than a threshold time, the telescopic display unit 20 at step 840 manipulates the laparoscope 5 to receive an image of the corresponding position (for example, to move or change the image input angle). An operation command is generated and transmitted to the slave robot 2. Here, the threshold time may be set to a time for preventing the laparoscopic 5 from being manipulated by the movement of the pupil for movement of the operator's pupils or general overview of the surgical site, and the time value is set experimentally and statistically. Or set by an operator or the like.

However, if the changed gaze direction is not maintained for more than the threshold time, the process proceeds to step 810 again.

9 is a flowchart illustrating a method of transmitting a laparoscopic manipulation command according to another embodiment of the present invention, and FIG. 10 is a view illustrating an image display form by a telescopic display unit according to an embodiment of the present invention.

9, in operation 810, when the contact of the operator's face is detected by the contact sensor 510, the telescopic display unit 20 activates the camera unit 710 to the operator's eye visible through the eyepiece 220. The digital image data is generated and stored in the storage unit 720.

In operation 820, the telescopic display unit 20 compares and determines digital image data stored in the storage unit 720 in real time or at predetermined intervals, and generates analysis information about changes in the pupil position and the gaze direction of the operator.

In operation 910, the telescopic display unit 20 determines whether the operator's gaze position is a predetermined setting.

10 illustrates an image display form by the telescopic display unit 20.

As illustrated in FIG. 10, through the eyepiece 220, the operator may check an image image 1010 provided through the laparoscope 5, and the image image may include a surgical part and an instrument 1020. . In addition, the image by the telescopic display unit 20 may be displayed by overlapping the gaze position 1030 of the operator, and the setting positions may be displayed together.

The setting position may include one or more of an outer edge 1040, a first rotation instruction position 1050, a second rotation instruction position 1060, and the like. For example, when the operator watches the side positioned in any direction of the outer edge 1040 or more for a threshold time, the laparoscope 5 may be controlled to move in the corresponding direction. That is, when the left side of the outer edge 1040 is watched for more than a threshold time, the laparoscope 5 may be controlled to be moved to the left to photograph the left side of the current display position. In addition, when the operator watches the first rotational instruction position 1050 for a threshold time or more, the laparoscope is controlled to rotate in a counterclockwise direction, and when the operator watches the second rotational instruction position 1060 for more than the threshold time, the laparoscope is in a clockwise direction. It may be controlled to rotate.

Referring back to FIG. 9, if the operator has a gaze position other than the above-described setting position, the operation proceeds to step 810 again.

However, referring back to FIG. 9, when the operator has the gaze position as described above, the controller proceeds to step 920 to determine whether the gaze of the operator is maintained for a predetermined time or more.

If the operator's attention to the set position is maintained for more than the threshold time, the telescopic display unit 20 generates an operation command so that the laparoscope 5 is operated according to the command specified for the set position in step 930 so that the slave robot 2 can be operated. To send.

However, if the operator's attention to the set position is not maintained for more than the threshold time, the process proceeds to step 810 again.

11 is a flowchart illustrating a method of transmitting a laparoscopy manipulation command according to another embodiment of the present invention.

Referring to FIG. 11, when the contact of the operator's face is detected by the contact sensor 510 in step 810, the telescopic display unit 20 activates the camera unit 710 to the eye of the operator visible through the eyepiece 220. The digital image data is generated and stored in the storage unit 720.

In operation 1110, the telescopic display unit 20 compares and determines image information stored in the storage unit 720 in real time or at a predetermined period to generate analysis information about a change in the shape of the driver's eyes. For example, the interpretation information may be information about how many times the operator's eyes blinked during a certain time, and which eyes blinked when the operator blinked.

In operation 1120, the telescopic display unit 20 determines whether analysis information regarding the change in eye shape satisfies a predetermined predetermined condition. For example, the designated condition according to the change of the eye shape may be set in advance, for example, whether the right eye blinks twice in a predetermined time or whether the left eye blinks twice in a predetermined time.

If the analysis information on the change in eye shape satisfies a predetermined condition, the flow advances to step 1130 and generates an operation command for manipulating the laparoscope 5 as a designated command in the case of satisfying the condition, thereby generating the slave robot 2. To send. For example, the designation command according to the change of the eye shape may be, for example, the laparoscopic (5) approach to the surgical site in the case of two consecutive blinks of the right eye, or the clockwise rotation in the case of two consecutive blinks of the left eye. It may be specified in advance.

However, if the interpretation information on the eye shape change does not satisfy the predetermined condition, the process proceeds to step 910.

12 is a conceptual diagram illustrating a master interface of a surgical robot according to another embodiment of the present invention.

Referring to FIG. 12, the master interface 4 of the master robot 1 includes a monitor unit 6, a master controller, and an imaging unit 1210. Although not shown, as described above, the slave robot 2 may include a robot arm 3 and a laparoscope 5.

The monitor unit 6 of the master interface 4 may be composed of one or more monitors as shown, and information necessary for the operation of each monitor (for example, an image taken by the laparoscope 5, the patient's Ecological information, etc.) may be displayed separately. Of course, the quantity of the monitor can be variously determined according to the type or type of information requiring display.

The patient's ecological information (for example, pulse, respiration, blood pressure, body temperature, etc.) displayed through the monitor 6 may be divided and output for each region, and such biometric information may be biometric information provided in the slave robot 2. It can be measured by the measuring unit and provided to the master robot 1.

The imaging means 1210 is a means for photographing the operator's appearance (eg, a face region) in a non-contact manner. The imaging means 1210 may be implemented as a camera device including an image sensor, for example.

The image photographed by the imaging means 1210 is provided to the laparoscopy manipulation unit 1200 (see FIG. 13A), and the laparoscopy manipulation unit 1200 uses the amount of change information on the subject through the interpretation of the image to determine the laparoscope 5. Control to rotate, move, or enlarge / reduce an image.

In addition, the master interface 4 is provided with a master controller so that the operator can be gripped and manipulated by both hands. The master controller may be implemented with two or more handles 10, and an operation signal according to the operator's manipulation of the handle 10 is transmitted to the slave robot 2 to control the robot arm 3. By operating the handle 10 of the operator, a surgical operation such as a position movement, rotation, and cutting operation of the robot arm 3 may be performed.

For example, the handle 10 may be configured to include a main handle and a sub handle, and the operator may operate the robot arm 3 or the laparoscope of the slave robot 2 implemented to have multiple degrees of freedom with only the main handle. 5) or a plurality of surgical equipment can be operated in real time by operating the sub handle. Of course, the master manipulator is not limited to the shape of the handle, and may be applied without any limitation as long as it can control the operation of the robot arm 3 through a network.

One or more slave robots 2 may be used to operate the patient, and the laparoscope 5 for allowing the surgical site to be displayed as an image (that is, an image image) that can be confirmed through the display unit 6 is an independent slave. It may be implemented by the robot (2). In addition, as described above, embodiments of the present invention may be used universally in operations in which various surgical endoscopes (eg, thoracoscopic, arthroscopy, parenteral, etc.) other than laparoscopic are used.

13A is a block diagram schematically illustrating a configuration of a laparoscopic manipulation unit according to another embodiment of the present invention, and FIG. 13B is a view illustrating an operation concept of a laparoscopic manipulation unit according to another embodiment of the present invention. 14 and 15 are views illustrating facial movements for laparoscopy manipulation according to another embodiment of the present invention, respectively.

Referring to FIG. 13, the laparoscopic manipulation unit 1200 includes a storage unit 1220, an angle and distance calculator 1230, an operation command generator 1240, and a transmitter 1250. One or more of the illustrated components may be implemented as a software program or may be a combination of hardware and software programs. In addition, one or more components shown may be omitted.

The storage unit 1220 stores the image image captured by the imaging unit 1210, the displacement amount information calculated by the angle and distance calculator 1230.

The amount of displacement information is, for example, information about the angle and rotation direction between the extension line of the two eyes and the center line of the screen (that is, the horizontal line passing through the horizontal and vertical center points of the image image) calculated by using two consecutive image images in the calculation cycle. The moving distance information between the reference points in the face and the distance change amount information between the center points of the two eyes may be included.

Of course, it is not limited to use two consecutive video images to calculate the displacement information, etc., it is natural that the current video image and the video image photographed at any point in time can be used. For example, when there is an n-3th video image, an n-2th video image, an n-1th video image, and an nth video image that is the current view point, an image used for analysis When the photographing period of the image is 3, the displacement information may be calculated using the n-3 th image image and the n th image image. Even if the description is omitted, it is natural that the principle can be equally applied to the angle and distance calculation as well as the displacement information. However, in the present specification, for the convenience of explanation and understanding, the case where the current image image and the image image immediately before is used when the image image is interpreted for generating specific information will be described as an example.

The angle and distance calculator 1230 is captured by the image pickup means 1210 and is captured immediately before two video images consecutively calculated in a calculation cycle among the image images stored in the storage unit 1220 (that is, immediately before the currently captured image image). Image information) to generate displacement information between the image images and store it in the storage unit 1220.

Here, the angle and distance calculator 1230 is the information about the angle and rotation direction between the extension line of the two eyes and the screen center line calculated using two consecutive image images, the movement distance information between the reference point in the face, the center point of both eyes Interval change amount information and the like can be generated.

Hereinafter, a method of generating displacement amount information in the angle and distance calculator 1230 will be described. However, an image analysis technique for recognizing an image image and recognizing a location of a face region, an eye position, and a reference point (for example, a center point of the nose) is obvious to those skilled in the art, and thus a detailed description thereof will be omitted.

First, in order to generate information related to the rotation of the face, as illustrated in FIG. 14, the angle and distance calculator 1230 is photographed by the imaging unit 1210 and stored in the image image stored in the storage unit 1220. 크기, the size of the angle between the extension line of the two eyes) and the screen center line is calculated, and information about the rotation direction and the angle of rotation is generated by comparing the size of the angle with the size of the angle analyzed for the previous video image.

In other words, the angle and distance calculator 1230 may be configured to include, for example, the position and shape of both eyes in an image image generated by the imaging unit 1210, as illustrated in FIGS. 14B and 14C. After recognition by image processing techniques such as an edge detection technique, the center point of each eye is obtained, and then the size of the inner angle formed by the virtual straight line connecting the center point of each eye and the screen center line and the direction of rotation of the face are calculated. For example, if the previous video image generated is the raw video image of (a) and the currently generated video image is the video image of (b), a straight line connecting the center line of the screen and the center of the eyes for each video image. By comparing the intersections formed at which positions of the face and the size of the cabinet according to each other, it is possible to recognize in which direction and by which angle the face is rotated.

Next, in order to generate information about the parallel movement of the face, the angle and distance calculator 1230 is photographed by the imaging means 1210 and stored in the image 12 stored in the storage unit 1220 in a preset reference point (eg, in the face). For example, it detects how much the center of the nose is moved from the horizontal and vertical centers of the video image.

That is, in the case of (d) of FIG. 14, in the raw image image indicated by the dotted line, the reference point is in agreement with the center point of the image image, but in the current image image, the reference point is moved in parallel to a right distance from the center point of the image image. It can be recognized as one. Of course, the direction of the parallel movement can be varied, such as up, down, left and right.

Next, in order to generate information related to enlargement and reduction of the screen, the angle and distance calculation unit 1230 is photographed by the imaging unit 1210 and the distance of both eyes in the image image stored in the storage unit 1220 (see FIG. 15). d1, d2 or d3), and the calculated distance is increased than the distance of both eyes in the image image taken and stored immediately before (i.e., the operator's face moves toward the imaging means 1210) or decreases. The face of the operator in a direction away from the imaging means 1210 is calculated. The distance between the two eyes may be variously designated and applied, for example, the distance between the center point of each eye or the shortest distance between the outlines of each eye.

The operation command generation unit 1240 uses the displacement information calculated by the angle and distance calculation unit 1230 to determine the position of the laparoscope 5 (eg, movement, rotation, etc.) and the image magnification (eg, enlargement). Create a control command to control

If it is recognized that the operator's face is rotated in an arbitrary direction, the operation command generation unit 1240 will generate an operation command for rotating the laparoscope 5, and the operator's face is moved in parallel in an arbitrary direction. If it is recognized, the operation command generation unit 1240 will generate an operation command for moving the laparoscope 5 by a corresponding direction and distance, and if it is recognized that the operator's face is closer or farther away from the imaging means 1210, the operation command. The generation unit 1240 may generate an operation command to enlarge or reduce the magnification of the video image generated by the laparoscope 5.

Of course, the operation command generated by the operation command generation unit 1240 may be generated to enable the control of two or more of the rotation, movement and magnification of the laparoscope.

The transmission unit 1250 transmits the operation command generated by the operation command generation unit 1240 to the slave robot 2 to be manipulated with respect to the position of the laparoscope 5, and provides an image accordingly. The transmission unit 1250 may be a transmission unit already provided in the master robot 1 to transmit an operation command for the operation of the robot arm 3.

In addition, although not shown in FIG. 13A, a determination unit may be further included to determine whether a user in the image image currently being captured by the imaging unit 1210 is an authenticated user.

That is, the determining unit determines whether the face image in the image image currently captured and the face image in the image image pre-stored in the storage unit 1220 as an authenticated user match within an error range, and then, if it matches, the method described above. According to the laparoscopy operation can be made. Of course, in determining whether the determination unit is an authenticated user, one or more of the features such as the shape and shape of the eyes / eyebrows / nose / mouth, eye color, skin color, skin wrinkles, and iris, as well as the features of the shape of the face. Of course it can also be used.

In this way, by additionally providing a determination unit, even when a plurality of people are photographed in the image image captured by the imaging unit 1210, the laparoscopic operation can be performed in association with only the change in the appearance of the operator's face.

In addition, the determination unit may further determine whether the above-described laparoscopic manipulation unit 1200 functions by determining whether the face shape in the video image is located in a predetermined area and whether the size of the face is greater than or equal to the predetermined size. . Of course, the determination unit may determine whether the size of the face corresponds to a predetermined size condition instead of determining whether the size of the face is greater than or equal to a predetermined size. This is because it may cause malfunction even if the operator presses the face too close.

For example, as illustrated in FIG. 13B, the determination unit determines whether the face is located in the predetermined area 1320 in the display area 1310 in which the image image photographed by the imaging unit 1210 is displayed. It may be determined whether or not the laparoscopic manipulation unit 1200 is to function. Of course, the face is not limited to be positioned to include all of the predetermined area 1320, and may be preset to be sufficient as long as a part of the face is included in the predetermined area 1320.

If the face is located in the predetermined area 1320, the laparoscopy manipulation unit 1200 may be processed to function. Otherwise, the laparoscopy manipulation unit 1200 may be processed to not function.

In addition, as illustrated in FIG. 13B, the determination unit may determine whether the size of the face included in the image image is larger than the size of the predetermined area 1320 to determine whether the laparoscopic manipulation unit 1200 is to function. have. Whether the size of the face is larger than the size of the predetermined area 1320 may be determined as, for example, how the area calculated by face outline detection has a relationship with the size of the predetermined area 1320. In this case, as described above, it may be further determined whether the size of the face is smaller than the specified threshold.

If the size of the face is smaller than the size of the predetermined area as shown in (a), even if the operator is a third person who is located far from the master interface 4 or the operator is located far from the master interface 4 Since the surgical operation is not in progress, the laparoscopic manipulation unit 1200 may be prevented from functioning.

As described above, the determination unit allows the laparoscopic manipulation unit 1200 to function only when a face having a predetermined size or more is displayed to correspond to the predetermined area 1320, thereby irrelevant to facial movements or surgical operations of a third person other than the operator. It is possible to prevent the laparoscopic 5 from being misoperated by the movement.

In addition, the determination unit may process the above-described laparoscopic manipulation unit 1200 to function only when the facial movement in the video image captured by the imaging unit 1210 is maintained for a predetermined time. For example, the laparoscopic operation unit 1200 is a laparoscope to the right only when it is recognized that the operator tilts the head to the right in a normal state facing the front and then maintains it for a predetermined time (for example, 2 seconds). It may be preset to generate and transmit an operation command to rotate or / and move by an angle. Of course, it is natural that the determination unit may process the aforementioned laparoscopic manipulation unit 1200 to function not only when the facial movement is maintained for a predetermined time but also when the facial movement exceeds a predetermined range.

In this way, when the operator takes an action such as consciously or unconsciously looking back or moving, the laparoscopic 5 inserted into the abdomen of the patient may be prevented from being rapidly manipulated according to the operator's movement. This may be handled similarly to the method of securing the safety of the patient by blocking the transfer of the operation of the slave robot 2 when the face is removed from the contact structure described above, for example.

In addition, the determination unit generates a corresponding operation command to the operation command generation unit 1240 when a change of a specific part (for example, eye blinking) in the face in the image image captured by the imaging unit 1210 is recognized. May further perform a function of requesting transmission to the slave robot 2. In the case of a specific part change, information on which manipulation command should be generated may be stored in advance in the storage unit 1220.

For example, if the operator blinks only the left eye, for example, when the image captured by the imaging unit 1210 is blurred by smoke during surgery, the operator blinks only the left eye, the operation command generation unit 1240 may provide a valve for discharging carbon dioxide in the abdominal cavity. The operation command to open the can be generated and transmitted to the slave robot (2). As another example, if the operator blinks only the right eye during the operation, a manipulation command for displaying the pre-operational image of the current operating area on the monitor 6 of the master interface 4 is displayed as augmented reality or disappearing. You can also print

Of course, in addition to the method of instructing the controller to generate a manipulation command by recognizing a change in a specific part of the face in the image, the image recognition technique recognizes a motion of nodding or shaking the head up and down, and a corresponding manipulation command is generated. Naturally, it can be instructed to be generated. For example, if it is recognized that the operator has taken a nodding motion, it may be recognized as a positive response, and if it is recognized that the operator has taken a head stroke, it may be recognized as a negative response.

Using this, in certain situations, when the surgical robot requires the operator to make a choice of yes / no to perform an action, the operator nods or intercepts the head without having to press a button on the console or the steering wheel. It is possible to provide a response to the surgical robot alone.

In addition, in the non-contact laparoscopic control method according to the present embodiment, when the 3D laparoscope is applied, the degree of overlapping the left and right images for 3D image processing may be adjusted according to the position of the face and the eye.

16 is a flowchart illustrating an operation process of a laparoscopic manipulation unit according to another embodiment of the present invention.

Referring to FIG. 16, in operation 1610, the laparoscopic manipulation unit 1200 analyzes an image of a face in which an operator's face is photographed, and detects a face region, a position of an eye, and a position of a reference point (eg, a center point of a nose).

Subsequently, in step 1620, the laparoscopic manipulation unit 1200 calculates the cabinet angle and rotation direction between the extension lines of the two eyes and the center line of the screen with respect to the corresponding video image, and the difference between the cabinet angle and the rotation direction calculated for the immediately previous image image. Calculate the face rotation direction and angle of rotation of the current operator.

In addition, in operation 1630, the laparoscopic manipulation unit 1200 calculates a distance change amount between the face reference points in the image image immediately before the reference point in the face is captured with respect to the corresponding image.

In addition, in step 1640, the laparoscopic manipulation unit 1200 calculates the distance between the center points of the two eyes with respect to the corresponding video image, and refers to the distance between the two eyes by referring to the distance between the center points of the two eyes calculated with respect to the immediately captured image image. Calculate the amount of change.

The above-described steps 1620 to 1640 may be performed sequentially or nonsequentially or simultaneously.

In step 1650, the laparoscopic manipulation unit 1200 generates an operation command corresponding to the displacement amount information calculated by steps 1620 through 1640, and provides the slave command 2 to the laparoscopic 5.

FIG. 17 is a flowchart specifically illustrating step 1610 of FIG. 16 according to another embodiment of the present invention.

Step 1610 of FIG. 16 relates to a step in which the laparoscopic manipulation unit 1200 detects the position of the face region, the position of the eye, and the reference point (for example, the center point of the nose) by analyzing the image image of the operator's face. will be.

Referring to FIG. 17 in which step 1610 is embodied and illustrated, the laparoscopic manipulation unit 1200 is provided with an image image in which the operator's face is photographed in step 1710, and proceeds to step 1720 to interpret the image image to analyze the face region and eyes. The position of and the position of the reference point (eg, the center point of the nose) are detected.

Subsequently, the laparoscopy manipulation unit 1200 determines whether the corresponding movement state of the operator is maintained for a predetermined time in step 1730.

If the movement is maintained for a certain time, it is recognized that the laparoscopic 5 is for manipulating and proceeds to steps 1620 to 1640.

However, if the movement is not maintained for a certain time, it is recognized that the movement is irrelevant to the operation of the laparoscope 5, and the flow proceeds to step 1710 again.

As such, step 1610 may be embodied in a plurality of steps to have an accuracy of whether or not the laparoscope 5 is manipulated.

In addition, step 1610 can be embodied in a variety of ways.

For example, it is determined whether the face image in the received video image is the authenticated user in the error range and the face shape predetermined in the storage unit 1220, and only if the face image matches within the error range, steps 1620 to 1640. It can be embodied to proceed.

As another example, it is determined whether the face in the provided video image is located in a predetermined area, and whether the face is larger than or equal to the predetermined size and less than or equal to the predetermined size. It may be embodied to proceed to step 1620 to step 1640 only when located.

In addition, it is obvious that step 1610 may be embodied in various ways, and one or more embodiments may be embodied in combination.

The above-described laparoscopic manipulation method may be implemented by a software program or the like. Codes and code segments constituting a program can be easily inferred by a computer programmer in the art. The program is also stored in a computer readable media, and read and executed by a computer to implement the method. The information storage medium includes a magnetic recording medium, an optical recording medium and a carrier wave medium.

It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit or scope of the invention as defined in the appended claims. It will be understood that the invention may be varied and varied without departing from the scope of the invention.

1: master robot 2: slave robot
3: robot arm 4: master interface
5: Laparoscope 6: Monitor
10: handle 20: telescopic display portion
210: contact portion 220: eyepiece
230, 240: support part 250: flow part
310: motion detection unit 320, 1240: operation command generation unit
330, 1250: transmission unit 510: contact detection unit
520: Restore the original state 710: Camera unit
720 and 1220: storage unit 730: eye tracker unit
740: control unit 1200: laparoscopic operation unit
1230: angle and distance calculator

Claims (14)

A surgical robot for controlling one or more of the position and the image input angle of the vision unit using the operation signal,
A contact portion flowing in a direction and a size corresponding to the movement direction and the size of the contacted operator's face;
A motion detector for outputting sensing information corresponding to a direction and a magnitude in which the contact part flows; And
And a manipulation command generation unit configured to generate and output a manipulation command for at least one of the position and the image input angle of the vision unit by using the sensing information.
The method of claim 1,
And the manipulation command relates to one or more of straight line operation and rotational operation of the vision unit, wherein the operation handle direction of the surgical robot is changed to correspond thereto.
The method of claim 1,
And the contact portion is formed as part of a console panel of the surgical robot.
The method of claim 1,
Surgical robot is characterized in that the support is formed to protrude at one or more points of the contact portion to fix the position of the operator's face.
The method of claim 1,
And the eyepiece is perforated in the contact portion such that the image obtained by the vision portion is viewed as visual information.
The method of claim 1,
And the contact portion is formed of a material of light transmitting material so that the image obtained by the vision portion is viewed as visual information.
The method of claim 4, wherein
A touch sensing unit configured to detect whether the operator's face is in contact with the contacting unit or the support unit; And
And a home state restoration unit configured to process the contact portion to return to a reference state which is a position and state designated as default when the contact release is recognized by the detection of the touch sensing unit.
The method of claim 7, wherein
And the original state restoration unit processes the contact portion to return to the reference state by reverse manipulation of the contact portion flow direction and size according to the sensing information.
The method of claim 1,
A surgical robot further comprising an eye tracker unit for generating analysis information by analyzing one or more of a change in eye position, a change in eye shape, and a gaze direction by comparing and sequentially determining sequentially generated image data.
10. The method of claim 9,
A camera unit for imaging the inner surface of the surgical robot toward the contact portion to generate the image data; And
Surgical robot further comprising a storage unit for storing the generated image data.
10. The method of claim 9,
And the manipulation command generation unit determines whether the analysis information satisfies a preset change as an arbitrary manipulation command, and outputs a corresponding manipulation command when the manipulation information is satisfied.
The method of claim 1,
The vision unit is a surgical robot, characterized in that any one of the microscope, the endoscope.
The method of claim 12,
The endoscope is a surgical robot, characterized in that at least one of the laparoscope, thoracoscopic, arthroscopy, parenteral, bladder, rectal, duodenum, mediastinoscope, cardiac.
The method of claim 3,
The contact portion is formed on the front of the console panel to be supported by an elastic body,
The elastic body is a surgical robot, characterized in that to provide a restoring force so that the contact portion returns to its original position when the external force for the movement of the contact portion is removed.

Images