KR20150135752A - Apparatus and Method for processing surgical image based on motion - Google Patents

Abstract

Disclosed are an apparatus and a method for processing a tangible surgical image. The apparatus for processing the tangible surgical image comprises: an image input unit for receiving an input of an endoscopic image provided from a surgical endoscopic instrument; a screen display unit for outputting the endoscopic image on a specific region; and a screen display control unit for changing the specific region of the screen display unit for outputting the endoscopic image corresponding to the viewpoint of the surgical endoscopic instrument. The apparatus enables a user to feel the sense of reality during an actual surgical situation, by changing an output position of the endoscopic image outputted to a monitor and watched by the user corresponding to the motion of the surgical endoscopic instrument.

Description

Technical Field [0001] The present invention relates to an image processing apparatus and method for a tactile type surgical operation,

The present invention relates to an image processing apparatus, and more particularly, to an image processing apparatus and method for a tactile type surgical operation.

The surgical robot refers to a robot having a function of substituting surgical operation performed by a surgeon. Such a surgical robot has an advantage that it can perform accurate and precise operation compared to a human and can perform a remote operation.

Surgical robots currently being developed around the world include bone surgery robots, laparoscopic surgery robots, and stereotactic robots. Here, a laparoscopic surgical robot is a robot that performs minimally invasive surgery using laparoscopic and small surgical instruments.

Laparoscopic surgery is an advanced surgical technique that inserts a laparoscope, which is an endoscope that penetrates a 1 cm hole in the umbilicus and looks into the abdomen.

A recent laparoscope is equipped with a computer chip that provides a sharper, larger image than what the naked eye can see, and the laparoscopic surgical instruments specially designed for viewing through the monitor have been developed to allow any surgery.

In addition, laparoscopic surgery is similar to laparotomy, but it has fewer complications than laparotomy, can start treatment within a short time after surgery, and has the advantage of maintaining the patient's physical strength and immune function have. Therefore, in the United States and Europe, laparoscopic surgery is becoming increasingly recognized as a standard surgery for colorectal cancer treatment and the like.

Surgical robot systems generally consist of a master robot and a slave robot. When the user manipulates a manipulator (e.g., a handle) provided in the master robot, the surgical tool that is coupled to the robot arm of the slave robot or held by the robot arm is operated to perform the operation. The master robot and the slave robot are connected through a communication network to perform network communication.

Currently, laparoscopic surgery images are output to the user during the laparoscopic surgery, which causes the user to feel the actual feeling rather than the operation performed directly by the user. The problem is that even when the laparoscope moves and revolves in the abdominal cavity to illuminate other areas, the image that the user sees is output from the monitor of the same position and size, so that the relative distance and movement between the controller and the image, Which is different from the relative distance and motion between the surgical instrument and the organ.

In addition, since the surgical image taken by the laparoscope includes only a part of the surgical tool, the user may not be able to perform the operation smoothly due to the difficulty of manipulation or the visual field when they collide or overlap with each other.

The above-described background technology is technical information that the inventor holds for the derivation of the present invention or acquired in the process of deriving the present invention, and can not necessarily be a known technology disclosed to the general public prior to the filing of the present invention.

The present invention relates to a bodily-sensible type which allows a user to feel an actual operation situation more realistically by changing an output position of an endoscopic image output to a monitor viewed by a user in accordance with the viewpoint of an endoscope which changes according to the motion of the endoscope for surgery. And a surgical image processing apparatus and method.

In addition, the present invention extracts an endoscopic image previously input and stored at the current time point, and outputs the endoscopic image together with the current endoscopic image to a screen display unit, thereby enabling a user to inform the user about the change of the endoscopic image. Processing apparatus and method.

In addition, the present invention can be applied to an endoscopic endoscopic surgery using an endoscope, and a method of correcting an endoscopic image that has been actually photographed and a modeling image stored in an image storage unit, And to provide an image processing apparatus and method for a tactile type surgical operation capable of outputting images to an observable monitor.

It is another object of the present invention to provide an image processing apparatus and method for a tactile type surgical operation which enables a user to feel the reality of a surgery more vividly by rotating and moving the monitor in accordance with various aspects of the endoscope.

The technical problems other than the present invention can be easily understood from the following description.

According to an aspect of the present invention, there is provided an endoscopic image display apparatus including an image input unit for receiving an endoscopic image provided from a surgical endoscope, a screen display unit for outputting the endoscopic image to a specific region, a screen display unit for outputting an endoscopic image in accordance with the viewpoint of the surgical endoscope, And a screen display control unit for changing a specific area of the screen.

Herein, the surgical endoscope may be at least one of a laparoscope, a thoracoscopic, an arthroscope, a nystagmus, a cystoscope, a rectum, a duodenal, a mediastinoscope, and a heart, and may be a stereoscopic endoscope.

Here, the screen display control unit may include an endoscopic viewpoint tracking unit that tracks the viewpoint information of the surgical endoscope in accordance with the movement and rotation of the endoscopic endoscope, and a display unit that extracts movement information of the endoscopic image using the viewpoint information of the endoscopic endoscope A movement information extracting unit and an image position setting unit for setting a specific area of the screen display unit on which the endoscopic image is output using the movement information.

Further, the screen display control section can move the center point of the endoscopic image in accordance with the coordinate change value of the viewpoint of the surgical endoscope.

According to another aspect of the present invention, there is provided a stereoscopic image display apparatus including an image input unit that receives a first endoscopic image and a second endoscopic image provided at different times from a surgical endoscope, and a second endoscopic image output unit that outputs the first endoscopic image and the second endoscopic image, An image storage unit for storing the first endoscopic image and the second endoscopic image and a display unit for outputting the first endoscopic image and the second endoscopic image to different areas in accordance with different viewpoints of the surgical endoscope, And an image display control unit for controlling the image display unit.

Here, the image input unit may receive the first endoscopic image before the second endoscopic image, and the display unit may output at least one of saturation, lightness, color, and screen pattern differently from the first endoscopic image and the second endoscopic image .

The screen display control unit may further include a stored image display unit for extracting the first endoscopic image stored in the image storage unit and outputting the extracted first endoscopic image to the screen display unit while the second display unit displays the second endoscopic image input in real time.

According to another aspect of the present invention, there is provided a surgical instrument comprising: an image input unit for receiving an endoscopic image provided from a surgical endoscope; a screen display unit for outputting an endoscopic image to a specific area; An image matching unit for generating an output image by matching the endoscopic image and the modeling image with each other, and a specific area of the screen display unit for outputting the endoscopic image corresponding to the viewpoint of the surgical endoscope is changed And a screen display control unit for outputting the matched endoscopic image and the modeling image to a screen display unit.

Here, the image matching unit can generate an output image by matching the actual surgical tool image included in the endoscopic image and the modeling operation tool image included in the modeling image with each other.

The image matching unit may further include a feature value calculation unit that calculates a feature value using at least one of an endoscopic image and positional coordinate information of an actual surgical tool coupled to the at least one robot arm; And a modeling image implementing unit for implementing the modeling image.

Here, the image matching unit may further include an overlapping image processing unit for removing the overlapping area between the modeling operation tool image and the actual operation tool image from the modeling operation tool image, and the position of the modeling operation tool image output to the modeling image may be And can be set using manipulation information of the surgical tool.

According to another aspect of the present invention, there is provided an endoscopic endoscope system including an image input unit for receiving an endoscopic image provided from a surgical endoscope, a screen display unit for outputting an endoscopic image, a screen driving unit for rotating and moving the screen display unit, And a screen driving control unit for controlling the screen driving unit so that the screen driving unit rotates and moves the screen display unit correspondingly.

Here, the screen drive control unit includes an endoscopic viewpoint tracking unit that tracks the viewpoint of the surgical endoscope in accordance with the movement and rotation of the endoscopic endoscope, and a display unit that extracts movement information of the endoscopic image using the perspective information of the endoscopic endoscope A movement information extracting unit and a drive information generating unit for generating screen drive information of the screen display unit using the movement information.

Here, the screen driving unit may be configured such that one end of the screen driving unit is coupled to the screen display unit to move the screen display unit along a predetermined moving groove, or the screen driving unit is coupled to the screen display unit to move and rotate the screen display unit And can be a space-shaped movement driving unit in the form of a robot arm.

Here, the screen display section may include a dome-shaped screen and a projector for projecting an endoscope image on a dome-shaped screen.

According to another aspect of the present invention, there is provided a method of outputting an endoscopic image, the method comprising: receiving an endoscopic image provided from a surgical endoscope; outputting an endoscopic image to a specific area of the screen display; There is provided an image processing method for a haptic surgical operation including a step of changing a specific area of a screen display unit outputting an endoscopic image in accordance with the viewpoint of the surgical endoscope.

Herein, the surgical endoscope may be at least one of a laparoscope, a thoracoscopic, an arthroscope, a nystagmus, a cystoscope, a rectum, a duodenal, a mediastinoscope, and a heart, and may be a stereoscopic endoscope.

The step of changing the specific area of the screen display section may include the steps of tracking the perspective information of the surgical endoscope corresponding to the movement and rotation of the surgical endoscope, extracting movement information of the endoscopic image using the perspective information of the surgical endoscope, And setting a specific region of the screen display unit on which the endoscopic image is output using the movement information.

Here, the step of changing the specific area of the screen display part may include a step of moving the center point of the endoscopic image in accordance with the coordinate change value of the viewpoint of the operation endoscope.

According to another aspect of the present invention, there is provided a method of outputting an endoscopic image, the method comprising: receiving a first endoscopic image and a second endoscopic image provided at different points in time from a surgical endoscope; Outputting the endoscopic image and the second endoscopic image to different areas of the screen display section; storing the first endoscopic image and the second endoscopic image; and storing the first endoscopic image and the second endoscopic image in accordance with different viewpoints of the surgical endoscope, And controlling the screen display unit to output the endoscopic images to different areas.

Herein, in the step of receiving the endoscopic image, the first endoscopic image may be input before the second endoscopic image, and the outputting step may output the first endoscopic image and the second endoscopic image in any of saturation, brightness, One or more can be output differently.

The screen display unit control step may further include extracting a first endoscopic image stored in the image storage unit and outputting the first endoscopic image to the screen display unit while the second display unit displays the second endoscopic image input in real time.

According to another aspect of the present invention, there is provided a method of outputting an endoscopic image, the method comprising: inputting an endoscopic image provided from a surgical endoscope; outputting an endoscopic image to a specific area of the screen display; A step of storing a modeling image related to a surgical tool to be operated by a surgical endoscope, a step of generating an output image by matching an endoscopic image with a modeling image, and a step of outputting an endoscopic image corresponding to the viewpoint of the surgical endoscope And outputting a matching endoscopic image and a modeling image to a screen display unit.

Here, in the step of generating an output image, an output image may be generated by matching the actual surgical tool image included in the endoscopic image and the modeling and surgical tool image included in the modeling image with each other.

The generating of the output image may include calculating a characteristic value using at least one of an endoscopic image and positional coordinate information of an actual surgical tool coupled to the at least one robot arm, And may further comprise a step of implementing.

Here, the step of generating an output image may further include removing an overlapping area between the modeling operation tool image and the actual operation tool image from the modeling operation tool image.

In addition, the position of the modeling surgical tool image output to the modeling image can be set using operation information of the surgical tool.

According to another aspect of the present invention, there is provided a method of outputting an endoscopic image by a surgical image processing apparatus, comprising: receiving an endoscopic image provided from a surgical endoscope; outputting an endoscopic image to a screen display; And a step of rotating and moving the screen display unit in accordance with the viewpoint of the user.

The step of rotating and moving the screen display unit may include the steps of tracking the perspective information of the surgical endoscope in accordance with the movement and rotation of the surgical endoscope, extracting movement information of the endoscopic image using the perspective information of the surgical endoscope, And generating motion information of the screen display unit using the motion information, the step, and the movement information.

Here, the screen display section may include a dome-shaped screen and a projector for projecting an endoscope image on a dome-shaped screen.

According to another aspect of the present invention, there is provided a program for tangibly embodying an image processing method for a tactile surgical operation, the program of instructions executable by the digital processing apparatus being tangibly embodied and readable by the digital processing apparatus, There is provided a recording medium on which recorded information is recorded.

According to yet another aspect of the present invention, there is provided a stereoscopic image display apparatus including an image input unit receiving a first endoscopic image and a second endoscopic image provided at different points in time from a surgical endoscope rotating at one end, And a screen display control section for controlling the screen display section to output the first endoscopic image and the second endoscopic image to different areas in accordance with different viewpoints of the surgical endoscope, An image processing apparatus is provided.

Here, the surgical endoscope may be periodically rotated. In this embodiment, the rotation direction, the rotational angular velocity, the acceleration / deceleration type, the rotational speed, the rotational time point, the rotational end point, And a turning radius setting unit that sets at least one of rotation-related information among the rotation-related information.

Here, the surgical endoscope may be rotated to form a closed figure, and may be rotated in the shape of a cone or pyramid, and the rotational locus of the surgical endoscope may be any one of a circle, an ellipse, a triangle and a rectangle.

In addition, the screen display control unit may include a continuous image generation unit that extracts overlapping regions of the first endoscopic image and the second endoscopic image to generate a continuous image, and a non-overlap region of the first endoscopic image and the second endoscopic image, And a peripheral image generating unit for generating an image.

According to another aspect of the present invention, there is provided an endoscope system including: a first image input unit that receives a first endoscopic image from a surgical endoscope; and a second image input unit that is provided at different points of time from an auxiliary endoscope that is coupled to one side of the endoscope, A second display unit for displaying the first endoscopic image and the second endoscopic image in different areas and a second display unit for displaying the first endoscopic image and the second endoscopic image in accordance with different views of the surgical endoscope and the endoscopic endoscope, And a screen display control unit for controlling the screen display unit to output the first endoscopic image and the second endoscopic image to different areas.

Here, the auxiliary endoscope can be periodically rotated, and the present embodiment can be applied to a case where the rotation direction, the rotational angular velocity, the acceleration / deceleration type, the rotational speed, the rotation time point, the rotation time end point, And a rotation operation section for setting at least one of the rotation related information among the plurality of rotation related information.

In addition, the screen display control unit may include a continuous image generation unit that extracts overlapping regions of the first endoscopic image and the second endoscopic image to generate a continuous image, and a non-overlap region of the first endoscopic image and the second endoscopic image, And a peripheral image generating unit for generating an image.

The screen display control unit may include a continuous image generating unit for generating a continuous image from the first endoscopic image and a peripheral image generating unit for extracting the second endoscopic image to generate a peripheral image of the continuous image.

Further, the auxiliary endoscope can be detachably coupled to the surgical endoscope.

According to another aspect of the present invention, there is provided a method for outputting an endoscopic image by a surgical image processing apparatus, the method comprising: receiving a first endoscopic image and a second endoscopic image provided at different points in time from a surgical endoscope, Outputting the first endoscopic image and the second endoscopic image to different areas of the screen display unit and outputting the first endoscopic image and the second endoscopic image to different areas in accordance with different viewpoints of the surgical endoscope, And a step of controlling the display unit.

Here, the surgical endoscope may be periodically rotated. In this embodiment, the rotation direction, the rotational angular velocity, the acceleration / deceleration type, the rotational speed, the rotational time point, the rotational end point, And a rotation radius of the at least one rotation-related information.

The endoscope for surgery can be rotated to form a closed shape at one end, and can be rotated in the shape of a cone or pyramid, and the rotation locus of the endoscope for surgery can be any one of a circle, an ellipse, a triangle and a rectangle.

Here, the screen display unit control step may include a step of generating a continuous image by extracting an overlapping area between the first endoscopic image and the second endoscopic image, and a step of extracting a non-overlapping area between the first endoscopic image and the second endoscopic image to generate a surrounding image .

According to still another aspect of the present invention, there is provided a method of operating an endoscope, including the steps of receiving a first endoscopic image from a surgical endoscope, inputting a plurality of images from an auxiliary endoscope, A step of outputting the first endoscopic image and the second endoscopic image to different areas, a step of outputting the first endoscopic image and the second endoscopic image to each other in accordance with different views of the surgical endoscope and the endoscopic endoscope, And controlling the screen display unit to output the image data to another area.

Here, the auxiliary endoscope can be periodically rotated, and the present embodiment can be applied to a case where the rotation direction, the rotational angular velocity, the acceleration / deceleration type, the rotational speed, the rotation time point, the rotation time end point, And setting the rotation-related information of any one or more of the radiuses.

In addition, the screen display unit control step may include a step of generating a continuous image by extracting overlapping regions of the first endoscopic image and the second endoscopic image, extracting a non-overlapping region between the first endoscopic image and the second endoscopic image, .

In addition, the screen display unit control step may include a step of generating a continuous image from the first endoscopic image, and a step of extracting the second endoscopic image to generate a surrounding image of the continuous image.

In addition, the present embodiment may further include a step of detachably coupling the auxiliary endoscope to the surgical endoscope.

Here, the screen display section may include a dome-shaped screen and a projector for projecting an endoscope image on a dome-shaped screen.

According to still another aspect of the present invention, there is provided a computer readable recording medium having a program of tangibly embodied program executable by a digital processing apparatus for performing the above-described tactile surgical image processing method, A recording medium on which a program is recorded is provided.

Other aspects, features, and advantages will become apparent from the following drawings, claims, and detailed description of the invention.

The image processing apparatus and method according to the present invention can change an output position of an endoscopic image output to a monitor viewed by a user in accordance with the viewpoint of an endoscope that changes according to the motion of the endoscope for surgery, There is an effect that makes the situation feel more realistic.

In addition, the present invention provides an image processing apparatus and method for a sensible-type surgical operation, which extracts an endoscopic image previously input and stored and outputs the same to a screen display unit together with a current endoscopic image to obtain information on a change in an endoscopic image There is an effect that can inform the user.

In addition, the image processing apparatus and method according to the present invention can be applied to an endoscopic image obtained by using an endoscope during surgery and a modeling image stored in an image storage unit, It is possible to modify the image by adjusting the size and output it to the monitor which the user can observe.

In addition, the image processing apparatus and method according to the present invention can rotate and move the monitor according to the viewpoints of various endoscopes, so that the user can feel the reality of surgery more vividly.

1 is a plan view showing the entire structure of a surgical robot according to an embodiment of the present invention.
2 is a conceptual diagram showing a master interface of a surgical robot according to the first embodiment of the present invention;
FIG. 3 is a block diagram of a surgical robot according to a first embodiment of the present invention; FIG.
FIG. 4 is a block diagram of an image processing apparatus for a sensible-type surgery according to a first embodiment of the present invention; FIG.
FIG. 5 is a flow chart of a method for image processing for a sensible type surgical operation according to the first embodiment of the present invention. FIG.
FIG. 6 is a block diagram of an output image according to an image processing method for a sensible type surgical operation according to a first embodiment of the present invention; FIG.
FIG. 7 is a block diagram of a surgical robot according to a second embodiment of the present invention; FIG.
FIG. 8 is a block diagram of an image processing apparatus for a sensible-type surgery according to a second embodiment of the present invention; FIG.
FIG. 9 is a flow chart of a method for image processing for a sensible-type surgery according to a second embodiment of the present invention.
FIG. 10 is a block diagram of an output image according to an image processing method for a sensible type surgical operation according to a second embodiment of the present invention. FIG.
11 is a block diagram of a surgical robot according to a third embodiment of the present invention.
FIG. 12 is a block diagram of an image processing apparatus for a sensible-type surgery according to a third embodiment of the present invention; FIG.
13 is a flowchart of a method of image processing for a sensible-type surgical operation according to a third embodiment of the present invention.
FIG. 14 is a block diagram of an output image according to an image processing method for a sensible type surgical operation according to a third embodiment of the present invention; FIG.
15 is a conceptual diagram showing a master interface of a surgical robot according to a fourth embodiment of the present invention;
16 is a block diagram of a surgical robot according to a fourth embodiment of the present invention.
17 is a block diagram of an image processing apparatus for a tactile sensation type according to a fourth embodiment of the present invention.
FIG. 18 is a flowchart of a method for image processing for a tactile sense according to a fourth embodiment of the present invention; FIG.
FIG. 19 is a conceptual view showing a master interface of a surgical robot according to a fifth embodiment of the present invention; FIG.
20 is a block diagram of an image processing apparatus for a tactile sensation type according to a sixth embodiment of the present invention;
FIG. 21 is a flowchart of a method for image processing for a tactile sense according to a sixth embodiment of the present invention; FIG.
22A is a view showing a rotation operation of the surgical endoscope according to the sixth embodiment of the present invention.
FIG. 22B is a view showing a rotation operation of the surgical endoscope according to the sixth embodiment of the present invention; FIG.
23A and 23B are views showing an endoscopic image according to the sixth embodiment of the present invention.
24 is a view showing a rotating operation of the auxiliary endoscope according to the seventh embodiment of the present invention;
25 is a conceptual diagram illustrating a master interface of a surgical robot according to an eighth embodiment of the present invention;

While the invention is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will herein be described in detail. It is to be understood, however, that the invention is not to be limited to the specific embodiments, but includes all modifications, equivalents, and alternatives falling within the spirit and scope of the invention.

Terms including ordinals, such as first, second, etc., may be used to describe various elements, but the elements are not limited to these terms. The terms are used only for the purpose of distinguishing one component from another. It is to be understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, .

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, the terms "comprises" or "having" and the like refer to the presence of stated features, integers, steps, operations, elements, components, or combinations thereof, But do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or combinations thereof.

In the following description of the present invention with reference to the accompanying drawings, the same components are denoted by the same reference numerals regardless of the reference numerals, and redundant explanations thereof will be omitted. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description, well-known functions or constructions are not described in detail since they would obscure the invention in unnecessary detail.

FIG. 1 is a plan view showing an overall structure of a surgical robot according to an embodiment of the present invention, and FIG. 2 is a conceptual view illustrating a master interface of a surgical robot according to the first embodiment of the present invention.

The present embodiment changes the output position of the endoscopic image 9 outputted to the monitor viewed by the user in accordance with the viewpoint of the endoscope which changes according to the motion of the endoscope for operation so that the user can feel the actual operation situation more realistically . That is, since the viewpoint of the endoscope can coincide with the viewpoint of the user performing the operation, the present embodiment can be realized by matching the position of the endoscope within the abdominal cavity with the position and the direction of the output of the monitor outputting the endoscope image 9 at the external surgery site , The motion of the system located at the external surgical site reflects the motion of the endoscope moving inside the patient, thereby providing a more realistic feeling.

The surgical endoscope according to the present embodiment may be various kinds of instruments used as an imaging tool in surgery such as a laparoscope, a thoracoscopic, an arthroscope, a non-diameter, a cystoscope, a rectoscope, a duodenoscope, a mediastinoscope, In addition, the surgical endoscope according to the present embodiment may be a stereoscopic endoscope. That is, the surgical endoscope according to the present embodiment may be a stereoscopic endoscope for generating stereoscopic image information, and the stereoscopic image information generating method may be implemented by various techniques. For example, the surgical endoscope according to the present embodiment may include a plurality of cameras to acquire a plurality of images having stereoscopic information, or may acquire a plurality of images using one camera, Images can be acquired. In addition to this method, it is needless to say that the surgical endoscope according to the present embodiment can generate a stereoscopic image by various other methods.

Further, the haptic surgical image processing apparatus according to the present embodiment is not necessarily implemented in the surgical robot system as shown in the drawing, and is a system that outputs an endoscopic image 9 during surgery and performs surgery using surgical instruments Applicable. Hereinafter, the case in which the surgical image processing apparatus according to the present embodiment is applied to the surgical robot system will be described.

1 and 2, the surgical robot system includes a slave robot 2 for performing surgery on a patient lying on the operating table and a master robot 1 for remotely controlling the slave robot 2 . The master robot 1 and the slave robot 2 are not necessarily separated from each other by a separate device that is physically independent and can be integrated into one unit. In this case, the master interface 4 is, for example, It can correspond to the interface part.

The master interface 4 of the master robot 1 includes a monitor unit 6 and a master controller and the slave robot 2 includes a robot arm 3 and an instrument 5. The instrument 5 is a surgical instrument such as an endoscope such as a laparoscope or a surgical instrument for directly manipulating the affected part.

The master interface 4 is provided with a master controller so that the operator can be grasped by both hands and operated. The master manipulator can be implemented with two handles 10 as illustrated in Figures 1 and 2, and an operation signal according to operation of the operating handle 10 is transmitted to the slave robot 2, Respectively. The robot arm 3 and / or the instrument 5 can be moved, rotated, cut, or the like by operating the handle 10 of the operator.

For example, the handle 10 may comprise a main handle and a sub handle. The slave robot arm 3 or the instrument 5 can be operated with only one handle or a plurality of surgical instruments can be operated simultaneously in real time by adding a sub handle. The main handle and the sub handle may have various mechanical configurations according to their operation modes. For example, a robot arm 3 of a slave robot 2 such as a joystick type, a keypad, a trackball, a touch screen, Various input means for operating the equipment can be used.

The master manipulator is not limited to the shape of the handle 10, and can be applied without any limitation as long as it can control the operation of the robot arm 3 via the network.

The endoscope image 9, the camera image, and the modeling image input by the instrument 5 are displayed as image images on the monitor unit 6 of the master interface 4. [ In addition, the information displayed on the monitor unit 6 may vary depending on the type of the selected image.

The monitor unit 6 may be constituted by one or more monitors, and information necessary for operation may be individually displayed on each monitor. 1 and 2 illustrate the case where the monitor unit 6 includes three monitors, but the number of monitors can be variously determined according to the type and kind of information required to be displayed. Also, when the monitor unit 6 includes a plurality of monitors, the screens can be interlocked and expanded. That is, the endoscopic image 9 can freely move each monitor like a window output to one monitor, and a whole image may be outputted by outputting some images connected to each monitor.

The slave robot 2 and the master robot 1 are coupled to each other via a wired or wireless communication so that the master robot 1 transmits an operation signal to the slave robot 2. The slave robot 2 transmits the operation signal to the slave robot 2, The endoscopic image 9 inputted through the instrument 5 can be transmitted to the endoscope apparatus. If two operation signals by the two handles 10 provided in the master interface 4 and / or operation signals for adjusting the position of the instrument 5 need to be transmitted at the same time and / , The respective operation signals can be transmitted to the slave robot 2 independently of each other. Here, the fact that each operation signal is transmitted 'independently' means that the operation signals do not interfere with each other, and that any one operation signal does not affect the other. In order to allow a plurality of operation signals to be independently transmitted, header information for each operation signal is added and transmitted in the generation of each operation signal, or each operation signal is transmitted in accordance with the generation order, or A variety of schemes may be used, such that priorities are given in advance regarding the transmission order of each operation signal and are transmitted in accordance with the priorities. In this case, the transmission path through which each operation signal is transmitted may be provided independently so that interference between each operation signal may be fundamentally prevented.

The robot arm 3 of the slave robot 2 can be implemented to be driven with multiple degrees of freedom. The robot arm 3 includes, for example, a surgical tool inserted into a surgical site of a patient, a swinging drive unit for rotating the surgical tool in a yaw direction according to a surgical position, a pitch direction , A pitch driving part for rotating the surgical tool, a feed driving part for moving the surgical tool in the longitudinal direction, a rotation driving part for rotating the surgical tool, and a surgical tool driving part for cutting or cutting the surgical lesion, . However, it should be understood that the configuration of the robot arm 3 is not limited thereto, and that these examples do not limit the scope of the present invention. The actual control process of rotating or moving the robot arm 3 in the corresponding direction by operating the handle 10 by the surgeon is somewhat distant from the gist of the present invention, and a detailed description thereof will be omitted.

The slave robot 2 can be used more than one to perform surgery on the patient and the instrument 5 for displaying the surgical site through the monitor unit 6 as an image image can be implemented as an independent slave robot 2 And the master robot 1 may also be integrated with the slave robot 2.

3 is a block diagram of a surgical robot according to a first embodiment of the present invention. 3, the master robot 1 includes an image input unit 310, a screen display unit 320, a dark operation unit 330, an operation signal generation unit 340, a screen display control unit 350, and a control unit 370. [ And a slave robot 2 including a robot arm 3 and an endoscope 8 are shown.

The haptic surgical image processing apparatus according to the present embodiment may be implemented as a module including an image input unit 310, a screen display unit 320 and a screen display control unit 350. Of course, An operation signal generating unit 340, and a control unit 370.

The image input unit 310 receives the image input through the endoscope 8 of the slave robot 2 through wired or wireless transmission. The endoscope 8 may also be one kind of surgical instruments according to the present embodiment, and the number thereof may be one or more.

The screen display unit 320 outputs visual images corresponding to the images received through the image input unit 310 as visual information. The screen display unit 320 may output the endoscopic image as it is in size or zoom in / zoom out, or may output the endoscopic image and the modeling image, which will be described later, as a separate image.

In addition, the screen display unit 320 may simultaneously output an endoscopic image and an image that illuminates the entire operation situation, for example, a camera image generated by the camera by photographing the outside of the operation target, It may be facilitated.

The screen display unit 320 outputs a reduced image of the entire image (endoscopic image, modeling image, camera image, etc.) to a window created on a part of the output image or on a separate screen, A function of moving or rotating the entire output image when selecting or rotating a specific point out of the reduced image output by using the master controller, that is, a so-called bird's eye view function of the CAD program. Functions such as zooming in / zooming out, moving, rotating, etc. of the image outputted to the screen display unit 320 as described above can be controlled by the control unit 370 in accordance with the operation of the master controller.

The screen display unit 320 may be implemented in the form of a monitor unit 6 and the like. The image processing process for causing the received image to be output as an image through the screen display unit 320 may include a control unit 370, (350) or a separate image processing unit (not shown). The screen display unit 320 according to the present exemplary embodiment may be a display implemented by various technologies, for example, an ultra high resolution monitor such as a multivision UHDTV (7380x4320). Also, the screen display unit 320 according to the present embodiment may be a 3D display. For example, the screen display unit 320 according to the present embodiment can allow the user to recognize the left eye image and the right eye image separately by using the principle of binocular parallax. Such a 3D image realization method can be implemented in various ways such as a spectacle type (for example, a red-eyeglass type (anaglyph), a polarized glasses type (passive glass), a shutter type glasses (active glass), etc.), a lenticular method, have.

The screen display unit 320 outputs the input endoscopic image to a specific area. Here, the specific area may be a screen area having a predetermined size and position. This specific area can be determined corresponding to the change of the viewpoint of the endoscope 8 as described above.

The screen display control section 350 can set this specific area in accordance with the viewpoint of the endoscope 8. [ That is, the screen display control unit 350 tracks the viewpoint corresponding to the motion of the endoscope 8, such as the rotation and the movement, and sets a specific area for outputting the endoscope image on the screen display unit 320,

The arm control unit 330 is a means for enabling the operator to operate the position and function of the robot arm 3 of the slave robot 2. [ 2, a shape of the handle 10 is not limited to the shape of the handle 10, but may be modified into various shapes to achieve the same purpose. In addition, for example, a part may be formed in a handle shape and the other part may be formed in another shape such as a clutch button or the like. In order to facilitate manipulation of the surgical tool, a finger insertion tube or insertion More rings may be formed.

The operation signal generator 340 generates a corresponding operation signal when the operator operates the arm control unit 330 to move the robot arm 3 and / or the endoscope 8 or to operate the robot arm 3 and / To the robot (2). The operation signal may be transmitted or received via a wired or wireless communication network.

The operation signal generating unit 340 generates an operation signal by using the operation information according to the operation of the operation member 340 of the operator and transmits the generated operation signal to the slave robot 2. As a result, As shown in FIG. In addition, the position and manipulation shape of the actual surgical tool manipulated by the manipulation signal can be confirmed by the operator through the image input by the endoscope 8. [

4 is a block diagram of an image processing apparatus for a haptic surgical method according to a first embodiment of the present invention. 4, the screen display control unit 350 may include an endoscopic viewpoint tracking unit 351, an image movement information extracting unit 353, and an image position setting unit 355.

The endoscope viewpoint tracking unit 351 tracks the viewpoint information of the endoscope 8 in accordance with the movement and rotation of the endoscope 8. [ Here, the point of view information means a view point viewed by the endoscope 8, and such point of view information can be extracted from the signal for operating the endoscope 8 in the above-described surgical robot system. In other words, the point of view information can be specified by a signal that manipulates the movement and rotational motion of the endoscope 8. [ The manipulation signal of the endoscope 8 is generated in the surgical robot system and transmitted to the robot arm 3 that manipulates the endoscope 8. Therefore, the direction of the endoscope 8 can be tracked using such a signal.

The image movement information extracting unit 353 extracts movement information of the endoscopic image using the perspective information of the endoscope 8. [ That is, the viewpoint information of the endoscope 8 may include information on the position change amount of the object to be captured of the obtained endoscopic image, and movement information of the endoscopic image can be extracted from this information.

The image position setting unit 355 sets a specific region of the screen display unit 320 on which the endoscopic image is output by using the extracted movement information. For example, if the point of view information of the endoscope 8 is changed by a predetermined vector A, the movement information of the endoscopic image of the patient's internal organs corresponding to the vector is specified, A specific area of the display unit 320 is set. If the endoscopic image has changed by a predetermined vector B, a specific area to be actually output from the endoscopic image to the screen display unit 320 may be set using this information and the size, shape, and resolution of the screen display unit 320.

FIG. 5 is a flowchart of a method for image processing for a sensible type surgical operation according to the first embodiment of the present invention. It is needless to say that each step to be performed below can be performed by the screen display control unit 350 as a subject, and that the steps need not always be executed in a time sequence in the described order.

In step S511, the viewpoint information of the endoscope 8, which is information on the viewpoint of the endoscope 8 in accordance with the movement and rotation of the endoscope 8, is tracked. The direction of the endoscope 8 can be tracked by specifying the point of view information by a signal that manipulates the movement and rotation of the endoscope 8. [

In step S513, the movement information of the endoscopic image corresponding to the positional change amount of the object of the endoscopic image is extracted using the viewpoint information of the endoscope 8. [

In step S515, a specific area of the screen display unit 320 on which the endoscopic image is output is set by using the extracted movement information. That is, when the view information of the endoscope 8 and the movement information of the endoscope image are specified as described above, the specific region to which the endoscope image is to be output is set on the screen display unit 320 using this movement information.

In step S517, the endoscope image is output to the specific area set by the screen display unit 320. [

6 is a configuration diagram of an output image according to the image processing method for a tactile sense according to the first embodiment of the present invention. The screen display unit 320 may be a full screen and the endoscope image 620 acquired by the endoscope 8 may be displayed on the endoscopic image 620 at a specific position of the screen display unit 320, ) At the center point of the image. The coordinates (X, Y) may be set corresponding to the amount of change of the viewpoint of the endoscope 8. [ For example, when the perspective information of the endoscope 8 and the amount of movement of the endoscopic image are changed by +1 and -1 vertically, the center point of the endoscopic image 620 is positioned at the position of the coordinates (X + 1, Y-1) Can be moved.

7 is a block diagram of a surgical robot according to a second embodiment of the present invention. 7, an image input unit 310, a screen display unit 320, a dark operation unit 330, an operation signal generation unit 340, a screen display control unit 350, an image storage unit 360, a control unit 370, A slave robot 2 including a robot arm 3 and an endoscope 8 is shown. The differences from the above will be mainly described.

The present embodiment is characterized in that at this point in time, an endoscopic image previously inputted and stored is extracted and output to the screen display unit 320 together with the current endoscopic image, thereby informing the user of the change in the endoscopic image .

The image input unit 310 receives the first endoscopic image and the second endoscopic image provided at different points in time from the surgical endoscope. Here, the ordinal numbers such as first, second, etc. may be identifiers for distinguishing between different endoscopic images, and the first endoscopic image and the second endoscopic image may be images obtained by the endoscope 8 at different viewpoints and points of view . In addition, the image input unit 310 may receive the first endoscopic image before the second endoscopic image.

The image storage unit 360 stores the first endoscopic image and the second endoscopic image. The image storage unit 360 stores not only image information that is the actual image content of the first endoscopic image and the second endoscopic image, but also information about a specific region to be output to the screen display unit 320.

The screen display unit 320 outputs the first endoscopic image and the second endoscopic image to different areas and the display control unit 350 displays the first endoscopic image and the second endoscopic image in accordance with different viewpoints of the endoscope 8. [ To the different areas.

Here, the screen display unit 320 may output at least one of saturation, lightness, color, and screen pattern differently from the first endoscopic image and the second endoscopic image. For example, the screen display unit 320 may output a second endoscopic image, which is currently input, as a color image, and a first endoscopic image, which is a past image, as a monochrome image or the like. 10, the second endoscopic image 622, which is the currently input image, is output as the color image at coordinates (X1, Y1), and the first endoscopic image 621, which is a previously input image, An example is shown in which a hatched pattern is formed and output to the coordinates (X2, Y2).

Also, the first endoscopic image, which is the previous image, can be continuously output or only for a predetermined time. In the latter case, the past image is output to the screen display unit 320 only for a predetermined time, so that the new endoscope image can be continuously updated on the screen display unit 320. [

FIG. 8 is a block diagram of a haptic surgical image processor according to a second embodiment of the present invention. 8, the screen display control unit 350 may include an endoscopic viewpoint tracking unit 351, an image movement information extraction unit 353, an image position setting unit 355, and a stored image display unit 357.

The endoscopic viewpoint tracking section 351 tracks the viewpoint information of the endoscope 8 in accordance with the movement and rotation of the endoscope 8. The image movement information extracting section 353 extracts, The movement information of the endoscopic image is extracted as described above.

The image position setting unit 355 sets a specific region of the screen display unit 320 on which the endoscopic image is output by using the extracted movement information.

The storage image display unit 357 extracts the first endoscopic image stored in the image storage unit 360 and outputs the first endoscopic image to the screen display unit 320 while the screen display unit 320 outputs the second endoscopic image input in real time. Since the first endoscopic image and the second endoscopic image have different output regions and image information, the stored image display unit 357 extracts the information from the image storage unit 360 and stores the first endoscopic image, And outputs it to the screen display unit 320.

FIG. 9 is a flowchart of a method for image processing for a sensible type surgical operation according to a second embodiment of the present invention. Each of the steps to be described below can be executed by the screen display control unit 350 as a subject and can be roughly divided into a step of outputting a first endoscopic image and a step of outputting a second endoscopic image, 1 endoscopic image may be output together with the second endoscopic image.

In step S511, the perspective information of the endoscope 8, which is information on the viewpoint of the endoscope 8 in accordance with the first movement and rotation information of the endoscope 8, is tracked.

In step S513, movement information of the first endoscopic image is extracted. In step S515, a specific area of the screen display unit 320 on which the endoscopic image is output is set using the extracted movement information. In step S517, And outputs the first endoscopic image.

In step S519, the information on the output endoscopic image and the first screen position are stored in the image storage unit 360. [

In step S521, the perspective information of the endoscope 8, which is information on the viewpoint of the endoscope 8 in accordance with the second movement and rotation information of the endoscope 8, is tracked.

In step S522, movement information of the second endoscopic image is extracted. In step S523, a specific area of the screen display unit 320 on which the endoscopic image is output is set using the extracted movement information. In step S524, And outputs a second endoscopic image.

In step S525, the information on the output second endoscopic image and the second screen position are stored in the image storage unit 360. [ In step S526, the first endoscopic image stored in the image storage unit 360 together with the second endoscopic image is output to the first screen position. Here, the first endoscopic image may output at least one of saturation, brightness, color, and screen pattern with the second endoscopic image.

11 is a block diagram of a surgical robot according to a third embodiment of the present invention. 11, an image input unit 310, a screen display unit 320, a dark operation unit 330, an operation signal generation unit 340, a screen display control unit 350, a control unit 370, an image matching unit 450, A slave robot 2 including a robot arm 3 and an endoscope 8 is shown. The differences from the above will be mainly described.

In the present embodiment, an endoscopic image actually photographed using an endoscope during surgery and a modeling image stored in the image storage unit 360 are generated for the surgical tool in advance, There is a feature that the user can output to the observable screen display unit 320.

The image matching unit 450 generates an output image by matching the endoscopic image received through the image input unit 310 with the modeling image of the surgical tool stored in the image storage unit 360, . The endoscopic image is an image obtained by photographing the inside of the patient's body using an endoscope and capturing only a limited area, and thus includes an image of a part of the surgical tool.

The modeling image is the image generated by implementing the shape of the whole surgical tool as 2D or 3D image. The modeling image may be an image relating to a surgical tool photographed at a specific time point before the start of surgery, for example, in an initial setting state. Since the modeling image is an image generated by the computer simulation technique, the image matching unit 450 may output the matching result of the surgical tool and the modeling image shown in the actual endoscopic image. The technique of acquiring an image by modeling an actual object is somewhat distant from the gist of the present invention, so a detailed description thereof will be omitted. The specific functions, various detailed configurations, and the like of the image matching unit 450 will be described in detail below with reference to related drawings.

The control unit 370 controls the operation of each component so that the above-described functions can be performed. The control unit 370 may perform a function of converting an image input through the image input unit 310 into an image image to be displayed through the screen display unit 320. [ The control unit 360 controls the image matching unit 450 so that the modeling image is output through the screen display unit 320 when the operation information according to the operation of the dark operation unit 330 is input.

The actual surgical tool included in the endoscopic image is a surgical tool included in the image input by the endoscope 8 and transmitted to the master robot 1, and is a surgical tool that directly applies a surgical operation to the patient's body. In contrast, the modeling and surgical tools included in the modeling image are mathematically modeled in advance for the entire surgical tool and stored in the image storage unit 360 as a 2D or 3D image. Modeling of Surgical Tool and Modeling Image of Endoscopic Image The surgical tool is controlled by operating information of the master robot 1 (that is, information about movement, rotation, etc. of the surgical tool) . The actual operation tool and the modeling operation tool can determine the position and the operation shape by the operation information. 14, the endoscopic image 620 is matched with the modeling image 610 and output to the coordinates (X, Y) of the screen display unit 320.

In addition, the modeling image can include reconstructed images by modeling not only the surgical tools but also the patient's organs. That is, the modeling image can be classified into CT (Computer Tomography), MR (Magnetic Resonance), PET (Positron Emission Tomography), SPECT (Single Photon Emission Computed Tomography) ), US (Ultrasonography, Ultrasound), and the reconstructed 2D or 3D image of the long-term surface of the patient. In this case, matching the actual endoscopic image with the computer modeling image It is possible to provide the operator with the entire image including the surgical site.

12 is a block diagram of an image processing apparatus for a haptic surgical method according to a third embodiment of the present invention. 12, the image matching unit 450 may include a feature value calculation unit 451, a modeling image implementation unit 453, and a superimposed image processing unit 455. FIG.

The characteristic value calculating unit 451 calculates the characteristic value of the slave robot 2 based on the image input by the laparoscope 8 of the slave robot 2 and / or the coordinate information on the position of the actual surgical tool coupled to the robot arm 3, . The position of the actual surgical tool can be recognized by referring to the position value of the robot arm 3 of the slave robot 2 and information on the position can be provided from the slave robot 2 to the master robot 1 .

The characteristic value calculation unit 451 calculates the characteristic value of the laparoscope 8 based on the image of the laparoscope 8 using the image of the laparoscope 8, for example, the field of view of the laparoscope 8, the magnification, the viewpoint , The type of the actual surgical tool, the direction, the depth, and the degree of deflection. In the case of calculating the characteristic value using the image by the laparoscope 8, an image recognition technique for recognizing the contour extraction, shape recognition, tilted angle, etc. of the subject included in the image may be used. In addition, the type of the actual surgical tool and the like may be input in advance in the process of coupling the surgical tool to the robot arm 3 or the like.

The modeling image implementing unit 453 implements a modeling image corresponding to the characteristic value calculated by the characteristic value calculating unit 451. [ Data related to the modeling image can be extracted from the image storage unit 360. [ That is, the modeling image implementing unit 453 may be configured to classify the characteristic values of the laparoscope 8 (FOV, field of view, enlargement ratio, perspective, viewing depth, and the type, direction, depth, Modeling image data for the corresponding surgical tool is extracted and the modeling image is implemented so as to be matched with the surgical tool of the endoscopic image.

The modeling image implementation unit 453 may extract the image corresponding to the characteristic value calculated by the characteristic value calculation unit 451 in various ways. For example, the modeling image implementation unit 453 can directly extract the modeling image corresponding to the characteristic value of the laparoscope 8. That is, the modeling image implementing unit 453 can refer to the data of the angle of view and the enlargement ratio of the laparoscope 8, and extract the 2D or 3D modeling operation tool image corresponding to the data, and match the obtained image with the endoscopic image. Here, the characteristic values such as the angle of view and the enlargement ratio can be calculated by comparing and analyzing the images of the laparoscope 8, which are calculated through comparison with the reference image according to the initial setting value or sequentially generated.

According to another embodiment, the modeling image implementing unit 453 can extract the modeling image using the operation information for determining the position and the operation shape for the laparoscope 8 and the robot arm 3. That is, as described above, since the surgical tool of the endoscopic image can be controlled by operation information recognized by the master robot 1 as the operator manipulates the operation unit 330, the modeling operation corresponding to the characteristic value of the endoscopic image The position and manipulation shape of the tool can be determined by the manipulation information.

The manipulation information may be stored in a separate database according to a temporal order, and the modeling image implementing unit 453 can recognize a characteristic value of the actual surgical tool by referring to the database, and correspondingly, Can be extracted. That is, the position of the surgical tool output to the modeling image can be set using the cumulative data of the position change signal of the surgical tool. For example, if the manipulation information for the surgical instrument, which is one of the surgical tools, includes information such as 90 degrees rotation in the clockwise direction and 1 cm movement in the extension direction, the modeling image implementation unit 453 calculates The image of the surgical instrument included in the modeling image can be converted and extracted.

Here, the surgical instrument is mounted on a distal end portion of a surgical robot arm provided with an actuator, and a driving wheel (not shown) provided to a driving unit (not shown) is actuated by receiving a driving force from an actuator, The operator who is inserted into the patient's body performs a predetermined operation to perform the operation. The driving wheel is formed in a disc shape and can be transmitted to the actuator in a clutched manner. Further, the number of driving wheels may be determined in accordance with the number of control objects, and the description of such driving wheels is obvious to those skilled in the art relating to surgical instruments.

The superimposed image processor 455 outputs a partial image of the modeling image in order to prevent the endoscopic image and the modeling image from overlapping with each other. That is, when the endoscopic image includes a part of the shape of the surgical tool and the modeling image implementing unit 453 outputs the corresponding modeling and operation tool, the superimposed image processing unit 455 displays the actual operation tool image of the endoscopic image, And deleting the overlapped portion from the modeling operation tool image so that the two images can be matched with each other. The overlapping image processing unit 455 can process the overlapping region by removing the overlapping region between the modeling and surgical tool image and the actual surgical tool image from the modeling and operation tool image.

For example, if the overall length of the surgical tool is 20 cm and the characteristics (FOV, Field of View, magnification, viewpoint, depth of view, and type, direction, depth, When the length of the actual surgical instrument image of the endoscopic image is 3 cm, the superimposed image processing unit 455 outputs the modeling operation tool image that has not been output to the endoscopic image, using the characteristic value, in the modeling image.

FIG. 13 is a flowchart of a method of image processing for a sensible type surgical operation according to a third embodiment of the present invention. The differences from the above will be mainly described.

In step S131, a modeling image is generated and stored in advance regarding the object to be operated and / or the surgical tool. The modeling image may be modeled by computer simulation, and the present embodiment may generate a modeling image using a separate modeling image generating apparatus.

In step S132, the characteristic value computing section 351 computes a characteristic value of the endoscopic image. As described above, the characteristic value calculator 351 calculates the characteristic value of the slave robot 2 based on the image input by the laparoscope 8 and / or coordinate information on the position of the actual surgical tool coupled to the robot arm 3 And the characteristic value is calculated based on the field of view (FOV) of the laparoscope 8, the magnification, the viewpoint (for example, the viewing direction), the viewing depth, , Degree of bending, and the like.

In step S133, the image matching unit 450 extracts a modeling image corresponding to the endoscopic image, processes the overlapping area, and aligns the two images to output the same to the screen display unit 320. [ It is needless to say that the output timing of the endoscopic image and the modeling image may be variously set such that the endoscopic image and the modeling image are outputted first at the same time, or after the endoscopic image is output, and then the modeling image is output together.

15 is a conceptual diagram illustrating a master interface of a surgical robot according to a fourth embodiment of the present invention. 15, the master interface 4 may include a monitor unit 6, a handle 10, a monitor driving means 12, and a moving groove 13. The differences from the above will be mainly described.

The present embodiment rotates and moves the monitor unit 6 of the master interface 4 in accordance with the viewpoint of the endoscope 8 varying as described above so that the user can feel the reality of the operation more vividly There are features.

The monitor driving means 12 is configured such that one end is coupled to the monitor portion 6 and the other end is coupled to the main body portion of the master interface 4 to apply a driving force to the monitor portion 6, . Here, the rotation may include rotation about various axes (X, Y, Z), i.e., rotation by pitch, roll, yaw axis. Referring to Fig. 15, rotation (A) by the yaw axis is shown.

The monitor driving unit 12 moves the monitor unit 6 along the moving groove 13 formed in the main body of the master interface 4 located at the lower end of the monitor unit 6 to move the monitor unit 6 toward the endoscope 8 ). ≪ / RTI > The moving groove 13 may be formed in a concave direction toward the user so that the front face of the monitor unit 6 always faces the user when the monitor unit 6 moves along the moving groove 13. [

16 is a block diagram of a surgical robot according to a fourth embodiment of the present invention. 16, an image input unit 310, a screen display unit 320, a dark operation unit 330, an operation signal generation unit 340, a control unit 370, a screen drive control unit 380, and a screen drive unit 390 A master robot 1 and a slave robot 2 including a robot arm 3 and an endoscope 8 are shown. The differences from the above will be mainly described.

The screen driving unit 390 may include a motor, a monitor unit 6 holding means, and the like as means for rotating and moving the screen display unit 320, such as the monitor unit 6 described above. The screen drive control unit 380 may control the screen drive unit 390 to rotate and move the screen display unit 320 in accordance with the viewpoint of the endoscope 8. [ The screen driving unit 390 may include the monitor driving unit 12 described above and may move the monitor unit 6 according to the moving groove 13. [

17, the screen drive control unit 380 includes an endoscope viewpoint tracking unit 381 that tracks the viewpoint information of the endoscope 8 in accordance with the movement and rotation of the endoscope 8, A motion information generation unit 385 that generates motion information (screen drive information) of the screen display unit 320 using the movement information, a motion information generation unit 385 that generates movement information of the endoscope image using motion information, . ≪ / RTI > The screen driving unit 390 can drive the screen display unit 320 as described above using the motion information of the screen display unit 320 generated by the driving information generating unit 385. [

Further, according to another embodiment, the screen driving unit 390 may be driven by a user's command. For example, the screen driving control unit 380 can be replaced with a user interface, for example, a switch (e.g., a foot switch) that can be operated by a user. In this case, May be controlled.

The motion of the screen driver 390 can also be controlled by a touch screen. For example, the screen display unit 320 is implemented as a touch screen. When the user touches the screen display unit 320 using a finger or the like and drags the screen display unit 320 in a predetermined direction, the screen display unit 320 rotates and It can also be moved. In addition, the motion of the screen display unit 320 may be controlled by using a rotation / movement signal generated according to the direction of movement of the face contact unit, a rotation / movement signal generated according to a voice command, or the like.

18 is a flowchart of a method for image processing for a sensible-type surgical operation according to the fourth embodiment of the present invention. Each of the steps to be performed below may be executed by the screen drive control unit 380 as a subject.

In step S181, the perspective information of the endoscope 8, which is information on the viewpoint of the endoscope 8 in accordance with the movement and rotation of the endoscope 8, is tracked.

In step S182, the movement information of the endoscopic image corresponding to the positional change amount of the object of the endoscopic image is extracted using the viewpoint information of the endoscope 8. [

In step S183, the above-described screen driving information is generated using the perspective information of the endoscope 8 and / or the extracted movement information. That is, when the view information of the endoscope 8 and the movement information of the endoscopic image are specified as described above, information for moving and rotating the screen display unit 320 is generated using the movement information.

In step S184, the screen display unit 320 is moved and rotated in accordance with the screen drive information.

19 is a conceptual diagram illustrating a master interface of a surgical robot according to a fifth embodiment of the present invention. 19, a dome screen 191, a projector 192, a workbench 193, a first endoscopic image 621, and a second endoscopic image 622 are shown.

As described above, the present embodiment implements the function of outputting an endoscopic image to a specific area of the screen display section 320 using the dome screen 191 and the projector 192, There is a feature that can be confirmed easily.

The projector 520 projects the endoscopic image on the dome screen 510. Here, the endoscopic image may be a spherical image having a spherical shape at the front end of the projected image. Here, the spherical shape does not mean only a shape having a strictly spherical shape mathematically, and may include various shapes such as an elliptical shape, a curved shape, and a partially spherical shape.

The dome screen 510 has an open front end and a hemispherical inner dome surface that reflects the image projected from the projector 520. The size of the dome screen 510 may be as large as the user can see, for example, the diameter of the dome screen 510 may be about 1 m to 2 m. The internal dome surface of the dome screen 510 may be surface-treated or semi-spherical. Also, the dome screen 510 may be formed axially symmetrically about its central axis, and the user's line of sight may be located on the central axis of the dome screen 510.

The projector 520 is positioned between the dome screen 510 and the user performing surgery, so that the image projected by the user can be blocked. In addition, the projector 520 may be attached to the underside of the work table 530 in order to secure the space of the work table without blocking the projected image at the time of the user's work. The inner dome may be formed of a highly reflective material or may be coated with such a material.

In the case where the dome screen 191 and the projector 192 are used, the first endoscope image 621 and the second endoscope image 621 are displayed in a specific area of the dome screen 191 in accordance with various aspects of the endoscope 8, 622).

20 is a block diagram of an image processing apparatus for a sensible-type surgery according to a sixth embodiment of the present invention. 20, the screen display control unit 350 includes an endoscope viewpoint tracking unit 351, an image movement information extracting unit 353, an image position setting unit 355, a stored image display unit 357, a continuous image generating unit 352 and a peripheral image generating unit 354. The differences from the above will be mainly described.

The present embodiment is characterized in that a plurality of images are acquired by rotating one end of the surgical endoscope so that a wide field of view of the user can be ensured. That is, in this embodiment, one end of the endoscope for surgery is rotated to form a predetermined locus, thereby acquiring not only a surgical site but also a peripheral image, thereby allowing a user to view a wider area.

The continuous image generation unit 352 extracts the overlap region between the first endoscopic image and the second endoscopic image acquired from the surgical endoscope to generate a continuous image. The first endoscopic image and the second endoscopic image may be images provided at different points in time from the rotating endoscope for surgery. Referring to FIG. 22A, the surgical endoscope 221 can be rotated by tilting about the rotational axis A to obtain a plurality of endoscopic images.

For example, the surgical endoscope 221, which is extended to a predetermined length, has a light incident portion (lens portion), which is a single end, forms a rotational locus And the other end rotates along the conical shape or the pyramid shape as a whole by being positioned on the rotary shaft so that the one rotary trajectory can be various shapes such as circle, ellipse, triangle, quadrangle, other polygon, closed curve, closed figure . Here, the closed figure may be understood as a concept including a closed curve.

In addition, the speed and time at which the surgical endoscope 221 rotates can be determined as needed. For example, one end of the surgical endoscope 221 may be rotated periodically or in accordance with an arbitrary method operated by a user. Here, the term " periodic " may include a case where the surgical endoscope 221 performs circular motion at the same speed. Also, the term " periodic " may include a case in which the rotating state and the non-rotating state are periodically repeated.

According to the embodiment shown in FIG. 22B, the surgical endoscope 221 is embodied in a curved shape so that when one end of the surgical endoscope 221 rotates about the rotation axis A, a predetermined locus is formed And rotate. In this case, the endoscope for surgery 221 includes a first shaft 222 extending in a state of overlapping with the rotation axis A, a first light- And a shaft connecting portion 224 extending in parallel with the direction of the rotational axis A and connecting one end of the first shaft 222 to the other end of the second shaft 223 have.

In addition, the rotation locus of the light incidence portion may be various shapes such as circle, ellipse, triangle, quadrangle, other polygon, closed curve, closed figure, and the like.

Here, the rotation-related attributes such as the rotation direction, the degree of rotation, the shape of the rotation locus, the size of the rotation locus, the rotation speed, and the like of the operation endoscope 221 may be previously programmed and stored in a storage unit (not shown).

The screen display control unit 350 can extract an overlapping area of a plurality of images by referring to the previously stored rotation related property and generate such a region as a continuous image. For example, when the angle of view of the endoscope for surgery 221 is 70 degrees, the circular locus is circular, and the radius of the locus of rotation is 2 centimeters, overlapping portions of images to be photographed are extracted, .

The peripheral image generating unit 354 extracts non-overlapping regions of the first endoscopic image and the second endoscopic image to generate a surrounding image. The non-overlapping region may be an area in which the first endoscopic image and the second endoscopic image do not overlap with each other. Also, this area can be a preset area. For example, an image of an area other than the continuous image 232 set in advance in FIG. 23A can be set as the surrounding image 231, which is a non-overlapping area.

Referring to FIG. 23A, an image 231 is shown which is superimposed and continuously shot so that it is not overlapped with a continuous image 232 that is continuously viewed and processed as a surrounding image. Each circle represents different endoscopic images and may be referred to as a first endoscopic image or a second endoscopic image to distinguish them from each other. The continuous image 232 is an image continuously and continuously displayed on the screen, and the peripheral image 231 is an image that is displayed only when the image is continuously captured. In order to display them separately, the continuous image 232 can be expressed brightly and clearly, and the peripheral image 231 can be expressed differently. That is, the continuous image 232 and the surrounding image 231 may be expressed differently in brightness, saturation, and color.

Referring to FIG. 23B, the continuous image 232 may be an image of a predetermined area. 23A, when the continuous image 232 is set to only the image of the area in which all the surrounding images 231 are overlapped, the size of the continuous image 232 that is mainly visible can be reduced. Therefore, ), For example, an image of an area in which about two to three peripheral images 231 overlap, may be set as the continuous image 232. In this case, the continuous image 232 includes information about a continuous image relative to the surrounding image 231, and the size of the area to be photographed may be larger than the size of the area in which all the surrounding images 231 overlap There are advantages.

FIG. 21 is a flowchart of a method of image processing for a sensible-type surgery according to a sixth embodiment of the present invention. Each step to be performed below can be executed mainly by the screen display control unit 350 as a main body.

In step S211, the image input unit 310 receives the first endoscopic image and the second endoscopic image, which are provided at different points in time from the rotating surgical endoscope 221. Here, the screen display control unit 350 displays the viewpoint information of the operation endoscope 221, which is information on the viewpoint of the operation endoscope 221, in accordance with the movement and rotation of the endoscope 221, ≪ / RTI >

In step S212, the movement information of the endoscopic image corresponding to the position change amount of the object to be imaged of the endoscopic image is extracted using the perspective information of the operation endoscope 221. [

In step S213, the screen position at which the first endoscopic image and the second endoscopic image are to be output is set using the perspective information of the surgical endoscope 221 and / or the extracted movement information. That is, when the viewpoint information of the operation endoscope 221 and the movement information of the endoscopic image are specified as described above, the screen display unit 320 displays the first endoscopic image and the second endoscopic image on the screen Set the position.

In step S214, the first endoscopic image and the second endoscopic image are output to different areas, which are set positions on the screen display unit 320. [

24 is a view showing a rotating operation of the auxiliary endoscope according to the seventh embodiment of the present invention. 24, a surgical endoscope 241, an auxiliary endoscope 242, and a coupling portion 243 are shown.

The present embodiment can further provide the auxiliary endoscope 242 that rotates around the surgical endoscope 241 to acquire a plurality of endoscope images to thereby generate the continuous image and the surrounding image as described above. That is, the auxiliary endoscope 242 acquires an endoscopic image while rotating around the surgical endoscope 241, thereby obtaining a continuous image and a surrounding image from the superimposed image and the non-superimposed image.

The auxiliary endoscope 242 is rotatably coupled to one side of the surgical endoscope 241, for example, a side surface. A general endoscope structure in which an image is acquired by receiving light through the lens can also be applied to the auxiliary endoscope 242 as well. Here, the image acquired by the surgical endoscope 241 may be referred to as a first endoscopic image, and the image acquired by the auxiliary endoscope 242 may be referred to as a second endoscopic image.

The auxiliary endoscope 242 may be detachably coupled to the surgical endoscope 241 around the central axis A or may be integrally coupled to the surgical endoscope 241. [ In the former case, the auxiliary endoscope 242 is coupled to the surgical endoscope 241 outside the patient's body, or inserted into the patient's body separately from the surgical endoscope 241, and then coupled to the surgical endoscope 241, There is an advantage that a first endoscopic image and a second endoscopic image can be obtained.

According to another embodiment, the first endoscopic image of the surgical endoscope 241 may be set as a continuous image, and the second endoscopic image of the auxiliary endoscope 242 may be set as a peripheral image. That is, this embodiment has an advantage that the spirituality processing time can be reduced by generating the continuous image and the surrounding image without extracting the overlapping area.

25 is a conceptual diagram illustrating a master interface of a surgical robot according to an eighth embodiment of the present invention. 25, the master interface 4 described above may include a monitor unit 6, a handle 10, and a spatial motion driver 25. The differences from the above will be mainly described.

The present embodiment is characterized in that the monitor unit 6 is coupled to a spatial motion drive unit 25 that can freely move the monitor unit 6 so that the monitor unit 6 can freely rotate and move in space. Here, the monitor unit 6 may be the screen display unit 320 described above.

The spatial movement drive unit 25 has one end coupled to the monitor unit 6 and the other end coupled to the main body of the master interface 4 to apply a driving force to the monitor unit 6, Lt; / RTI > In this case, the rotation is performed by rotating the center of the various axes X, Y, Z corresponding to the number of joints provided in the spatial motion drive unit 25, that is, pitch, roll, yaw, And may include rotation by an axis.

The spatial movement drive unit 25 may be embodied in the form of a robot arm and may be mounted on the monitor unit 4 of the master interface 4 in accordance with the viewpoint of the surgical endoscope operated by the handle 10 or variously changed as described above. The user can feel the reality of the operation more vividly by rotating and moving the camera 6.

Further, another embodiment of the present invention may further include a rotation operation unit (not shown) for rotating the surgical endoscope 221 and / or the auxiliary endoscope 242 described above. The rotation operation unit is a rotation operation unit that allows the user to rotate the endoscope in accordance with information about rotation of the endoscope such as rotation direction, rotational angular velocity, acceleration / deceleration type, rotational speed, rotation time point, rotation time end point, rotation time length, rotation direction, And may be an operation unit for determining information.

Here, the rotation direction is a direction in which the endoscope rotates, such as clockwise or counterclockwise, and the acceleration / deceleration type means a shape in which the endoscope rotates in various forms such as a straight line, an S curve, and an exponential function type. The rotational angular velocity and the rotational speed are the speed at which one end of the surgical endoscope 221 or the auxiliary endoscope 242 rotates. The rotational time point is time information for starting rotation, and the rotational end point is time information to be. The radius of rotation is determined by the distance between the rotation axis and one end of the surgical endoscope 221 when the endoscope 221 is rotated in a conical shape, the length of the shaft connection portion 224 when the endoscope is a curved endoscope, And the endoscope for surgery 241 may be spaced apart from each other.

The rotating operation portion may include a user-operable interface, for example, the interface may be implemented in various forms for operating a robot arm and / or other surgical equipment, such as a joystick form, a button form, a keypad, a trackball, . When the user sets and inputs rotation related information using the interface, the surgical endoscope 221 can be rotated in a conical shape or rotated around the first shaft 222, , And the auxiliary endoscope 242 can also rotate around the surgical endoscope 241 as a center axis.

Other standard specifications for the haptic surgical image processing device according to the embodiment of the present invention, an interface standard technology such as an embedded system, a common platform technology such as O / S, a communication protocol, an I / O interface, and an actuator, , Sensors, and other parts standardization techniques are obvious to those skilled in the art and will not be described here.

The sensory processing image processing method according to the present invention may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer readable medium. That is, the recording medium may be a computer-readable recording medium having recorded thereon a program for causing a computer to execute the steps described above.

The computer-readable medium may include program instructions, data files, data structures, etc., singly or in combination. The program instructions recorded on the medium may be those specially designed and configured for the present invention or may be available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic recording media such as magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magnetic recording media such as floppy disks, Optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like.

The medium may be a transmission medium such as an optical or metal line, a wave guide, or the like, including a carrier wave for transmitting a signal designating a program command, a data structure, or the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention.

It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined in the appended claims.

1: Master robot 2: Slave robot
3: Robot arm 4: Master interface
5: Instrument 6: Monitor section
8: ENDOSCOPY 9: ENDOSCOPE IMAGING
10: Handle 12: Monitor driving means
13: moving groove 25:
191: Dome screen 192: Projector
193: Work table 221: Surgical endoscope
222: first shaft 223: second shaft
224: shaft connecting portion 231: peripheral image
232: continuous image 241: surgical endoscope
242: auxiliary endoscope 243:
310: image input unit 320:
330: Cancer control unit 340: Operation signal generation unit
350: screen display control unit 351: endoscope viewpoint tracking unit
352: Continuous image generation unit 353: Image movement information extraction unit
354: peripheral image generating unit 355: image position setting unit
357: Storage image display unit 360: Image storage unit
380: Screen drive control unit 381: Endoscope viewpoint tracking unit
383: image movement information extracting unit 385: drive information generating unit
390: Screen driving unit 450:
451: characteristic value computing unit 453: modeling image implementing unit
455: Overlaid image processor 610: Modeling image
620: endoscopic image 621: first endoscopic image
622: second endoscopic image

Claims (14)

A method for outputting an endoscopic image by a surgical image processing apparatus,
Receiving an endoscopic image provided from a surgical endoscope;
Outputting the endoscopic image to a specific area of the screen display unit; And
And changing a specific area of the screen display unit that outputs the endoscopic image in accordance with the viewpoint of the surgical endoscope.
The method according to claim 1,
Wherein the surgical endoscope is at least one of a laparoscope, a thoracoscopic, an arthroscope, a nystagmus, a cystoscope, a rectum, a duodenal, a mediastinum, and a heart.
The method according to claim 1,
Wherein the surgical endoscope is a stereoscopic endoscope.
The method according to claim 1,
The step of changing a specific area of the screen display unit includes:
Tracking perspective information of the surgical endoscope corresponding to movement and rotation of the surgical endoscope;
Extracting movement information of the endoscopic image using the perspective information of the operation endoscope; And
And setting a specific region of the screen display unit to which the endoscopic image is output using the movement information.
The method according to claim 1,
The step of changing a specific area of the screen display unit includes:
And moving the center point of the endoscopic image in accordance with a coordinate change value of the viewpoint of the surgical endoscope.
A method for outputting an endoscopic image by a surgical image processing apparatus,
Receiving an endoscopic image provided from a surgical endoscope;
Outputting the endoscopic image to a specific area of the screen display unit;
Storing a modeling image related to a surgical tool for operating a surgical target taken by the surgical endoscope;
Generating an output image by matching the endoscopic image and the modeling image with each other; And
Changing a specific region of the screen display unit outputting the endoscopic image in accordance with the viewpoint of the surgical endoscope and outputting the registered endoscopic image and the modeling image to the screen display unit, Way.
The method according to claim 6,
Wherein the generating the output image comprises:
Wherein the output image is generated by matching the actual surgical tool image included in the endoscopic image and the modeling surgical tool image included in the modeling image with each other.
8. The method of claim 7,
Wherein the generating the output image comprises:
Computing a characteristic value using at least one of an endoscopic image and positional coordinate information of an actual surgical tool coupled to the at least one robot arm; And
And implementing a modeling image corresponding to the calculated characteristic value.
8. The method of claim 7,
Wherein the generating the output image comprises:
Further comprising the step of removing an overlapping area between the modeling operation tool image and the actual operation tool image from the modeling operation tool image.
8. The method of claim 7,
Wherein the position of the modeling and operation tool image output to the modeling image is set using operation information of the surgical tool.
A method for outputting an endoscopic image by a surgical image processing apparatus,
Receiving an endoscopic image provided from a surgical endoscope;
Outputting the endoscopic image to a screen display unit; And
And rotating and moving the screen display unit in accordance with the viewpoint of the surgical endoscope.
12. The method of claim 11,
The step of rotating and moving the screen display unit includes:
Tracking perspective information of the surgical endoscope corresponding to movement and rotation of the surgical endoscope;
Extracting movement information of the endoscopic image using the perspective information of the operation endoscope;
And generating motion information of the screen display unit using the movement information.
The method according to any one of claims 1, 6, and 11,
The screen display unit displays,
A dome-shaped screen;
And a projector for projecting the endoscope image onto the dome-shaped screen.
A program for a tactile sensation surgical image processing method according to any one of claims 1 to 12, wherein a program of instructions executable by the digital processing apparatus is tangibly embodied and readable by a digital processing apparatus Is recorded.

Images