KR20110114421A - Apparatus and method for processing surgical image based on motion - Google Patents

Abstract

Disclosed are a tangible surgical image processing apparatus and method. An image input unit for receiving an endoscope image provided from a surgical endoscope, a screen display unit for outputting an endoscope image in a specific area, and a screen display for changing a specific area of the screen display unit for outputting an endoscope image in accordance with the viewpoint of the surgical endoscope The immersive surgical image processing apparatus including a control unit has an effect of changing the output position of the endoscope image output to the monitor viewed by the user according to the movement of the surgical endoscope, so that the user can feel the actual surgical situation more realistically. There is.

Description

Apparatus and Method for processing surgical image based on motion}

The present invention relates to an image processing apparatus, and more particularly, to an image processing apparatus and method for tangible surgery.

Surgical robot refers to a robot having a function that can replace a surgical operation performed by a surgeon. Such a surgical robot has the advantage of being capable of accurate and precise operation and remote surgery compared to humans.

Surgical robots currently being developed worldwide include bone surgery robots, laparoscopic surgery robots, and stereotactic surgery robots. The laparoscopic surgical robot is a robot that performs minimally invasive surgery using a laparoscope and a small surgical tool.

Laparoscopic surgery is an advanced surgical technique that involves surgery after inserting a laparoscope, an endoscope that looks into the belly and drills a hole about 1 cm in the navel area.

Recent laparoscopic is equipped with a computer chip to obtain a clearer and enlarged image than the naked eye, and has been developed so that any operation can be performed using specially designed laparoscopic surgical instruments while viewing the screen through the monitor.

Moreover, laparoscopic surgery has the same range of surgery as laparotomy, but has fewer complications than laparotomy, and can start treatment much faster after the procedure, and it has the advantage of maintaining the stamina or immune function of the patient. have. As a result, laparoscopic surgery is increasingly recognized as a standard surgery for treating colorectal cancer in the US and Europe.

The surgical robot system generally consists of a master robot and a slave robot. When a user manipulates a manipulator (for example, a handle) provided in the master robot, a surgical tool coupled to the robot arm of the slave robot or held by the robot arm is operated to perform surgery. The master robot and the slave robot are combined through a communication network to perform network communication.

At the time of laparoscopic surgery, the surgical image taken by the laparoscopic is output to the user and the user sees this. This problem is that even when the laparoscope moves and rotates in the abdominal cavity to illuminate other parts of the body, the image seen by the user is output from the monitor of the same position and size, so that the relative distance and movement between the manipulator and the image as described above may be reduced. This can be caused by a difference in the relative distance and movement between the surgical tool and the organ.

In addition, since the surgical image taken by the laparoscopic includes only some shapes of the surgical tool, the user may be difficult to operate when they collide or overlap each other, or may not be able to perform the operation smoothly because of obscured vision.

The above-described background technology is technical information that the inventor holds for the derivation of the present invention or acquired in the process of deriving the present invention, and can not necessarily be a known technology disclosed to the general public prior to the filing of the present invention.

The present invention changes the output position of the endoscope image output to the monitor to the user in response to the endoscope changing according to the movement of the surgical endoscope, so that the user can feel the actual surgical situation more realistic It is to provide a surgical image processing apparatus and method.

In addition, the present invention, by extracting the previously stored endoscope image at the present time point and outputs to the screen display unit along with the current endoscope image, haptic surgical image that can inform the user about the change in the endoscope image It is to provide a processing apparatus and method.

In addition, the present invention by modifying the image, such as to match the image or adjust the size of each of the modeling image stored in the image storage and generated in advance for the endoscopic image and the surgical instrument actually taken using the endoscope during surgery It is to provide a tangible surgical image processing apparatus and method that can output to an observable monitor.

In addition, the present invention is to provide a tangible surgical image processing apparatus and method that can make the user feel more vivid to the operation by rotating and moving the monitor in accordance with the viewpoint of varying endoscopes.

The technical problems other than the present invention can be easily understood from the following description.

According to an aspect of the present invention, an image input unit for receiving an endoscope image provided from the surgical endoscope, a screen display unit for outputting the endoscope image in a specific region, and a screen display unit for outputting the endoscope image corresponding to the viewpoint of the surgical endoscope There is provided a tangible surgical image processing apparatus including a screen display control unit for changing a specific area of the.

Here, the surgical endoscope may be one or more of laparoscopic, thoracoscopic, arthroscopic, parenteral, cystoscopic, rectal, duodenum, mediastinal, cardiac, or may be a stereoscopic endoscope.

Here, the screen display control unit, the endoscope perspective tracking unit for tracking the perspective information of the surgical endoscope corresponding to the movement and rotation of the surgical endoscope, and the image to extract the movement information of the endoscope image using the perspective information of the surgical endoscope The apparatus may include a movement information extracting unit and an image position setting unit for setting a specific region of the screen display unit on which the endoscope image is output using the movement information.

In addition, the screen display controller may move the center point of the endoscope image corresponding to the coordinate change value of the viewpoint of the surgical endoscope.

According to another aspect of the invention, the image input unit for receiving the first endoscope image and the second endoscope image provided at different time points from the surgical endoscope, and outputting the first endoscope image and the second endoscope image to different areas A screen display unit, an image storage unit for storing the first endoscope image and the second endoscope image, and a screen display unit for outputting the first endoscope image and the second endoscope image to different areas according to different viewpoints of the surgical endoscope There is provided an immersive surgical image processing apparatus including a screen display control unit for controlling the same.

The image input unit may receive the first endoscope image before the second endoscope image, and the screen display unit may differently output one or more of the saturation, brightness, color, and screen pattern of the first endoscope image and the second endoscope image. Can be.

The screen display controller may further include a storage image display unit which extracts the first endoscope image stored in the image storage unit and outputs the first endoscope image stored in the image storage unit while the screen display unit outputs the second endoscope image input in real time.

According to another aspect of the invention, the image input unit for receiving an endoscope image provided from the surgical endoscope, a screen display unit for outputting the endoscope image to a specific area, and a surgical tool for operating a surgical target taken by the surgical endoscope Change a specific area of an image storage unit for storing a modeling image, an image matching unit for generating an output image by matching an endoscope image and a modeling image, and a screen display unit for outputting an endoscope image in accordance with a surgical endoscope In addition, there is provided a tangible surgical image processing apparatus including a screen display control unit for outputting the matched endoscope image and modeling image to the screen display unit.

Here, the image matching unit may generate an output image by matching the actual surgical tool image included in the endoscope image with the modeling surgical tool image included in the modeling image.

The image matching unit may further include a characteristic value calculator configured to calculate a characteristic value using at least one of an endoscope image and position coordinate information of an actual surgical tool coupled to at least one robot arm, and a characteristic value calculated by the characteristic value calculator. The apparatus may further include a modeling image implementation unit for implementing a modeling image.

Here, the image matching unit may further include an overlapping image processor which removes an overlapping region between the modeling surgical tool image and the actual surgical tool image from the modeling surgical tool image, and the position of the modeling surgical tool image output on the modeling image is It can be set using the operation information of the surgical instrument.

According to another aspect of the invention, the image input unit for receiving the endoscope image provided from the surgical endoscope, the screen display unit for outputting the endoscope image, the screen drive unit for rotating and moving the screen display, and the endoscope for surgery Correspondingly, there is provided a immersive surgical image processing apparatus including a screen driving control unit which controls the screen driving unit so that the screen driving unit rotates and moves the screen display unit.

Here, the screen driving control unit, the endoscope perspective tracking unit for tracking the perspective information of the surgical endoscope corresponding to the movement and rotation of the surgical endoscope, and the image to extract the movement information of the endoscope image using the perspective information of the surgical endoscope The apparatus may include a movement information extracting unit and a driving information generating unit generating screen driving information of the screen display unit using the movement information.

Here, the screen driving unit may be coupled to the screen display unit to move the screen display unit along a predetermined movement groove, or the screen driving unit may be coupled to the screen display unit to move and rotate the screen display unit in space. It may be a spatial movement driver in the form of a robot arm.

Here, the screen display unit may include a dome-shaped screen and a projector that projects an endoscope image on the dome-shaped screen.

According to another aspect of the invention, in the method for outputting the endoscope image by the surgical image processing apparatus, receiving an endoscope image provided from the surgical endoscope, outputting the endoscope image to a specific area of the screen display unit and There is provided a tangible surgical image processing method comprising the step of changing a specific area of the screen display unit for outputting the endoscope image corresponding to the viewpoint of the surgical endoscope.

Here, the surgical endoscope may be one or more of laparoscopic, thoracoscopic, arthroscopic, parenteral, cystoscopic, rectal, duodenum, mediastinal, cardiac, or may be a stereoscopic endoscope.

In addition, the step of changing a specific area of the screen display unit, the step of tracking the perspective information of the surgical endoscope corresponding to the movement and rotation of the surgical endoscope, extracting the movement information of the endoscope image using the perspective information of the surgical endoscope And setting a specific area of the screen display unit on which the endoscope image is output using the movement information.

Here, the changing of the specific area of the screen display unit may include moving the center point of the endoscope image corresponding to the coordinate change value of the viewpoint of the surgical endoscope.

According to another aspect of the present invention, in the method for outputting the endoscope image by the surgical image processing apparatus, receiving the first endoscope image and the second endoscope image provided at different time points from the surgical endoscope, the first Outputting the endoscope image and the second endoscope image to different areas of the screen display unit, storing the first endoscope image and the second endoscope image, and the first endoscope image and the second corresponding to different perspectives of the surgical endoscope; There is provided a tangible surgical image processing method comprising controlling the screen display unit to output an endoscope image to different areas.

Here, the receiving of the endoscope image may include receiving the first endoscope image before the second endoscope image, and the outputting step may include any one of the saturation, brightness, color, and screen pattern of the first endoscope image and the second endoscope image. You can print one or more differently.

The controlling of the screen display unit may further include extracting and outputting the first endoscope image stored in the image storage unit to the screen display unit while the screen display unit outputs the second endoscope image input in real time.

According to another aspect of the invention, in the method for outputting the endoscope image by the surgical image processing apparatus, receiving an endoscope image provided from the surgical endoscope, outputting the endoscope image to a specific area of the screen display unit, Storing a modeling image of a surgical tool for operating a surgical target photographed by the surgical endoscope; generating an output image by matching the endoscope image and the modeling image; and outputting the endoscope image corresponding to the perspective of the surgical endoscope There is provided a tangible surgical image processing method including changing a specific region of a screen display unit, and outputting a matched endoscope image and a modeling image to a screen display unit.

Here, in the generating of the output image, the output image may be generated by matching the actual surgical tool image included in the endoscope image with the modeling surgical tool image included in the modeling image.

The generating of the output image may include calculating a characteristic value using at least one of an endoscope image and position coordinate information of an actual surgical tool coupled to at least one robot arm, and generating a modeling image corresponding to the calculated characteristic value. It may further comprise the step of implementing.

The generating of the output image may further include removing an overlapping region between the modeling surgical tool image and the actual surgical tool image from the modeling surgical tool image.

In addition, the position of the modeling surgical tool image output to the modeling image may be set using the operation information of the surgical tool.

According to another aspect of the invention, in the method for outputting the endoscope image by the surgical image processing apparatus, receiving an endoscope image provided from the surgical endoscope, outputting the endoscope image on the screen display and the endoscope for surgery According to the aspect of the present invention, there is provided a haptic surgical image processing method comprising the step of rotating and moving the screen display.

Here, the rotating and moving the screen display unit, tracking the perspective information of the surgical endoscope corresponding to the movement and rotation of the surgical endoscope, extracting the movement information of the endoscope image using the perspective information of the surgical endoscope The method may include generating motion information of the screen display unit using the movement information.

Here, the screen display unit may include a dome-shaped screen and a projector that projects an endoscope image on the dome-shaped screen.

According to another aspect of the present invention, a program of instructions that can be executed by a digital processing apparatus is tangibly embodied in order to perform the above-described immersive surgical image processing method, and a program that can be read by a digital processing apparatus. Recorded recording medium is provided.

According to another aspect of the present invention, an image input unit for receiving a first endoscope image and a second endoscope image provided at different points of time from one endoscope rotating surgical end, and the first endoscope image and the second endoscope image For a haptic surgery including a screen display for outputting to another area and a screen display control unit for controlling the screen display to output the first endoscope image and the second endoscope image to different areas according to different viewpoints of the surgical endoscope An image processing apparatus is provided.

Here, the surgical endoscope can be rotated periodically, the present embodiment is a rotational direction, rotational angular velocity, acceleration and deceleration form, rotational speed, rotation time point, rotation time end point, rotation time length associated with the rotation of one end of the surgical endoscope And it may further include a rotation operation unit for setting the rotation-related information of any one or more of the radius of rotation.

Here, the endoscope for surgery can be rotated to form a closed figure, one end can rotate in the shape of a cone or a pyramid, the rotational trajectory of the endoscope may be any one of a circle, an ellipse, a triangle and a square.

The screen display control unit may further include a continuous image generation unit configured to extract an overlapping region of the first endoscope image and the second endoscope image to generate a continuous image, and extract and extract a non-overlapping region of the first endoscope image and the second endoscope image. It may include a peripheral image generating unit for generating an image.

According to another aspect of the present invention, the first image input unit for receiving a first endoscope image from the surgical endoscope, and coupled to one side of the surgical endoscope provided at different times from the auxiliary endoscope rotating around the surgical endoscope A second image input unit configured to receive a plurality of second endoscope images, a screen display unit for outputting a first endoscope image and a second endoscope image to different regions, and corresponding to different views of a surgical endoscope and an auxiliary endoscope; There is provided a tangible surgical image processing apparatus including a screen display control unit for controlling the screen display unit to output the first endoscope image and the second endoscope image to different areas.

Here, the auxiliary endoscope can be rotated periodically, this embodiment is a rotation direction, rotational angular velocity, acceleration and deceleration form, rotational speed, rotation time point, rotation time end point, rotation time length and rotation associated with the rotation of one end of the auxiliary endoscope The apparatus may further include a rotation manipulation unit configured to set rotation related information of at least one of the radiuses.

The screen display control unit may further include a continuous image generation unit configured to extract an overlapping region of the first endoscope image and the second endoscope image to generate a continuous image, and extract and extract a non-overlapping region of the first endoscope image and the second endoscope image. It may include a peripheral image generating unit for generating an image.

The screen display controller may include a continuous image generator that generates a continuous image from the first endoscope image, and a peripheral image generator that extracts a second endoscope image to generate a peripheral image of the continuous image.

In addition, the auxiliary endoscope may be detachably coupled to the surgical endoscope.

According to another aspect of the present invention, in the method for outputting the endoscope image by the surgical image processing apparatus, the first endoscope image and the second endoscope image provided at different points in time from the surgical endoscope rotating one end And outputting the first endoscope image and the second endoscope image to different areas of the screen display unit, and outputting the first endoscope image and the second endoscope image to different areas according to different viewpoints of the surgical endoscope. There is provided a tangible surgical image processing method comprising the step of controlling the display unit.

Here, the surgical endoscope can be rotated periodically, the present embodiment is a rotational direction, rotational angular velocity, acceleration and deceleration form, rotational speed, rotation time point, rotation time end point, rotation time length associated with the rotation of one end of the surgical endoscope And setting the rotation related information of any one or more of the radius of rotation.

The surgical endoscope can be rotated so that one end forms a lung diagram, and can be rotated into a cone or pyramid shape, and the rotational trajectory of the surgical endoscope can be any one of a circle, an ellipse, a triangle, and a rectangle.

The controlling of the screen display unit may include generating a continuous image by extracting an overlapping region of the first endoscope image and the second endoscope image, and generating a surrounding image by extracting a non-overlapping region of the first endoscope image and the second endoscope image. It may include the step.

According to another aspect of the invention, the step of receiving the first endoscope image from the surgical endoscope, a plurality of agents provided at different times from the auxiliary endoscope to rotate around the surgical endoscope coupled to one side of the surgical endoscope 2) receiving an endoscope image, outputting the first endoscope image and the second endoscope image in different areas, and correspondingly to the different viewpoints of the endoscope for the operation and the endoscope, the first endoscope image and the second endoscope image There is provided a immersive surgical image processing method comprising controlling the screen display unit to output to another area.

Here, the auxiliary endoscope can be rotated periodically, this embodiment is a rotation direction, rotational angular velocity, acceleration and deceleration form, rotational speed, rotation time point, rotation time end point, rotation time length and rotation associated with the rotation of one end of the auxiliary endoscope The method may further include setting rotation related information of at least one of the radiuses.

The controlling of the screen display unit may include generating a continuous image by extracting an overlapping region of the first endoscope image and the second endoscope image, and extracting a non-overlapping region of the first endoscope image and the second endoscope image to generate a surrounding image. It may include the step.

The controlling of the screen display unit may include generating a continuous image from the first endoscope image and generating a peripheral image of the continuous image by extracting the second endoscope image.

In addition, the present embodiment may further include a step of detachably attaching the auxiliary endoscope to the surgical endoscope.

Here, the screen display unit may include a dome-shaped screen and a projector that projects an endoscope image on the dome-shaped screen.

In addition, according to another aspect of the present embodiment, a program of instructions that can be executed by a digital processing apparatus for performing the above-described haptic surgical image processing method is tangibly implemented and can be read by the digital processing apparatus. A recording medium for recording the program is provided.

Other aspects, features, and advantages other than those described above will become apparent from the following drawings, claims, and detailed description of the invention.

The bodily-type surgical image processing apparatus and method according to the present invention changes the output position of the endoscope image output to the monitor according to the viewpoint of the endoscope that changes according to the movement of the surgical endoscope, thereby allowing the user to actually perform the surgery. It has the effect of making the situation more realistic.

In addition, the apparatus and method for immersive surgical image processing according to the present invention extracts a previously input and stored endoscope image at the present time point and outputs the information on the change of the endoscope image by outputting it to the screen display together with the current endoscope image. There is an effect that can inform the user.

In addition, the bodily-type surgical image processing apparatus and method according to the present invention match each or each of the modeling images that are actually generated by using an endoscope during surgery and modeling images previously generated for a surgical tool and stored in an image storage unit or the same. It is effective to modify the image such as adjusting the size and output it to the monitor that the user can observe.

In addition, the bodily-type surgical image processing apparatus and method according to the present invention has an effect that allows the user to feel more realistic about the surgery by rotating and moving the monitor in accordance with the viewpoint of various changes in the endoscope.

1 is a plan view showing the overall structure of a surgical robot according to an embodiment of the present invention.
2 is a conceptual diagram showing a master interface of a surgical robot according to a first embodiment of the present invention.
Figure 3 is a block diagram of a surgical robot according to a first embodiment of the present invention.
Figure 4 is a block diagram of the immersive surgical image processing apparatus according to the first embodiment of the present invention.
Figure 5 is a flow chart of a tangible surgical image processing method according to a first embodiment of the present invention.
6 is a block diagram of an output image according to the tangible surgical image processing method according to the first embodiment of the present invention.
Figure 7 is a block diagram of a surgical robot according to a second embodiment of the present invention.
8 is a block diagram of an apparatus for processing immersion-type surgery according to a second embodiment of the present invention.
9 is a flow chart of a tangible surgical image processing method according to a second embodiment of the present invention.
10 is a block diagram of an output image according to the tangible surgical image processing method according to a second embodiment of the present invention.
11 is a block diagram of a surgical robot according to a third embodiment of the present invention.
12 is a block diagram illustrating an apparatus for immersive surgical image processing according to a third embodiment of the present invention.
Figure 13 is a flow chart of a tangible surgical image processing method according to a third embodiment of the present invention.
14 is a block diagram of an output image according to the immersive surgical image processing method according to a third embodiment of the present invention.
15 is a conceptual diagram illustrating a master interface of a surgical robot according to a fourth embodiment of the present invention.
Figure 16 is a block diagram of a surgical robot according to a fourth embodiment of the present invention.
FIG. 17 is a block diagram illustrating an apparatus for immersive surgical image processing according to a fourth embodiment of the present invention. FIG.
18 is a flow chart of a immersive surgical image processing method according to a fourth embodiment of the present invention.
19 is a conceptual diagram illustrating a master interface of a surgical robot according to a fifth embodiment of the present invention.
20 is a block diagram illustrating an apparatus for immersive surgical image processing according to a sixth embodiment of the present invention.
21 is a flow chart of a immersive surgical image processing method according to a sixth embodiment of the present invention.
Figure 22a is a view showing a rotation operation of the surgical endoscope according to the sixth embodiment of the present invention.
Figure 22b is a view showing a rotation operation of the surgical endoscope according to the sixth embodiment of the present invention.
23A and 23B illustrate an endoscope image according to a sixth embodiment of the present invention.
24 illustrates a rotation operation of the auxiliary endoscope according to the seventh embodiment of the present invention.
25 is a conceptual diagram illustrating a master interface of a surgical robot according to an eighth embodiment of the present invention.

As the invention allows for various changes and numerous embodiments, particular embodiments will be illustrated in the drawings and described in detail in the written description. However, this is not intended to limit the present invention to specific embodiments, it should be understood to include all changes, equivalents, and substitutes included in the spirit and scope of the present invention.

Terms including ordinal numbers such as first and second may be used to describe various components, but the components are not limited by the terms. The terms are used only for the purpose of distinguishing one component from another. When a component is said to be "connected" or "connected" to another component, it may be directly connected to or connected to that other component, but it may be understood that another component may exist in between. Should be.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. As used herein, the terms "comprise" or "have" are intended to indicate that there is a feature, number, step, action, component, part, or combination thereof described on the specification, and one or more other features. It is to be understood that the present invention does not exclude the possibility of the presence or the addition of numbers, steps, operations, components, components, or a combination thereof.

In the following description of the present invention with reference to the accompanying drawings, the same components are denoted by the same reference numerals regardless of the reference numerals, and redundant explanations thereof will be omitted. In the following description of the present invention, if it is determined that the detailed description of the related known technology may unnecessarily obscure the subject matter of the present invention, the detailed description thereof will be omitted.

1 is a plan view showing the overall structure of a surgical robot according to an embodiment of the present invention, Figure 2 is a conceptual diagram showing a master interface of a surgical robot according to a first embodiment of the present invention.

According to the present embodiment, the output position of the endoscope image 9 output to the monitor viewed by the user corresponds to the viewpoint of the endoscope changing according to the movement of the surgical endoscope, so that the user can feel the actual surgical situation more realistically. There is a characteristic to make. That is, since the viewpoint of the endoscope may coincide with the viewpoint of the user performing the surgery, the present embodiment may match the view of the endoscope in the abdominal cavity with the position and output direction of the monitor outputting the endoscope image 9 at the external surgery site. In addition, the motion of the system located at the external surgery site reflects the motion of the endoscope moving inside the actual patient.

Surgical endoscopes according to the present embodiment may be a variety of tools used as an imaging tool during surgery, such as laparoscopic as well as thoracoscopic, arthroscopy, parenteral, bladder, rectal, duodenum, mediastinal, cardiac. In addition, the surgical endoscope according to the present embodiment may be a stereoscopic endoscope. That is, the surgical endoscope according to the present embodiment may be a stereoscopic endoscope for generating stereoscopic image information, and the stereoscopic image information generating method may be implemented by various techniques. For example, the surgical endoscope according to the present embodiment includes a plurality of cameras to acquire a plurality of images having stereoscopic information, or acquire a plurality of images using a single camera. An image can be obtained. In addition to this method, the surgical endoscope according to the present embodiment may of course generate a stereoscopic image by various other methods.

In addition, the bodily-type surgical image processing apparatus according to the present embodiment is not necessarily implemented to be limited to the surgical robot system as shown, and if the system outputs an endoscope image 9 during surgery and operates using a surgical tool. Applicable. Hereinafter, the case where the surgical image processing apparatus according to the present embodiment is applied to the surgical robot system will be described.

1 and 2, the surgical robot system includes a slave robot 2 performing surgery on a patient lying on an operating table and a master robot 1 remotely controlling the slave robot 2. . The master robot 1 and the slave robot 2 are not necessarily separated into separate devices that are physically independent, but may be integrated into one and integrally formed, in which case the master interface 4 may be, for example, of an integrated robot. May correspond to an interface portion.

The master interface 4 of the master robot 1 comprises a monitor 6 and a master manipulator, and the slave robot 2 comprises a robot arm 3 and an instrument 5. The instrument 5 is an endoscopic, such as a laparoscope, or a surgical instrument, such as a surgical instrument for directly manipulating the affected part.

The master interface 4 is provided with a master controller so that the operator can be gripped and manipulated by both hands. As illustrated in FIGS. 1 and 2, the master controller may be implemented with two handles 10. An operation signal according to the manipulation of the operator's handle 10 is transmitted to the slave robot 2 so that the robot arm 3 may be moved. Controlled. By operating the handle 10 of the operator, the position movement, rotation, and cutting of the robot arm 3 and / or the instrument 5 may be performed.

For example, the handle 10 may be composed of a main handle and a sub handle. The slave robot arm 3, the instrument 5, etc. can be operated with only one handle, or a plurality of surgical equipment can be operated at the same time by adding a sub handle. The main handle and the sub handle may have various mechanical configurations depending on the operation method thereof. For example, the robot arm 3 and / or other surgery of the slave robot 2, such as a joystick type, a keypad, a trackball, and a touch screen, may be used. Various input means for operating the equipment can be used.

The master controller is not limited to the shape of the handle 10 and may be applied without any limitation as long as it can control the operation of the robot arm 3 through a network.

The monitor 6 of the master interface 4 displays an endoscope image 9, a camera image, and a modeling image input by the instrument 5 as image images. In addition, the information displayed on the monitor unit 6 may vary according to the type of the selected image.

The monitor unit 6 may be composed of one or more monitors, and may display information necessary for surgery on each monitor separately. 1 and 2 illustrate a case in which the monitor unit 6 includes three monitors, the quantity of the monitors may be variously determined according to the type or type of information requiring display. In addition, when the monitor unit 6 includes a plurality of monitors, the screen may be extended in cooperation with each other. That is, the endoscope image 9 may freely move each monitor, such as a window output to one monitor, and the entire image may be output by outputting some images connected to each monitor.

The slave robot 2 and the master robot 1 are coupled to each other via wired or wireless, so that the master robot 1 transmits an operation signal to the slave robot 2, and the slave robot 2 is the master robot 1. The endoscope image 9 input through the instrument 5 may be transmitted to the. If two operation signals by the two handles 10 provided in the master interface 4 and / or operation signals for adjusting the position of the instrument 5 need to be transmitted at the same time and / or at a similar time point, Each operation signal may be independently transmitted to the slave robot 2. Herein, when each operation signal is 'independently' transmitted, it means that the operation signals do not interfere with each other and one operation signal does not affect the other signal. As described above, in order to transmit the plurality of operation signals independently of each other, in the generation step of each operation signal, header information for each operation signal is added and transmitted, or each operation signal is transmitted in the generation order thereof, or Various methods may be used such as prioritizing each operation signal in advance and transmitting the operation signal accordingly. In this case, the transmission path through which each operation signal is transmitted may be provided independently so that interference between each operation signal may be fundamentally prevented.

The robot arm 3 of the slave robot 2 can be implemented to be driven with multiple degrees of freedom. The robot arm 3 includes, for example, a surgical tool inserted into a surgical site of a patient, a rocking drive unit for rotating the surgical tool in the yaw direction according to a surgical position, and a pitch direction perpendicular to the rotational drive of the rocking drive unit. It comprises a pitch drive unit for rotating the surgical tool, a transfer drive for moving the surgical tool in the longitudinal direction, a rotation drive for rotating the surgical tool, the surgical tool drive is installed on the end of the surgical tool to cut or cut the surgical lesion Can be. However, the configuration of the robot arm 3 is not limited thereto, and it should be understood that this example does not limit the scope of the present invention. In addition, the actual control process, such as the operator rotates the robot arm 3 in the corresponding direction by operating the handle 10 is somewhat distanced from the subject matter of the present invention, so a detailed description thereof will be omitted.

One or more slave robots 2 may be used to operate the patient, and the instrument 5 for allowing the surgical site to be displayed as an image image through the monitor unit 6 may be implemented as an independent slave robot 2. In addition, the master robot 1 may also be implemented integrally with the slave robot 2.

3 is a block diagram of a surgical robot according to a first embodiment of the present invention. Referring to FIG. 3, a master robot 1 including an image input unit 310, a screen display unit 320, an arm operation unit 330, an operation signal generator 340, a screen display control unit 350, and a control unit 370. And a slave robot 2 comprising a robot arm 3, an endoscope 8.

The haptic surgical image processing apparatus according to the present embodiment may be implemented as a module including an image input unit 310, a screen display unit 320, and a screen display control unit 350. Of course, such a module may be an arm manipulation unit 330. ), The operation signal generator 340 and the controller 370 may be included.

The image input unit 310 receives an image input through the endoscope 8 of the slave robot 2 through wired or wireless transmission. The endoscope 8 may also be one type of surgical tool according to the present embodiment, and the number thereof may be one or more.

The screen display unit 320 outputs an image image corresponding to the image received through the image input unit 310 as visual information. The screen display 320 may output the endoscope image as it is or zoom in / zoom out, or may match the endoscope image and a modeling image to be described later, or output each as a separate image.

In addition, the screen display unit 320 outputs the endoscope image and the image reflecting the entire surgical situation, for example, a camera image generated by the camera photographing the outside of the surgical target simultaneously and / or matched with each other and outputted to identify the surgical situation. It may be easy.

In addition, the screen display unit 320 outputs a reduced image of the entire image (endoscopic image, modeling image, camera image, etc.) in a portion of the output image or a window generated on a separate screen, and the operator described above When selecting or rotating a specific point among the reduced images output using the master controller, the entire output image may be moved or rotated, so-called bird's eye view function of the CAD program. Functions such as zooming in / out, moving, and rotating the image output to the screen display unit 320 as described above may be controlled by the controller 370 according to the operation of the master controller.

The screen display unit 320 may be implemented in the form of a monitor unit 6. An image processing process for outputting a received image as an image image through the screen display unit 320 may include a control unit 370 and a screen display control unit. 350 or by a separate image processor (not shown). The screen display unit 320 according to the present exemplary embodiment may be a displayer implemented by various technologies, and may be, for example, an ultra high resolution monitor such as a multi-vision or UHDTV (7380x4320). In addition, the screen display 320 according to the present embodiment may be a 3D display. For example, the screen display 320 according to the present exemplary embodiment may allow the user to separately recognize the left eye and right eye images by using the principle of binocular disparity. Such a 3D image implementation method may be implemented in various ways such as glasses (for example, red blue glasses (anaglyph), polarized glasses (passive glass), shutter glasses (active glass), etc.), lenticular method, barrier method. have.

The screen display unit 320 outputs the input endoscope image to a specific region. Here, the specific area may be an area on the screen having a predetermined size and location. This particular area may be determined in correspondence with the change of view of the endoscope 8 as described above.

The screen display control unit 350 may set this specific area according to the viewpoint of the endoscope 8. That is, the screen display control unit 350 tracks the point of view corresponding to the motion of the endoscope 8, such as rotation and movement, and sets the specific area for outputting the endoscope image on the screen display unit 320 by reflecting the viewpoint.

The arm manipulation unit 330 is a means for allowing the operator to manipulate the position and function of the robot arm 3 of the slave robot 2. The arm manipulation unit 330 may be formed in the shape of the handle 10 as illustrated in FIG. 2, but the shape is not limited thereto and may be modified in various shapes for achieving the same purpose. Further, for example, some may be formed in the shape of a handle, others may be formed in other shapes such as a clutch button, and a finger cannula or insertion may be inserted to fix the operator's finger to facilitate manipulation of the surgical tool. More rings may be formed.

The operation signal generator 340 generates a corresponding operation signal when the operator manipulates the arm operation unit 330 for the movement of the robot arm 3 and / or the endoscope 8 or the operation for surgery. Transfer to the robot (2). The operation signal may be transmitted and received via a wired or wireless communication network.

The operation signal generator 340 generates an operation signal by using the operation information according to the operation of the operator's arm operation unit 340, and transmits the generated operation signal to the slave robot 2. To be manipulated accordingly. In addition, the position and operation shape of the actual surgical instrument operated by the operation signal can be confirmed by the operator by the image input by the endoscope (8).

FIG. 4 is a block diagram of an apparatus for treating haptic surgery according to a first embodiment of the present invention. Referring to FIG. 4, the screen display controller 350 may include an endoscope perspective tracker 351, an image movement information extractor 353, and an image position setter 355.

The endoscope perspective tracking unit 351 tracks the perspective information of the endoscope 8 in correspondence with the movement and rotation of the endoscope 8. Here, the view point information refers to a view point viewed by the endoscope 8, and the view point information may be extracted from signals for manipulating the endoscope 8 in the above-described surgical robot system. That is, the viewpoint information can be specified by signals for manipulating the movement and rotational movement of the endoscope 8. Since the endoscope 8 manipulation signal is generated in the surgical robot system and transmitted to the robot arm 3 for manipulating the endoscope 8, the signal can be used to track the direction of the endoscope 8.

The image movement information extractor 353 extracts movement information of the endoscope image by using the viewpoint information of the endoscope 8. That is, the viewpoint information of the endoscope 8 may include information on the position change amount of the target object of the acquired endoscope image, and the movement information of the endoscope image may be extracted from this information.

The image position setting unit 355 sets a specific area of the screen display unit 320 on which the endoscope image is output using the extracted movement information. For example, if the viewpoint information of the endoscope 8 has been changed by a predetermined vector A, the endoscope image of the patient's internal organs has its movement information corresponding to the corresponding vector, and the screen using the movement information A specific area of the display unit 320 is set. If the endoscope image is changed by a predetermined vector B, a specific area in which the endoscope image is actually output to the screen display unit 320 may be set using this information and the size, shape, and resolution of the screen display unit 320.

5 is a flowchart of a bodily-type surgical image processing method according to a first embodiment of the present invention. Each step to be performed below may be executed by the screen display control unit 350 as a subject, and each step may not necessarily be executed in time series in the described order.

In step S511, the viewpoint information of the endoscope 8, which is information about the viewpoint viewed by the endoscope 8, is tracked corresponding to the movement and rotation of the endoscope 8. The viewpoint information is specified by a signal for manipulating the movement and rotational movement of the endoscope 8, so that the direction of the endoscope 8 can be tracked.

In step S513, the movement information of the endoscope image corresponding to the amount of change in the position of the image capturing object of the endoscope image is extracted using the viewpoint information of the endoscope 8.

In operation S515, a specific area of the screen display unit 320 on which the endoscope image is output is set using the extracted movement information. That is, when the viewpoint information of the endoscope 8 and the movement information of the endoscope image are specified as described above, a specific area for outputting the endoscope image on the screen display unit 320 is set using the movement information.

In step S517, the endoscope image is output to the specific area set by the screen display unit 320.

6 is a block diagram of an output image according to the tangible surgical image processing method according to the first embodiment of the present invention. The screen display unit 320 may be a full screen, and the endoscope image 620 acquired by the endoscope 8 may include the endoscope image 620 at a specific position of the screen display unit 320, for example, coordinates (X, Y). ) Can be output at the centered position. Coordinates (X, Y) can be set corresponding to the amount of change in the viewpoint of the endoscope (8). For example, when the viewpoint information of the endoscope 8 and the movement amount of the endoscope image change by +1 to the left and -1 to the vertical, the center point of the endoscope image 620 is moved to the position of coordinates (X + 1, Y-1). I can move it.

7 is a block diagram of a surgical robot according to a second embodiment of the present invention. Referring to FIG. 7, the image input unit 310, the screen display unit 320, the arm operation unit 330, the operation signal generator 340, the screen display control unit 350, the image storage unit 360, and the control unit 370. A master robot 1 and a robot arm 3 comprising a slave robot 2 comprising an endoscope 8 are shown. The differences from the above will be explained mainly.

The present embodiment has a feature of extracting a previously input and stored endoscope image at the present time point and outputting the endoscope image together with the current endoscope image to inform the user of the change of the endoscope image. .

The image input unit 310 receives a first endoscope image and a second endoscope image provided at different time points from the surgical endoscope. Here, the ordinal numbers, such as the first and the second, may be identifiers for distinguishing different endoscope images, and the first endoscope image and the second endoscope image may be images captured by the endoscope 8 from different viewpoints and viewpoints. Can be. In addition, the image input unit 310 may receive the first endoscope image before the second endoscope image.

The image storage unit 360 stores the first endoscope image and the second endoscope image. The image storage unit 360 stores not only image information which is actual image content of the first endoscope image and the second endoscope image, but also information on a specific region to be output to the screen display unit 320.

The screen display unit 320 outputs the first endoscope image and the second endoscope image to different regions, and the screen display control unit 350 corresponds to the first and second endoscope images according to different viewpoints of the endoscope 8. The screen display 320 may be controlled to output the data to different areas.

Here, the screen display unit 320 may differently output one or more of the saturation, brightness, color, and screen pattern of the first endoscope image and the second endoscope image. For example, the screen display unit 320 may output a second endoscope image currently input as a color image, and output a first endoscope image, which is a past image, as a black and white image, so that the user may distinguish the images from each other. Referring to FIG. 10, the second endoscope image 622, which is the currently input image, is output as the color image at the coordinates X1 and Y1, and the first endoscope image 621, which is the image input in the past, is a screen pattern. An example in which hatched patterns are formed and output at coordinates X2 and Y2 is shown.

In addition, the first endoscope image, which is a previous image, may be output only for a continuous or preset time. In the latter case, the past image is output to the screen display unit 320 only for a predetermined time, so that the new endoscope image may be continuously updated on the screen display unit 320.

8 is a block diagram illustrating an apparatus for immersive surgical image processing according to a second embodiment of the present invention. Referring to FIG. 8, the screen display controller 350 may include an endoscope perspective tracker 351, an image movement information extractor 353, an image position setter 355, and a stored image display 357.

The endoscope perspective tracking unit 351 tracks the perspective information of the endoscope 8 in correspondence with the movement and rotation of the endoscope 8, and the image movement information extracting unit 353 uses the perspective information of the endoscope 8 to describe the details. As described above, the movement information of the endoscope image is extracted.

The image position setting unit 355 sets a specific area of the screen display unit 320 on which the endoscope image is output using the extracted movement information.

The storage image display unit 357 extracts the first endoscope image stored in the image storage unit 360 and outputs the image to the screen display unit 320 while the screen display unit 320 outputs the second endoscope image input in real time. Since the output area and the image information of the first endoscope image and the second endoscope image are different from each other, the storage image display unit 357 extracts the information from the image storage unit 360 to store the first endoscope image, which is a past image. Output to the screen display unit 320.

9 is a flowchart of an image processing method for immersion type surgery according to a second embodiment of the present invention. Each step to be described below may be executed by the screen display control unit 350 as a main subject, and may be classified into a step of outputting a first endoscope image and a step of outputting a second endoscope image. The first endoscope image may be output together with the second endoscope image.

In step S511, the viewpoint information of the endoscope 8, which is information about the viewpoint viewed by the endoscope 8, is tracked in correspondence with the first movement and rotation information of the endoscope 8.

In step S513, the movement information of the first endoscope image is extracted. In step S515, a specific area of the screen display unit 320 on which the endoscope image is output is set using the extracted movement information. In step S517, the specific position of the first endoscope image is extracted. The first endoscope image is output.

In operation S519, the information about the outputted first endoscope image and the first screen position are stored in the image storage unit 360.

In step S521, the viewpoint information of the endoscope 8, which is information about the viewpoint viewed by the endoscope 8, is tracked corresponding to the second movement and rotation information of the endoscope 8.

In step S522, the movement information of the second endoscope image is extracted. In step S523, a specific area of the screen display unit 320 on which the endoscope image is output is set using the extracted movement information. In step S524, the movement information of the second endoscope image is set. The second endoscope image is output.

In operation S525, the information about the outputted second endoscope image and the second screen position are stored in the image storage unit 360. In operation S526, the first endoscope image stored in the image storage unit 360 together with the second endoscope image is output to the first screen position. Here, the first endoscope image may be output by differently performing any one or more of saturation, brightness, color, and screen pattern from the second endoscope image.

11 is a block diagram of a surgical robot according to a third embodiment of the present invention. Referring to FIG. 11, the image input unit 310, the screen display unit 320, the arm operation unit 330, the operation signal generation unit 340, the screen display control unit 350, the control unit 370, and the image matching unit 450. A master robot 1 and a robot arm 3 comprising a slave robot 2 comprising an endoscope 8 are shown. The differences from the above will be explained mainly.

In this embodiment, the endoscope image actually photographed using an endoscope during surgery and a modeling image generated in advance for a surgical tool and stored in the image storage unit 360 are matched with each other, or the size thereof is corrected. There is a feature that can be output to the screen display unit 320 that the user can observe.

The image matching unit 450 generates an output image by matching the endoscope image received through the image input unit 310 with the modeling image of the surgical tool stored in the image storage unit 360 and generating the output image. Output to. The endoscope image is an image of the inside of the patient's body using the endoscope. Since the image is obtained by capturing only a limited area, the endoscope image includes an image of a part of the surgical instrument.

The modeling image is an image generated by realizing the shape of the entire surgical tool as a 2D or 3D image. The modeling image may be an image of a surgical tool photographed at a specific time point before the start of surgery, for example, an initial setting state. Since the modeling image is an image generated by a computer simulation technique of the surgical tool, the image matching unit 450 may output the registered surgical tool and the modeling image shown in the actual endoscope image. Since a technique of obtaining an image by modeling a real object has a little distance from the gist of the present invention, a detailed description thereof will be omitted. In addition, specific functions, various detailed configurations, and the like of the image matching unit 450 will be described in detail with reference to the accompanying drawings.

The controller 370 controls the operation of each component so that the above-described function can be performed. The controller 370 may perform a function of converting an image input through the image input unit 310 into an image image to be displayed through the screen display unit 320. In addition, the controller 360 controls the image matching unit 450 to output the modeling image through the screen display unit 320 in response to the manipulation information according to the manipulation of the arm manipulation unit 330.

The actual surgical tool included in the endoscope image is a surgical tool included in the image inputted by the endoscope 8 and transmitted to the master robot 1 and is a surgical tool that applies a surgical operation directly to the patient's body. In contrast, the modeling surgical tool included in the modeling image is mathematically modeled with respect to the entire surgical tool in advance and stored in the image storage unit 360 as a 2D or 3D image. Surgical tools and modeling images of the endoscope image Surgical tools are controlled by the operation information (that is, information about the movement, rotation, etc. of the surgical tool) recognized by the master robot 1 as the operator operates the cancer operation unit 330 Can be. In actual surgical tools and modeling surgical tools, their position and manipulation shape may be determined by the manipulation information. Referring to FIG. 14, the endoscope image 620 is matched with the modeling image 610 and output to the coordinates (X, Y) of the screen display unit 320.

In addition, the modeling image may include an image reconstructed by modeling not only the surgical instrument but also the organ of the patient. That is, the modeled images are CT (Computer Tomography), MR (Magnetic Resonance), PET (Positron Emission Tomography), SPECT (Single Photon Emission Computed Tomography), and single photon emission tomography (SPECT). ), Which may include 2D or 3D images of the organ surface of the patient, reconstructed with reference to images acquired from imaging equipment such as US (Ultrasonography), in which case the actual endoscope image and the computer modeling image are matched. It is more effective to provide the operator with a full image including the surgical site.

12 is a block diagram of an apparatus for processing immersion-type surgery according to a third embodiment of the present invention. Referring to FIG. 12, the image matcher 450 may include a feature value calculator 451, a modeled image implementer 453, and an overlapped image processor 455.

The characteristic value calculator 451 uses the characteristic value by using the image inputted by the laparoscope 8 of the slave robot 2 and / or coordinate information on the position of the actual surgical tool coupled to the robot arm 3. Calculate The actual position of the surgical tool can be recognized by referring to the position value of the robot arm 3 of the slave robot 2, the information on the position may be provided from the slave robot 2 to the master robot (1). .

The characteristic value calculator 451 may use, for example, an image of the laparoscope 8, for example, a field of view (FOV), an enlargement ratio, a viewpoint (for example, a viewing direction), a viewing depth, etc. of the laparoscope 8. In addition, it is possible to calculate characteristic values such as the type, direction, depth, and degree of bending of the actual surgical instrument. When the characteristic value is calculated using the image of the laparoscope 8, an image recognition technique for recognizing the outline of the subject included in the image, shape recognition, tilt angle, or the like may be used. In addition, the type of the actual surgical tool may be input in advance in the process of coupling the corresponding surgical tool to the robot arm (3).

The modeling image implementer 453 implements a modeling image corresponding to the feature value calculated by the feature value calculator 451. Data related to the modeled image may be extracted from the image storage unit 360. That is, the modeling image implementer 453 may determine the characteristic values of the laparoscope 8 (field of view (FOV), magnification, perspective, viewing depth, etc., type, direction, depth, degree of bending, etc. of the actual surgical instrument). The modeling image is implemented to extract the modeling image data of the corresponding surgical tool and the like to match the surgical tool of the endoscope image.

The modeling image implementer 453 may extract various images according to the feature values calculated by the feature value calculator 451. For example, the modeling image implementer 453 may extract a modeling image corresponding to the characteristic value of the laparoscope 8 directly. That is, the modeling image implementer 453 may extract the 2D or 3D modeling surgical tool image corresponding to the above-described data such as the angle of view and the magnification of the laparoscope 8 and match it with the endoscope image. Here, the characteristic values such as the angle of view and the magnification may be calculated by comparing and analyzing the images of the laparoscope 8, which are calculated or sequentially generated through comparison with the reference image according to the initial set value.

In addition, according to another exemplary embodiment, the modeling image implementer 453 may extract the modeling image by using manipulation information for determining the position and the manipulation shape of the laparoscope 8 and the robot arm 3. That is, as described above, since the surgical tool of the endoscope image may be controlled by the operation information recognized by the master robot 1 as the operator manipulates the cancer operation unit 330, the modeling surgery corresponding to the characteristic value of the endoscope image is performed. The position and manipulation shape of the tool can be determined by the manipulation information.

Such manipulation information may be stored in a separate database according to the temporal order, and the modeling image implementer 453 may recognize the characteristic values of the actual surgical tool by referring to the database, and correspondingly, the information about the modeling image. Can be extracted. That is, the position of the surgical tool output on the modeling image may be set using cumulative data of the position change signal of the surgical tool. For example, if the operation information for the surgical instrument, which is one of the surgical instruments, includes the information that it is rotated 90 degrees clockwise and 1 cm in the extending direction, the modeling image implementer 453 may correspond to the operation information. An image of a surgical instrument included in the modeling image may be converted and extracted.

Here, the surgical instrument is mounted to the front end of the surgical robot arm is provided with an actuator, the driving wheel (not shown) provided in the drive unit (not shown) by receiving the driving force from the actuator is operated, connected to the drive wheel and surgery The operator inserted into the patient's body performs the operation by predetermined operation. The driving wheel is formed in a disc shape, and may be clutched to the actuator to receive the driving force. In addition, the number of driving wheels may be determined corresponding to the number of objects to be controlled, and the description of such driving wheels will be apparent to those skilled in the art related to surgical instruments, and thus detailed description thereof will be omitted.

The superimposed image processor 455 outputs a partial image of the modeled image so that the actually captured endoscope image and the modeled image do not overlap. That is, when the endoscope image includes some shape of the surgical tool and the modeling image implementer 453 outputs the corresponding modeling surgical tool, the superimposed image processor 455 may perform the actual surgical tool image and the modeling surgical tool image of the endoscope image. By checking the overlapping region of, and deleting the overlapping portion from the modeling surgical tool image, the two images can be matched with each other. The overlapping image processor 455 may process the overlapping region by removing the overlapping region of the modeling surgical tool image and the actual surgical tool image from the modeling surgical tool image.

For example, the total length of an actual surgical instrument is 20 cm, and characteristics values (field of view (FOV), magnification, perspective, depth of view, etc., type, direction, depth, degree of bending, etc. of the actual surgical instrument) are considered. When the length of the actual surgical tool image of the endoscope image is 3cm, the superimposed image processor 455 outputs a modeling surgical tool image which is not output to the endoscope image by using the characteristic value.

FIG. 13 is a flowchart illustrating a immersive surgical image processing method according to a third exemplary embodiment of the present invention. FIG. The differences from the above will be explained mainly.

In step S131, a modeling image is generated and stored in advance with respect to the surgical target and / or the surgical tool. The modeled image may be modeled by computer simulation, and the embodiment may generate a modeled image by using a separate modeling image generating apparatus.

In operation S132, the characteristic value calculator 351 calculates a characteristic value of the endoscope image. As described above, the characteristic value calculator 351 may input the image inputted by the laparoscope 8 of the slave robot 2 and / or coordinate information on the position of the actual surgical tool coupled to the robot arm 3. The characteristic value is calculated using the field of view (FOV, field of view), magnification, perspective (for example, viewing direction), viewing depth, and the type, direction, and depth of the actual surgical tool. , Bend, and so on.

In operation S133, the image matching unit 450 extracts the modeling image corresponding to the endoscope image, processes the overlapping region, and matches the two images to be output to the screen display unit 320. Here, the output time point may be variously set such that the endoscope image and the modeling image are initially output at the same time point or the endoscope image is output and the modeling image is also output together.

15 is a conceptual diagram illustrating a master interface of a surgical robot according to a fourth embodiment of the present invention. Referring to FIG. 15, the master interface 4 may include a monitor unit 6, a handle 10, a monitor driving unit 12, and a moving groove 13. The differences from the above will be explained mainly.

According to the present embodiment, the monitor unit 6 of the master interface 4 is rotated and moved in accordance with the viewpoint of variously changing endoscopes 8 as described above, so that the user can feel more realistic about the surgery. There is a characteristic.

One end of the monitor driving means 12 is coupled to the monitor portion 6, and the other end thereof is coupled to the main body portion of the master interface 4 to rotate the monitor portion 6 by applying a driving force to the monitor portion 6. Move it. Here, the rotation may include rotation about various axes (X, Y, Z), that is, rotation by a pitch, roll, yaw axis. Referring to FIG. 15, rotation A by the yaw axis is shown.

In addition, the monitor driving means 12 moves (B direction) along the moving groove 13 formed in the main body of the master interface 4 located at the lower end of the monitor 6 to endoscope 8. ) Can be moved according to the point of view. The moving groove 13 may have a concave direction toward the user so that the front surface of the monitor 6 always faces the user when the monitor 6 moves along the moving groove 13.

16 is a block diagram of a surgical robot according to a fourth embodiment of the present invention. Referring to FIG. 16, the image input unit 310, the screen display unit 320, the arm operation unit 330, the operation signal generation unit 340, the control unit 370, the screen driving control unit 380, and the screen driving unit 390 may be used. A master robot 1 and robot arm 3 comprising, a slave robot 2 comprising an endoscope 8 is shown. The differences from the above will be explained mainly.

The screen driver 390 is a means for rotating and moving the screen display 320 such as the monitor 6 described above, and may include, for example, a motor, a monitor 6 support means, and the like. The screen driving controller 380 may control the screen driving unit 390 so that the screen driving unit 390 rotates and moves the screen display unit 320 according to the viewpoint of the endoscope 8. The screen driver 390 may include the above-described monitor driving means 12 and may move the monitor 6 according to the moving groove 13.

Referring to FIG. 17, the screen driving control unit 380 may include an endoscope viewpoint tracking unit 381 for tracking perspective information of the endoscope 8 and a viewpoint of the endoscope 8 according to movement and rotation of the endoscope 8. An image movement information extraction unit 383 for extracting movement information of the endoscope image using the information, and a driving information generation unit 385 for generating motion information (screen driving information) of the screen display unit 320 using the movement information. It may include. The screen driver 390 may drive the screen display 320 as described above using the motion information of the screen display 320 generated by the drive information generator 385.

According to another embodiment, the screen driver 390 may be driven by a user's command. For example, the screen driving controller 380 may be replaced with a user interface, for example, a switch (eg, a foot switch) that is operable by a user, and in this case, the screen driving unit 390 may be rotated and moved by a user's manipulation. This may be controlled.

The motion of the screen driver 390 can also be controlled by the touch screen. For example, the screen display unit 320 is implemented as a touch screen, and when the user touches the screen display unit 320 using a finger or the like and drags in a predetermined direction, the screen display unit 320 rotates accordingly. You can also move. In addition, the motion of the screen display 320 may be controlled by using the rotation / movement signal generated according to the user's eyes or the rotation / movement signal generated according to the moving direction of the face contact unit or the voice command.

18 is a flowchart illustrating a immersive surgical image processing method according to a fourth embodiment of the present invention. Each step to be performed below may be performed by the screen driving controller 380 as a main agent.

In step S181, the viewpoint information of the endoscope 8, which is information about the viewpoint viewed by the endoscope 8, is tracked corresponding to the movement and rotation of the endoscope 8.

In step S182, the movement information of the endoscope image corresponding to the amount of change in the position of the image capturing object of the endoscope image is extracted using the viewpoint information of the endoscope 8.

In step S183, the above-described screen driving information is generated using the viewpoint information of the endoscope 8 and / or the extracted movement information. That is, when the viewpoint information of the endoscope 8 and the movement information of the endoscope image are specified as described above, information for moving and rotating the screen display unit 320 is generated using the movement information.

In step S184, the screen display unit 320 is moved and rotated according to the screen driving information.

19 is a conceptual diagram illustrating a master interface of a surgical robot according to a fifth embodiment of the present invention. Referring to FIG. 19, a dome screen 191, a projector 192, a work table 193, a first endoscope image 621, and a second endoscope image 622 are illustrated.

The present embodiment implements a function of outputting an endoscope image to a specific region of the screen display unit 320 using the dome screen 191 and the projector 192 as described above, so that the user can quickly operate the surgery on a wide screen. There is a characteristic which can be confirmed conveniently.

The projector 520 projects the endoscope image onto the dome screen 510. The endoscope image may be a spherical image having a spherical shape in front of the projected image. Here, the spherical shape does not mean mathematically strictly a shape having a spherical shape, and may include various shapes such as an ellipse, a curved shape of a cross section, and some spherical shapes.

The dome screen 510 has an open front end and a hemispherical inner dome surface that reflects the image projected from the projector 520. The size of the dome screen 510 may be a size that is easy for a user to see, for example, about 1m to about 2m in diameter. The inner dome surface of the dome screen 510 may be face-to-block or hemispherical in shape. In addition, the dome screen 510 may be axially symmetric about its central axis, and the user's line of sight may be located at the central axis of the dome screen 510.

The projector 520 may be located between the user performing the surgery and the dome screen 510 so that the image projected by the user may not be blocked. In addition, the projector 520 may be attached to the bottom surface of the work bench 530 in order to secure a space of the work table without covering the projected image during the user's work. The inner dome surface may be formed of or coated with a material with high reflectivity.

When the dome screen 191 and the projector 192 are used, the first endoscope image 621 and the second endoscope image (i.e., the first endoscope image 621 and the second endoscope image) in a specific area of the dome screen 191 may correspond to various viewpoints of the endoscope 8 as described above. 622 may be projected.

20 is a block diagram illustrating an apparatus for immersive surgical image processing according to a sixth embodiment of the present invention. Referring to FIG. 20, the screen display controller 350 may include an endoscope perspective tracker 351, an image movement information extractor 353, an image position setter 355, a stored image display 357, and a continuous image generator ( 352 and the surrounding image generator 354. The differences from the above will be explained mainly.

This embodiment has a feature that can secure a wider field of view by obtaining a plurality of images by rotating one end of the surgical endoscope. In other words, the present embodiment is characterized by allowing one end of the surgical endoscope to rotate to form a predetermined trajectory to acquire not only the surgical site but also the surrounding image so that the user can see a wider area.

The continuous image generator 352 generates a continuous image by extracting an overlapping region of the first endoscope image and the second endoscope image obtained from the surgical endoscope. The first endoscope image and the second endoscope image may be images provided at different points of time from the rotating surgical endoscope. Referring to FIG. 22A, the surgical endoscope 221 may acquire a plurality of endoscope images by tilting and rotating about the rotation axis A. FIG.

One end of the surgical endoscope 221 may be rotated to form a variety of trajectories, for example, the surgical endoscope 221 extended to a predetermined length is one end of the light incident portion (lens portion) forms a rotation trajectory The other end is rotated along the cone shape or the whole cone shape by being located on the axis of rotation, so that one end of the rotation trajectory may be various shapes such as circles, ellipses, triangles, squares, other polygons, closed curves, closed figures, and the like. Can be. Here, the closed figure may be understood as a concept including a closed curve.

In addition, the speed and time at which the surgical endoscope 221 rotates may be determined as necessary. For example, one end of the surgical endoscope 221 may rotate periodically or correspond to any manner in which the user manipulates. Here, the periodic meaning may include a case in which the surgical endoscope 221 makes a circular motion at the same speed. In addition, the term "cyclic" may include a case in which a state in which rotation is performed and a state in which rotation is not repeated are periodically repeated.

In addition, according to the embodiment shown in Figure 22b, the surgical endoscope 221 is implemented in a curved shape, when one end of the surgical endoscope 221 when the rotation around the axis of rotation (A) forms a predetermined trajectory Can rotate. In this case, the surgical endoscope 221 is a first shaft 222 extending in a state overlapping with the rotation axis (A), the light incident portion is coupled to one end in the direction of the rotation axis (A) spaced apart from the first shaft 222 It may include a second shaft 223 extending in the direction, the shaft connecting portion 224 extending not parallel to the direction of the rotation axis (A) and connecting one end of the first shaft 222 and the other end of the second shaft 223. have.

In addition, the rotation trajectory of the light incident part may have various shapes such as a circle, an ellipse, a triangle, a rectangle, other polygons, a closed curve, a closed figure, and the like.

Here, rotation-related attributes such as the rotation direction, the degree of rotation, the shape of the rotation trajectory, the size of the rotation trajectory, the rotation speed, etc. of the surgical endoscope 221 may be pre-programmed and stored in a storage unit (not shown).

The screen display controller 350 may extract overlapping regions of the plurality of images by referring to the rotation-related attributes stored in advance, and generate the regions as continuous images. For example, when the angle of view of the surgical endoscope 221 is 70 degrees, the rotational trajectory is circular, and the radius of the rotational trajectory is 2 cm, an overlapping portion of the captured image is extracted, and the extracted overlapping image is a continuous image that is continuously visible. Can be.

The peripheral image generator 354 extracts a non-overlapping region of the first endoscope image and the second endoscope image to generate the peripheral image. The non-overlapped area may be an area in which the first endoscope image and the second endoscope image do not overlap each other. In addition, this area may be a predetermined area. For example, an image of a region other than the continuous image 232 preset in FIG. 23A may be set as a peripheral image 231 which is a non-overlapping region.

Referring to FIG. 23A, an image 231 that is overlapped and continuously photographed, and that is not overlapped with a continuous image 232 that is continuously visible and is processed as a surrounding image is illustrated. Each circle represents a different endoscope image and may be referred to as a first endoscope image or a second endoscope image to distinguish each other. The continuous image 232 is an image which is continuously seen on the screen, and the peripheral image 231 is an image which is visible only when the image is taken continuously. In order to display them separately, the continuous image 232 may be expressed brightly and clearly, and the peripheral image 231 may be expressed differently. That is, the brightness, saturation, and color of the continuous image 232 and the surrounding image 231 may be different from each other.

Referring to FIG. 23B, the continuous image 232 may be an image of a preset area. That is, when only the image of the region where all the surrounding images 231 overlap with each other is set as the continuous image 232 as shown in FIG. For example, an image of a region where two or three surrounding images 231 overlap may be set as the continuous image 232. In this case, the continuous image 232 includes information about a relatively continuous image compared to the surrounding image 231, and the size of the region to be photographed may be larger than the size of the region where all the peripheral images 231 overlap. There is an advantage.

21 is a flowchart illustrating a immersive surgical image processing method according to a sixth embodiment of the present invention. Each step to be performed below may be performed mainly by the screen display controller 350 as a subject.

In operation S211, the image input unit 310 receives a first endoscope image and a second endoscope image provided at different views from the rotating surgical endoscope 221. Here, the screen display control unit 350, as described above, the perspective information of the surgical endoscope 221, which is information on the viewpoint viewed by the surgical endoscope 221 corresponding to the movement and rotation of one end of the surgical endoscope 221, as described above. To track.

In step S212, the movement information of the endoscope image corresponding to the amount of change in the position of the target of the endoscope image is extracted using the viewpoint information of the surgical endoscope 221.

In operation S213, the screen position at which the first endoscope image and the second endoscope image are output is set using the perspective information and / or the extracted movement information of the surgical endoscope 221. That is, when the viewpoint information of the surgical endoscope 221 and the movement information of the endoscope image are specified as described above, a screen on which the first endoscope image and the second endoscope image are output to the screen display unit 320 using the movement information. Set the location.

In operation S214, the first endoscope image and the second endoscope image are output to different areas that are set positions of the screen display unit 320.

24 is a diagram illustrating a rotation operation of the auxiliary endoscope according to the seventh embodiment of the present invention. Referring to FIG. 24, a surgical endoscope 241, an auxiliary endoscope 242, and a coupling portion 243 are shown.

In this embodiment, the auxiliary endoscope 242 which rotates around the surgical endoscope 241 may be further provided to obtain a plurality of endoscope images, thereby generating the continuous images and the surrounding images as described above. That is, the auxiliary endoscope 242 may acquire an endoscope image while rotating around the surgical endoscope 241, thereby obtaining a continuous image and a peripheral image from the overlapping image and the non-overlapping image.

The auxiliary endoscope 242 is rotatably coupled to one side, for example, the side of the surgical endoscope 241. A general endoscope structure for receiving an image through a lens and obtaining an image may also be applied to the auxiliary endoscope 242. Here, the image acquired by the surgical endoscope 241 may be referred to as the first endoscope image and the image obtained by the auxiliary endoscope 242 may be referred to as a second endoscope image.

In addition, the auxiliary endoscope 242 may be coupled in a detachable form with the surgical endoscope 241 around the central axis (A) or may be integrally coupled with the surgical endoscope 241. In the former case, the auxiliary endoscope 242 is coupled to the surgical endoscope 241 outside the patient's body or inserted into the patient's body separately from the surgical endoscope 241 and then coupled to the surgical endoscope 241 therein. There is an advantage in that one first endoscope image and the second endoscope image can be obtained.

According to another embodiment, the first endoscope image of the surgical endoscope 241 may be set as a continuous image, and the second endoscope image of the auxiliary endoscope 242 may be set as a peripheral image. That is, the present embodiment has an advantage of reducing the spiritual processing time by generating the continuous image and the surrounding image without extracting the overlapped region.

25 is a conceptual diagram illustrating a master interface of a surgical robot according to an eighth embodiment of the present invention. Referring to FIG. 25, the above-described master interface 4 may include a monitor unit 6, a handle 10, and a spatial movement driver 25. The differences from the above will be explained mainly.

The present embodiment is characterized in that the monitor unit 6 is coupled to a spatial movement driver 25 which can freely move in space so that the monitor unit 6 can freely rotate and move in space. Here, the monitor 6 may be the screen display 320 described above.

The spatial movement driver 25 has one end coupled to the monitor 6 and the other end coupled to the main body of the master interface 4 to apply the driving force to the monitor 6 to space the monitor 6. Rotate and move on the phase. Here, the rotation is the rotation around the various axes (X, Y, Z) corresponding to the number of joints provided in the spatial movement driver 25, that is, pitch, roll, yaw Rotation by an axis.

The spatial movement driving unit 25 may be implemented in the form of a robot arm, and the monitor unit of the master interface 4 corresponds to the viewpoint of various surgical endoscopes operated by the handle 10 or varying as described above. By rotating and moving (6), there is a feature that allows the user to feel more realistic about the surgery.

In addition, another embodiment of the present invention may further include a rotation manipulation unit (not shown) for rotating the above-described surgical endoscope 221 and / or auxiliary endoscope 242. The rotation operation unit allows the user to rotate information such as information about rotation of the endoscope, for example, rotation direction, rotation angular velocity, acceleration / deceleration form, rotation speed, rotation time point, rotation time end point, rotation time length, rotation direction, rotation radius, and the like. It may be an operation unit for determining information.

Here, the rotation direction is a direction in which the endoscope rotates, such as clockwise or counterclockwise, and the acceleration / deceleration form refers to a form in which the endoscope rotates in various forms such as a straight line, an S-curve, and an exponential function. In addition, the rotational angular velocity and the rotational speed are speeds at which one end of the surgical endoscope 221 or the auxiliary endoscope 242 rotates, the rotation time point is time information for starting rotation, and the rotation time end point is time information for ending rotation. to be. In addition, the rotation radius is the distance between the rotation axis and one end of the surgical endoscope 221 when the surgical endoscope 221 is rotated in a conical shape, the length of the shaft connecting portion 224 in the case of a curved endoscope, the auxiliary endoscope 242 It may be a distance between the surgical endoscope 241 and.

The rotary manipulation unit may include a user operable interface, for example, the interface may be implemented in various forms for operating the robot arm and / or other surgical equipment such as a joystick form, a button form, a keypad, a trackball, a touch screen, and the like. Can be. When the user sets the rotation-related information using the corresponding interface and inputs it, the surgical endoscope 221 may rotate in a cone shape or rotate about the first shaft 222 so that one end thereof is rotated. The auxiliary endoscope 242 may also rotate about the surgical endoscope 241 with the central axis.

In addition, specific standards for immersive surgical image processing apparatus according to an embodiment of the present invention, common platform technology such as embedded system, O / S, communication protocol, interface standardization technology such as I / O interface, actuator, battery, camera Details of parts standardization technology such as sensors and the like will be omitted since it is obvious to those skilled in the art.

The immersive surgical image processing method according to the present invention may be implemented in the form of program instructions that can be executed by various computer means and recorded in a computer readable medium. That is, the recording medium may be a computer-readable recording medium having recorded thereon a program for causing a computer to execute the steps described above.

The computer readable medium may include a program command, a data file, a data structure, etc. alone or in combination. Program instructions recorded on the media may be those specially designed and constructed for the purposes of the present invention, or they may be of the kind well-known and available to those having skill in the computer software arts. Examples of the computer-readable recording medium include magnetic media such as a hard disk, a floppy disk and a magnetic tape, optical recording media such as CD-ROM and DVD, magnetic recording media such as a floppy disk Optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like.

The medium may be a transmission medium such as an optical or metal line, a wave guide, or the like, including a carrier wave for transmitting a signal designating a program command, a data structure, or the like. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate as one or more software modules to perform the operations of the present invention.

Those skilled in the art will appreciate that various modifications and changes can be made in the present invention without departing from the spirit and scope of the invention as set forth in the claims below.

1: Master Robot 2: Slave Robot
3: robot arm 4: master interface
5: instrument 6: monitor
8: endoscope 9: endoscope image
10 handle 12 monitor driving means
13: moving groove 25: moving drive unit in space
191: dome screen 192: projector
193: work bench 221: surgical endoscope
222: first shaft 223: second shaft
224: shaft connection 231: surrounding image
232: continuous image 241: surgical endoscope
242: secondary endoscope 243: coupling portion
310: image input unit 320: screen display unit
330: arm operation unit 340: operation signal generation unit
350: screen display control unit 351: endoscope perspective tracking unit
352: continuous image generator 353: image movement information extractor
354: Peripheral image generating unit 355: Image position setting unit
357: storage image display unit 360: image storage unit
380: screen driving control unit 381: endoscope perspective tracking unit
383: Image movement information extraction unit 385: Driving information generation unit
390: screen driver 450: image matching unit
451: characteristic value calculator 453: modeling image implementer
455: Superimposed image processor 610: modeling image
620: Endoscope image 621: First endoscope image
622: second endoscope image

Claims (64)

An image input unit for receiving an endoscope image provided from a surgical endoscope;
A screen display unit which outputs the endoscope image to a specific area;
And a screen display control unit for changing a specific area of the screen display unit for outputting the endoscope image corresponding to the viewpoint of the surgical endoscope.
The method of claim 1,
The surgical endoscope is a laparoscopic, thoracoscopic, arthroscopy, parenteral, bladder, rectal, duodenum, mediastinum, cardiac girdle, characterized in that at least one of the surgical image processing device.
The method of claim 1,
The surgical endoscope is a tangible surgical image processing device, characterized in that the three-dimensional endoscope.
The method of claim 1,
The screen display control unit,
An endoscope perspective tracker for tracking perspective information of the surgical endoscope in response to movement and rotation of the surgical endoscope;
An image movement information extracting unit which extracts movement information of the endoscope image by using perspective information of the surgical endoscope;
And an image position setting unit configured to set a specific area of the screen display unit on which the endoscope image is output using the movement information.
The method of claim 1,
The screen display control unit,
Sensory surgical image processing apparatus, characterized in that for moving the center point of the endoscope image corresponding to the coordinate change value of the perspective of the surgical endoscope.
An image input unit configured to receive a first endoscope image and a second endoscope image provided at different time points from the surgical endoscope;
A screen display unit configured to output the first endoscope image and the second endoscope image to different regions;
An image storage unit for storing the first endoscope image and the second endoscope image;
And a screen display controller configured to control the screen display unit to output the first endoscope image and the second endoscope image to different areas according to different viewpoints of the surgical endoscope.
The method of claim 6,
And the image input unit receives the first endoscope image before the second endoscope image.
The method of claim 6,
The screen display unit is a haptic surgical image processing device, characterized in that for outputting any one or more of the saturation, brightness, color, and screen pattern differently the first endoscope image and the second endoscope image.
The method of claim 6,
The screen display control unit,
And a stored image display unit for extracting and outputting a first endoscope image stored in the image storage unit to the screen display unit while the screen display unit outputs the second endoscope image input in real time. .
An image input unit for receiving an endoscope image provided from a surgical endoscope;
A screen display unit which outputs the endoscope image to a specific area;
An image storage unit for storing a modeling image of a surgical tool for operating a surgical target photographed by the surgical endoscope;
An image matching unit which matches the endoscope image and the modeling image to generate an output image;
And a screen display control unit configured to change a specific area of the screen display unit to output the endoscope image in accordance with the viewpoint of the surgical endoscope, and to output the matched endoscope image and modeling image to the screen display unit. Image processing device.
The method of claim 10,
And the image matching unit generates the output image by matching the actual surgical tool image included in the endoscope image and the modeling surgical tool image included in the modeling image to each other.
The method of claim 11,
The image matching unit,
A characteristic value calculator for calculating characteristic values using at least one of the endoscope image and position coordinate information of an actual surgical tool coupled to at least one robot arm;
And a modeling image implementer configured to implement a modeling image corresponding to the characteristic value calculated by the characteristic value calculator.
The method of claim 11,
The image matching unit,
And a superimposed image processing unit for removing an overlapping region of the modeling surgical tool image and the actual surgical tool image from the modeling surgical tool image.
The method of claim 11,
The position of the modeling surgical tool image output to the modeling image is a haptic surgical image processing apparatus, characterized in that set using the operation information of the surgical tool.
An image input unit for receiving an endoscope image provided from a surgical endoscope;
A screen display unit for outputting the endoscope image;
A screen driver for rotating and moving the screen display;
And a screen driving control unit configured to control the screen driving unit to rotate and move the screen display unit in accordance with the viewpoint of the surgical endoscope.
16. The method of claim 15,
The screen drive control unit,
An endoscope perspective tracker for tracking perspective information of the surgical endoscope in response to movement and rotation of the surgical endoscope;
An image movement information extracting unit which extracts movement information of the endoscope image by using perspective information of the surgical endoscope;
And a driving information generator for generating screen driving information of the screen display unit using the movement information.
16. The method of claim 15,
One end of the screen driving unit is coupled to the screen display unit, the immersive surgical image processing apparatus, characterized in that for moving the screen display unit along a predetermined moving groove.
16. The method of claim 15,
The screen driving unit is a haptic surgical image processing apparatus, characterized in that the robot arm-like spatial movement driving unit for one end coupled to the screen display to move and rotate the screen display in space.
In the surgical image processing apparatus outputs an endoscope image,
Receiving the endoscope image provided from a surgical endoscope;
Outputting the endoscope image to a specific area of a screen display unit; And
And changing a specific area of the screen display unit for outputting the endoscope image in correspondence with the viewpoint of the surgical endoscope.
The method of claim 19,
The surgical endoscope is a laparoscopic, thoracoscopic, arthroscopy, parenteral, bladder, rectal, duodenum, mediastinum, cardiac mirror image processing method characterized in that at least one.
The method of claim 19,
The surgical endoscope is a tangible surgical image processing method, characterized in that the stereoscopic endoscope.
The method of claim 19,
Changing the specific area of the screen display unit,
Tracking perspective information of the surgical endoscope in correspondence with the movement and rotation of the surgical endoscope;
Extracting movement information of the endoscope image by using perspective information of the surgical endoscope; And
And setting a specific area of the screen display unit on which the endoscope image is output using the movement information.
The method of claim 19,
Changing the specific area of the screen display unit,
And moving the center point of the endoscope image corresponding to the coordinate change value of the viewpoint of the surgical endoscope.
In the surgical image processing apparatus outputs an endoscope image,
Receiving a first endoscope image and a second endoscope image provided at different time points from the surgical endoscope;
Outputting the first endoscope image and the second endoscope image to different regions of a screen display unit;
Storing the first endoscope image and the second endoscope image; And
And controlling the screen display unit to output the first endoscope image and the second endoscope image to different regions according to different viewpoints of the surgical endoscope.
25. The method of claim 24,
The step of receiving the endoscope image,
Sensory surgery image processing method, characterized in that the first endoscope image is received before the second endoscope image.
25. The method of claim 24,
The output step,
The first endoscopic image and the second endoscopic image differently outputting any one or more of the saturation, brightness, color, and screen pattern.
25. The method of claim 24,
The screen display control step,
And extracting a first endoscope image stored in the image storage unit and outputting the first endoscope image stored in the image storage unit while the screen display unit outputs the second endoscope image input in real time.
In the surgical image processing apparatus outputs an endoscope image,
Receiving an endoscope image provided from a surgical endoscope;
Outputting the endoscope image to a specific area of a screen display unit;
Storing a modeling image of a surgical tool for operating a surgical target photographed by the surgical endoscope;
Generating an output image by matching the endoscope image with the modeling image; And
Changing the specific region of the screen display unit for outputting the endoscope image in accordance with the perspective of the surgical endoscope, and outputting the matched endoscope image and modeling image to the screen display immersive surgical image processing Way.
The method of claim 28,
Generating the output image,
3. The method of claim 1, wherein the output image is generated by matching an actual surgical tool image included in the endoscope image with a modeling surgical tool image included in the modeling image.
The method of claim 29,
Generating the output image,
Calculating a characteristic value using at least one of the endoscope image and position coordinate information of an actual surgical tool coupled to one or more robot arms; And
And implementing a modeling image corresponding to the calculated characteristic value.
The method of claim 29,
Generating the output image,
And removing the overlapping region of the modeling surgical tool image and the actual surgical tool image from the modeling surgical tool image.
The method of claim 29,
The position of the modeling surgical tool image output to the modeling image is a tangible surgical image processing method, characterized in that the setting using the operation information of the surgical tool.
In the surgical image processing apparatus outputs an endoscope image,
Receiving an endoscope image provided from a surgical endoscope;
Outputting the endoscope image to a screen display unit; And
The immersive surgical image processing method comprising the step of rotating and moving the screen display in accordance with the perspective of the surgical endoscope.
The method of claim 33, wherein
Rotating and moving the screen display unit,
Tracking perspective information of the surgical endoscope in correspondence with the movement and rotation of the surgical endoscope;
Extracting movement information of the endoscope image by using perspective information of the surgical endoscope;
And generating motion information of the screen display unit using the movement information.
The method according to any one of claims 19, 24, 28 and 33,
The screen display unit,
A dome shaped screen;
And a projector for projecting the endoscope image onto the dome-shaped screen.
A program that is tangibly embodied and can be read by a digital processing device in order to perform the immersive surgical image processing method according to any one of claims 19 to 35. Recording medium.
An image input unit configured to receive a first endoscope image and a second endoscope image provided at different points of time from one endoscope to which the surgical endoscope rotates;
A screen display unit configured to output the first endoscope image and the second endoscope image to different regions;
And a screen display controller configured to control the screen display unit to output the first endoscope image and the second endoscope image to different areas according to different viewpoints of the surgical endoscope.
The method of claim 37,
The surgical endoscope is a tangible surgical image processing device, characterized in that to rotate periodically.
The method of claim 37,
Rotation operation unit for setting the rotation-related information of any one or more of the rotation direction, rotation angular velocity, acceleration / deceleration form, rotation speed, rotation time point, rotation time end point, rotation time length and rotation radius related to one end rotation of the surgical endoscope A haptic surgical image processing apparatus further comprising.
The method of claim 37,
The surgical endoscope is a tangible surgical image processing device, characterized in that one end rotates to form a lung diagram.
The method of claim 37,
The surgical endoscope is a tangible surgical image processing device, characterized in that rotating in the shape of a cone or a pyramid.
The method of claim 37,
Rotational trajectory of the surgical endoscope is a tangible surgical image processing device, characterized in that any one of a circle, ellipse, triangle and rectangle.
The method of claim 37,
The screen display control unit,
A continuous image generator configured to extract a region overlapping the first endoscope image and the second endoscope image to generate a continuous image;
And a peripheral image generating unit configured to extract a non-overlapping region of the first endoscope image and the second endoscope image to generate a peripheral image.
A first image input unit configured to receive a first endoscope image from a surgical endoscope;
A second image input unit coupled to one side of the surgical endoscope to receive a plurality of second endoscope images provided at different views from an auxiliary endoscope rotating around the surgical endoscope;
A screen display unit configured to output the first endoscope image and the second endoscope image to different regions;
An immersive surgical image including a screen display control unit for controlling the screen display unit to output the first endoscope image and the second endoscope image to different areas corresponding to different viewpoints of the surgical endoscope and the auxiliary endoscope Processing unit.
The method of claim 44,
The auxiliary endoscope is a haptic surgical image processing device, characterized in that to rotate periodically.
The method of claim 44,
A rotation operation unit for setting rotation related information of any one or more of a rotation direction, rotation angular velocity, acceleration / deceleration form, rotation speed, rotation time point, rotation time end point, rotation time length and rotation radius associated with one end rotation of the auxiliary endoscope; Sensory type surgical image processing apparatus comprising.
The method of claim 44,
The screen display control unit,
A continuous image generator configured to extract a region overlapping the first endoscope image and the second endoscope image to generate a continuous image;
And a peripheral image generating unit for extracting a non-overlapping region of the first endoscope image and the second endoscope image to generate a peripheral image.
The method of claim 44,
The screen display control unit,
A continuous image generating unit generating a continuous image from the first endoscope image;
And a peripheral image generator configured to extract the second endoscope image to generate a peripheral image of the continuous image.
The method of claim 44,
The auxiliary endoscope is a haptic surgical image processing device, characterized in that coupled to the surgical endoscope detachably.
In the surgical image processing apparatus outputs an endoscope image,
Receiving a first endoscope image and a second endoscope image provided at different points in time from a surgical endoscope rotating at one end;
Outputting the first endoscope image and the second endoscope image to different regions of a screen display unit; And
And controlling the screen display unit to output the first endoscope image and the second endoscope image to different regions according to different viewpoints of the surgical endoscope.
51. The method of claim 50,
The surgical endoscope is a tangible surgical image processing method, characterized in that the rotation periodically.
51. The method of claim 50,
Setting rotation related information of any one or more of rotation direction, rotation angular velocity, acceleration / deceleration form, rotation speed, rotation time point, rotation time end point, rotation time length and rotation radius related to one end rotation of the surgical endoscope. An immersive surgical image processing method comprising.
51. The method of claim 50,
The surgical endoscope is a tangible surgical image processing device, characterized in that one end rotates to form a lung diagram.
51. The method of claim 50,
The surgical endoscope is a tangible surgical image processing method, characterized in that rotating in the shape of a cone or a pyramid.
51. The method of claim 50,
Rotational trajectory of the surgical endoscope is a tangible surgical image processing method, characterized in that any one of a circle, ellipse, triangle and square.
51. The method of claim 50,
The screen display control step,
Generating a continuous image by extracting an overlapping region of the first endoscope image and the second endoscope image; And
And extracting a non-overlapping region of the first endoscope image and the second endoscope image to generate a peripheral image.
Receiving a first endoscope image from a surgical endoscope;
Receiving a plurality of second endoscope images provided at different points in time from an auxiliary endoscope which is coupled to one side of the surgical endoscope and rotates around the surgical endoscope;
Outputting the first endoscope image and the second endoscope image to different regions; And
And controlling the screen display unit to output the first endoscope image and the second endoscope image to different areas according to different viewpoints of the surgical endoscope and the auxiliary endoscope. .
The method of claim 57,
The auxiliary endoscope is a haptic surgical image processing method characterized in that the rotation periodically.
The method of claim 57,
Setting rotation related information of any one or more of a rotation direction, rotation angular velocity, acceleration / deceleration form, rotation speed, rotation time point, rotation time end point, rotation time length, and rotation radius associated with one end of rotation of the auxiliary endoscope. Image processing method for haptic surgery.
The method of claim 57,
The screen display control step,
Generating a continuous image by extracting an overlapping region of the first endoscope image and the second endoscope image; And
And extracting a non-overlapping region of the first endoscope image and the second endoscope image to generate a peripheral image.
The method of claim 57,
The screen display control step,
Generating a continuous image from the first endoscope image; And
And extracting the second endoscope image to generate a peripheral image of the continuous image.
The method of claim 57,
And immersing the auxiliary endoscope in a detachable manner to the surgical endoscope.
The method according to any one of claims 1, 6, 10, 15, 37 and 44,
The screen display unit,
A dome shaped screen;
And a projector for projecting the endoscope image onto the dome-shaped screen.
63. A program that is tangibly embodied and can be read by a digital processing device in order to perform the immersive surgical image processing method according to any one of claims 50 to 62. Recording medium.

Images