KR102013866B1 - Method and apparatus for calculating camera location using surgical video - Google Patents

Abstract

Provided is a method for calculating a camera position using an actual surgical video. The method can comprise steps of: obtaining a reference object from the actual surgical video captured by a camera entering into the body of a patient to set a reference position with respect to the camera; calculating a change amount of the camera position as the camera moves; and calculating a current position of the camera based on the change amount of the camera position with respect to the reference position. According to the present invention, it is possible to obtain a position of a surgical site and a surgical tool.

Description

Method and device for calculating camera position using real surgical image {METHOD AND APPARATUS FOR CALCULATING CAMERA LOCATION USING SURGICAL VIDEO}

The present invention relates to a method and apparatus for calculating a camera position using an actual surgical image.

Recently, researches on applying virtual reality to the field of medical surgery simulation have been actively conducted.

Medical surgery can be classified into open surgery, laparoscopic surgery, minimally invasive surgery (MIS) including radiologic surgery, radio surgery, and the like. Laparoscopic surgery refers to surgery performed by medical staff to see and touch the part to be treated. Minimally invasive surgery is also known as keyhole surgery, and laparoscopic surgery and robotic surgery are typical. In laparoscopic surgery, a small hole is made in a necessary part without opening, and a laparoscopic with a special camera is attached and a surgical tool is inserted into the body and observed through a video monitor. Microsurgery is performed using a laser or a special instrument. In addition, robot surgery is to perform minimally invasive surgery using a surgical robot. Furthermore, radiation surgery refers to surgical treatment with radiation or laser light outside the body.

In the case of such medical surgery, especially in the case of minimally invasive surgery, it is important to precisely specify the location of the surgical site and the surgical tool to perform the surgery. In addition, it is important to accurately identify the location of each organ, blood vessel, etc. in the patient's body continuously during the surgery.

The problem to be solved by the present invention is to provide a method and apparatus for calculating a camera position using an actual surgical image.

Problems to be solved by the present invention are not limited to the above-mentioned problems, and other problems not mentioned will be clearly understood by those skilled in the art from the following description.

In the method of calculating a camera position using an actual surgical image performed by a computer according to an embodiment of the present invention, a reference object is obtained by acquiring a reference object from an actual surgical image photographed by a camera entering the body of a surgical subject. Setting a position, calculating a position change amount of the camera as the camera moves, and calculating a current position of the camera based on the position change amount of the camera with respect to the reference position.

In an embodiment of the present disclosure, the setting of the reference position with respect to the camera may include obtaining a virtual image including the reference object from the virtual body model of the surgery subject, and the reference object included in the virtual image. And matching the reference object included in the actual surgical image, and setting the reference position by using the position of the reference object obtained from the virtual body model based on the matching of the reference object. have.

In one embodiment of the present invention, the step of calculating the change amount of the position of the camera, the step of acquiring the first image and the second image from which the movement of the camera is detected from the actual surgical image, the first image and the Detecting a change between images by matching a second image, and calculating an amount of position change of the camera based on the change between the images.

In an embodiment of the present invention, the method may further include deriving a position of a surgical part or a position of a surgical tool based on the current position of the camera.

In an embodiment of the present disclosure, the method may further include matching a surgery position in the virtual body model corresponding to the surgery position of the surgery subject based on the current position of the camera.

In one embodiment of the present invention, obtaining a new reference object from the actual surgery image to reset the reference position based on the new reference object, and based on the reset reference position to the current position of the camera The method may further include reloading.

In an embodiment of the present disclosure, the resetting of the reference position may reset the reference position when the new reference object acquired from the actual surgery image corresponds to a preset reference object.

In one embodiment of the present invention, the reference object may be an organ that satisfies a predetermined condition among organs located inside the body of the subject.

In one embodiment of the present invention, the actual surgical image is a stereoscopic 3D image, the virtual body model may be 3D modeling data generated based on the medical image data taken inside the body of the surgical target have.

An apparatus according to an embodiment of the present invention includes a memory for storing one or more instructions, and a processor for executing the one or more instructions stored in the memory, wherein the processor executes the one or more instructions, thereby executing the body of the surgical subject. Acquiring a reference object from an actual surgical image photographed by a camera entering inside to set a reference position with respect to the camera, calculating a position change amount of the camera as the camera moves, and at the reference position Calculating a current position of the camera based on an amount of change in the position of the camera.

The computer program according to an embodiment of the present invention is combined with a computer, which is hardware, and stored in a computer-readable recording medium to perform a camera position calculation method using the actual surgical image.

According to the present invention, the position of the surgical site and the surgical tool can be obtained by calculating the current position of the camera in real time during the actual surgery.

Effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a schematic diagram of a system capable of performing robot surgery according to an embodiment of the present invention.
2 is a flowchart illustrating a camera position calculation method using an actual surgical image according to an embodiment of the present invention.
3 is a flowchart illustrating a process of setting a reference position for a camera according to an embodiment of the present invention.
4 is a flowchart illustrating a process of calculating a position change amount of a camera according to an embodiment of the present invention.
5 is a view showing the position of the camera derived according to an embodiment of the present invention on a coordinate plane.
FIG. 6 is a diagram schematically illustrating a configuration of an apparatus 300 for performing a camera position calculating method using an actual surgical image according to an embodiment of the present invention.

Advantages and features of the present invention and methods for achieving them will be apparent with reference to the embodiments described below in detail with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but can be embodied in various different forms, and the present embodiments only make the disclosure of the present invention complete, and those of ordinary skill in the art to which the present invention belongs. It is provided to fully inform the skilled worker of the scope of the invention, which is defined only by the scope of the claims.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. In this specification, the singular also includes the plural unless specifically stated otherwise in the phrase. As used herein, "comprises" and / or "comprising" does not exclude the presence or addition of one or more other components in addition to the mentioned components. Like reference numerals refer to like elements throughout, and "and / or" includes each and all combinations of one or more of the mentioned components. Although "first", "second", etc. are used to describe various components, these components are of course not limited by these terms. These terms are only used to distinguish one component from another. Therefore, of course, the first component mentioned below may be a second component within the technical spirit of the present invention.

Unless otherwise defined, all terms used in the present specification (including technical and scientific terms) may be used in a sense that can be commonly understood by those skilled in the art. In addition, terms that are defined in a commonly used dictionary are not ideally or excessively interpreted unless they are specifically defined clearly.

As used herein, the term "part" or "module" refers to a hardware component such as software, FPGA, or ASIC, and the "part" or "module" plays certain roles. However, "part" or "module" is not meant to be limited to software or hardware. The “unit” or “module” may be configured to be in an addressable storage medium or may be configured to play one or more processors. Thus, as an example, a "part" or "module" may include components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, Procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables. Functions provided within components and "parts" or "modules" may be combined into smaller numbers of components and "parts" or "modules" or into additional components and "parts" or "modules". Can be further separated.

As used herein, “image” may mean multidimensional data composed of discrete image elements (eg, pixels in a 2D image and voxels in a 3D image). For example, the image may include a medical image of the object obtained by the CT imaging apparatus.

As used herein, an "object" may be a person or an animal, or part or all of a person or an animal. For example, the subject may include at least one of organs such as the liver, heart, uterus, brain, breast, abdomen, and blood vessels.

As used herein, a "user" may be a doctor, a nurse, a clinical pathologist, a medical imaging professional, or the like, and may be a technician who repairs a medical device, but is not limited thereto.

In the present specification, "medical image data" is a medical image photographed by a medical imaging apparatus, and includes all medical images that can be implemented as a three-dimensional model of the body of an object. "Medical image data" may include computed tomography (CT) images, magnetic resonance imaging (MRI), positron emission tomography (PET) images, and the like.

As used herein, the term "virtual body model" refers to a model generated according to the actual patient's body based on medical image data. The "virtual body model" may be generated by modeling medical image data in three dimensions as it is, or may be corrected as in actual surgery after modeling.

As used herein, "virtual surgery data" refers to data including rehearsal or simulation actions performed on a virtual body model. The "virtual surgery data" may be image data for which rehearsal or simulation has been performed on the virtual body model in the virtual space, or may be data recorded for a surgical operation performed on the virtual body model.

As used herein, the term "actual surgery data" refers to data obtained by performing a surgery by an actual medical staff. "Real surgery data" may be image data photographing the surgical site in the actual surgical procedure, or may be data recorded for the surgical operation performed in the actual surgical procedure.

As used herein, the term "computer" includes all the various devices capable of performing arithmetic processing to provide a result to a user. For example, a computer can be a desktop PC, a notebook, as well as a smartphone, a tablet PC, a cellular phone, a PCS phone (Personal Communication Service phone), synchronous / asynchronous The mobile terminal of the International Mobile Telecommunication-2000 (IMT-2000), a Palm Personal Computer (PC), a Personal Digital Assistant (PDA), and the like may also be applicable. In addition, when a head mounted display (HMD) device includes a computing function, the HMD device may be a computer. Also, the computer may correspond to a server that receives a request from a client and performs information processing.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a schematic diagram of a system capable of performing robot surgery according to an embodiment of the present invention.

According to FIG. 1, the robotic surgical system includes a medical imaging apparatus 10, a server 100, a control unit 30 provided in an operating room, a display 32, and a surgical robot 34. According to an embodiment, the medical imaging apparatus 10 may be omitted in the robot surgery system according to the disclosed embodiment.

In one embodiment, surgical robot 34 includes imaging device 36 and surgical instrument 38.

In one embodiment, the robot surgery is performed by the user controlling the surgical robot 34 using the control unit 30. In one embodiment, the robot surgery may be automatically performed by the controller 30 without the user's control.

The server 100 is a computing device including at least one processor and a communication unit.

The controller 30 includes a computing device including at least one processor and a communication unit. In one embodiment, the control unit 30 includes hardware and software interfaces for controlling the surgical robot 34.

The imaging device 36 includes at least one image sensor. That is, the imaging device 36 includes at least one camera device and is used to photograph an object, that is, a surgical site. In one embodiment, the imaging device 36 includes at least one camera coupled with a surgical arm of the surgical robot 34.

In one embodiment, the image captured by the imaging device 36 is displayed on the display 32.

In one embodiment, surgical robot 34 includes one or more surgical tools 38 that can perform cutting, clipping, fixing, grabbing operations, and the like, of the surgical site. Surgical tool 38 is used in conjunction with the surgical arm of the surgical robot 34.

The controller 30 receives information necessary for surgery from the server 100 or generates information necessary for surgery and provides the information to the user. For example, the controller 30 displays the information necessary for surgery, generated or received, on the display 32.

For example, the user performs the robot surgery by controlling the movement of the surgical robot 34 by manipulating the control unit 30 while looking at the display 32.

The server 100 generates information necessary for robotic surgery using medical image data of an object previously photographed from the medical image photographing apparatus 10, and provides the generated information to the controller 30.

The controller 30 displays the information received from the server 100 on the display 32 to provide the user, or controls the surgical robot 34 by using the information received from the server 100.

In one embodiment, the means that can be used in the medical imaging apparatus 10 is not limited, for example, other various medical image acquisition means such as CT, X-Ray, PET, MRI may be used.

In the disclosed embodiment, the surgical image obtained by the imaging device 36 is transmitted to the controller 30.

Hereinafter, a camera position calculation method using an actual surgical image will be described in detail with reference to the accompanying drawings.

2 is a flowchart illustrating a camera position calculation method using an actual surgical image according to an embodiment of the present invention.

Each step shown in FIG. 2 is performed in time series in the server 100 or the controller 30 shown in FIG. 1. Hereinafter, for the convenience of description, each step is described as being performed by a computer, but the performing agent of each step is not limited to a specific device, and all or part thereof may be performed by the server 20 or the controller 30. Can be.

Referring to FIG. 2, in the camera position calculation method using an actual surgery image according to an embodiment of the present invention, a computer acquires a reference object from an actual surgery image photographed by a camera that enters into the body of a surgery subject. Setting a reference position with respect to the camera (S100), calculating a position change amount of the camera as the camera moves (S200), and a current position of the camera based on the position change amount of the camera with respect to the reference position It may include the step (S300) to calculate. Hereinafter, a detailed description of each step will be described.

The computer may acquire a reference object from an actual surgery image photographed by a camera that enters the body of the surgery subject and set a reference position with respect to the camera (S100).

The actual surgery image refers to data obtained by the actual medical staff performing the surgery, for example, may be an image of the actual surgery scene actually performed by the surgical robot 34. In addition, the actual surgical image may be a stereoscopic 3D image, and thus the actual surgical image may be an image having a three-dimensional stereoscopic sense, that is, a depth. Therefore, it is possible to accurately grasp the position of the surgical tool in the three-dimensional space through the depth map of the actual surgical image. In addition, the camera may be a device capable of capturing 3D images, for example, may be a stereo camera. Accordingly, the actual surgical image may be captured as a 3D image by a stereo camera.

According to an embodiment, the depth information of the actual surgical image does not need to have a depth value for every pixel of the entire screen area, and a depth value may be obtained for a minimum reference point that can sufficiently express geometrical features. For example, if there is no reference 3D data model (eg, virtual body model), a depth value may be required for all pixels, but if there is already a 3D data model (eg, virtual body model), depth is required for all pixels. It does not have to have a value.

In one embodiment, when performing minimally invasive surgery, such as laparoscopic surgery, robot surgery, surgery is performed by inserting only a surgical tool and a camera in the surgical site of the patient (patient). In this case, the computer may acquire an image taken by a camera entering the patient's body as an actual surgical image. For example, the actual surgery image is taken of the process of moving from the inside of the patient's body to the surgical site of the body, the process from the start of the action by the surgical tool to the end of the body to the surgical site, etc. It may be image data.

As described above, the object (object) may be an organ located inside the body of the patient to be operated, or another part other than the organ. In one embodiment of the present invention, the reference object may be a specific organ or a specific part or region that satisfies a predetermined condition among objects located inside the body of the surgery subject. That is, the reference object is easy to detect features from the image, is present in a fixed position inside the body, there is no movement or very little at the time of surgery, no deformation of the shape does not occur, is not affected by surgical instruments, medical images Organs or specific internal parts that satisfy at least one of the conditions, such as acquisition of data, such as data captured by CT, PET, etc., may be used. For example, a portion having little movement during surgery such as liver and abdominal disease, and a portion that can be obtained from medical image data such as stomach, esophagus, gallbladder, etc. may be determined as the reference object.

3 is a flowchart illustrating a process of setting a reference position for a camera according to an embodiment of the present invention. 3 illustrates the operation of the step S100 in detail.

Referring to FIG. 3, the computer may acquire a virtual image including the reference object from the virtual body model of the surgery subject (S110).

As described above, the virtual body model may be 3D modeling data generated based on medical image data photographing the inside of the body of the patient in advance. In addition, the virtual image may be an image of a viewpoint having the same information as that captured by an actual camera in a virtual body model constructed in 3D.

In one embodiment, the camera entering the patient's body during surgery is to shoot all the objects in the direction the camera looks. In this case, the computer may select an image of a desired viewpoint, that is, an image of a viewpoint including a reference object, from among the actual surgical images photographed by the camera. Thereafter, the computer may acquire a virtual image in the virtual body model corresponding to the actual surgical image of the viewpoint including the reference object.

The computer may match the reference object included in the virtual image with the reference object included in the actual surgical image (S120).

In one embodiment, the registration between the two images may be performed by extracting and comparing features of the reference object. For example, a depth map may be obtained or features of a reference object may be extracted by applying image segmentation techniques of deep learning.

The computer may obtain a depth map of the reference object from the actual surgery image, and compare the depth map information with respect to the reference object obtained from the virtual image to determine whether the reference object between the two images is the same object. Depth maps can be used to compare surface information of objects (organs). Some objects are difficult to identify by surface features alone. In this case, the computer may apply a segmentation technique of deep learning to obtain a segmentation image of a reference object and perform registration between the two images. According to an exemplary embodiment, the computer may use the segmentation image information together with the depth map information to perform registration between the two images. For example, if an object (long-term) has a smooth surface feature, it is difficult to identify the object through the depth map, so the image segmentation of deep learning can be applied together. In addition, when the object (long-term) has a three-dimensional structure, since image segmentation of deep learning is performed in two dimensions, the surface feature of the object may apply image segmentation, and the depth map may be applied to the three-dimensional structure.

In addition, the virtual body model is generated by including information on the type of the object and location information for each object, such information can be configured as an image map. Accordingly, the computer may detect an object region that is expected to be matched in the virtual image based on the feature information of the reference object obtained from the actual surgical image. The computer may obtain depth information and segmentation information based on the detected object region in the virtual image, and may construct the map image. The computer may compare the map image information of the object obtained from the virtual image with the feature information (depth information and segmentation image information) of the reference object obtained from the actual surgical image to determine whether the reference object is identical between the two images.

Meanwhile, in performing matching between two images, if the matching is performed at a random position, the execution time may be long. However, because the position of the camera does not change suddenly due to the nature of the surgical image, the change in the image does not occur much. Therefore, the computer may acquire an actual surgical image of a predetermined viewpoint of the camera, and acquire a virtual image of the virtual body model corresponding to the viewpoint, and perform registration between the two images. In one embodiment, since the surgical procedure is constant, the computer may recognize the location of the surgical tool or organ (object) with respect to the surgical procedure through deep learning, and thereby obtain an actual surgical image of a predetermined viewpoint of the camera. . That is, the computer recognizes the position of the reference object through deep learning and obtains the reference position of the camera by acquiring the actual surgical image of the recognized viewpoint.

In addition, according to an embodiment, the computer first recognizes the position of the reference object through deep learning to obtain an actual surgery image of the viewpoint including the reference object, and then corresponds to the actual surgery image of the viewpoint including the reference object. Virtual images of the virtual body model at the time point may be matched. In this case, the matching between the actual surgical image and the virtual image may apply a depth map or an image segmentation technique of deep learning as described above. In such an embodiment, the position of the reference object may be accurately found by first finding an image of a viewpoint including the reference object in the actual surgery image and narrowing the area, and then matching the actual surgery image and the virtual image of the viewpoint.

The computer may acquire the position of the reference object from the virtual body model based on the result of registration of the reference object between the two images, and set the reference position with respect to the camera based on this.

As described above, since the virtual body model is generated including the position information of the object, the position of the reference object photographed by the camera can be derived from the virtual body model. Thus, the computer can determine the reference position of the camera based on the position of the reference object.

Referring back to FIG. 2, the computer may calculate the position change amount of the camera as the camera moves (S200).

Since the camera is inserted into the body of the surgical target to photograph the surgical procedure while moving, the position of the camera may continue to change. Therefore, after setting the reference position of the camera, it is possible to calculate the position of the camera during surgery by calculating the amount of change in accordance with the movement of the camera in real time.

4 is a flowchart illustrating a process of calculating a position change amount of a camera according to an embodiment of the present invention. 4 illustrates the operation of the step S200 in detail.

Referring to FIG. 4, the computer may acquire a first image and a second image from which the camera movement is detected from the actual surgical image (S210). That is, the computer detects an image in which the movement of the camera occurs from the actual surgical image photographed by the camera after setting the reference position.

To this end, it is necessary to detect the optical flow related to the camera movement in the image and filter out the optical flow that is not related to the camera movement, for example, the movement of the surgical tool, but it is very difficult to distinguish it from the general image. However, during surgery, there is a limitation that the surgical tool does not move when the camera moves. Therefore, by using this constraint, only the image of the camera moving effectively can be detected from the actual surgical image. That is, the characteristic of the image having only the movement of the camera is that all parts of the image move in a constant direction. On the other hand, the movements of the surgical instruments and organs are also different in each moving direction in the image. Therefore, by using this feature, it is possible to obtain an image with only the camera movement from the actual surgical image. In one embodiment, the computer may detect the first image and the second image in which the camera movement is generated from the actual surgical image by applying the features of the surgical image as described above, such as deep learning.

In the present embodiment, it is considered that there is a constraint that the surgical tool does not move when the camera moves as described above during surgery, but this is only an example and the present invention is not limited thereto. According to an embodiment, both the camera and the surgical tool may consider a case in which a motion occurs. In this case, when the deep learning technique is applied, the first and second images in which the camera motion is detected are detected from the actual surgical image. can do.

Next, the computer may detect a change between the images by matching the first image and the second image (S220).

In an embodiment, the computer may extract and match the feature points of each of the first and second images. For example, the computer may extract each feature point for each of the first image and the second image by using an algorithm such as scale-invariant feature transform (SIFT). In this case, various feature point extraction algorithms may be used in addition to the SIFT. The computer may match the feature points of the first image and the feature points of the second image with each other, and detect a change between the matched feature points. In this case, a feature point matching algorithm may be used, for example, brute force, FLANN (Fast approximate nearest neighbor) search, or the like. For example, the computer may construct a point cloud by calculating a depth map for each of the first image and the second image. In this case, corresponding points in the point cloud for each of the two images may also be matched by matching the feature points between the first image and the second image. Here, since the feature point matching is performed in 2D coordinates, the 3D coordinates are converted into 3D coordinates using depth maps of the first and second images, and the matching is performed in the point cloud between the two images based on the converted 3D coordinates. Can be. Through the matching between the two images, it is possible to detect that all the corresponding points of the point cloud for each of the two images in the same direction. Therefore, the computer can know the position change amount of the camera based on the position change of the feature point (corresponding point) between the two images.

According to an embodiment, the computer may obtain a depth map through matching of stereo images when constructing a point cloud for the first image and the second image, but this is only one example, and the first image and the second image may be obtained using different methods. The point cloud for the second image may be configured and matched. For example, in limited circumstances, a single image may be used to obtain a depth map. Since the size of the surgical tool is known correctly and the size of the surgical tool does not change during surgery, the characteristics of the camera can be kept constant. In this case, a depth value for a point of interest (feature point) may be obtained using a single image.

Next, the computer may calculate the position change amount of the camera based on the change between images (S230).

As an example, as described above, since the computer detects a position change amount of a feature point between two images, the change amount may be estimated as a change amount according to the movement of the camera.

Meanwhile, in calculating the position change amount of the camera according to the embodiment, the computer may calculate the position change amount of the camera based on the surgical tool according to whether the surgical tool is included in the actual surgical image.

In one embodiment, the computer may determine whether a surgical tool is included in the actual surgical image acquired as the camera movement occurs. For example, the computer may determine whether the surgical tool is included in the first image acquired at the start of the movement of the camera and the second image acquired at the end of the movement of the camera. In this case, the computer may determine whether a surgical tool exists from the first image and the second image by using an image recognition method.

When the surgical tool is included in the first image and the second image according to the determination result, the computer compares the information of the surgical tool in the first image with the information of the surgical tool in the second image and positions the camera based on the comparison result. The amount of change can be calculated relatively. As described above, since the surgical tool does not move when the camera moves during the operation, the surgical tool does not move in the images (first image and second image) acquired from the start of the movement of the camera to the end of the movement. It is fixed without. Therefore, if a surgical tool is included in the first image and the second image, the corresponding surgical tool may be changed only by the movement of the camera. That is, the computer derives a change in the position, size, direction, etc. of the surgical tool included in each of the first image and the second image, and calculates a relative change in the position of the camera based on the change information between the surgical tools in each image. Can be.

Alternatively, the computer may determine whether at least one surgical tool is included in the images acquired from the start of the movement of the camera to the end of the movement of the camera. For example, the computer may determine whether a surgical tool is included in all or a part of the image frame acquired at the start of the movement of the camera, and determine whether the surgical tool is included in the image frame at the end of the movement of the camera. Subsequently, the amount of position change of the camera may be relatively calculated by comparing information of the surgical tool between the image frames including the surgical tool according to the determination result.

Alternatively, the computer may determine whether the surgical tool is included in at least one image frame of the image acquired at the start of the movement of the camera or at the end of the movement of the camera, and relatively calculate the position change of the camera. For example, when the surgical tool is included in the first image acquired when the camera starts to move, or when the surgical tool is included in the second image obtained when the camera stops moving, the computer refers to the image including the surgical tool. In addition, the moving distance of the camera can be calculated relatively. That is, since the surgical tool is fixed when the camera is moved, the distance to the position where the camera is moved (that is, the start position of the camera or the end position of the camera) is calculated based on the position of the fixed surgical tool. The amount of change in the position of the camera can also be calculated relatively.

In addition, if the surgical tool is not included in the images (eg, the first image and the second image) acquired from the start of the movement of the camera to the end of the movement of the camera as a result of the determination, the computer performs the above-described process of FIG. 4. The amount of change in position of the camera may be calculated based on the change between images.

Referring back to FIG. 2, the computer may calculate the current position of the camera based on the position change amount of the camera with respect to the reference position (S300).

That is, since the position change amount of the camera is calculated by the change amount according to the movement from the reference position, the current position of the camera can be finally calculated by applying the position change amount of the camera at the reference position.

In an embodiment, the computer may calculate the current position of the camera by deriving a change between the first image and the second image through a rotation or translation matrix and reflecting the derived value in a reference position. have.

According to an embodiment of the present invention, the computer may derive the current position of the camera, and may calculate the position of the surgical region or the position of the surgical tool of the currently-operated subject based on this.

In addition, according to an embodiment of the present invention, by deriving the current position of the camera, the computer may match the surgical position in the virtual body model corresponding to the surgical position of the surgical subject based on this. That is, by providing the actual surgical position in the virtual body model has the effect of providing more accurate surgical guide information to the current medical staff.

In addition, according to an embodiment of the present invention, since the camera may continuously move or move during the surgery, the computer may acquire an actual surgical image including a new reference object photographed while the camera moves or moves. . In this case, the computer may acquire a new reference object from the actual surgery image and reset the reference position based on the acquired new reference object. The computer can also recalculate the current position of the camera based on the reset reference position. This may be applied in the same manner as described in the above steps S100 to S300.

In an embodiment, when the actual surgical image including the new reference object is acquired, the computer may recalculate the current position of the camera by resetting the reference position only when the new reference object corresponds to the preset reference object. For example, whenever a new reference object is photographed and acquired according to the movement of the camera, the current position of the camera may be updated by performing the above steps S100 to S300, or a new object is acquired based on a preset time period. In this case, the current position of the camera may be updated by performing the above steps S100 to S300.

On the other hand, when the reference position for the camera is initially set, and the current position of the camera is continuously calculated based on this, an error may accumulate if the position change amount of the camera cannot be accurately detected over time. However, in the case of resetting the reference position by acquiring a new reference object as in the above-described embodiment, even if an error occurs in the process of calculating the position change amount of the camera, the current position of the camera is corrected by resetting the reference position so that it is more accurate. Surgical information can be provided.

5 is a view showing the position of the camera derived according to an embodiment of the present invention on a coordinate plane.

It is important to accurately specify the location of the surgical site and the surgical tool during the actual surgery. Therefore, in the present invention, in order to accurately provide the position of the surgical site and the surgical tool in real time at the time of the actual operation to calculate the position of the camera and through this again to calculate the position of the surgical site and the surgical tool, the method for this It has been described in detail.

That is, according to the method of FIGS. 2 to 4, as shown in FIG. 5, by setting a reference position through a specific organ of the patient (ie, a reference object), the actual coordinate space (200, absolute coordinate space) of the camera is set. The position 220 of may be derived.

In this case, it is possible to know what positional relationship the coordinate space 210 of the camera has on the actual coordinate space 200. Therefore, the actual position of the surgical site and the surgical tool can be calculated based on the position of the camera. First, the relative positions of the surgical site and the surgical tool may be obtained based on the coordinate space 210 of the camera, and the actual positions of the surgical site and the surgical tool may be derived by converting the surgical site and the surgical tool back into a positional relationship on the actual coordinate space 200. .

FIG. 6 is a diagram schematically illustrating a configuration of an apparatus 300 for performing a camera position calculating method using an actual surgical image according to an embodiment of the present invention.

Referring to FIG. 6, the processor 310 may include a connection passage (for example, a bus or the like) that transmits and receives signals with one or more cores (not shown) and a graphic processor (not shown) and / or other components. ) May be included.

The processor 310 according to an embodiment executes one or more instructions stored in the memory 320 to perform a camera position calculation method using the actual surgical image described with reference to FIGS. 2 to 4.

For example, the processor 310 acquires a reference object from an actual surgery image taken by a camera entering the body of the surgery subject by executing one or more instructions stored in the memory 320 to obtain a reference position for the camera. In this case, the position change amount of the camera may be calculated as the camera moves, and the current position of the camera may be calculated based on the position change amount of the camera with respect to the reference position.

Meanwhile, the processor 310 may read random access memory (RAM) and read-only memory (ROM) for temporarily and / or permanently storing a signal (or data) processed in the processor 310. , Not shown) may be further included. In addition, the processor 310 may be implemented in the form of a system on chip (SoC) including at least one of a graphic processor, a RAM, and a ROM.

The memory 320 may store programs (one or more instructions) for processing and controlling the processor 310. Programs stored in the memory 320 may be divided into a plurality of modules according to their functions.

The camera position calculation method using the actual surgical image according to the embodiment of the present invention described above may be implemented as a program (or an application) to be executed in combination with a computer which is hardware and stored in a medium.

The above-described program includes C, C ++, JAVA, machine language, etc. which can be read by the computer's processor (CPU) through the computer's device interface so that the computer reads the program and executes the methods implemented as the program. Code may be coded in the computer language of. Such code may include functional code associated with a function or the like that defines the necessary functions for executing the methods, and includes control procedures related to execution procedures necessary for the computer's processor to execute the functions according to a predetermined procedure. can do. In addition, the code may further include memory reference code for additional information or media required for the computer's processor to execute the functions at which location (address address) of the computer's internal or external memory should be referenced. have. Also, if the processor of the computer needs to communicate with any other computer or server remotely in order to execute the functions, the code may be used to communicate with any other computer or server remotely using the communication module of the computer. It may further include a communication related code for whether to communicate, what information or media should be transmitted and received during communication.

The stored medium is not a medium for storing data for a short time such as a register, a cache, a memory, but semi-permanently, and means a medium that can be read by the device. Specifically, examples of the storage medium include, but are not limited to, a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like. That is, the program may be stored in various recording media on various servers to which the computer can access or various recording media on the computer of the user. The media may also be distributed over network coupled computer systems so that the computer readable code is stored in a distributed fashion.

The steps of a method or algorithm described in connection with an embodiment of the present invention may be implemented directly in hardware, in a software module executed by hardware, or by a combination thereof. The software module may include random access memory (RAM), read only memory (ROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, hard disk, removable disk, CD-ROM, or It may reside in any form of computer readable recording medium well known in the art.

In the above, embodiments of the present invention have been described with reference to the accompanying drawings, but those skilled in the art to which the present invention pertains may implement the present invention in other specific forms without changing the technical spirit or essential features. I can understand that. Therefore, it should be understood that the embodiments described above are exemplary in all respects and not restrictive.

Claims (11)

In the camera position calculation method using a real operation image,
Setting a reference position with respect to the camera by acquiring a reference object from an actual surgery image photographed by a camera entering the body of the patient;
Calculating a position change amount of the camera as the camera moves; And
Calculating a current position of the camera based on a change amount of the position of the camera relative to the reference position;
Calculating the position change amount of the camera,
Acquiring a first image and a second image from which the movement of the camera is detected from the actual surgical image;
Detecting a change between images by matching the first image and the second image; And
And calculating the position change amount of the camera based on the change between the images. The method of claim 1,
Setting a reference position for the camera,
Obtaining a virtual image including the reference object from the virtual body model of the surgery subject;
Matching the reference object included in the virtual image with the reference object included in the actual surgical image; And
And setting the reference position by using the position of the reference object obtained from the virtual body model based on the registration of the reference object. delete The method of claim 1,
And calculating a position of a surgical site or a position of a surgical tool on the basis of the current position of the camera. The method of claim 1,
And matching the surgical position in the virtual body model corresponding to the surgical position of the surgical subject based on the current position of the camera. The method of claim 1,
Acquiring a new reference object from the actual surgical image and resetting the reference position based on the new reference object; And
And recalculating the current position of the camera based on the reset reference position. The method of claim 6,
Resetting the reference position,
And resetting the reference position when the new reference object acquired from the actual surgery image corresponds to a preset reference object. The method of claim 1,
The reference object,
Camera position calculation method using an actual surgical image, characterized in that the organ satisfying a predetermined condition among the organs located inside the body of the operation target. The method of claim 2,
The actual surgery image,
Stereoscopic 3D video,
The virtual body model,
Camera position calculation method using the actual surgical image, characterized in that the 3D modeling data generated based on the medical image data taken inside the body of the subject. Memory for storing one or more instructions; And
A processor for executing the one or more instructions stored in the memory,
The processor executes the one or more instructions,
Setting a reference position with respect to the camera by acquiring a reference object from an actual surgery image photographed by a camera entering the body of the patient;
Calculating a position change amount of the camera as the camera moves; And
Calculating a current position of the camera based on an amount of change in the position of the camera relative to the reference position;
Calculating the position change amount of the camera,
Acquiring a first image and a second image from which the movement of the camera is detected from the actual surgical image;
Detecting a change between images by matching the first image and the second image; And
And calculating a position change amount of the camera based on the change between the images. A computer program, coupled to a computer, which is hardware, stored on a recording medium readable by a computer so as to perform the method of claim 1.

Images