WO2020181498A1 - 体内导航系统和方法 - Google Patents

体内导航系统和方法 Download PDF

Info

Publication number
WO2020181498A1
WO2020181498A1 PCT/CN2019/077865 CN2019077865W WO2020181498A1 WO 2020181498 A1 WO2020181498 A1 WO 2020181498A1 CN 2019077865 W CN2019077865 W CN 2019077865W WO 2020181498 A1 WO2020181498 A1 WO 2020181498A1
Authority
WO
WIPO (PCT)
Prior art keywords
target
dimensional image
video
optical mark
coordinate system
Prior art date
Application number
PCT/CN2019/077865
Other languages
English (en)
French (fr)
Inventor
杨永生
Original Assignee
上海复拓知达医疗科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 上海复拓知达医疗科技有限公司 filed Critical 上海复拓知达医疗科技有限公司
Priority to PCT/CN2019/077865 priority Critical patent/WO2020181498A1/zh
Publication of WO2020181498A1 publication Critical patent/WO2020181498A1/zh

Links

Images

Classifications

    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis

Definitions

  • the present invention relates to the technical field of computer-assisted medical treatment, and in particular to an in-vivo navigation system and method.
  • percutaneous extracorporeal puncture can be used to obtain trace samples of diseased tissues in the human body.
  • accurate navigation of medical devices is very important.
  • image-based puncture positioning and navigation has always been a key requirement in clinical work.
  • the existing image navigation systems are mainly divided into two categories: optical navigation systems based on infrared reflective points and electromagnetic field navigation systems based on micro-coils (referred to as magnetic navigation). In actual use, these two systems have their own problems, as shown in Table 1:
  • an in-vivo navigation system including:
  • a video capture device is used to capture a video of a target in real time, where the video includes an optical mark fixed on the target, and the angle at which the video capture device captures the video is consistent with the user's viewing direction, wherein the target Including equipment and/or objects;
  • the positioning device is used to identify the optical mark in the video, and anchor the three-dimensional image of the target to the corresponding one in the video according to the identified optical mark and the optical mark in the three-dimensional image of the target Location;
  • the display is used to three-dimensionally display the part of the target in the object in the video.
  • the video capture device is a head-mounted optical camera.
  • the display is a head-mounted display.
  • the optical identification is a two-dimensional code and/or a checkerboard.
  • the optical mark is a three-dimensional figure of the target surface.
  • the positioning device anchors the three-dimensional image of the target to the corresponding position in the video according to the recognized optical mark and the optical mark in the three-dimensional image of the target in the following manner:
  • the identified optical mark and the characteristic points in the optical mark in the three-dimensional image determine the first conversion parameter for converting the three-dimensional image to the user coordinate system, wherein the video is the user In the coordinate system;
  • the three-dimensional image is anchored to the corresponding position in the video by using the first conversion parameter.
  • the positioning device anchors the three-dimensional image of the target to the corresponding position in the video according to the recognized optical mark and the optical mark in the three-dimensional image of the target in the following manner:
  • the optical mark in the image of the target and the characteristic points in the optical mark in the three-dimensional image determine the second conversion parameter for converting the three-dimensional image to the target coordinate system, wherein the image of the target Is in the target coordinate system, and the image of the target also includes the optical mark;
  • the identified optical mark and the characteristic points in the optical mark in the three-dimensional image in the target coordinate system determine a third transformation for converting the three-dimensional image in the target coordinate system to the user coordinate system Parameters, where the video is in the user coordinate system;
  • the third conversion parameter is used to anchor the three-dimensional image in the target coordinate system to a corresponding position in the video.
  • the video acquisition device includes a positioning sensor for detecting the position of the video acquisition device in real time;
  • the positioning device anchors the three-dimensional image of the target to the corresponding position in the video in the following manner:
  • the fourth conversion parameter for converting the three-dimensional image to the target coordinate system is determined, wherein the image of the target Is the image in the target coordinate system collected by the video collecting device at the first moment;
  • a fifth conversion parameter for converting the three-dimensional image in the target coordinate system to the user coordinate system is determined, wherein the video is In the user coordinate system;
  • the fifth conversion parameter is used to anchor the three-dimensional image in the target coordinate system to the corresponding position in the video.
  • the in-vivo navigation system further includes an input device for receiving input from the user;
  • the positioning device is further configured to adjust the corresponding position of the three-dimensional image of the target in the video according to the input.
  • the target is an object
  • the three-dimensional image of the object is a three-dimensional image obtained by a reconstruction operation after scanning the object with a CT, MRI or ultrasound equipment.
  • the target is a device
  • the three-dimensional image of the device is a three-dimensional image drawn by a mapping software for the device, scanned and mapped by a three-dimensional scanner, or obtained after a reconstruction operation after a CT scan.
  • an in-vivo navigation method including:
  • Collect a video of a target in real time where the video includes an optical mark fixed on the target, and the collection angle of the video is consistent with the user's viewing direction, where the target includes an instrument and/or an object;
  • the part of the target in the subject is displayed in three dimensions.
  • the anchoring the three-dimensional image of the target to the corresponding position in the video according to the recognized optical mark and the optical mark in the three-dimensional image of the target further includes:
  • the identified optical mark and the characteristic points in the optical mark in the three-dimensional image determine the first conversion parameter for converting the three-dimensional image to the user coordinate system, wherein the video is the user In the coordinate system;
  • the three-dimensional image is anchored to the corresponding position in the video by using the first conversion parameter.
  • the anchoring the three-dimensional image of the target to the corresponding position in the video according to the identified optical mark and the optical mark in the three-dimensional image of the target further includes:
  • the optical mark in the image of the target and the characteristic points in the optical mark in the three-dimensional image determine the second conversion parameter for converting the three-dimensional image to the target coordinate system, wherein the image of the target Is in the target coordinate system, and the image of the target also includes the optical mark;
  • the identified optical mark and the characteristic points in the optical mark in the three-dimensional image in the target coordinate system determine a third transformation for converting the three-dimensional image in the target coordinate system to the user coordinate system Parameters, where the video is in the user coordinate system;
  • the third conversion parameter is used to anchor the three-dimensional image in the target coordinate system to a corresponding position in the video.
  • the method further includes: real-time detection of the position of a video acquisition device for real-time acquisition of the target;
  • the anchoring the three-dimensional image of the target to the corresponding position in the video according to the recognized optical mark and the optical mark in the three-dimensional image of the target further includes:
  • the fourth conversion parameter for converting the three-dimensional image to the target coordinate system is determined, wherein the image of the target Is the image in the target coordinate system collected by the video collecting device at the first moment;
  • a fifth conversion parameter for converting the three-dimensional image in the target coordinate system to the user coordinate system is determined, wherein the video is In the user coordinate system;
  • the fifth conversion parameter is used to anchor the three-dimensional image in the target coordinate system to the corresponding position in the video.
  • the method further includes:
  • the user will not have equipment or consumables that can interfere with the line of sight within the operating space of the subject, thus not only effectively avoiding the visual field problem of fixed infrared optical navigation, but also avoiding magnetic navigation
  • the navigation hardware in the system occupies the operation space, and the electromagnetic field is susceptible to interference to affect the positioning accuracy.
  • the above-mentioned in-vivo navigation system uses visible light optical markings, which greatly reduces the cost of consumables for navigation surgery while ensuring the practicability of navigation.
  • the implementation of the solution is simple, the user only needs to keep paying attention to the patient or the surgical instrument at all times, without changing the traditional use habits, and the difficulty of learning and promotion is low. Allows more operating objects to benefit from in vivo navigation technology.
  • Fig. 1 shows a schematic block diagram of an in-vivo navigation system and its working environment according to an embodiment of the present invention
  • Fig. 2 shows a schematic diagram of an object and an apparatus operated thereon according to an embodiment of the present invention
  • Figure 3 shows a three-dimensional image of an object according to an embodiment of the invention
  • Figure 4 shows a three-dimensional image of an instrument according to an embodiment of the invention
  • Figure 5 shows a frame in a video according to an embodiment of the present invention
  • Fig. 6 shows a manner in which a positioning device according to an embodiment of the present invention anchors a three-dimensional image of an object to a corresponding position in a video
  • Fig. 7 shows a manner in which a positioning device according to another embodiment of the present invention anchors a three-dimensional image of an object to a corresponding position in a video
  • FIG. 8 shows the coordinate conversion process in the embodiment of FIG. 7
  • Fig. 9 shows a manner in which a positioning device according to another embodiment of the present invention anchors a three-dimensional image of an object to a corresponding position in a video
  • Fig. 10 shows the manner in which the positioning device according to an embodiment of the present invention anchors the three-dimensional image of the instrument to the corresponding position in the video.
  • the in-vivo navigation system is used to provide users with navigation for the tissues and/or parts of the device in the body of the subject.
  • the user is the observer of the entire body navigation process, and he is also the operator who penetrates the instrument into the subject.
  • the object can be a person or other animal that the user needs to operate on.
  • the instrument can be any tool that can be penetrated into the subject.
  • the instrument can be, for example, a puncture needle, a biopsy needle, a radiofrequency or microwave ablation needle, an ultrasound probe, a rigid endoscope, an oval forceps under endoscopic surgery, an electrosurgical knife, or a stapler.
  • the position of the target is first determined based on the visible light image, and then the target is displayed three-dimensionally in the visible light image.
  • the target may be an instrument and/or object.
  • the internal organs, lesions, and/or medical equipment of the object that are actually invisible are displayed to the user in three dimensions to guide the user to operate the equipment in the real environment.
  • an in-vivo navigation system In the video displayed to the user by the display in the in-vivo navigation system, both the object and the device can be displayed at the corresponding positions therein, which provides the user with more comprehensive navigation information.
  • both the object and the device can be displayed at the corresponding positions therein, which provides the user with more comprehensive navigation information.
  • a person of ordinary skill in the art can understand that only objects or instruments can also be displayed therein, and for undisplayed targets, the instrument operation can be performed based on the user's experience. According to the following description, those of ordinary skill in the art can understand that only embodiments of objects or devices are displayed, and for the sake of brevity, details are not repeated here.
  • Fig. 1 shows a schematic block diagram of an in-vivo navigation system and its working environment according to an embodiment of the present invention.
  • the in-vivo navigation system shown in FIG. 1 includes a video capture device 110, a positioning device 120 and a display 130.
  • the video capture device 110 is used to capture videos of objects and equipment in real time.
  • the user can watch the display 130, which is used to display a video, in which not only the surface parts of the objects and instruments collected by the video capture device 110 are displayed, but also the internal organs and lesions of the objects that are actually invisible in the corresponding position are displayed in three dimensions. And the part of the device inside the subject.
  • the invisible internal organs, lesions, and parts of the device in the body are aligned with the human body and the actual device, thereby guiding the user to operate the device in a virtual three-dimensional scene similar to the real environment.
  • the collection angle of the video collection device 110 is consistent with the user's viewing direction.
  • the video capture device is a head-mounted optical camera.
  • the collection angle of the head-mounted optical camera can be kept consistent with its viewing direction.
  • the first optical mark is fixed on the surface of the object
  • the second optical mark is fixed on the surface of the instrument.
  • the skin of the subject may be pasted with a first optical mark.
  • a second optical mark may be printed on the part of the instrument that is close to the handheld, and this part is outside the subject's body when the in-vivo navigation system is working. Both the first optical mark and the second optical mark are identifiable in the visible light image.
  • Fig. 2 shows an object according to an embodiment of the present invention and an instrument operated on it.
  • both the object and the instrument surface have optical markings, namely the two-dimensional code in Figure 2.
  • a two-dimensional code is a black-and-white flat graphic distributed on a plane. The dots on it are very easy to identify. By identifying at least three of them, the positioning of the two-dimensional code can be realized. Because the two-dimensional code is fixed to the object or device, the positioning of the object or device to which the two-dimensional code is fixed can be realized.
  • the optical mark may also be another plane figure such as a checkerboard.
  • the use of two-dimensional codes or checkerboards as optical markings makes locating objects or instruments more accurate and fast. Thus, it is possible to more accurately navigate fast-moving equipment.
  • the optical mark fixed on the surface of the object and the instrument can also be a three-dimensional figure.
  • the second optical mark may be the handle of the device, or the second optical mark may be a structure fixed on the side of the handle.
  • the video captured by the video capture device 110 includes the first optical mark and the second optical mark, which, as described above, are respectively used to locate objects and instruments in the video.
  • the positioning device 120 is used to identify the first optical mark and the second optical mark in the video. That is, the first optical mark and the second optical mark are identified from each frame of the video.
  • the recognition operation can be based on existing mature image recognition algorithms, such as recognition methods based on texture features, frequency domain analysis, and machine learning.
  • the positioning device 120 is also used to anchor the three-dimensional image of the object to the corresponding position in the video captured by the video capture device 110 according to the recognized first optical mark and the first optical mark in the three-dimensional image of the object, and according to The identified second optical mark and the second optical mark in the three-dimensional image of the instrument anchor the three-dimensional image of the instrument to the corresponding position in the video.
  • the three-dimensional image of the object includes the characteristic points in the first optical mark.
  • the object is first scanned with CT, MRI, or ultrasound equipment to obtain a tomographic image of the object. Then the tomographic image is reconstructed to obtain the three-dimensional image.
  • the three-dimensional image is in the image coordinate system.
  • a logo picture or logo structure can be fixed at a specific position on the subject, and it can be scanned together with the subject.
  • the image material contains marking points that can be recognized by scanning equipment (such as CT, MRI, ultrasound equipment, etc.).
  • the way to obtain the three-dimensional image of the object is low in cost, easy to implement and high in accuracy.
  • a person of ordinary skill in the art can understand that the manner of obtaining a three-dimensional image of an object given in the above example is only for illustration and not for limitation.
  • a three-dimensional image of the object can be segmented from the original tomographic image data, and reconstructed to generate a three-dimensional image of the object.
  • the three-dimensional image of the object can three-dimensionally show the target element of the object and the characteristic points of the first optical marker.
  • Fig. 3 shows a three-dimensional image of an object according to an embodiment of the invention. The bone and liver of the object and the three characteristic points of the first optical marker are shown.
  • the three-dimensional image of the above-mentioned device includes a second optical mark.
  • the three-dimensional image of the device contains the spatial relationship between the second optical mark and the device itself.
  • Figure 4 shows a three-dimensional image of an instrument according to an embodiment of the invention.
  • the three-dimensional image of the device may be a three-dimensional image drawn by a mapping software for the device, a three-dimensional scanner scanning the device, or a three-dimensional image obtained through a reconstruction operation after a CT scan of the device.
  • the three-dimensional image of the object includes the first optical mark, and both the object and the first optical mark can be considered to be approximately rigid, so the positioning device 120 compares the first optical mark in the three-dimensional image of the object with the video The identified first optical marks are completely aligned, and the three-dimensional image of the object can be anchored to the corresponding position in the video frame collected by the video collection device 110.
  • the above-mentioned alignment operation may only use the characteristic points in the first optical mark. Multiple feature points can replace the entire first optical mark to complete the positioning of the object.
  • the use of the feature points to realize the positioning of the object is only for illustration, and other graphics such as straight lines on the first optical mark may also be used to realize the positioning of the object.
  • the three-dimensional image of the object may only include a part of the target element of the object, and the target element may be the human body tissue of the object, such as various organs, trachea, blood vessels, and bones, and may also be the focus of the object.
  • the three-dimensional image of the object is projected to a three-dimensional position in real space that is consistent with the object itself, so that the position of the three-dimensional image of the object and its video are matched.
  • the positioning device 120 is also used to anchor the three-dimensional image of the instrument to the corresponding position in the video according to the second optical identification identified and the second optical identification in the three-dimensional image of the instrument. .
  • the principle and detailed process will not be repeated here.
  • the display 130 is used to three-dimensionally display the part inside the target body and the part inside the target body of the instrument in the video.
  • FIG. 5 shows a frame in the video displayed by the display 130 according to an embodiment of the present invention.
  • the silhouette of the human body in Figure 5 is the content in the original frame of the video.
  • a three-dimensional image of the object is also displayed at the corresponding position of the human body. Only the target elements of the object that are of interest to the user are displayed, including bones, liver, and lesions.
  • instruments puncture needles
  • the user can "see” the real position of the three-dimensional organ extracted from the three-dimensional image in real time, and choose to avoid bones, large blood vessels and other important organs. Operation path.
  • the user can rely on the continuous recognition of the device’s second optical mark, and the display of the device’s three-dimensional image through the display can "see” the part hidden in the subject and the device’s
  • the direction of the extension line corresponding to the head ensures that the instrument can aim at the target at any time and proceed along the established route. All the anatomical parts, target areas, instruments, advancement path (surgical plan) and other prompt images or information of the subject can be displayed on the independent display 130 for the user to observe in real time.
  • the display 130 may be an ordinary display erected within the user's field of view.
  • the display 130 is a head-mounted display.
  • the head-mounted display is kept in the user's field of view at any time, so that it can focus on the object and the instrument, and the user does not need to frequently switch between the two actions of looking up at the display and looking down at the surgical instrument, reducing its operational risk .
  • the in-vivo navigation system When the in-vivo navigation system according to the present invention is used for operation, the user will not have equipment or consumables that can interfere with the line of sight within the operating space of the target body. Therefore, it not only effectively avoids the visual field problem of fixed infrared optical navigation, but also avoids The navigation hardware in the existing magnetic navigation system occupies the problem of operating space.
  • the above-mentioned in-vivo navigation system uses visible light to track optical signs, which greatly reduces the cost of consumables for navigation surgery, while ensuring that the user's desired target area is displayed three-dimensionally in the video, and the practicability of navigation is ensured.
  • the solution is simple to implement, does not change the user's traditional usage habits, and has low learning and promotion difficulties. Allows more operating objects to benefit from in vivo navigation technology.
  • the in-vivo navigation system further includes an input device for receiving user input.
  • the input device is, for example, a mouse, a keyboard, a voice control input device, etc.
  • the user can directly observe the overlap of the real target and the three-dimensional image of the target on the display 130 to confirm the positioning accuracy of the target, and at the same time can use the input device to input instructions.
  • the positioning device 120 is also used to adjust the corresponding position of the three-dimensional image of the target in the video according to the user's input received by the input device, so that the three-dimensional image of the target on the display 130 is translated or rotated to obtain more High-precision positioning effect.
  • the positioning device 120 can anchor the three-dimensional image of the object to the corresponding position in the video in the following manner.
  • S11 Determine a first conversion parameter for converting the three-dimensional image of the object to the user coordinate system based on the first optical mark recognized from the video and the characteristic points in the first optical mark in the three-dimensional image of the object.
  • the three-dimensional image of the object is in the image coordinate system
  • the video is in the user coordinate system.
  • the first conversion parameter can be used to unify the three-dimensional image and video of the object under the same coordinate system.
  • virtual objects can be presented at appropriate positions in the video frame in the video, so that the user seems to see the target elements in the virtual, three-dimensional object in the video, such as organs and bones.
  • the matching operation uses an iterative closest point algorithm to achieve registration, and uses a mean square error function to find the optimal solution, that is, to find the best matching result.
  • the formula (1) can be used to calculate the mean square error f(R, T) of the video frame and the three-dimensional image of the object in the video.
  • R 3d and T 3d represent the rotation matrix and the conversion matrix, respectively.
  • f(R, T) represents the mean square error of the three-dimensional image of the video frame and the object in the video
  • N represents the total number of feature points in the first optical mark
  • S12 Anchor the three-dimensional image of the object to the corresponding position in the video by using the determined first conversion parameter.
  • this operation can be realized using formula (2).
  • the user coordinates of the pixel can be calculated The coordinates X O , Y O and Z O under the system , that is, the position of the pixel in the video is obtained.
  • the three-dimensional image of the object is anchored to the video according to the characteristic points of the first optical mark, which has a small amount of calculation and good real-time performance.
  • Another embodiment of the present invention supports faster positional movement of the video capture device 110 relative to the object, that is, allows the user to move faster relative to the object.
  • the user may wear the video capture device 110 and manipulate the object in its comfortable and convenient position.
  • the positioning device 120 can anchor the three-dimensional image of the object to the corresponding position in the video in the following manner, which will be described in detail below with reference to FIG. 7.
  • S21 Obtain an image of an object that also includes the aforementioned first optical mark.
  • the image of the object is in the object coordinate system.
  • the image of the object may be an initial video frame in the video captured by the video capturing device 110.
  • the image of the object may also be collected by other devices other than the video collection device 110.
  • the second conversion parameter for converting the three-dimensional image of the object into the object coordinate system is determined according to the first optical mark in the image of the object and the characteristic points in the first optical mark in the three-dimensional image of the object.
  • the second conversion parameter can be used to unify the three-dimensional image of the object to the coordinate system where the image of the object is located, that is, the object coordinate system.
  • virtual and three-dimensional target elements such as organs, bones, etc., can be presented in a suitable position in the image of the object.
  • S22 Convert the three-dimensional image of the object into the three-dimensional image of the object in the object coordinate system by using the second conversion parameter.
  • the above two operations S21 and S22 are respectively similar to the above operations S11 and S12, and only the video frame of the video therein needs to be replaced with the image of the object in the operation. For the sake of brief description, it will not be repeated here.
  • X P , Y P and Z P respectively represent the X-axis, Y-axis and Z-axis coordinates of a feature point in the first optical mark in the three-dimensional image in the object coordinate system
  • X O , Y O and Z O respectively Represents the X-axis, Y-axis and Z-axis coordinates of the aforementioned feature points in the video frame of the video.
  • the above formula (3) can be used to determine the third conversion parameter: with
  • the image of the object in the object coordinate system involved in operation S21 may be a video frame captured at the first moment
  • the video frame of the video involved in operation S24 may be a video captured after the first moment.
  • Frame, through operation S21 and operation S22 the three-dimensional image of the object in the image coordinate system is converted to the three-dimensional image in the object coordinate system.
  • the three-dimensional image in the object coordinate system is converted into the three-dimensional image in the user coordinate system.
  • Fig. 8 shows the coordinate conversion process according to the above embodiment of the present invention.
  • real-time accurate tracking of the position of the object is achieved through two coordinate transformations, and the body position of the object and the user is no longer restricted, and the comfort of the object and the convenience of the user are improved.
  • the video capture device 110 includes a positioning sensor, which is used to detect the location of the video capture device 110 in real time.
  • the positioning sensor such as a gyroscope and an accelerometer, can record its own spatial displacement.
  • the positioning device 120 can anchor the three-dimensional image of the object to the corresponding position in the video in the following manner, which is described in detail below with reference to FIG. 9.
  • S31 Determine a fourth conversion parameter for converting the three-dimensional image into the object coordinate system according to the first optical mark in the image of the object and the characteristic points in the first optical mark in the three-dimensional image.
  • the image of the object is an image in the object coordinate system collected by the video collecting device 110.
  • the image of the object also includes the first optical mark.
  • the above two operations S31 and S32 are respectively similar to the above operations S11 and S12, and only the video frame of the video therein needs to be replaced with the image of the object in the operation. For the sake of brief description, it will not be repeated here.
  • S33 Determine a fifth conversion parameter for converting the three-dimensional image in the object coordinate system to the user coordinate system according to the detected position changes of the video capture device 110 at the first moment and the current moment.
  • the first moment is the moment when the video capture device 110 captures the image of the object.
  • the aforementioned video is in the user coordinate system, which is collected by the video collection device 110 at the current moment.
  • the position change of the video capture device 110 at the first moment and the current moment corresponds to the conversion between the object coordinate system and the user coordinate system.
  • the fifth conversion parameter used to convert the three-dimensional image in the object coordinate system to the user coordinate system the rotation matrix and the conversion matrix is determined according to the position changes of the video capture device at different times.
  • the fifth transformation parameter the rotation matrix and the transformation matrix can be used to anchor the three-dimensional image in the object coordinate system to the aforementioned video. This operation is similar to the above-mentioned operation S12. For brevity, it will not be repeated here.
  • the positioning sensor on the video capture device 110 is used to determine its position in real time, thereby realizing the tracking of the object in the video.
  • the user does not need to keep paying attention to the first optical mark of the object at any time, but to observe the target position in the object body more, thereby improving the user experience.
  • the above operations S21 to S24 and the above operations S31 to S34 may be performed in different time periods, respectively.
  • the object is tracked according to the first optical mark of the object and the position of the video capture device 110 detected by the positioning sensor in different time periods.
  • the positioning sensor has an error in detecting the position of the video capture device 110. If an object is tracked based on the detected position for a long time, the error may gradually increase. Using the above two methods successively can achieve more accurate tracking and positioning.
  • the positioning device 120 anchors the three-dimensional image of the instrument to the corresponding position in the video in the following manner. The process is described in detail below in conjunction with FIG. 10.
  • the three-dimensional image of the instrument includes the second optical mark.
  • the sixth conversion parameter used to convert the three-dimensional image of the instrument to the user coordinate system can be determined based on the second optical mark recognized from the video and the characteristic points of the second optical mark in the three-dimensional image.
  • the video is in the user coordinate system.
  • the above operations S41 and S42 are respectively similar to the above operations S11 and S12, and only the three-dimensional image of the object therein needs to be replaced with the three-dimensional image of the instrument in the operation. For the sake of brief description, it will not be repeated here.
  • the three-dimensional image of the instrument is anchored to the video according to the characteristic points of the second optical mark, with a small amount of calculation and good real-time performance.
  • an in-vivo navigation method includes:
  • Collect a video of a target in real time where the video includes an optical mark fixed on the target, and the collection angle of the video is consistent with the user's viewing direction, where the target includes an instrument and/or an object;
  • the part of the target in the subject is displayed in three dimensions.
  • the anchoring the three-dimensional image of the target to the corresponding position in the video according to the recognized optical mark and the optical mark in the three-dimensional image of the target further includes:
  • the identified optical mark and the characteristic points in the optical mark in the three-dimensional image determine the first conversion parameter for converting the three-dimensional image to the user coordinate system, wherein the video is the user In the coordinate system;
  • the three-dimensional image is anchored to the corresponding position in the video by using the first conversion parameter.
  • the anchoring the three-dimensional image of the target to the corresponding position in the video according to the identified optical mark and the optical mark in the three-dimensional image of the target further includes:
  • the optical mark in the image of the target and the characteristic points in the optical mark in the three-dimensional image determine the second conversion parameter for converting the three-dimensional image to the target coordinate system, wherein the image of the target Is in the target coordinate system, and the image of the target also includes the optical mark;
  • the identified optical mark and the characteristic points in the optical mark in the three-dimensional image in the target coordinate system determine a third transformation for converting the three-dimensional image in the target coordinate system to the user coordinate system Parameters, where the video is in the user coordinate system;
  • the third conversion parameter is used to anchor the three-dimensional image in the target coordinate system to a corresponding position in the video.
  • the method further includes: real-time detection of the position of a video acquisition device for real-time acquisition of the target;
  • the anchoring the three-dimensional image of the target to the corresponding position in the video according to the recognized optical mark and the optical mark in the three-dimensional image of the target further includes:
  • the fourth conversion parameter for converting the three-dimensional image to the target coordinate system is determined, wherein the image of the target Is the image in the target coordinate system collected by the video collecting device at the first moment;
  • a fifth conversion parameter for converting the three-dimensional image in the target coordinate system to the user coordinate system is determined, wherein the video is In the user coordinate system;
  • the fifth conversion parameter is used to anchor the three-dimensional image in the target coordinate system to the corresponding position in the video.
  • the method further includes:
  • the target in the above in-vivo navigation algorithm may be an object and/or a device.
  • the implementation and function of each device have been described in detail. Those skilled in the art can understand the specific steps and advantages of the in-vivo navigation method in conjunction with the above description of FIGS. 1 to 10 For the sake of brevity, this article will not go into details.
  • first”, “second”, etc. are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with “first”, “second”, etc. may explicitly or implicitly include one or more of these features.

Landscapes

  • Health & Medical Sciences (AREA)
  • Surgery (AREA)
  • Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Robotics (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Medical Informatics (AREA)
  • Molecular Biology (AREA)
  • Animal Behavior & Ethology (AREA)
  • General Health & Medical Sciences (AREA)
  • Public Health (AREA)
  • Veterinary Medicine (AREA)
  • Image Analysis (AREA)

Abstract

本发明提供一种体内导航系统和方法。该系统包括：视频采集装置，用于实时采集目标的视频，其中视频中包括目标上所固定的光学标识，视频采集装置采集视频的角度与用户的观察方向保持一致，其中，目标包括器械和/或对象；定位装置，用于识别视频中的光学标识，根据所识别的光学标识和目标的三维图像中的光学标识，将目标的三维图像锚定到视频中的对应位置；以及显示器，用于在视频中三维显示目标在对象体内的部分。上述方案不仅有效避免了固定式红外光学导航的视野问题，避免了磁导航系统中的导航装置占用手术空间，电磁场易受干扰影响定位精度的问题，同时可以大幅降低导航手术的耗材成本，提高导航的实用性。

Description

体内导航系统和方法 技术领域
本发明涉及计算机辅助医疗技术领域，具体地，涉及一种体内导航系统和方法。
背景技术
在医疗临床实践过程中，常需要将医疗器械探入人体内，进行某种操作。例如，可通过经皮体外穿刺来获取人体内病变组织的微量标本。这类操作中，医疗器械的准确导航是非常重要的。在如上示例中，基于影像的穿刺定位导航一直是临床工作中的重点需求。
现有的影像导航系统主要分为两类：基于红外反光点的光学导航系统和基于微线圈的电磁场导航系统(简称磁导航)。在实际使用中，这两种系统各有问题，如表1所示：
表1 现有的导航系统简介
Figure PCTCN2019077865-appb-000001
因此，需要提供一种体内导航系统和方法，以至少部分地解决现有技术中存在的上述问题。
发明内容
为了至少部分地解决现有技术中存在的问题，根据本发明的一个方面，提供一种体内导航系统，包括：
视频采集装置，用于实时采集目标的视频，其中所述视频中包括所述目标上所固定的光学标识，所述视频采集装置采集视频的角度与用户的观察方向保持一致，其中，所述目标包括器械和/或对象；
定位装置，用于识别所述视频中的所述光学标识，根据所识别的光学标识和所述目标的三维图像中的光学标识，将所述目标的三维图像锚定到所述视频中的对应位置；以及
显示器，用于在所述视频中三维显示所述目标在对象体内的部分。
示例性地，所述视频采集装置是头戴式光学摄像头。
示例性地，所述显示器是头戴式显示器。
示例性地，所述光学标识是二维码和/或棋盘格。
示例性地，所述光学标识是所述目标表面的立体图形。
示例性地，所述定位装置通过以下方式根据所识别的光学标识和所述目标的三维图像中的光学标识将所述目标的三维图像锚定到所述视频中的对应位置：
根据所述所识别的光学标识和所述三维图像中的光学标识中的特征点，确定用于将所述三维图像转换到用户坐标系下的第一转换参数，其中所述视频是所述用户坐标系下的；以及
利用所述第一转换参数将所述三维图像锚定到所述视频中的对应位置。
示例性地，所述定位装置通过以下方式根据所识别的光学标识和所述目标的三维图像中的光学标识将所述目标的三维图像锚定到所述视频中的对应位置：
根据所述目标的图像中的光学标识和所述三维图像中的光学标识中的特征点，确定用于将所述三维图像转换到目标坐标系下的第二转换参数，其中所述目标的图像是所述目标坐标系下的，并且所述目标的图像中也包 括所述光学标识；
利用所述第二转换参数将所述三维图像转换为所述目标坐标系下的三维图像；
根据所述所识别的光学标识和所述目标坐标系下的三维图像中的光学标识中的特征点，确定用于将所述目标坐标系下的三维图像转换到用户坐标系下的第三转换参数，其中所述视频是所述用户坐标系下的；以及
利用所述第三转换参数将所述目标坐标系下的三维图像锚定到所述视频中的对应位置。
示例性地，所述视频采集装置包括定位传感器，用于实时检测所述视频采集装置的位置；
所述定位装置通过以下方式将所述目标的三维图像锚定到所述视频中的对应位置：
根据所述目标的图像中的光学标识和所述三维图像中的光学标识中的特征点，确定用于将所述三维图像转换到目标坐标系下的第四转换参数，其中所述目标的图像是由所述视频采集装置在第一时刻采集的、所述目标坐标系下的图像；
利用所述第四转换参数将所述三维图像转换为所述目标坐标系下的三维图像；
根据所检测的所述视频采集装置在第一时刻和当前时刻的位置变化，确定用于将所述目标坐标系下的三维图像转换到用户坐标系下的第五转换参数，其中所述视频是所述用户坐标系下的；以及
利用所述第五转换参数将所述目标坐标系下的三维图像锚定到所述视频中的对应位置。
示例性地，所述体内导航系统还包括输入装置，用于接收所述用户的输入；
所述定位装置还用于根据所述输入调整所述目标的三维图像在所述视频中的对应位置。
示例性地，所述目标是对象，所述对象的三维图像是利用CT、MRI或超声设备对所述对象进行扫描后，经重建操作所获得的三维图像。
示例性地，所述目标是器械，所述器械的三维图像是通过制图软件针对所述器械绘制的、三维扫描仪对所述器械扫描测绘的或CT扫描后经重 建操作所获得的三维图像。
根据本发明另一方面，还提供了一种体内导航方法，包括：
实时采集目标的视频，其中所述视频中包括所述目标上所固定的光学标识，所述视频的采集角度与用户的观察方向保持一致，其中，所述目标包括器械和/或对象；
识别所述视频中的所述光学标识，根据所识别的光学标识和所述目标的三维图像中的光学标识，将所述目标的三维图像锚定到所述视频中的对应位置；以及
在所述视频中三维显示所述目标在对象体内的部分。
示例性地，所述根据所识别的光学标识和所述目标的三维图像中的光学标识，将所述目标的三维图像锚定到所述视频中的对应位置进一步包括：
根据所述所识别的光学标识和所述三维图像中的光学标识中的特征点，确定用于将所述三维图像转换到用户坐标系下的第一转换参数，其中所述视频是所述用户坐标系下的；以及
利用所述第一转换参数将所述三维图像锚定到所述视频中的对应位置。
示例性地，所述根据所识别的光学标识和所述目标的三维图像中的光学标识将所述目标的三维图像锚定到所述视频中的对应位置进一步包括：
根据所述目标的图像中的光学标识和所述三维图像中的光学标识中的特征点，确定用于将所述三维图像转换到目标坐标系下的第二转换参数，其中所述目标的图像是所述目标坐标系下的，并且所述目标的图像中也包括所述光学标识；
利用所述第二转换参数将所述三维图像转换为所述目标坐标系下的三维图像；
根据所述所识别的光学标识和所述目标坐标系下的三维图像中的光学标识中的特征点，确定用于将所述目标坐标系下的三维图像转换到用户坐标系下的第三转换参数，其中所述视频是所述用户坐标系下的；以及
利用所述第三转换参数将所述目标坐标系下的三维图像锚定到所述视频中的对应位置。
示例性地，所述方法还包括：实时检测用于实时采集所述目标的视频采集装置的位置；
所述根据所识别的光学标识和所述目标的三维图像中的光学标识将所述目标的三维图像锚定到所述视频中的对应位置进一步包括：
根据所述目标的图像中的光学标识和所述三维图像中的光学标识中的特征点，确定用于将所述三维图像转换到目标坐标系下的第四转换参数，其中所述目标的图像是由所述视频采集装置在第一时刻采集的、所述目标坐标系下的图像；
利用所述第四转换参数将所述三维图像转换为所述目标坐标系下的三维图像；
根据所检测的所述视频采集装置在第一时刻和当前时刻的位置变化，确定用于将所述目标坐标系下的三维图像转换到用户坐标系下的第五转换参数，其中所述视频是所述用户坐标系下的；以及
利用所述第五转换参数将所述目标坐标系下的三维图像锚定到所述视频中的对应位置。
示例性地，所述方法还包括：
接收所述用户的输入；
根据所述输入调整所述目标的三维图像在所述视频中的对应位置。
根据本发明提供的体内导航系统和方法，用户在对象体内的操作空间范围内不会出现可干扰视线的设备或耗材，因此不仅有效避免了固定式红外光学导航的视野问题，还避免了磁导航系统中的导航硬件占用手术空间，电磁场易受干扰影响定位精度的问题。上述体内导航系统使用可见光光学标识，在大幅降低导航手术的耗材成本的同时，保证了导航的实用性。同时方案实施简单，用户只需要时刻保持注意力在患者或手术器械上，不改变传统使用习惯，学习推广难度低。可以让更多操作对象受益于体内导航技术。
在发明内容中引入了一系列简化的概念，这些概念将在具体实施方式部分中进一步详细说明。本发明内容部分并不意味着要试图限定所要求保护的技术方案的关键特征和必要技术特征，更不意味着试图确定所要求保护的技术方案的保护范围。
以下结合附图，详细说明本发明的优点和特征。
附图说明
本发明的下列附图在此作为本发明的一部分用于理解本发明。附图中示出了本发明的实施方式及其描述，用来解释本发明的原理。在附图中，
图1示出根据本发明一个实施例的体内导航系统以及其工作环境的示意性框图；
图2示出了根据本发明一个实施例的对象和针对其进行操作的器械的示意图；
图3示出了根据本发明一个实施例的对象的三维图像；
图4示出了根据本发明一个实施例的器械的三维图像；
图5示出了根据本发明一个实施例的视频中的一帧；
图6示出了根据本发明一个实施例的定位装置将对象的三维图像锚定到视频中的对应位置的方式；
图7示出了根据本发明另一个实施例的定位装置将对象的三维图像锚定到视频中的对应位置的方式；
图8示出了图7的实施例中的坐标转换过程；
图9示出了根据本发明又一个实施例的定位装置将对象的三维图像锚定到视频中的对应位置的方式；以及
图10示出了根据本发明一个实施例的定位装置将器械的三维图像锚定到视频中的对应位置的方式。
具体实施方式
在下文的描述中，提供了大量的细节以便能够彻底地理解本发明。然而，本领域技术人员可以了解，如下描述仅涉及本发明的较佳实施例，本发明可以无需一个或多个这样的细节而得以实施。此外，为了避免与本发明发生混淆，对于本领域公知的一些技术特征未进行描述。
本申请所提供的体内导航系统用于为用户提供针对对象体内的组织和/或器械位于对象体内的部分的导航。其中，用户是整个体内导航过程的观察者，其也是将器械探入对象体内的操作者。对象可以是用户需要对其进行操作的人或其他动物。器械可以是任意可探入对象体内的工具。器械可以例如是穿刺针、活检针、射频或微波消融针、超声探头、硬质内窥镜、内窥镜手术下卵圆钳、电刀或吻合器等医疗器械。
在上述体内导航系统中，首先基于可见光影像来确定目标的位置，然 后在该可见光影像中三维地显示前述目标。示意性地，该目标可以是器械和/或对象。从而，将实际不可见的对象的体内器官、病变和/或医疗器械三维地显示给用户，以指引用户在真实环境操作器械。
根据本发明的一个方面，提供一种体内导航系统。该体内导航系统中的显示器显示给用户的视频中，能够在其中的对应位置显示对象和器械这二者，为用户提供了更全面的导航信息。本领域普通技术人员可以理解，也可以在其中只显示对象或器械，对于未显示的目标，凭用户的经验进行器械操作。根据以下描述，本领域普通技术人员能够理解，只显示对象或器械的实施例，为了简洁，在此不再赘述。
图1示出根据本发明一个实施例的体内导航系统以及其工作环境的示意性框图。图1所示的体内导航系统包括视频采集装置110、定位装置120和显示器130。该视频采集装置110用于实时采集对象以及器械的视频。用户可以观看显示器130，其用于显示视频，在该视频中不仅显示视频采集装置110所采集的对象和器械的表面部分，而且还在对应位置三维地显示实际不可见的对象的体内器官、病变以及器械在对象体内的部分。换言之，在视频中，将实际不可见的体内器官、病变以及器械位于体内的部分与人体及实际器械对准，从而指引用户在类似于真实环境的虚拟三维场景中操作器械。
视频采集装置110的采集角度与用户的观察方向保持一致。当用户使用该体内导航系统时，其可以将视频采集装置110佩戴于身体上，例如头部。可选地，视频采集装置是头戴式光学摄像头。在用户使用体内导航系统时，无论其采用何种姿势，都可以很好地保持头戴式光学摄像头的采集角度与其观察方向一致。由此，不仅保证了显示器130所显示的视频的角度是用户所观看的角度，保证了器械导航的精准度，而且避免了体内导航系统的使用对用户的各种操作的干扰。从而显著提高了用户体验。
在体内导航系统工作时，对象的表面固定有第一光学标识，并且器械的表面固定有第二光学标识。例如，对象的皮肤上可以黏贴有第一光学标识。在器械的、靠近手持的部分可以印有第二光学标识，该部分在体内导航系统工作时处于对象的体外。第一光学标识和第二光学标识都是在可见光图像中可识别的。
图2示出了根据本发明一个实施例的对象和针对其进行操作的器械。 如图2所示，对象和器械表面都具有光学标识，即图2中的二维码。二维码是在平面上分布的黑白相间的平面图形，其上面的点非常易于识别，通过识别其中的至少3个点，可以实现该二维码的定位。因为二维码固定于对象或器械，所以，可以实现固定有该二维码的对象或器械的定位。可选地，光学标识还可以是诸如棋盘格的其他平面图形。利用二维码或棋盘格作为光学标识，使得定位对象或器械更准确且快速。从而，可以对快速移动器械进行更精准地导航。
可选地，对象和器械表面上所固定的光学标识还可以是立体图形。例如，在器械设计生产过程中，第二光学标识可以是该器械的手柄，或者第二光学标识可以是固定于手柄侧面的某个结构。使用立体图形进行空间定位虽然识别所需的计算时间相对平面图形长，但对固定不动或慢速移动的目标空间定位精度较高。
在视频采集装置110所采集的视频中包括该第一光学标识和该第二光学标识，如前所述，其分别用于对视频中的对象和器械进行定位。
定位装置120用于识别视频中的第一光学标识和第二光学标识。即从视频的各个帧中识别其中的第一光学标识和第二光学标识。该识别操作可以基于现有成熟的图像识别算法，例如基于纹理特征、频域分析和机器学习等识别方式。
定位装置120还用于根据所识别的第一光学标识和对象的三维图像中的第一光学标识，将该对象的三维图像锚定到视频采集装置110所采集的视频中的对应位置，并且根据所识别的第二光学标识和器械的三维图像中的第二光学标识，将该器械的三维图像锚定到该视频中的对应位置。
上述对象的三维图像中包括第一光学标识中的特征点。示例性地，首先利用CT、MRI或超声设备等对对象进行扫描，以获得对象的断层图像。然后对该断层图像进行重建，以获得该三维图像。该三维图像是影像坐标系下的。可以在扫描前，在对象身上的特定位置固定一个标识图片或标识结构，其与对象一同接受扫描。图片材质包含可以被扫描设备(如CT、MRI、超声设备等)识别的标识点。获得该对象的三维图像的方式成本低、易实现而且准确度高。本领域普通技术人员可以理解，上述示例给出的获得对象的三维图像的方式仅为示意，而非限制。
为了更好地为用户呈现对象体内的其感兴趣的目标元素，例如骨骼、血 管、脏器及病变目标等，避免其他无关元素的干扰，在获得包含第一光学标识的特征点的断层图像后，可利用常规图像后处理方法，从断层图像原始数据中分割出特征点以及对象的特定的解剖结构，例如骨骼、血管、脏器及病变等目标元素，并且经重建来生成对象的三维图像。该对象的三维图像中可以立体地示出对象的目标元素和第一光学标识的特征点。图3示出了根据本发明一个实施例的对象的三维图像。其中示出了对象的骨骼和肝脏以及第一光学标识的三个特征点。
上述器械的三维图像中包括第二光学标识。器械的三维图像中包含第二光学标识与器械本身的空间关系。图4示出了根据本发明一个实施例的器械的三维图像。示例性地，器械的三维图像可以是通过制图软件针对器械绘制的、三维扫描仪对器械进行扫描所测绘的或对器械进行CT扫描后经重建操作所获得的三维图像。
如前所述，对象的三维图像中包括第一光学标识，而且对象和第一光学标识都可以认为是近似刚性的，所以定位装置120将对象的三维图像中的第一光学标识与从视频中识别的第一光学标识完全对齐，即可将该对象的三维图像锚定到视频采集装置110所采集的视频帧中的对应位置。
可以理解，上述对齐操作，可以仅利用第一光学标识中的特征点。多个特征点可以替代整个第一光学标识，来完成对对象的定位。上述利用特征点来实现对象的定位仅为示意，还可以利用第一光学标识上的直线等其他图形来实现对象的定位。
可以理解，该对象的三维图像可以仅包括对象的一部分目标元素，该目标元素可以是对象的人体组织，例如各种脏器、气管、血管和骨骼等，还可以是对象的病灶部位。由此，将对象的三维图像投影至与对象自身一致的现实空间三维位置，实现了对象的三维图像与其视频的位置匹配。
与对象的前述定位方式类似地，定位装置120还用于根据所识别的第二光学标识和器械的三维图像中的第二光学标识，将该器械的三维图像锚定到该视频中的对应位置。为了简洁，其原理和详细过程在此不再赘述。
显示器130用于在视频中三维显示对象体内的部分和器械在对象体内的部分。图5示出了根据本发明一个实施例的显示器130所显示的视频中的一帧。图5中的人体轮廓是视频的原始帧中的内容。在该原始帧的基础上，还在人体的对应位置处显示了对象的三维图像。其中仅显示了用户感 兴趣的、对象的目标元素，包括骨骼、肝脏以及病灶部位等。对于器械(穿刺针)也是类似的，不仅显示了视频的原始帧中的真实器械本身，即对象体外的部分，还在器械的对应位置处虚拟了器械在对象体内的部分。
因为在视频中，对象的虚拟的三维图像已与真实对象配准，用户可实时“看”到从三维图像中提取的三维器官在体内的真实位置，选择避开骨骼，大血管等重要器官的操作路径。当器械的头部插入对象体内，无法看到时，用户可依靠持续识别器械的第二光学标识，通过显示器对器械的三维图像的显示，“看”到隐藏在对象体内的部分，以及器械的头部对应的延长线方向，确保器械随时瞄准目标，沿既定路线前进。对象的所有解剖部位、目标区域，器械，前进路径(手术计划)等提示图像或信息，可显示于独立的显示器130中，供用户实时观察。
显示器130可以是在用户视野范围内竖立的普通显示器。可选地，显示器130是头戴式显示器。在用户使用体内导航系统时，头戴式显示器随时保持于用户的视野，便于其聚焦到对象和器械，不用用户频繁在抬头观察显示器与低头看手术器械这两个动作中切换，降低其操作风险。
利用根据本发明的上述体内导航系统进行操作时，用户在对象体内的操作空间范围内不会出现可干扰视线的设备或耗材，因此不仅有效避免了固定式红外光学导航的视野问题，还避免了现有的磁导航系统中的导航硬件占用手术空间的问题。上述体内导航系统使用可见光追踪光学标识，在大幅降低导航手术的耗材成本的同时，确保了视频中三维地显示用户的期望目标区域，保证了导航的实用性。同时方案实施简单，不改变用户传统使用习惯，学习推广难度低。可以让更多操作对象受益于体内导航技术。
示例性地，体内导航系统还包括输入装置，用于接收用户的输入。该输入装置例如鼠标、键盘、声控输入装置等。用户可通过直接观察显示器130上真实目标与目标的三维图像的重叠情况，确认目标的定位精度，同时可以利用输入装置输入指令。在该示例中，定位装置120还用于根据输入装置所接收的用户的输入调整目标的三维图像在视频中的对应位置，从而使得显示器130上的目标的三维图像发生平移或旋转，以获得更高精度的定位效果。
根据本发明的一个实施例，如图6所示，定位装置120可以通过以下方式将对象的三维图像锚定到视频中的对应位置。
S11，根据从视频中所识别的第一光学标识和对象的三维图像中的第 一光学标识中的特征点，确定用于将对象的三维图像转换到用户坐标系下的第一转换参数。如前所述对象的三维图像是影像坐标系下的，视频是用户坐标系下的。该第一转换参数可以用于将对象的三维图像和视频统一到同一坐标系下。由此，可以在视频中视频帧的合适的位置呈现虚拟的对象，使得用户看起来好像看到了视频中虚拟的、立体的对象体内的目标元素，例如脏器、骨骼等。
对视频中的第一光学标识与对象的三维图像中的第一光学标识进行匹配。示例性地，该匹配操作使用迭代最近点算法实现配准，并通过均方误差函数来求取最优解，即寻找最佳匹配结果。可以利用公式(1)计算视频中视频帧和对象的三维图像的均方误差f(R,T)。当均方误差f(R,T)小于某一特定阈值时即可认为获得了期望的第一转换参数R 3d和T 3d。其中R 3d和T 3d分别表示旋转矩阵和转换矩阵。
Figure PCTCN2019077865-appb-000002
其中f(R,T)表示视频中视频帧和对象的三维图像的均方误差，N表示第一光学标识中的特征点的总数，
Figure PCTCN2019077865-appb-000003
和
Figure PCTCN2019077865-appb-000004
分别表示对象的三维图像中的第一光学标识中的特征点和视频帧中的第一光学标识的对应特征点。
S12，利用所确定的第一转换参数将对象的三维图像锚定到视频中的对应位置。可选地，利用公式(2)实现该操作。对于对象的三维图像中每个像素点，基于其在三维图像中的坐标X P、Y P和Z P以及所确定的第一转换参数R 3d和T 3d，即可计算该像素点在用户坐标系下的坐标X O、Y O和Z O，即获得了该像素点在视频中的位置。
Figure PCTCN2019077865-appb-000005
上述实施例中，根据第一光学标识的特征点将对象的三维图像锚定到视频中，计算量小，实时性好。
本发明的另一实施例支持视频采集装置110相对于对象的较快速的位置移动，即允许用户相对于对象的较快速移动。例如，用户可以佩戴视频采集装置110，以其舒适、方便的体位操作对象。在该实施例中，定位装置120可以通过以下方式将对象的三维图像锚定到视频中的对应位置，下面结合图7进行详细描述。
S21，获得也包括前述第一光学标识的对象的图像。设该对象的图像 是对象坐标系下的。可以理解，该对象的图像可以是利用视频采集装置110所采集的视频中的一个初始视频帧。本领域普通技术人员可以理解，该对象的图像也可以是视频采集装置110以外的其他装置所采集的。并且，根据该对象的图像中的第一光学标识和对象的三维图像中的第一光学标识中的特征点，确定用于将对象的三维图像转换到对象坐标系下的第二转换参数。由此，第二转换参数可以用于将对象的三维图像统一到对象的图像所在的坐标系下，即对象坐标系。由此，可以在对象的图像中的合适位置呈现虚拟的、立体的对象体内的目标元素，例如脏器、骨骼等。
S22，利用第二转换参数将对象的三维图像转换为对象坐标系下的对象的三维图像。
上述两个操作S21和S22分别与如上操作S11和S12类似，只需将其中的视频的视频帧更换为本操作中的对象的图像，为了简介，在此不再赘述。
S23，根据从视频中识别的第一光学标识和所述对象坐标系下的三维图像中的第一光学标识，确定用于将对象坐标系下的三维图像转换到用户坐标系下的第三转换参数，其中前述视频是该用户坐标系下的。
可选地，利用如下公式(3)确定第三转换参数：
Figure PCTCN2019077865-appb-000006
和
Figure PCTCN2019077865-appb-000007
其中
Figure PCTCN2019077865-appb-000008
和
Figure PCTCN2019077865-appb-000009
分别表示旋转矩阵和转换矩阵。
Figure PCTCN2019077865-appb-000010
其中，X P、Y P和Z P分别表示对象坐标系下的三维图像中的第一光学标识中的一个特征点的X轴、Y轴和Z轴坐标，X O、Y O和Z O分别表示前述特征点在视频的视频帧中的X轴、Y轴和Z轴坐标。
根据已知的多个特征点的坐标，利用如上公式(3)即可以确定第三转换参数：
Figure PCTCN2019077865-appb-000011
和
Figure PCTCN2019077865-appb-000012
S24，利用第三转换参数将对象坐标系下的三维图像锚定到所述视频中的对应位置。该操作与如上操作S12类似，只需将其中的影像坐标系下的三维图像更换为本操作中的对象坐标系下的三维图像，为了简介，在此不再赘述。
在该实施例中，操作S21中所涉及的对象坐标系下的对象的图像可以是在第一时刻采集的视频帧，操作S24所涉及的视频的视频帧可以是在第一时刻之后采集的视频帧，通过操作S21和操作S22，将影像坐标系下的对象的三维图像转换为对象坐标系下的三维图像。然后根据对象坐标系和 用户坐标系的转换，其对应于用户在前后时刻的观察位置的变化，将对象坐标系下的三维图像转换为用户坐标系下的三维图像。图8示出了根据本发明上述实施例的坐标转换过程。该实施例中，通过两次坐标变换实现了对对象位置的实时精准追踪，不再对对象和用户的体位做限制，提高了对象的舒适度和用户的方便性。
本发明的又一实施例也支持用户相对于对象的较快速位置移动，即允许视频采集装置110相对于对象的较快速移动。在该实施例中，视频采集装置110包括定位传感器，其用于实时检测视频采集装置110的位置。该定位传感器例如陀螺仪和加速度计等，其能够记录自身的空间位移。定位装置120可以通过以下方式将对象的三维图像锚定到视频中的对应位置，下面结合图9进行详细描述。
S31，根据对象的图像中的第一光学标识和三维图像中的第一光学标识中的特征点，确定用于将三维图像转换到对象坐标系下的第四转换参数。所述对象的图像是由视频采集装置110采集的、对象坐标系下的图像。所述对象的图像中也包括第一光学标识。
S32，利用所述第四转换参数将所述三维图像转换为所述对象坐标系下的三维图像。
上述两个操作S31和S32分别与如上操作S11和S12类似，只需将其中的视频的视频帧更换为本操作中的对象的图像，为了简介，在此不再赘述。
S33，根据所检测的视频采集装置110在第一时刻和当前时刻的位置变化，确定用于将对象坐标系下的三维图像转换到用户坐标系下的第五转换参数。所述第一时刻是视频采集装置110采集所述对象的图像的时刻。前述视频是该用户坐标系下的，其是视频采集装置110在当前时刻采集的。视频采集装置110在第一时刻和当前时刻的位置变化对应于对象坐标系和用户坐标系的转换。在一个示例中，根据视频采集装置在不同时刻的位置变化确定用于将对象坐标系下的三维图像转换到用户坐标系下的第五转化参数：旋转矩阵和转换矩阵。
S34，利用第五转换参数将对象坐标系下的三维图像锚定到前述视频中的对应位置。在如上示例中，可以利用第五转换参数：旋转矩阵和转换矩阵，将对象坐标系下的三维图像锚定到前述视频中。该操作与如上所述的操作S12类似，为了简洁，在此不再赘述。
该实施例中，利用视频采集装置110上的定位传感器来实时确定其位置，从而实现对视频中对象的追踪。用户可不必随时保持注意力在对象的第一光学标识附近，而多去观察对象体内的目标位置，从而提高了用户体验。
根据本发明再一个实施例，可以在不同时间段分别执行上述操作S21至S24和上述操作S31至S34。换言之，在不同时间段分别根据对象的第一光学标识和定位传感器检测的视频采集装置110的位置来进行对象的追踪。定位传感器对于视频采集装置110的位置的检测有误差，如果长时间基于所检测的位置进行对象追踪，误差有可能逐渐累积增加。先后采用上述两种方式可以实现更准确的追踪定位。
根据本发明的一个实施例，所述定位装置120通过以下方式将器械的三维图像锚定到视频中的对应位置。下面结合图10详细描述该过程。
S41，如前所述，器械的三维图像中包括第二光学标识。可以根据从视频中识别的第二光学标识和该三维图像中的第二光学标识的特征点，确定用于将器械的三维图像转换到用户坐标系下的第六转换参数。视频是所述用户坐标系下的。
S42，利用所述第六转换参数将器械的三维图像锚定到视频中的对应位置。
上述操作S41和S42分别与如上操作S11和S12类似，只需将其中的对象的三维图像更换为本操作中的器械的三维图像，为了简介，在此不再赘述。根据第二光学标识的特征点将器械的三维图像锚定到视频中，计算量小，实时性好。
根据本发明另一方面，还提供了一种体内导航方法。该体内导航方法包括：
实时采集目标的视频，其中所述视频中包括所述目标上所固定的光学标识，所述视频的采集角度与用户的观察方向保持一致，其中，所述目标包括器械和/或对象；
识别所述视频中的所述光学标识，根据所识别的光学标识和所述目标的三维图像中的光学标识，将所述目标的三维图像锚定到所述视频中的对应位置；以及
在所述视频中三维显示所述目标在对象体内的部分。
示例性地，所述根据所识别的光学标识和所述目标的三维图像中的光 学标识，将所述目标的三维图像锚定到所述视频中的对应位置进一步包括：
根据所述所识别的光学标识和所述三维图像中的光学标识中的特征点，确定用于将所述三维图像转换到用户坐标系下的第一转换参数，其中所述视频是所述用户坐标系下的；以及
利用所述第一转换参数将所述三维图像锚定到所述视频中的对应位置。
示例性地，所述根据所识别的光学标识和所述目标的三维图像中的光学标识将所述目标的三维图像锚定到所述视频中的对应位置进一步包括：
根据所述目标的图像中的光学标识和所述三维图像中的光学标识中的特征点，确定用于将所述三维图像转换到目标坐标系下的第二转换参数，其中所述目标的图像是所述目标坐标系下的，并且所述目标的图像中也包括所述光学标识；
利用所述第二转换参数将所述三维图像转换为所述目标坐标系下的三维图像；
根据所述所识别的光学标识和所述目标坐标系下的三维图像中的光学标识中的特征点，确定用于将所述目标坐标系下的三维图像转换到用户坐标系下的第三转换参数，其中所述视频是所述用户坐标系下的；以及
利用所述第三转换参数将所述目标坐标系下的三维图像锚定到所述视频中的对应位置。
示例性地，所述方法还包括：实时检测用于实时采集所述目标的视频采集装置的位置；
所述根据所识别的光学标识和所述目标的三维图像中的光学标识将所述目标的三维图像锚定到所述视频中的对应位置进一步包括：
根据所述目标的图像中的光学标识和所述三维图像中的光学标识中的特征点，确定用于将所述三维图像转换到目标坐标系下的第四转换参数，其中所述目标的图像是由所述视频采集装置在第一时刻采集的、所述目标坐标系下的图像；
利用所述第四转换参数将所述三维图像转换为所述目标坐标系下的三维图像；
根据所检测的所述视频采集装置在第一时刻和当前时刻的位置变化，确定用于将所述目标坐标系下的三维图像转换到用户坐标系下的第 五转换参数，其中所述视频是所述用户坐标系下的；以及
利用所述第五转换参数将所述目标坐标系下的三维图像锚定到所述视频中的对应位置。
示例性地，所述方法还包括：
接收所述用户的输入；
根据所述输入调整所述目标的三维图像在所述视频中的对应位置。
可以理解，上述体内导航算法中的目标可以对象和/或器械。在以上关于体内导航系统的描述中，已经详细描述了其中各装置的实施方式和功能作用等，本领域技术人员结合以上关于图1至图10的描述能够理解体内导航方法的具体步骤及其优点等，为了简洁，本文不对此进行赘述。
在本发明的描述中，术语“第一”、“第二”等仅用于描述目的，而不能理解为指示或暗示相对重要性或者隐含指明所指示的技术特征的数量。由此，限定有“第一”、“第二”等的特征可以明示或者隐含地包括一个或者更多个该特征。
本发明已经通过上述实施例进行了说明，但应当理解的是，上述实施例只是用于举例和说明的目的，而非意在将本发明限制于所描述的实施例范围内。此外本领域技术人员可以理解的是，本发明并不局限于上述实施例，根据本发明的教导还可以做出更多种的变型和修改，这些变型和修改均落在本发明所要求保护的范围以内。本发明的保护范围由附属的权利要求书及其等效范围所界定。

Claims (16)

  1. 一种体内导航系统，包括：
    视频采集装置，用于实时采集目标的视频，其中所述视频中包括所述目标上所固定的光学标识，所述视频采集装置采集视频的角度与用户的观察方向保持一致，其中，所述目标包括器械和/或对象；
    定位装置，用于识别所述视频中的所述光学标识，根据所识别的光学标识和所述目标的三维图像中的光学标识，将所述目标的三维图像锚定到所述视频中的对应位置；以及
    显示器，用于在所述视频中三维显示所述目标在对象体内的部分。
  2. 如权利要求1所述的体内导航系统，其中，所述视频采集装置是头戴式光学摄像头。
  3. 如权利要求1或2所述的体内导航系统，其中，所述显示器是头戴式显示器。
  4. 如权利要求1或2所述的体内导航系统，其中，所述光学标识是二维码和/或棋盘格。
  5. 如权利要求1或2所述的体内导航系统，其中，所述光学标识是所述目标表面的立体图形。
  6. 如权利要求1或2所述的体内导航系统，其中，所述定位装置通过以下方式根据所识别的光学标识和所述目标的三维图像中的光学标识将所述目标的三维图像锚定到所述视频中的对应位置：
    根据所述所识别的光学标识和所述三维图像中的光学标识中的特征点，确定用于将所述三维图像转换到用户坐标系下的第一转换参数，其中所述视频是所述用户坐标系下的；以及
    利用所述第一转换参数将所述三维图像锚定到所述视频中的对应位置。
  7. 如权利要求1或2所述的体内导航系统，其中，所述定位装置通过以下方式根据所识别的光学标识和所述目标的三维图像中的光学标识将所述目标的三维图像锚定到所述视频中的对应位置：
    根据所述目标的图像中的光学标识和所述三维图像中的光学标识中的特征点，确定用于将所述三维图像转换到目标坐标系下的第二转换参数， 其中所述目标的图像是所述目标坐标系下的，并且所述目标的图像中也包括所述光学标识；
    利用所述第二转换参数将所述三维图像转换为所述目标坐标系下的三维图像；
    根据所述所识别的光学标识和所述目标坐标系下的三维图像中的光学标识中的特征点，确定用于将所述目标坐标系下的三维图像转换到用户坐标系下的第三转换参数，其中所述视频是所述用户坐标系下的；以及
    利用所述第三转换参数将所述目标坐标系下的三维图像锚定到所述视频中的对应位置。
  8. 如权利要求1或2所述的体内导航系统，其中，所述视频采集装置包括定位传感器，用于实时检测所述视频采集装置的位置；
    所述定位装置通过以下方式将所述目标的三维图像锚定到所述视频中的对应位置：
    根据所述目标的图像中的光学标识和所述三维图像中的光学标识中的特征点，确定用于将所述三维图像转换到目标坐标系下的第四转换参数，其中所述目标的图像是由所述视频采集装置在第一时刻采集的、所述目标坐标系下的图像；
    利用所述第四转换参数将所述三维图像转换为所述目标坐标系下的三维图像；
    根据所检测的所述视频采集装置在第一时刻和当前时刻的位置变化，确定用于将所述目标坐标系下的三维图像转换到用户坐标系下的第五转换参数，其中所述视频是所述用户坐标系下的；以及
    利用所述第五转换参数将所述目标坐标系下的三维图像锚定到所述视频中的对应位置。
  9. 如权利要求1或2所述的体内导航系统，其中，还包括输入装置，用于接收所述用户的输入；
    所述定位装置还用于根据所述输入调整所述目标的三维图像在所述视频中的对应位置。
  10. 如权利要求1或2所述的体内导航系统，其中，所述目标是对象，所述对象的三维图像是利用CT、MRI或超声设备对所述对象进行扫描后，经重建操作所获得的三维图像。
  11. 如权利要求1或2所述的体内导航系统，其中，所述目标是器械，所述器械的三维图像是通过制图软件针对所述器械绘制的、三维扫描仪对所述器械扫描测绘的或CT扫描后经重建操作所获得的三维图像。
  12. 一种体内导航方法，包括：
    实时采集目标的视频，其中所述视频中包括所述目标上所固定的光学标识，所述视频的采集角度与用户的观察方向保持一致，其中，所述目标包括器械和/或对象；
    识别所述视频中的所述光学标识，根据所识别的光学标识和所述目标的三维图像中的光学标识，将所述目标的三维图像锚定到所述视频中的对应位置；以及
    在所述视频中三维显示所述目标在对象体内的部分。
  13. 如权利要求12所述的体内导航方法，所述根据所识别的光学标识和所述目标的三维图像中的光学标识，将所述目标的三维图像锚定到所述视频中的对应位置进一步包括：
    根据所述所识别的光学标识和所述三维图像中的光学标识中的特征点，确定用于将所述三维图像转换到用户坐标系下的第一转换参数，其中所述视频是所述用户坐标系下的；以及
    利用所述第一转换参数将所述三维图像锚定到所述视频中的对应位置。
  14. 如权利要求12所述的体内导航方法，所述根据所识别的光学标识和所述目标的三维图像中的光学标识将所述目标的三维图像锚定到所述视频中的对应位置进一步包括：
    根据所述目标的图像中的光学标识和所述三维图像中的光学标识中的特征点，确定用于将所述三维图像转换到目标坐标系下的第二转换参数，其中所述目标的图像是所述目标坐标系下的，并且所述目标的图像中也包括所述光学标识；
    利用所述第二转换参数将所述三维图像转换为所述目标坐标系下的三维图像；
    根据所述所识别的光学标识和所述目标坐标系下的三维图像中的光学标识中的特征点，确定用于将所述目标坐标系下的三维图像转换到用户坐标系下的第三转换参数，其中所述视频是所述用户坐标系下的；以及
    利用所述第三转换参数将所述目标坐标系下的三维图像锚定到所述视频中的对应位置。
  15. 如权利要求12所述的体内导航方法，其中，
    所述方法还包括：实时检测用于实时采集所述目标的视频采集装置的位置；
    所述根据所识别的光学标识和所述目标的三维图像中的光学标识将所述目标的三维图像锚定到所述视频中的对应位置进一步包括：
    根据所述目标的图像中的光学标识和所述三维图像中的光学标识中的特征点，确定用于将所述三维图像转换到目标坐标系下的第四转换参数，其中所述目标的图像是由所述视频采集装置在第一时刻采集的、所述目标坐标系下的图像；
    利用所述第四转换参数将所述三维图像转换为所述目标坐标系下的三维图像；
    根据所检测的所述视频采集装置在第一时刻和当前时刻的位置变化，确定用于将所述目标坐标系下的三维图像转换到用户坐标系下的第五转换参数，其中所述视频是所述用户坐标系下的；以及
    利用所述第五转换参数将所述目标坐标系下的三维图像锚定到所述视频中的对应位置。
  16. 如权利要求12所述的体内导航方法，其中，所述方法还包括：
    接收所述用户的输入；
    根据所述输入调整所述目标的三维图像在所述视频中的对应位置。
PCT/CN2019/077865 2019-03-12 2019-03-12 体内导航系统和方法 WO2020181498A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/CN2019/077865 WO2020181498A1 (zh) 2019-03-12 2019-03-12 体内导航系统和方法

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2019/077865 WO2020181498A1 (zh) 2019-03-12 2019-03-12 体内导航系统和方法

Publications (1)

Publication Number Publication Date
WO2020181498A1 true WO2020181498A1 (zh) 2020-09-17

Family

ID=72426314

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/077865 WO2020181498A1 (zh) 2019-03-12 2019-03-12 体内导航系统和方法

Country Status (1)

Country Link
WO (1) WO2020181498A1 (zh)

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106560163A (zh) * 2015-09-30 2017-04-12 合肥美亚光电技术股份有限公司 手术导航系统及手术导航系统的配准方法
US20170116729A1 (en) * 2015-04-17 2017-04-27 Clear Guide Medical, Inc. System and method for fused image based navigation with late marker placement
CN107088091A (zh) * 2017-06-08 2017-08-25 广州技特电子科技有限公司 一种辅助骨科手术的手术导航系统及导航方法
CN107536643A (zh) * 2017-08-18 2018-01-05 北京航空航天大学 一种前交叉韧带重建的增强现实手术导航系统
CN108601628A (zh) * 2015-11-23 2018-09-28 R.A.W.责任有限公司 将操作器械定位在患者身体内的导航、跟踪和引导系统
CN109833092A (zh) * 2017-11-29 2019-06-04 上海复拓知达医疗科技有限公司 体内导航系统和方法

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170116729A1 (en) * 2015-04-17 2017-04-27 Clear Guide Medical, Inc. System and method for fused image based navigation with late marker placement
CN106560163A (zh) * 2015-09-30 2017-04-12 合肥美亚光电技术股份有限公司 手术导航系统及手术导航系统的配准方法
CN108601628A (zh) * 2015-11-23 2018-09-28 R.A.W.责任有限公司 将操作器械定位在患者身体内的导航、跟踪和引导系统
CN107088091A (zh) * 2017-06-08 2017-08-25 广州技特电子科技有限公司 一种辅助骨科手术的手术导航系统及导航方法
CN107536643A (zh) * 2017-08-18 2018-01-05 北京航空航天大学 一种前交叉韧带重建的增强现实手术导航系统
CN109833092A (zh) * 2017-11-29 2019-06-04 上海复拓知达医疗科技有限公司 体内导航系统和方法

Similar Documents

Publication Publication Date Title
US11871913B2 (en) Computed tomography enhanced fluoroscopic system, device, and method of utilizing the same
US11896414B2 (en) System and method for pose estimation of an imaging device and for determining the location of a medical device with respect to a target
US11464578B2 (en) Systems, methods, apparatuses, and computer-readable media for image management in image-guided medical procedures
EP2637593B1 (en) Visualization of anatomical data by augmented reality
JP6348130B2 (ja) デュアルデータ同期を用いた解剖学的部位の位置の再特定
US10575755B2 (en) Computer-implemented technique for calculating a position of a surgical device
EP3081184A1 (en) System and method for fused image based navigation with late marker placement
US20070225553A1 (en) Systems and Methods for Intraoperative Targeting
CN109998678A (zh) 在医学规程期间使用增强现实辅助导航
US20080123910A1 (en) Method and system for providing accuracy evaluation of image guided surgery
CN108140242A (zh) 视频摄像机与医学成像的配准
US20080234570A1 (en) System For Guiding a Medical Instrument in a Patient Body
CN109833092A (zh) 体内导航系统和方法
JP2010274044A (ja) 手術支援装置、手術支援方法及び手術支援プログラム
CN112168346A (zh) 三维医学影像与患者实时重合的方法及手术辅助系统
EP3911262A1 (en) System and method for medical navigation
CN208017582U (zh) 计算机辅助微创手术装置
WO2020181498A1 (zh) 体内导航系统和方法
Meinzer et al. Computer-assisted soft tissue interventions

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19918624

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19918624

Country of ref document: EP

Kind code of ref document: A1