WO2022206408A1 - 增强现实信息显示装置、方法、可读存储介质及设备 - Google Patents

增强现实信息显示装置、方法、可读存储介质及设备 Download PDF

Info

Publication number
WO2022206408A1
WO2022206408A1 PCT/CN2022/081472 CN2022081472W WO2022206408A1 WO 2022206408 A1 WO2022206408 A1 WO 2022206408A1 CN 2022081472 W CN2022081472 W CN 2022081472W WO 2022206408 A1 WO2022206408 A1 WO 2022206408A1
Authority
WO
WIPO (PCT)
Prior art keywords
information
surgical
augmented reality
area
module
Prior art date
Application number
PCT/CN2022/081472
Other languages
English (en)
French (fr)
Inventor
孙非
朱奕
郭晓杰
崔芙粒
单莹
Original Assignee
上海复拓知达医疗科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 上海复拓知达医疗科技有限公司 filed Critical 上海复拓知达医疗科技有限公司
Publication of WO2022206408A1 publication Critical patent/WO2022206408A1/zh

Links

Images

Classifications

    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/10—Computer-aided planning, simulation or modelling of surgical operations
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/25—User interfaces for surgical systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/006—Mixed reality
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/10—Computer-aided planning, simulation or modelling of surgical operations
    • A61B2034/107—Visualisation of planned trajectories or target regions
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/10—Computer-aided planning, simulation or modelling of surgical operations
    • A61B2034/108—Computer aided selection or customisation of medical implants or cutting guides
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
    • A61B2034/2046—Tracking techniques
    • A61B2034/2065—Tracking using image or pattern recognition
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
    • A61B2034/2068—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis using pointers, e.g. pointers having reference marks for determining coordinates of body points
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/25—User interfaces for surgical systems
    • A61B2034/252—User interfaces for surgical systems indicating steps of a surgical procedure

Definitions

  • Embodiments of the present invention relate to the technical field of computer image processing, and in particular, to an augmented reality information display apparatus, method, readable storage medium, and device.
  • the operation steps of surgery are necessary knowledge for doctors to learn surgical skills.
  • the traditional surgical learning process requires repeated training such as simulated surgical training, on-site surgical teaching and independent practice, so that doctors can memorize and master the correct surgical procedures.
  • the whole learning process is complex and completely dependent on memory, and the implementation risk is high.
  • recitation of texts that rely on surgical steps is not enough to reflect the actual specific surgical steps related to space and instruments. During the operation (especially the more complicated operation), it will be difficult for the operator to perform the operation solely by relying on the memory of the operation procedure.
  • embodiments of the present invention provide an augmented reality information display apparatus, method, readable storage medium, and device, so as to reduce the difficulty of an operator's surgical operation process.
  • an augmented reality information display device includes:
  • the operation planning information acquisition module is used to acquire the operation planning information based on the virtual operation scene
  • the real image information acquisition module is used to acquire the real image information of the operation site
  • an operation area initialization module configured to obtain identifiable identification information set on the operation site to obtain initialized operation area position information
  • the prompt information module is configured to superimpose and display prompt information for guiding the surgical operator at the corresponding position of the surgical area according to the surgical planning information and the virtual surgical scene.
  • the operation area initialization module initializes the position information of the operation area based on the position of the identifiable identification information in the operation site as a reference.
  • the operation area initialization module includes:
  • the world coordinate acquisition module is used to acquire the world coordinates of the identifiable identification information in the operation site;
  • the operation area position determination module is configured to determine, according to a preset corresponding relationship, an area location that conforms to a specific positional relationship with the world coordinates of the identifiable identification information as the operation area.
  • the prompt information module includes:
  • the registration prompt module is used for superimposing and displaying prompt information for collecting the registration point position at the corresponding position of the operation area, so as to register the operation area with the virtual operation scene.
  • the registration prompting module includes:
  • a prompt information display module used for prompting the surgical operator to perform the registration point position collection step in the form of text, animation or pattern
  • the registration point position indication module is used for superimposing and displaying text, pattern or animation at the position of the registration point in the operation area to indicate the position of the registration point.
  • the prompt information module includes:
  • a direction adjustment prompt module is used to prompt the operator in the form of text, animation or pattern the direction that the camera should be adjusted when the surgical area exceeds the current field of view of the camera.
  • the prompt information module includes:
  • the operation step position prompting module is used for prompting the operation operator the position of the next operation step in the form of text, animation or pattern when it is recognized that the operation operator has completed the current operation step.
  • the operation step position prompting module includes:
  • a surgical instrument planning state information acquisition module used to acquire planning state information of the surgical instrument
  • the actual state information acquisition module of the surgical instrument is used to obtain the actual state information of the surgical instrument according to the obtained real image information of the operation site;
  • a comparison and judgment module configured to compare the planned state information with the actual state information, and if the planned state information is consistent with the actual state information, judge that the current operation step has been completed.
  • the planned state information includes at least one of a planned path, a planned stay time, a planned movement speed, and a planned angle relative to the skin;
  • the actual state information includes an actual route, an actual stay time, and an actual movement. At least one of speed and actual angle relative to the skin.
  • the augmented reality information display device further includes:
  • the actual position acquisition module of the current operating area is used to acquire the actual position information corresponding to the current operating area determined according to the real image information of the operating site acquired in the current operation step;
  • a position correction module configured to correct the initialized surgical area position information based on the actual position information
  • the operation step prompting module is used for prompting the operation operator where the next operation step is in the form of text, animation or pattern based on the corrected position information of the operation area.
  • the augmented reality information display device further includes:
  • An initialization operation area prompting module configured to prompt the operation operator to obtain the initialized operation area position information by using the recognizable identification information in the form of text, animation or pattern.
  • the identifiable identification information includes a characteristic pattern set on the operation site or a characteristic gesture made by the user.
  • a method for displaying augmented reality information includes:
  • prompt information for guiding the operation operator is superimposed and displayed at the corresponding position of the operation area.
  • the obtaining of the identifiable identification information set on the operation site to obtain the initialized operation area position information includes:
  • the position information of the operation area is initialized.
  • the initializing the location information of the operation area based on the location of the identifiable identification information in the operation site includes:
  • a region location that conforms to a specific location relationship with the world coordinates of the identifiable identification information is determined as the surgical region.
  • superimposing and displaying prompt information for guiding the operation operator at the corresponding position of the operation area includes:
  • the prompt information for performing registration point location acquisition is superimposed and displayed at the corresponding position of the operation area, so as to register the operation area with the virtual operation scene.
  • the prompt information for superimposing and displaying the registration point position collection at the corresponding position of the operation area includes:
  • Text, pattern or animation is superimposed and displayed at the position of the registration point in the operation area, indicating the position of the registration point.
  • superimposing and displaying prompt information for guiding the operation operator at the corresponding position of the operation area includes:
  • the operation operator is prompted in the form of text, animation or pattern in which direction the camera should be adjusted.
  • superimposing and displaying prompt information for guiding the operation operator at the corresponding position of the operation area includes:
  • the operation operator is prompted with the position of the next operation step in the form of text, animation or pattern.
  • prompting the operation operator to the location of the next operation step in the form of text, animation or pattern includes:
  • the planned state information is compared with the actual state information, and if the planned state information is consistent with the actual state information, it is determined that the current surgical step has been completed.
  • the planned state information includes at least one of a planned path, a planned stay time, a planned movement speed, and a planned angle relative to the skin;
  • the actual state information includes an actual route, an actual stay time, and an actual movement. At least one of speed and actual angle relative to the skin.
  • the augmented reality information display method further includes:
  • the operation operator is prompted with the position of the next operation step in the form of text, animation or pattern.
  • the method before the acquiring the identifiable identification information set on the operation site to acquire the initialized operation area position information, the method further includes:
  • the operation operator is prompted to obtain the initialized operation area position information by using the recognizable identification information in the form of text, animation or pattern.
  • the identifiable identification information includes a characteristic pattern set on the operation site or a characteristic gesture made by the user.
  • a readable storage medium has executable instructions thereon, and when the executable instructions are executed, causes a computer to execute the steps in the augmented reality information display method according to any one of the above .
  • an electronic device in another aspect of the embodiments of the present invention, includes a processor and a memory, the memory stores computer program instructions suitable for execution by the processor, and the computer program instructions are processed by the processor The steps in the augmented reality information display method described in any one of the above are executed when the server is running.
  • the augmented reality information display device and method based on the augmented reality technology, use a headset to display a virtual surgical scene, and use a camera to obtain real image information of the surgical site, and identify the identifiable identification information set on the surgical site, Obtain the initial position information of the surgical area, and then use the augmented reality technology to simulate the ability of three-dimensional space display, according to the surgical planning information and the virtual surgical scene, during the surgical process in real time in the corresponding position of the surgical area in the form of text, animation or pattern Overlays display prompts to guide the operator.
  • the surgical operator performs the surgical operation by obtaining the prompt information, which reduces the dependence on the mechanical memory of the surgical operation process and reduces the difficulty of the surgical operation process.
  • FIG. 1 is a schematic structural diagram of a module of an augmented reality information display device according to an embodiment of the present invention.
  • FIG. 2 is a schematic flowchart of a method for displaying augmented reality information according to an embodiment of the present invention.
  • FIG. 3 is another schematic flowchart of an augmented reality information display method according to an embodiment of the present invention.
  • FIG. 4 is a schematic diagram of an application scenario of the augmented reality information display method according to an embodiment of the present invention.
  • FIG. 5 is a schematic diagram of an exemplary structure of an electronic device according to an embodiment of the present invention.
  • Augmented Reality is a technical content that promotes the integration of real world information and virtual world information content.
  • the virtual information content can be effectively applied in the real world, and in this process, it can be perceived by human senses, so as to achieve a sensory experience beyond reality.
  • a device based on augmented reality technology generally includes a main computing device and a head mounted device 3; the main computing device runs augmented reality virtual software, and the head mounted device 3 is worn on the operator's head and is provided with a camera.
  • the head mounted device 3 is communicatively connected with the main computing device to transfer information and data to each other.
  • an augmented reality information display device is provided, as shown in FIG. 1 , including the following modules:
  • the operation planning information acquisition module is used to acquire the operation planning information formulated based on the virtual operation scene.
  • the virtual surgical scene includes a virtual three-dimensional model established based on things in the real surgical scene, such as a patient's anatomical structure model (obtained from a patient's preoperative image) and/or a surgical instrument model corresponding to the operation.
  • the virtual surgical scene usually runs on the main computing device, and transmits the data to the head mounted device 3 for display, so that the operator can see the displayed virtual image.
  • the real image information acquisition module is used to acquire the real image information of the operation site.
  • the real image information may be the real scene of the operation table area.
  • the real image information is obtained by continuously photographing the operation site in real time through the camera 2 provided on the head mounted device 3 worn on the operator's head.
  • the operation area initialization module is used to obtain the identifiable identification information 1 set on the operation site, so as to obtain the initialized operation area position information.
  • the identifiable identification information 1 is usually provided in an area of the operation site associated with the operation position, such as the side of the operation table 4 and the like.
  • the identifiable identification information 1 includes a characteristic pattern or a characteristic gesture made by the user, and the characteristic pattern can be a marker with a specific pattern, such as a barcode, a two-dimensional code, etc., and the camera 2 provided on the head mounted device 3 can be used. to be captured and identified by the system.
  • the surgical area refers to an area determined within the field of view of the camera 2 where a surgical operation is to be performed, and the operator can view the currently determined surgical area that is virtually displayed in the head mounted device 3 .
  • the prompt information module is used for superimposing and displaying prompt information for guiding the operation of the surgical operator at the corresponding position of the surgical area according to the surgical planning information and the virtual surgical scene.
  • the display mode of the prompt information may include superimposing and displaying text, animation or pattern at the corresponding position in the real scene, and may be supplemented by voice prompts.
  • the inventive concept of the augmented reality information display device of the embodiment of the present invention is to improve the imaging technology of the virtual scene, not to obtain a better or optimal surgical plan.
  • the technical solution of the embodiment of the present invention is essentially a computer image processing technology, and the improvement lies in the imaging technology itself.
  • the augmented reality information display device of the embodiment of the present invention is to reduce the degree of difficulty of the operator during the surgical operation and reduce the need for surgery. Dependence of process mechanical memory.
  • the augmented reality information display device of the embodiment of the present invention is based on augmented reality technology, uses the head mounted device 3 to display the human anatomical structure model and/or surgical instrument model corresponding to the operation, and uses the camera 2 to obtain the real image information of the operation site, and identifies the settings.
  • the identifiable identification information 1 at the surgical site to obtain the initialized surgical area position information, and then use the augmented reality technology to simulate the ability of three-dimensional space display, according to the surgical planning information and the virtual surgical scene, in the form of text, animation or pattern.
  • the corresponding position of the operation area is superimposed to display prompt information for guiding the operation operator.
  • the surgical operator performs the surgical operation by obtaining the prompt information, which reduces the dependence on the mechanical memory of the surgical operation process and reduces the difficulty of the surgical operation process.
  • the operation area initialization module initializes the position information of the operation area based on the position of the identifiable identification information 1 in the operation site as a reference. For example, set the identifiable identification information 1 such as a two-dimensional code on a sign, and set the sign on one side of the operating table 4, and the position of the sign is determined with the position of the operating table 4, then the camera 2 scans the After the two-dimensional code information on the sign, the information is transmitted to the host computer, and the host computer determines a specific area on the operating table 4 as the operating area based on the location of the sign. Wherein, the relationship between the determined position of the surgical area and the position of the sign is set in the host computing device in advance.
  • the operation area initialization module includes:
  • the world coordinate acquisition module is used to acquire the world coordinate of the identifiable identification information 1 in the operation site.
  • the world coordinate system is the absolute coordinate system of the system, and the coordinates of all points are determined by the origin (absolute reference point) of the coordinate system. That is to say, since the camera 2 of the head mounted device 3 can be in any position in the environment, a reference coordinate system is selected in the environment to describe the position of the camera, and it is used to describe the position of any object in the environment, this coordinate system is called the world Coordinate System.
  • the operation area position determination module is configured to determine, according to a preset corresponding relationship, an area location that conforms to a specific positional relationship with the world coordinates of the identifiable identification information 1 as an operation area. After the world coordinate of the identifiable identification information 1 is determined, since its positional relationship with the operating table 4 is fixed and the corresponding relationship between the two is set in the main computing device in advance, the main computing device is based on the world coordinate of the identifiable identification information 1 and the As for the correspondence between the positions of the operating table 4, an area on the operating table 4 that has a specific positional relationship with the position of the recognizable identification information 1 is determined as an operating area. The specific positional relationship is set in the main computing device in advance.
  • a rectangular area behind the position of the identifiable identification information 1 can be set as the operation area in advance, and the main computing device obtains the world coordinates of the identifiable identification information 1 in advance. Then, according to the preset specific position relationship, a rectangular area behind the position of the recognizable identification information 1 is determined as the current surgical area, and the initialization of the surgical area is completed.
  • the prompt information module includes:
  • the registration prompt module is used to superimpose and display prompt information for collecting the registration point position at the corresponding position of the operation area, so as to register the operation area with the virtual operation scene. That is to say, by displaying the prompt information for performing registration point location acquisition, the operator is guided to perform the registration point location acquisition steps.
  • the purpose of the registration point location acquisition is to register the surgical area with the virtual surgical scene, that is, to make the two benchmarks consistent.
  • the registration prompt module includes:
  • the prompt information display module is used to prompt the operator to collect the registration point position in the form of text, animation or pattern.
  • the registration point position indication module is used for superimposing and displaying text, pattern or animation at the position of the registration point in the operation area to indicate the position of the registration point.
  • the location of the registration point to be acquired is indicated, for example, by a static or dynamic arrow image displayed virtually.
  • the prompt information module includes:
  • the direction adjustment prompt module is used to prompt the operator in the form of text, animation or pattern the direction that the camera 2 should adjust when the surgical area exceeds the current field of view of the camera 2 .
  • the current field of view of the camera 2 is constantly changing, and the determined position of the surgical area is relatively fixed.
  • the field of view is inconsistent with the determined surgical area, and the direction that the camera 2 should be adjusted is indicated in the form of text, animation or pattern.
  • the direction in which the camera 2 should be adjusted is indicated by a static or dynamic arrow image displayed virtually.
  • the prompt information module includes:
  • the operation step position prompting module is used for prompting the operation operator the position of the next operation step in the form of text, animation or pattern when it is recognized that the operation operator has completed the current operation step.
  • the operation step position prompt module includes:
  • the surgical instrument planning state information acquisition module is used to acquire the planning state information of the surgical instrument; the planning state information of the surgical instrument is preset in the main computing device.
  • the actual state information acquisition module of the surgical instrument is used to obtain the actual state information of the surgical instrument according to the obtained real image information of the surgical site; the surgical instrument is preset with an identifiable mark, and the state of the surgical instrument can be identified through the camera 2 information.
  • the comparison and judgment module is used to compare the planned state information with the actual state information, and if the planned state information is consistent with the actual state information, it is judged that the current operation step has been completed. If the planned state information is inconsistent with the actual state information, it is determined that the current surgical step has not been completed.
  • the planned state information may include at least one of the planned path, the planned stay time, the planned movement speed, and the planned angle relative to the skin; the actual state information may include the actual route, the actual stay time, the actual movement speed, and the actual angle relative to the skin. at least one.
  • the planned state information and the actual state information may adopt various combinations of the above-mentioned different forms, but the types of the combination of the two need to correspond to each other. That is, when the planned state information is the planned path, the actual state information should be the corresponding actual path. When the planned state information is the planned movement speed and the relative skin planning angle, the actual state information should be the corresponding actual movement speed and the relative skin actual angle.
  • the planned path is the preset movement path of the surgical instrument
  • the relative skin planning angle is the preset inclination angle of the surgical instrument relative to the skin; the same movement path of the surgical instrument and different inclination angles are two different states.
  • the actual path of the movement of the surgical instrument is identified by the camera 2, transmitted to the main computing device and recorded.
  • the main computing device compares the actual path with the planned path. If the actual path is consistent with the planned path, it is determined that the current operation has been completed. step. Or, if the planned stay time of the surgical instrument in a certain area is 1s, and the actual stay time of the surgical instrument in the area is more than 1s recognized by the camera 2, it is determined that the current surgical step has been completed.
  • the parameters of the movement speed of the surgical instrument and the relative skin angle are sometimes required, and the main computing device records and compares whether the planned movement speed and the relative skin planned angle are consistent with the actual movement speed and relative skin angle. If the difference is within a predetermined range, it is judged that the current surgical step has been completed.
  • the augmented reality information display device further includes:
  • the actual position acquisition module of the current operation area is used to acquire the actual position information corresponding to the current operation area determined according to the real image information of the operation site acquired in the current operation step.
  • the actual position information corresponding to the determined operation area may be obtained based on the actual position of the identified and recorded surgical instrument. After the operation area is initialized, a corresponding operation area is determined as the operation area of the current operation step for each operation step.
  • the position correction module is used for correcting the initialized surgical area position information based on the actual position information.
  • the currently determined surgical area is compared with the actual position, and the currently determined surgical area is adjusted to be consistent with the actual position.
  • the operation step prompting module is used for prompting the operator of the position of the next operation step in the form of text, animation or pattern based on the corrected position information of the operation area. That is to say, the suggested position of the next surgical step is performed based on the corrected position information of the surgical area, which avoids the accumulation of position errors after performing multiple surgical steps.
  • the augmented reality information display device further includes;
  • the initializing operation area prompting module is used to prompt the operation operator to obtain the initialized operation area position information by using the recognizable identification information 1 in the form of text, animation or pattern.
  • An identification board is placed on one side of the operating table 4 (the identification board has identifiable identification information 1 such as a two-dimensional code), and the position of the operating table 4 area behind the identification board is preset in the host computing device as the initial position Determine the location of the surgical field.
  • the operator is instructed to scan the two-dimensional code on the identification board with the camera 2 through the prompt message to initialize the location information of the surgical area.
  • the position where a certain spatial range on the operating table 4 is located is determined as the position of the initial operating area. After that, prompt information is displayed to instruct the operator to collect the registration point position.
  • the surgical area is spatially registered with the virtual surgical scene.
  • the subsequent prompt information display is performed based on the registration.
  • corresponding prompt information is displayed step by step to guide the operator to perform the operation position of the corresponding surgical step.
  • prompt information is given to guide the operator in which direction the camera 2 should be adjusted.
  • a method for displaying augmented reality information is provided, as shown in FIG. 2 , including the following steps:
  • the real image information may be the real scene of the operation table area.
  • the display mode of the prompt information may include superimposing and displaying text, animation or pattern at the corresponding position in the real scene, and may be supplemented by voice prompts.
  • obtaining the identifiable identification information 1 set at the operation site to obtain the initialized operation area position information may include:
  • the position information of the operation area is initialized.
  • initializing the position information of the operation area includes:
  • the location of the region that conforms to the specific positional relationship with the world coordinates of the identifiable identification information 1 is determined as the surgical region.
  • superimposing and displaying prompt information for guiding the operation operator at the corresponding position of the operation area includes:
  • the prompt information for collecting the position of the registration point is superimposed and displayed at the corresponding position of the operation area, so as to register the operation area with the virtual operation scene.
  • the prompt information for superimposing and displaying the registration point position collection at the corresponding position of the operation area includes:
  • Text, pattern or animation is superimposed and displayed at the position of the registration point in the operation area, indicating the position of the registration point.
  • the positions of the registration points to be acquired are indicated by static or dynamic arrow images displayed virtually.
  • superimposing and displaying prompt information for guiding the operation operator at the corresponding position of the operation area includes:
  • the operator will be prompted in the form of text, animation or pattern in which direction the camera 2 should be adjusted.
  • superimposing and displaying prompt information for guiding the operation operator at the corresponding position of the operation area includes:
  • the operator will be prompted for the location of the next operation step in the form of text, animation or pattern.
  • prompting the operator in the form of text, animation or pattern of the location of the next surgical step includes:
  • the planning state information of the surgical instrument is acquired; the planning state information of the surgical instrument is preset in the host computing device.
  • the actual state information of the surgical instrument is obtained; the surgical instrument is pre-set with an identifiable mark, and the state information of the surgical instrument can be identified through the camera 2 .
  • the planned state information and the actual state information are compared, and if the planned state information is consistent with the actual state information, it is determined that the current surgical step has been completed. If the planned state information is inconsistent with the actual state information, it is determined that the current surgical step has not been completed.
  • the augmented reality information display method further includes:
  • the initialized surgical area location information is corrected based on the actual location information.
  • the currently determined surgical area is compared with the actual position, and the currently determined surgical area is adjusted to be consistent with the actual position.
  • the operator is prompted with the position of the next surgical step in the form of text, animation or pattern.
  • the method before obtaining the identifiable identification information 1 set at the operation site to obtain the initialized operation area position information, the method further includes:
  • augmented reality information display methods described in the above embodiments have the same beneficial effects as those of the augmented reality information display device, which will not be repeated here.
  • a readable storage medium having executable instructions thereon, and when the executable instructions are executed, the computer is made to execute the augmented reality information display method described in any one of the foregoing steps in .
  • an electronic device is also provided. As shown in FIG. 5 , the device includes a communication interface 1000 , a memory 2000 and a processor 3000 .
  • the communication interface 1000 is used to communicate with external devices and perform data interactive transmission.
  • a computer program executable on the processor 3000 is stored in the memory 2000 .
  • the number of the memory 2000 and the processor 3000 may be one or more.
  • the bus can be an industry standard architecture (ISA, Industry Standard Architecture) bus, a peripheral device interconnect (PCI, Peripheral Component) bus or an extended industry standard architecture (EISA, Extended Industry Standard Component) bus and the like.
  • ISA Industry Standard Architecture
  • PCI peripheral device interconnect
  • EISA Extended Industry Standard Component
  • the bus can be divided into an address bus, a data bus, a control bus, and the like. For ease of presentation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.
  • the communication interface 1000, the memory 2000, and the processor 3000 are integrated on one chip, the communication interface 1000, the memory 2000, and the processor 3000 can communicate with each other through an internal interface.
  • the processor is configured to execute one or more steps in the augmented reality information display method described in any of the foregoing embodiments.
  • the processor can be a central processing unit (Central Processing Unit, CPU for short), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other Programming logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
  • the general-purpose processor can be a microprocessor or the processor can also be any conventional processor or the like.
  • Computer program instructions suitable for execution by the processor are stored in the memory, and when the computer program instructions are executed by the processor, one or more steps in the augmented reality information display method described in any of the foregoing embodiments are executed. .
  • the memory can be Read-Only Memory (ROM) or other types of static storage devices that can store static information and instructions, Random Access Memory (RAM) or other types of storage devices that can store information and instructions
  • a dynamic storage device can also be an Electrically Erasable Programmable Read-Only Memory (EEPROM), a Compact Disc Read-Only Memory (CD-ROM), or other optical disk storage, optical disk storage (including compact discs, laser discs, compact discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or capable of carrying or storing desired program code in the form of instructions or data structures and capable of being stored by a computer any other medium taken, but not limited to this.
  • the memory may exist independently and be connected to the processor through a communication bus.
  • the memory can also be integrated with the processor.
  • references to the terms “one embodiment/mode”, “some embodiments/modes”, “example”, “specific example”, or “some examples”, etc. are intended to be combined with the description of the embodiment/mode
  • a particular feature, structure, material, or characteristic described by way of example or example is included in at least one embodiment/mode or example of the present application.
  • schematic representations of the above terms are not necessarily directed to the same embodiment/mode or example.
  • the particular features, structures, materials or characteristics described may be combined in any suitable manner in any one or more embodiments/means or examples.
  • those skilled in the art may combine and combine the different embodiments/modes or examples described in this specification and the features of the different embodiments/modes or examples without conflicting each other.
  • first and second are only used for descriptive purposes, and should not be construed as indicating or implying relative importance or implying the number of indicated technical features. Thus, a feature delimited with “first”, “second” may expressly or implicitly include at least one of that feature.
  • plurality means at least two, such as two, three, etc., unless expressly and specifically defined otherwise.

Landscapes

  • Health & Medical Sciences (AREA)
  • Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Surgery (AREA)
  • Medical Informatics (AREA)
  • Biomedical Technology (AREA)
  • Veterinary Medicine (AREA)
  • Public Health (AREA)
  • General Health & Medical Sciences (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Robotics (AREA)
  • Animal Behavior & Ethology (AREA)
  • Molecular Biology (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Physics & Mathematics (AREA)
  • Computer Graphics (AREA)
  • Computer Hardware Design (AREA)
  • General Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Processing Or Creating Images (AREA)

Abstract

一种增强现实信息显示装置、方法、可读存储介质及设备。增强现实信息显示装置包括：手术规划信息获取模块，用于获取基于虚拟手术场景制定的手术规划信息；真实图像信息获取模块，用于获取手术现场的真实图像信息；手术区域初始化模块，用于获取设置于手术现场的可识别标识信息，以获得初始化的手术区域位置信息；提示信息模块，用于根据手术规划信息和虚拟手术场景，在手术区域的相应位置叠加显示用于对手术操作者进行指导的提示信息。增强现实信息显示装置可以减少手术操作者手术操作过程的困难度，降低对手术流程机械记忆的依赖性。

Description

增强现实信息显示装置、方法、可读存储介质及设备 技术领域
本发明实施例涉及计算机图像处理技术领域，尤其涉及一种增强现实信息显示装置、方法、可读存储介质及设备。
背景技术
手术的操作步骤是医生学习手术技能的必要知识。传统的手术学习过程需要通过模拟手术培训，现场手术带教及独立实习等反复训练，才可能让医生记忆并掌握正确的手术流程。整个学习过程复杂而且完全依赖记忆，实施风险高。而依赖手术步骤的文字背诵，不足以反应实际与空间、器械相关的具体手术操作步骤。手术操作者在手术过程中(特别是较为复杂的手术)单纯依靠对手术流程的记忆来进行手术操作会存在较大困难。
发明内容
为了解决或者缓解至少一个上述技术问题，本发明实施例提供了一种增强现实信息显示装置、方法、可读存储介质及设备，以减少手术操作者手术操作过程的困难度。
本发明实施例的一个方面，一种增强现实信息显示装置，包括：
手术规划信息获取模块，用于获取基于虚拟手术场景制定的手术规划信息；
真实图像信息获取模块，用于获取手术现场的真实图像信息；
手术区域初始化模块，用于获取设置于所述手术现场的可识别标识信息，以获得初始化的手术区域位置信息；
提示信息模块，用于根据所述手术规划信息和所述虚拟手术场景，在所述手术区域的相应位置叠加显示用于对手术操作者进行指导的提示信息。
本发明实施例的一个实施方式，所述手术区域初始化模块以所述可识别标识信息在所述手术现场中的位置为基准，初始化所述手术区域的位置信息。
本发明实施例的一个实施方式，所述手术区域初始化模块包括：
世界坐标获取模块，用于获取可识别标识信息在手术现场中的世界坐标；
手术区域位置确定模块，用于根据预设的对应关系，确定与所述可识别标识信息的世界坐标符合特定位置关系的区域位置为所述手术区域。
本发明实施例的一个实施方式，所述提示信息模块包括：
配准提示模块，用于在所述手术区域的相应位置叠加显示进行配准点位置采集的提示信息，以将所述手术区域与所述虚拟手术场景进行配准。
本发明实施例的一个实施方式，所述配准提示模块包括：
提示信息显示模块，用于以文字、动画或图案的方式提示所述手术操作者进行配准点位置采集步骤；
配准点位置指示模块，用于在所述手术区域中的所述配准点的位置处叠加显示文字、图案或动画，指示出所述配准点的位置。
本发明实施例的一个实施方式，所述提示信息模块包括：
调整方向提示模块，用于在所述手术区域超出摄像头的当前视野范围时，以文字、动画或图案的方式提示手术操作者摄像头应调整的方向。
本发明实施例的一个实施方式，所述提示信息模块包括：
手术步骤位置提示模块，用于在识别到手术操作者已完成当前手术步骤时，以文字、动画或图案的方式提示所述手术操作者下一手术步骤所在位置。
本发明实施例的一个实施方式，所述手术步骤位置提示模块包括：
手术器械规划状态信息获取模块，用于获取手术器械的规划状态信息；
手术器械实际状态信息获取模块，用于根据所获取的手术现场的 真实图像信息，获取手术器械的实际状态信息；
比较判断模块，用于比较所述规划状态信息与所述实际状态信息，如果所述规划状态信息与所述实际状态信息一致，则判断所述当前手术步骤已完成。
本发明实施例的一个实施方式，所述规划状态信息包括规划路径、规划停留时间、规划运动速度和相对皮肤规划角度的至少一种；所述实际状态信息包括实际路径、实际停留时间、实际运动速度和相对皮肤实际角度的至少一种。
本发明实施例的一个实施方式，所述增强现实信息显示装置还包括：
当前手术区域实际位置获取模块，用于获取根据当前手术步骤中获取的手术现场的真实图像信息确定的当前手术区域对应的实际位置信息；
位置修正模块，用于基于所述实际位置信息修正所述初始化的手术区域位置信息；
手术步骤提示模块，用于基于修正后的所述手术区域位置信息，以文字、动画或图案的方式提示所述手术操作者下一手术步骤所在位置。
本发明实施例的一个实施方式，所述增强现实信息显示装置还包括：
初始化手术区域提示模块，用于以文字、动画或图案的方式提示所述手术操作者使用所述可识别标识信息获得初始化的手术区域位置信息。
本发明实施例的一个实施方式，所述可识别标识信息包括设置于所述手术现场的特征图案或用户做出的特征手势。
本发明实施例的另一个方面，一种增强现实信息显示方法，包括：
获取基于虚拟手术场景制定的手术规划信息；
获取手术现场的真实图像信息；
获取设置于所述手术现场的可识别标识信息，以获得初始化的手术区域位置信息；
根据所述手术规划信息和所述虚拟手术场景，在所述手术区域的相应位置叠加显示用于对手术操作者进行指导的提示信息。
本发明实施例的一个实施方式，所述获取设置于所述手术现场的可识别标识信息，以获得初始化的手术区域位置信息包括：
以所述可识别标识信息在所述手术现场中的位置为基准，初始化所述手术区域的位置信息。
本发明实施例的一个实施方式，所述以所述可识别标识信息在所述手术现场中的位置为基准，初始化所述手术区域的位置信息包括：
获取所述可识别标识信息在所述手术现场中的世界坐标；
根据预设的对应关系，确定与所述可识别标识信息的世界坐标符合特定位置关系的区域位置为所述手术区域。
本发明实施例的一个实施方式，所述根据所述手术规划信息和所述虚拟手术场景，在所述手术区域的相应位置叠加显示用于对手术操作者进行指导的提示信息包括：
在所述手术区域的相应位置叠加显示进行配准点位置采集的提示信息，以将所述手术区域与所述虚拟手术场景进行配准。
本发明实施例的一个实施方式，所述在所述手术区域的相应位置叠加显示进行配准点位置采集的提示信息包括：
以文字、动画或图案的方式提示操作者开始进行配准点位置采集步骤；
在所述手术区域中的所述配准点的位置处叠加显示文字、图案或动画，指示出所述配准点的位置。
本发明实施例的一个实施方式，所述根据所述手术规划信息和所述虚拟手术场景，在所述手术区域的相应位置叠加显示用于对手术操作者进行指导的提示信息包括：
如果所述手术区域超出摄像头的当前视野范围，则以文字、动画或图案的方式提示所述手术操作者所述摄像头应调整的方向。
本发明实施例的一个实施方式，所述根据所述手术规划信息和所述虚拟手术场景，在所述手术区域的相应位置叠加显示用于对手术操作者进行指导的提示信息包括：
如果识别到所述手术操作者已完成当前手术步骤，则以文字、动画或图案的方式提示所述手术操作者下一手术步骤所在位置。
本发明实施例的一个实施方式，所述如果识别到所述手术操作者已完成当前手术步骤，则以文字、动画或图案的方式提示所述手术操作者下一手术步骤所在位置包括：
获取手术器械的规划状态信息；
根据所获取的手术现场的真实图像信息，获取所述手术器械的实际状态信息；
比较所述规划状态信息与所述实际状态信息，如果所述规划状态信息与所述实际状态信息一致，则判断所述当前手术步骤已完成。
本发明实施例的一个实施方式，所述规划状态信息包括规划路径、规划停留时间、规划运动速度和相对皮肤规划角度的至少一种；所述实际状态信息包括实际路径、实际停留时间、实际运动速度和相对皮肤实际角度的至少一种。
本发明实施例的一个实施方式，所述增强现实信息显示方法还包括：
获取根据当前手术步骤中获取的手术现场的真实图像信息确定的当前手术区域对应的实际位置信息；
基于所述实际位置信息修正所述初始化的手术区域位置信息；
基于修正后的所述手术区域位置信息，以文字、动画或图案的方式提示所述手术操作者下一手术步骤所在位置。
本发明实施例的一个实施方式，所述获取设置于所述手术现场的可识别标识信息，以获得初始化的手术区域位置信息之前，还包括；
以文字、动画或图案的方式提示所述手术操作者使用所述可识别标识信息获得初始化的手术区域位置信息。
本发明实施例的一个实施方式，所述可识别标识信息包括设置于所述手术现场的特征图案或用户做出的特征手势。
本发明实施例的另一个方面，一种可读存储介质，其上具有可执行指令，当可执行指令被执行时，使得计算机执行如上述任一项所述的增强现实信息显示方法中的步骤。
本发明实施例的另一个方面，一种电子设备，所述设备包括处理器和存储器，所述存储器中存储有适于所述处理器执行的计算机程序指令，所述计算机程序指令被所述处理器运行时执行如上述任一项所述 的增强现实信息显示方法中的步骤。
本发明实施例的增强现实信息显示装置、方法，基于增强现实技术，利用头戴设备显示出虚拟手术场景，并利用摄像头获取手术现场的真实图像信息，识别设置于手术现场的可识别标识信息，以获得初始化的手术区域位置信息，然后利用增强现实技术可以模拟三维空间显示的能力，根据手术规划信息和虚拟手术场景，在手术过程中实时以文字、动画或图案的方式在手术区域的相应位置叠加显示用于对手术操作者进行指导的提示信息。手术操作者通过获取这些提示信息来进行手术操作，减少了对手术操作流程机械记忆的依赖性，降低了手术操作过程的困难程度。
附图说明
附图示出了本发明实施例的示例性实施方式，并与其说明一起用于解释本发明实施例的原理，其中包括了这些附图以提供对本发明实施例的进一步理解，并且附图包括在本说明书中并构成本说明书的一部分。
图1是本发明实施例的增强现实信息显示装置的一种模块结构示意图。
图2是本发明实施例的增强现实信息显示方法的一种流程示意图。
图3是本发明实施例的增强现实信息显示方法的另一种流程示意图。
图4是本发明实施例的增强现实信息显示方法的一种应用场景示意图。
图5是本发明实施例的电子设备的一种示例性结构示意图。
附图标记说明：
1-可识别标识信息；2-摄像头；3-头戴设备；4-手术台。
具体实施方式
下面结合附图和实施方式对本发明实施例作进一步的详细说明。可以理解的是，此处所描述的具体实施方式仅用于解释相关内容，而非对本发明实施例的限定。另外还需要说明的是，为了便于描述，附图中仅示出了与本发明实施例相关的部分。
需要说明的是，在不冲突的情况下，本发明实施例中的实施方式及实施方式中的特征可以相互组合。下面将参考附图并结合实施方式来详细说明本发明实施例。
增强现实(Augmented Reality，简称AR)，是促使真实世界信息和虚拟世界信息内容之间综合在一起的技术内容，将原本在现实世界的空间范围中比较难以进行体验的实体信息在电脑等科学技术的基础上，实施模拟仿真处理，将虚拟信息内容在真实世界中加以有效应用，并且在这一过程中能够被人类感官所感知，从而实现超越现实的感官体验。真实环境和虚拟场景之间重叠之后，能够在同一个画面以及空间中同时存在。基于增强现实技术的设备通常包括主计算设备和头戴设备3；主计算设备上运行增强现实虚拟软件，头戴设备3佩戴在操作者头上，上面设有摄像头。头戴设备3与主计算设备通信连接，相互传递信息数据。
根据本发明实施例的一方面，提出一种增强现实信息显示装置，参见图1所示，包括以下模块：
手术规划信息获取模块，用于获取基于虚拟手术场景制定的手术规划信息。其中，虚拟手术场景包括基于真实手术场景中的事物建立的虚拟三维模型，例如手术对应的患者的解剖结构模型(根据患者的术前影像获得)和/或手术器械模型。虚拟手术场景通常运行在主计算设备上，并将数据传输给头戴设备3进行显示，使操作者能看到显示的虚拟图像。
真实图像信息获取模块，用于获取手术现场的真实图像信息，举例来说，真实图像信息可以是手术台区域的真实场景。真实图像信息通过操作者头上佩戴的头戴设备3上设置的摄像头2，对手术现场实时连续拍摄得到。
手术区域初始化模块，用于获取设置于手术现场的可识别标识信息1，以获得初始化的手术区域位置信息。可识别标识信息1通常设置在手术现场的与手术位置相关联的区域，如手术台4一侧等。可选的，可识别标识信息1包括特征图案或用户做出的特征手势，特征图案可以是具有特定图案的标志物，例如条形码、二维码等，可以利用 头戴设备3上设置的摄像头2来进行捕获并由系统进行识别。手术区域是指在摄像头2的视野范围内确定出的将要进行手术操作的区域，操作者在头戴设备3中可以看到虚拟显示出的当前确定的手术区域。
提示信息模块，用于根据手术规划信息和虚拟手术场景，在手术区域的相应位置叠加显示用于对手术操作者的操作进行指导的提示信息。提示信息的显示方式可以包括将文字、动画或图案叠加显示在真实场景中的相应位置，并可辅以语音提示。
需要说明的是，本发明实施例的增强现实信息显示装置的发明构思在于改善虚拟场景的成像技术，并不在于获得更好的或最佳的手术方案。本发明实施例的技术方案实质上是一种计算机图像处理技术，其改进在于成像技术本身，本发明实施例的增强现实信息显示装置是为了降低操作者手术操作过程中的困难程度，减少对手术流程机械记忆的依赖性。
本发明实施例的增强现实信息显示装置基于增强现实技术，利用头戴设备3显示出手术对应的人体解剖结构模型和/或手术器械模型，并利用摄像头2获取手术现场的真实图像信息，识别设置于手术现场的可识别标识信息1，以获得初始化的手术区域位置信息，然后利用增强现实技术可以模拟三维空间显示的能力，根据手术规划信息和虚拟手术场景，以文字、动画或图案的方式在手术区域的相应位置叠加显示用于对手术操作者进行指导的提示信息。手术操作者通过获取这些提示信息来进行手术操作，减少了对手术操作流程机械记忆的依赖性，降低手术操作过程的困难程度。
在本发明一实施方式中，手术区域初始化模块以可识别标识信息1在手术现场中的位置为基准，初始化手术区域的位置信息。例如，将诸如二维码的可识别标识信息1设置在一个标牌上，将标牌设置在手术台4的一侧，该标牌所在的位置与手术台4的位置是确定的，则摄像头2扫描到该标牌上的二维码信息后，将信息传输给主计算机，主计算机则根据标牌所在的位置为基准，将手术台4上的特定区域确定为手术区域。其中，确定的手术区域的位置与标牌的位置之间的关系事先在主计算设备中进行了设定。
进一步的，手术区域初始化模块包括：
世界坐标获取模块，用于获取可识别标识信息1在手术现场中的世界坐标。世界坐标系是系统的绝对坐标系，所有点的坐标都是以该坐标系的原点(绝对参考点)来确定。也就是说，由于头戴设备3的摄像头2可在环境中的任意位置，在环境中选择一个基准坐标系来描述摄像机的位置，并用它描述环境中任何物体的位置，该坐标系称为世界坐标系。
手术区域位置确定模块，用于根据预设的对应关系，确定与可识别标识信息1的世界坐标符合特定位置关系的区域位置为手术区域。可识别标识信息1的世界坐标确定后，由于其与手术台4的位置关系固定且两者的对应关系事先设定在主计算设备中，主计算设备根据可识别标识信息1的世界坐标以及与手术台4的位置的对应关系，将手术台4上与可识别标识信息1的位置具有特定位置关系的区域确定为手术区域。该特定位置关系预先在主计算设备中设置好，例如可以事先将可识别标识信息1位置后方的一矩形区域设定成手术区域，则主计算设备在获取到该可识别标识信息1的世界坐标后，按照该预先设定的特定位置关系，将可识别标识信息1位置后方的一矩形区域确定为当前手术区域，完成手术区域的初始化。
在本发明一实施方式中，提示信息模块包括：
配准提示模块，用于在手术区域的相应位置叠加显示进行配准点位置采集的提示信息，以将手术区域与虚拟手术场景进行配准。也就是说，通过显示进行配准点位置采集的提示信息，指引操作者进行配准点位置采集的步骤。配准点位置采集的目的是为了将手术区域与虚拟手术场景进行配准，也就是使两者的基准相一致。
进一步的，配准提示模块包括：
提示信息显示模块，用于以文字、动画或图案的方式提示操作者进行配准点位置采集。
配准点位置指示模块，用于在所述手术区域中的所述配准点的位置处叠加显示文字、图案或动画，指示出配准点的位置。例如通过虚拟显示出的静态或动态的箭头图像来指出需要进行采集的配准点的位 置。
在本发明一实施方式中，提示信息模块包括：
调整方向提示模块，用于在手术区域超出摄像头2的当前视野范围时，以文字、动画或图案的方式提示手术操作者摄像头2应调整的方向。根据手术操作者头部的位置和转动角度，摄像头2的当前视野范围是在不断变化的，而确定的手术区域的位置是相对固定的，当手术操作者由于转头或移动使摄像头2的当前视野范围与确定的手术区域不一致，以文字、动画或图案的方式指示出摄像头2应调整的方向。例如通过虚拟显示出的静态或动态的箭头图像来指出摄像头2应调整的方向。
在本发明一实施方式中，提示信息模块包括：
手术步骤位置提示模块，用于在识别到手术操作者已完成当前手术步骤时，以文字、动画或图案的方式提示手术操作者下一手术步骤所在位置。
进一步的，手术步骤位置提示模块包括：
手术器械规划状态信息获取模块，用于获取手术器械的规划状态信息；手术器械的规划状态信息预先设置在主计算设备中。
手术器械实际状态信息获取模块，用于根据所获取的手术现场的真实图像信息，获取手术器械的实际状态信息；手术器械上预先设置有可识别的标志，通过摄像头2可以识别出手术器械的状态信息。
比较判断模块，用于比较规划状态信息与实际状态信息，如果规划状态信息与实际状态信息一致，则判断当前手术步骤已完成。如果规划状态信息与实际状态信息不一致，则判断当前手术步骤还没有完成。
可选的，规划状态信息可以包括规划路径、规划停留时间、规划运动速度和相对皮肤规划角度的至少一种；实际状态信息可以包括实际路径、实际停留时间、实际运动速度和相对皮肤实际角度的至少一种。规划状态信息与实际状态信息可以采用上述不同的多种组合形式，但两者组合时的种类需要对应。也就是说，规划状态信息为规划路径时，实际状态信息应为对应的实际路径。规划状态信息为规划运动速 度和相对皮肤规划角度时，实际状态信息应为对应的实际运动速度和相对皮肤实际角度。规划路径是预先设定的手术器械的移动路径，相对皮肤规划角度是预先设定的手术器械相对皮肤的倾斜角度；手术器械移动路径相同而倾斜角度不同是两种不同的状态。
举例来说，通过摄像头2识别手术器械移动的实际路径，传送给主计算设备并记录，主计算设备中将实际路径与规划路径进行比较，如果实际路径与规划路径一致，则判断已完成当前手术步骤。或者，手术器械在某一区域规划停留时间为1s，且通过摄像头2识别手术器械在该区域实际停留时间超过1s，则判断当前手术步骤已完成。类似的，有时需要手术器械的运动速度和相对皮肤角度的参数，主计算设备记录并比较规划运动速度和相对皮肤规划角度是否与实际运动速度和相对皮肤实际角度一致。如果差异在预定范围内，则判断已完成当前手术步骤。
在本发明一实施方式中，增强现实信息显示装置还包括：
当前手术区域实际位置获取模块，用于获取根据当前手术步骤中获取的手术现场的真实图像信息确定的当前手术区域对应的实际位置信息。可以基于识别和记录的手术器械的实际位置来得到确定的手术区域对应的实际位置信息。从进行手术区域的初始化之后，每对应一个手术步骤均会确定一个相应的手术区域作为当前手术步骤的手术区域。
位置修正模块，用于基于实际位置信息修正初始化的手术区域位置信息。将当前确定的手术区域与实际位置进行比较，将当前确定的手术区域调整到与实际位置相一致。
手术步骤提示模块，用于基于修正后的手术区域位置信息，以文字、动画或图案的方式提示手术操作者下一手术步骤所在位置。也就是说，提示的下一手术步骤所在的位置是基于修正后的手术区域位置信息进行的，避免了在进行多个手术步骤后位置误差的积累。
在本发明一实施方式中，增强现实信息显示装置还包括；
初始化手术区域提示模块，用于以文字、动画或图案的方式提示手术操作者使用可识别标识信息1获得初始化的手术区域位置信息。
以下参照图4说明本发明实施例的增强现实信息显示装置的一种应用场景。在手术台4的一侧摆放识别板(识别板上有诸如二维码的可识别标识信息1)，在主计算设备中预先设定该识别板后方的手术台4区域所在的位置为初始确定的手术区域的位置。通过提示信息指示操作者将摄像头2扫描识别板上的二维码进行手术区域位置信息的初始化操作。以该识别板的位置为基准，确定手术台4上的一定空间范围所在的位置为初始的手术区域的位置。之后，显示提示信息指示操作者进行配准点位置采集。将手术区域与虚拟手术场景进行空间位置配准。基于该配准进行后续的提示信息显示。根据主计算设备中预先设定的不同的手术流程，逐步显示相应的提示信息，指引操作者进行相应的手术步骤的操作位置。期间，如果手术区域超出摄像头2的当前视野范围，则给出提示信息指引操作者摄像头2应调整的方向。并通过识别手术器械的实际状态信息是否与规划状态信息相一致来判断当前手术步骤是否完成。如果已完成，则根据当前步骤中获取的手术现场的真实图像信息确定的当前手术区域对应的实际位置信息修正初始化的手术区域位置信息，并基于修正后的手术区域位置信息叠加显示提示信息，指引下一手术步骤的操作位置。
根据本发明实施例的另一方面，提出一种增强现实信息显示方法，参见图2所示，包括以下步骤：
S10、获取基于虚拟手术场景制定的手术规划信息。
S20、获取手术现场的真实图像信息，举例来说，真实图像信息可以是手术台区域的真实场景。
S30、获取设置于手术现场的可识别标识信息1，以获得初始化的手术区域位置信息。
S40、根据手术规划信息和虚拟手术场景，在手术区域的相应位置叠加显示用于对手术操作者的操作进行指导的提示信息。提示信息的显示方式可以包括将文字、动画或图案叠加显示在真实场景中的相应位置，并可辅以语音提示。
在本发明一实施方式中，S30、获取设置于手术现场的可识别标识信息1，以获得初始化的手术区域位置信息可以包括：
以可识别标识信息1在手术现场中的位置为基准，初始化手术区域的位置信息。
进一步的，以可识别标识信息1在手术现场中的位置为基准，初始化手术区域的位置信息包括：
获取可识别标识信息1在手术现场中的世界坐标。
根据预设的对应关系，确定与可识别标识信息1的世界坐标符合特定位置关系的区域位置为手术区域。
在本发明一实施方式中，S40、根据手术规划信息和虚拟手术场景，在手术区域的相应位置叠加显示用于对手术操作者进行指导的提示信息包括：
在手术区域的相应位置叠加显示进行配准点位置采集的提示信息，以将手术区域与虚拟手术场景进行配准。
进一步的，在手术区域的相应位置叠加显示进行配准点位置采集的提示信息包括：
以文字、动画或图案的方式提示操作者进行配准点位置采集。
在所述手术区域中的所述配准点的位置处叠加显示文字、图案或动画，指示出配准点的位置。例如通过虚拟显示出的静态或动态的箭头图像来指出需要进行采集的配准点的位置。
在本发明一实施方式中，S40、根据手术规划信息和虚拟手术场景，在手术区域的相应位置叠加显示用于对手术操作者进行指导的提示信息包括：
如果手术区域超出摄像头2的当前视野范围，则以文字、动画或图案的方式提示手术操作者摄像头2应调整的方向。
在本发明一实施方式中，S40、根据手术规划信息和虚拟手术场景，在手术区域的相应位置叠加显示用于对手术操作者进行指导的提示信息包括：
如果识别到手术操作者已完成当前手术步骤，则以文字、动画或图案的方式提示手术操作者下一手术步骤所在位置。
进一步的，如果识别到手术操作者已完成当前手术步骤，则以文字、动画或图案的方式提示手术操作者下一手术步骤所在位置包括：
获取手术器械的规划状态信息；手术器械的规划状态信息预先设置在主计算设备中。
根据所获取的手术现场的真实图像信息，获取手术器械的实际状态信息；手术器械上预先设置有可识别的标志，通过摄像头2可以识别出手术器械的状态信息。
比较规划状态信息与实际状态信息，如果规划状态信息与实际状态信息一致，则判断当前手术步骤已完成。如果规划状态信息与实际状态信息不一致，则判断当前手术步骤还没有完成。
在本发明一实施方式中，增强现实信息显示方法还包括：
获取根据当前手术步骤中获取的手术现场的真实图像信息确定的当前手术区域对应的实际位置信息。
基于实际位置信息修正初始化的手术区域位置信息。将当前确定的手术区域与实际位置进行比较，将当前确定的手术区域调整到与实际位置相一致。
基于修正后的手术区域位置信息，以文字、动画或图案的方式提示手术操作者下一手术步骤所在位置。
在本发明一实施方式中，参见图3所示，在S30、获取设置于手术现场的可识别标识信息1，以获得初始化的手术区域位置信息之前，还包括；
S21、以文字、动画或图案的方式提示手术操作者使用可识别标识信息1获得初始化的手术区域位置信息。
以上各实施方式所述的增强现实信息显示方法具有与增强现实信息显示装置相同的有益效果，再此不再赘述。
根据本发明实施例的另一方面，还提供了一种可读存储介质，其上具有可执行指令，当可执行指令被执行时，使得计算机执行前述任一项所述的增强现实信息显示方法中的步骤。
根据本发明实施例的另一方面，还提供了一种电子设备，如图5所示，设备包括通信接口1000、存储器2000和处理器3000。通信接口1000用于与外界设备进行通信，进行数据交互传输。存储器2000内存储有可在处理器3000上运行的计算机程序。所述存储器2000和 处理器3000的数量可以为一个或多个。
如果通信接口1000、存储器2000及处理器3000独立实现，则通信接口1000、存储器2000及处理器3000可以通过总线相互连接并完成相互间的通信。所述总线可以是工业标准体系结构(ISA，Industry Standard Architecture)总线、外部设备互连(PCI，Peripheral Component)总线或扩展工业标准体系结构(EISA，Extended Industry Standard Component)总线等。所述总线可以分为地址总线、数据总线、控制总线等。为便于表示，该图中仅用一条粗线表示，但并不表示仅有一根总线或一种类型的总线。
可选的，在具体实现上，如果通信接口1000、存储器2000、及处理器3000集成在一块芯片上，则通信接口1000、存储器2000、及处理器3000可以通过内部接口完成相互间的通信。
处理器用于执行上述任一实施例所述增强现实信息显示方法中的一个或多个步骤。处理器可以是中央处理单元(Central Processing Unit，简称CPU)，还可以是其他通用处理器、数字信号处理器(DSP)、专用集成电路(ASIC)、现场可编程门阵列(FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件等。其中，通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。
所述存储器中存储有适于所述处理器执行的计算机程序指令，所述计算机程序指令被所述处理器运行时执行上述任一实施例所述增强现实信息显示方法中的一个或多个步骤。
存储器可以是只读存储器(Read-Only Memory，ROM)或可存储静态信息和指令的其他类型的静态存储设备，随机存取存储器(Random Access Memory，RAM)或者可存储信息和指令的其他类型的动态存储设备，也可以是电可擦可编程只读存储器(Electrically Erasable Programmable Read-Only Memory，EEPROM)、只读光盘(Compact Disc Read-Only Memory，CD-ROM)或其他光盘存储、光碟存储(包括压缩光碟、激光碟、光碟、数字通用光碟、蓝光光碟等)、磁盘存储介质或者其他磁存储设备、或者能够用于携带或存储具有指令或数据结构形式的期望的程序代码并能够由计算机存取的任何其他 介质，但不限于此。存储器可以是独立存在，通过通信总线与处理器相连接。存储器也可以和处理器集成在一起。
在本说明书的描述中，参考术语“一个实施例/方式”、“一些实施例/方式”、“示例”、“具体示例”、或“一些示例”等的描述意指结合该实施例/方式或示例描述的具体特征、结构、材料或者特点包含于本申请的至少一个实施例/方式或示例中。在本说明书中，对上述术语的示意性表述不必须针对的是相同的实施例/方式或示例。而且，描述的具体特征、结构、材料或者特点可以在任一个或多个实施例/方式或示例中以合适的方式结合。此外，在不相互矛盾的情况下，本领域的技术人员可以将本说明书中描述的不同实施例/方式或示例以及不同实施例/方式或示例的特征进行结合和组合。
此外，术语“第一”、“第二”仅用于描述目的，而不能理解为指示或暗示相对重要性或者隐含指明所指示的技术特征的数量。由此，限定有“第一”、“第二”的特征可以明示或者隐含地包括至少一个该特征。在本申请的描述中，“多个”的含义是至少两个，例如两个，三个等，除非另有明确具体的限定。
本领域的技术人员应当理解，上述实施方式仅仅是为了清楚地说明本发明实施例，而并非是对本发明实施例的范围进行限定。对于所属领域的技术人员而言，在上述公开的基础上还可以做出其它变化或变型，并且这些变化或变型仍处于本发明实施例的范围内。

Claims (15)

  1. 一种增强现实信息显示装置，其特征在于，包括：
    手术规划信息获取模块，用于获取基于虚拟手术场景制定的手术规划信息；
    真实图像信息获取模块，用于获取手术现场的真实图像信息；
    手术区域初始化模块，用于获取设置于所述手术现场的可识别标识信息，以获得初始化的手术区域位置信息；
    提示信息模块，用于根据所述手术规划信息和所述虚拟手术场景，在所述手术区域的相应位置叠加显示用于对手术操作者进行指导的提示信息。
  2. 根据权利要求1所述的增强现实信息显示装置，其特征在于，所述手术区域初始化模块以所述可识别标识信息在所述手术现场中的位置为基准，初始化所述手术区域的位置信息。
  3. 根据权利要求2所述的增强现实信息显示装置，其特征在于，所述手术区域初始化模块包括：
    世界坐标获取模块，用于获取可识别标识信息在手术现场中的世界坐标；
    手术区域位置确定模块，用于根据预设的对应关系，确定与所述可识别标识信息的世界坐标符合特定位置关系的区域位置为所述手术区域。
  4. 根据权利要求1所述的增强现实信息显示装置，其特征在于，所述提示信息模块包括：
    配准提示模块，用于在所述手术区域的相应位置叠加显示进行配准点位置采集的提示信息，以将所述手术区域与所述虚拟手术场景进行配准。
  5. 根据权利要求4所述的增强现实信息显示装置，其特征在于，所 述配准提示模块包括：
    提示信息显示模块，用于以文字、动画或图案的方式提示所述手术操作者进行配准点位置采集步骤；
    配准点位置指示模块，用于在所述手术区域中的所述配准点的位置处叠加显示文字、图案或动画，指示出所述配准点的位置。
  6. 根据权利要求1所述的增强现实信息显示装置，其特征在于，所述提示信息模块包括：
    调整方向提示模块，用于在所述手术区域超出摄像头的当前视野范围时，以文字、动画或图案的方式提示手术操作者摄像头应调整的方向。
  7. 根据权利要求1所述的增强现实信息显示装置，其特征在于，所述提示信息模块包括：
    手术步骤位置提示模块，用于在识别到手术操作者已完成当前手术步骤时，以文字、动画或图案的方式提示所述手术操作者下一手术步骤所在位置。
  8. 根据权利要求7所述的增强现实信息显示装置，其特征在于，所述手术步骤位置提示模块包括：
    手术器械规划状态信息获取模块，用于获取手术器械的规划状态信息；
    手术器械实际状态信息获取模块，用于根据所获取的手术现场的真实图像信息，获取手术器械的实际状态信息；
    比较判断模块，用于比较所述规划状态信息与所述实际状态信息，如果所述规划状态信息与所述实际状态信息一致，则判断所述当前手术步骤已完成。
  9. 根据权利要求8所述的增强现实信息显示装置，其特征在于，所述规划状态信息包括规划路径、规划停留时间、规划运动速度和相对 皮肤规划角度的至少一种；所述实际状态信息包括实际路径、实际停留时间、实际运动速度和相对皮肤实际角度的至少一种。
  10. 根据权利要求1所述的增强现实信息显示装置，其特征在于，所述增强现实信息显示装置还包括：
    当前手术区域实际位置获取模块，用于获取根据当前手术步骤中获取的手术现场的真实图像信息确定的当前手术区域对应的实际位置信息；
    位置修正模块，用于基于所述实际位置信息修正所述初始化的手术区域位置信息；
    手术步骤提示模块，用于基于修正后的所述手术区域位置信息，以文字、动画或图案的方式提示所述手术操作者下一手术步骤所在位置。
  11. 根据权利要求1所述的增强现实信息显示装置，其特征在于，所述增强现实信息显示装置还包括：
    初始化手术区域提示模块，用于以文字、动画或图案的方式提示所述手术操作者使用所述可识别标识信息获得初始化的手术区域位置信息。
  12. 根据权利要求1-11任一项所述的增强现实信息显示装置，其特征在于，所述可识别标识信息包括设置于所述手术现场的特征图案或用户做出的特征手势。
  13. 一种增强现实信息显示方法，其特征在于，包括：获取基于虚拟手术场景制定的手术规划信息；
    获取手术现场的真实图像信息；
    获取设置于所述手术现场的可识别标识信息，以获得初始化的手术区域位置信息；
    根据所述手术规划信息和所述虚拟手术场景，在所述手术区域的 相应位置叠加显示用于对手术操作者进行指导的提示信息。
  14. 一种可读存储介质，其特征在于，其上具有可执行指令，当可执行指令被执行时，使得计算机执行如权利要求13所述的增强现实信息显示方法中的步骤。
  15. 一种电子设备，其特征在于，所述设备包括处理器和存储器，所述存储器中存储有适于所述处理器执行的计算机程序指令，所述计算机程序指令被所述处理器运行时执行如权利要求13所述的增强现实信息显示方法中的步骤。
PCT/CN2022/081472 2021-04-01 2022-03-17 增强现实信息显示装置、方法、可读存储介质及设备 WO2022206408A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110358156.7 2021-04-01
CN202110358156.7A CN113509266A (zh) 2021-04-01 2021-04-01 增强现实信息显示装置、方法、可读存储介质及设备

Publications (1)

Publication Number Publication Date
WO2022206408A1 true WO2022206408A1 (zh) 2022-10-06

Family

ID=78062185

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/081472 WO2022206408A1 (zh) 2021-04-01 2022-03-17 增强现实信息显示装置、方法、可读存储介质及设备

Country Status (2)

Country Link
CN (1) CN113509266A (zh)
WO (1) WO2022206408A1 (zh)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113509266A (zh) * 2021-04-01 2021-10-19 上海复拓知达医疗科技有限公司 增强现实信息显示装置、方法、可读存储介质及设备
CN117160029A (zh) * 2023-08-31 2023-12-05 江西格如灵科技股份有限公司 一种vr手柄检测方法及系统

Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030220541A1 (en) * 1998-12-08 2003-11-27 Intuitive Surgical, Inc. Devices and methods for presenting and regulating auxiliary information on an image display of a telesurgical system to assist an operator in performing a surgical procedure
CN105193503A (zh) * 2014-06-18 2015-12-30 柯惠Lp公司 增强的手术实践环境系统
WO2018052966A1 (en) * 2016-09-16 2018-03-22 Zimmer, Inc. Augmented reality surgical technique guidance
US20180357825A1 (en) * 2017-06-09 2018-12-13 Siemens Healthcare Gmbh Output of position information of a medical instrument
CN109996511A (zh) * 2016-09-22 2019-07-09 美敦力导航股份有限公司 用于引导进程的系统
CN110603002A (zh) * 2017-03-10 2019-12-20 拜欧米特制造有限责任公司 增强现实支持的膝盖手术
CN110650703A (zh) * 2017-05-05 2020-01-03 斯科皮斯有限公司 手术导航系统
WO2020186194A1 (en) * 2019-03-13 2020-09-17 Smith & Nephew Inc. Augmented reality assisted surgical tool alignment
WO2021007418A2 (en) * 2019-07-09 2021-01-14 Materialise N.V. Augmented reality assisted joint arthroplasty
CN113509266A (zh) * 2021-04-01 2021-10-19 上海复拓知达医疗科技有限公司 增强现实信息显示装置、方法、可读存储介质及设备

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
TWI741359B (zh) * 2019-08-30 2021-10-01 國立中央大學 與手術導航系統整合之混合實境系統

Patent Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030220541A1 (en) * 1998-12-08 2003-11-27 Intuitive Surgical, Inc. Devices and methods for presenting and regulating auxiliary information on an image display of a telesurgical system to assist an operator in performing a surgical procedure
CN105193503A (zh) * 2014-06-18 2015-12-30 柯惠Lp公司 增强的手术实践环境系统
WO2018052966A1 (en) * 2016-09-16 2018-03-22 Zimmer, Inc. Augmented reality surgical technique guidance
CN109996511A (zh) * 2016-09-22 2019-07-09 美敦力导航股份有限公司 用于引导进程的系统
CN110603002A (zh) * 2017-03-10 2019-12-20 拜欧米特制造有限责任公司 增强现实支持的膝盖手术
CN110650703A (zh) * 2017-05-05 2020-01-03 斯科皮斯有限公司 手术导航系统
US20180357825A1 (en) * 2017-06-09 2018-12-13 Siemens Healthcare Gmbh Output of position information of a medical instrument
WO2020186194A1 (en) * 2019-03-13 2020-09-17 Smith & Nephew Inc. Augmented reality assisted surgical tool alignment
WO2021007418A2 (en) * 2019-07-09 2021-01-14 Materialise N.V. Augmented reality assisted joint arthroplasty
CN113509266A (zh) * 2021-04-01 2021-10-19 上海复拓知达医疗科技有限公司 增强现实信息显示装置、方法、可读存储介质及设备

Also Published As

Publication number Publication date
CN113509266A (zh) 2021-10-19

Similar Documents

Publication Publication Date Title
WO2022206408A1 (zh) 增强现实信息显示装置、方法、可读存储介质及设备
US11806085B2 (en) Guidance for placement of surgical ports
US11925423B2 (en) Guidance for positioning a patient and surgical robot
US11918424B2 (en) System and method for improved electronic assisted medical procedures
US20160063707A1 (en) Image registration device, image registration method, and image registration program
JP2007236701A (ja) 医用画像の表示方法およびそのプログラム
JPWO2007102509A1 (ja) 医用画像の処理方法
Heinrich et al. Comparison of augmented reality display techniques to support medical needle insertion
US10325380B2 (en) Precise, low-cost orthopaedic surgical simulator
Benmahdjoub et al. Multimodal markers for technology-independent integration of augmented reality devices and surgical navigation systems
US11800966B2 (en) Medical system and medical system operating method
Forte et al. Design of interactive augmented reality functions for robotic surgery and evaluation in dry‐lab lymphadenectomy
IL260781B (en) Visualization of medical device navigation in a patient's organ using a dummy device and a physical 3D model
US20230260427A1 (en) Method and system for generating a simulated medical image
US10631948B2 (en) Image alignment device, method, and program
JP6461024B2 (ja) 画像位置合せ装置、方法およびプログラム
US20210192845A1 (en) Virtual augmentation of anatomical models
JP2021003379A (ja) リハビリ支援装置、及びプログラム
US10049480B2 (en) Image alignment device, method, and program
Nonaka et al. Capturing a surgical area using multiple depth cameras mounted on a robotic mechanical system
Condino et al. Single feature constrained manual registration method for Augmented Reality applications in gynecological laparoscopic interventions
EP3889738A1 (en) A system and a method for calibrating a user interface
US20210153959A1 (en) Physical medical element affixation systems, methods, and materials
CN115399875A (zh) 用于手术导航的导航工具校准方法、设备及存储介质
Glas Image guided surgery and the added value of augmented reality

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22778591

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 22778591

Country of ref document: EP

Kind code of ref document: A1