WO2022237787A1 - 机器人定位和位姿调整方法和系统 - Google Patents

机器人定位和位姿调整方法和系统 Download PDF

Info

Publication number
WO2022237787A1
WO2022237787A1 PCT/CN2022/092003 CN2022092003W WO2022237787A1 WO 2022237787 A1 WO2022237787 A1 WO 2022237787A1 CN 2022092003 W CN2022092003 W CN 2022092003W WO 2022237787 A1 WO2022237787 A1 WO 2022237787A1
Authority
WO
WIPO (PCT)
Prior art keywords
target
image
coordinate system
acquisition device
pose
Prior art date
Application number
PCT/CN2022/092003
Other languages
English (en)
French (fr)
Inventor
吴童
吴博
邓勇
汪全全
黄钦
Original Assignee
武汉联影智融医疗科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from CN202110505732.6A external-priority patent/CN113397704B/zh
Priority claimed from CN202111400891.6A external-priority patent/CN114098980B/zh
Application filed by 武汉联影智融医疗科技有限公司 filed Critical 武汉联影智融医疗科技有限公司
Priority to EP22806750.0A priority Critical patent/EP4321121A1/en
Publication of WO2022237787A1 publication Critical patent/WO2022237787A1/zh
Priority to US18/506,980 priority patent/US20240075631A1/en

Links

Images

Classifications

    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Programme-controlled manipulators
    • B25J9/16—Programme controls
    • B25J9/1694—Programme controls characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
    • B25J9/1697—Vision controlled systems
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/30—Surgical robots
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
    • A61B90/36—Image-producing devices or illumination devices not otherwise provided for
    • A61B90/361—Image-producing devices, e.g. surgical cameras
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Programme-controlled manipulators
    • B25J9/16—Programme controls
    • B25J9/1628—Programme controls characterised by the control loop
    • B25J9/163—Programme controls characterised by the control loop learning, adaptive, model based, rule based expert control
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Programme-controlled manipulators
    • B25J9/16—Programme controls
    • B25J9/1628—Programme controls characterised by the control loop
    • B25J9/1653—Programme controls characterised by the control loop parameters identification, estimation, stiffness, accuracy, error analysis
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Programme-controlled manipulators
    • B25J9/16—Programme controls
    • B25J9/1656—Programme controls characterised by programming, planning systems for manipulators
    • B25J9/1664—Programme controls characterised by programming, planning systems for manipulators characterised by motion, path, trajectory planning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
    • G06V10/75—Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
    • G06V10/759—Region-based matching
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/10—Computer-aided planning, simulation or modelling of surgical operations
    • A61B2034/101—Computer-aided simulation of surgical operations
    • A61B2034/105—Modelling of the patient, e.g. for ligaments or bones
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
    • A61B2034/2046—Tracking techniques
    • A61B2034/2065—Tracking using image or pattern recognition
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
    • A61B90/36—Image-producing devices or illumination devices not otherwise provided for
    • A61B2090/364—Correlation of different images or relation of image positions in respect to the body

Definitions

  • This specification relates to the field of robots, and in particular to a method and system for robot positioning and posture adjustment.
  • a robot includes a mechanical arm with a multi-degree-of-freedom structure, and the mechanical arm may include a base joint where the base of the mechanical arm is located and an end joint where the flange of the mechanical arm is located.
  • the flange plate of the mechanical arm is fixedly connected with terminal tools, such as various surgical tools such as electrode needles, puncture needles, syringes, and ablation needles.
  • the method may include acquiring a first image and a second image of the target object, the first image is acquired by an image acquisition device, and the second image is acquired by a medical imaging device; At least one target area, at least one target part is less affected by physiological motion than other parts; and based on the at least one target area and the second image, determining positioning information of the robot.
  • One of the embodiments of this specification provides a method for adjusting the pose of an image acquisition device.
  • the method may include acquiring a target image of the target object using an image acquisition device; determining at least one target feature point of the target object from the target image; determining at least one reference feature point corresponding to the at least one target feature point from a reference model of the target object, The reference model corresponds to the shooting angle of the target; based on at least one target feature point and at least one reference feature point, the first target pose of the image acquisition device in the base coordinate system is determined.
  • the system may include a storage device storing computer instructions; a processor connected to the storage device, and when the computer instructions are executed, the processor causes the system to perform the following operations: acquire a first image and a second image of the target object, the first image Acquired by an image acquisition device, the second image is acquired by a medical imaging device; determining at least one target region corresponding to at least one target part of the target object from the first image, at least one target part is less affected by physiological motion than other parts; and Positioning information of the robot is determined based on the at least one target area and the second image.
  • Fig. 1A is a schematic diagram of an application scenario of an exemplary robot control system according to some embodiments of the present specification
  • Fig. 1B is a schematic diagram of an application scenario of an exemplary robot control system according to some embodiments of the present specification
  • Figure 1C is a schematic diagram of an exemplary server according to some embodiments of the present specification.
  • Figure 2 is a block diagram of an exemplary processor according to some embodiments of the present specification.
  • Fig. 3 is a flowchart of an exemplary robot positioning process according to some embodiments of the present specification
  • Fig. 4 is a flowchart of an exemplary process of determining at least one target area according to some embodiments of the present specification
  • Fig. 5 is a flowchart of an exemplary process for determining a registration relationship according to some embodiments of the present specification
  • Fig. 6A is a schematic diagram of an exemplary set of reference points of a second image according to some embodiments of the present specification
  • Fig. 6B is a schematic diagram of candidate target point groups in an exemplary target area according to some embodiments of the present specification
  • Fig. 7 is a schematic diagram of exemplary facial landmarks according to some embodiments of the present specification.
  • FIG. 8 is a schematic diagram of an exemplary two-dimensional facial image according to some embodiments of the present specification.
  • Figure 9 is a block diagram of an exemplary processor according to some embodiments of the present specification.
  • Fig. 10 is a flowchart of an exemplary process for adjusting the pose of an image acquisition device according to some embodiments of the present specification
  • Fig. 11 is a flowchart of an exemplary process of determining a first target pose according to some embodiments of the present specification
  • Fig. 12 is a flowchart of an exemplary process for adjusting the pose of an image acquisition device according to some embodiments of the present specification
  • Fig. 13A is a front view of an exemplary standard human face model according to some embodiments of the present specification.
  • Fig. 13B is a side view of an exemplary standard human face model according to some embodiments of the present specification.
  • Fig. 14 is a schematic diagram of exemplary facial contour points according to some embodiments of the present specification.
  • 15A-15C are schematic diagrams of exemplary adjustment of the pose of an image acquisition device according to some embodiments of the present specification.
  • Fig. 16A is a schematic diagram of an exemplary image acquisition device before pose adjustment according to some embodiments of the present specification
  • Fig. 16B is a schematic diagram of an exemplary pose-adjusted image acquisition device according to some embodiments of the present specification
  • Fig. 16C, Fig. 16E and Fig. 16G are schematic diagrams of image data collected by an exemplary image collection device before pose adjustment according to some embodiments of the present specification
  • Fig. 16D, Fig. 16F and Fig. 16H are schematic diagrams of image data collected by an exemplary pose-adjusted image collection device according to some embodiments of the present specification
  • Fig. 17 is a schematic diagram of an exemplary adjustment of the pose of an image acquisition device according to some embodiments of the present specification.
  • FIG. 18 is a schematic diagram of an exemplary robot control system according to some embodiments of the present specification.
  • Figure 19 is a schematic diagram of an exemplary computer device according to some embodiments of the present specification.
  • Fig. 20 is a flowchart of an exemplary robot control process according to some embodiments of the present specification.
  • system means for distinguishing different components, elements, parts, parts or assemblies of different levels.
  • the words may be replaced by other expressions if other words can achieve the same purpose.
  • Fig. 1A is a schematic diagram of an application scenario of an exemplary robot control system 100 according to some embodiments of the present specification.
  • the robot control system 100 can be used for ground positioning and posture adjustment of the robot.
  • the robot control system 100 may include a server 110 , a medical imaging device 120 and an image acquisition device 130 .
  • Multiple components in the robot control system 100 may be connected to each other through a network.
  • the server 110 and the medical imaging device 120 may be connected or communicate through a network.
  • the server 110 and the image acquisition device 130 may be connected or communicate through a network.
  • connections between components in robotic control system 100 are variable.
  • the medical imaging device 120 may be directly connected to the image acquisition device 130 .
  • Server 110 may be used to process data and/or information from at least one component of robotic control system 100 (eg, medical imaging device 120 , image acquisition device 130 ) or an external data source (eg, cloud data center). For example, the server 110 may acquire the first image acquired by the image acquisition device 130 and the second image acquired by the medical imaging device 120, and determine the positioning information of the robot based on the first image and the second image. For another example, the server 110 may use the image acquisition device 130 to acquire a target image of the target object, and determine the first target pose of the image acquisition device 130 in the base coordinate system based on the target image and the reference model of the target object. In some embodiments, server 110 may be a single server or a group of servers.
  • the set of servers can be centralized or distributed (eg, server 110 can be a distributed system).
  • server 110 may be regional or remote.
  • the server 110 may be implemented on a cloud platform, or provided in a virtual manner.
  • a cloud platform may include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an internal cloud, a multi-tier cloud, etc., or any combination thereof.
  • server 110 may include one or more components. As shown in FIG. 1C , server 110 may include one or more (only one is shown in FIG. 1C ) processors 102 , memory 104 , transmission device 106 , and input/output device 108 . Those of ordinary skill in the art can understand that the structure shown in FIG. 1C is only illustrative and does not limit the structure of the server 110 . For example, server 110 may also include more or fewer components than shown in FIG. 1C , or have a different configuration than that shown in FIG. 1C .
  • Processor 102 may process data and/or information obtained from other devices or system components.
  • the processor 102 may execute program instructions based on these data, information and/or processing results to perform one or more functions described in this specification.
  • the processor 102 may include one or more sub-processing devices (eg, a single-core processing device or a multi-core multi-core processing device).
  • processor 102 may include a microprocessor (MCU), a central processing unit (CPU), an application specific integrated circuit (ASIC), an application specific instruction processor (ASIP), a graphics processing unit (GPU), a physical processing unit (PPU) ), digital signal processor (DSP), field programmable gate array (FPGA), programmable logic circuit (PLD), controller, microcontroller unit, reduced instruction set computer (RISC), etc., or any combination of the above.
  • processor 102 may be integrated or included in one or more other components of robotic control system 100 (eg, medical imaging device 120 , image acquisition device 130 , or other possible components).
  • Memory 104 may store data, instructions and/or any other information.
  • the memory 104 may be used to store computer programs, for example, software programs and modules of application software, such as computer programs corresponding to the positioning method and pose adjustment method in this embodiment.
  • the processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104 , that is, realizes the above-mentioned method.
  • the memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory.
  • the memory 104 may further include a memory that is remotely located relative to the processor 102, and these remote memories may be connected to the terminal through a network. Examples of the aforementioned networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.
  • storage 104 can be implemented on a cloud platform.
  • the transmission device 106 may be used to implement communication functions.
  • transmission device 106 may be used to receive or transmit data via a network.
  • the transmission device 106 may include a network adapter (Network Interface Controller, NIC for short), which can be connected to other network devices through a base station so as to communicate with the Internet.
  • the transmission device 106 may be a radio frequency (Radio Frequency, RF for short) module, which is used to communicate with the Internet in a wireless manner.
  • RF Radio Frequency
  • Input-output devices 108 may be used to input or output signals, data or information.
  • input and output devices 108 may enable a user to communicate with robotic control system 100 .
  • Exemplary input devices may include a keyboard, mouse, touch screen, microphone, etc., or any combination thereof.
  • Exemplary output devices may include a display device, speakers, printer, projector, etc., or any combination thereof.
  • Exemplary display devices may include liquid crystal displays (LCDs), light emitting diode (LED) based displays, flat panel displays, curved displays, television sets, cathode ray tubes (CRTs), etc., or any combination thereof.
  • LCDs liquid crystal displays
  • LED light emitting diode
  • CRTs cathode ray tubes
  • the server 110 can be set at any location (for example, the room where the robot is located, the room where the server is specially placed, etc.), and it is only necessary to ensure that the server 110 can communicate with the medical imaging equipment 120 and the image acquisition device 130 normally.
  • the medical imaging device 120 may be used to scan a target object in a detection area or a scanning area to obtain imaging data of the target object.
  • objects may include biological objects and/or non-biological objects.
  • an object may be animate or inanimate organic and/or inorganic matter.
  • medical imaging device 120 may be a non-invasive imaging device used for disease diagnosis or research purposes.
  • medical imaging device 120 may include a single modality scanner and/or a multimodal scanner.
  • Single modality scanners may include, for example, ultrasound scanners, X-ray scanners, computed tomography (CT) scanners, magnetic resonance imaging (MRI) scanners, sonography, positron emission tomography (PET) scanners , optical coherence tomography (OCT) scanner, ultrasound (US) scanner, intravascular ultrasound (IVUS) scanner, near-infrared spectroscopy (NIRS) scanner, far-infrared (FIR) scanner, etc., or any combination thereof.
  • CT computed tomography
  • MRI magnetic resonance imaging
  • PET positron emission tomography
  • OCT optical coherence tomography
  • US ultrasound
  • IVUS intravascular ultrasound
  • NIRS near-infrared spectroscopy
  • FIR far-infrared
  • Multimodal scanners may include, for example, X-ray imaging-magnetic resonance imaging (X-ray-MRI) scanners, positron emission tomography-X-ray imaging (PET-X-ray) scanners, single photon emission computed tomography-MRI Resonance imaging (SPECT-MRI) scanner, positron emission tomography-computed tomography (PET-CT) scanner, digital subtraction angiography-magnetic resonance imaging (DSA-MRI) scanner, etc. or any combination thereof.
  • X-ray imaging-magnetic resonance imaging X-ray-MRI
  • PET-X-ray positron emission tomography-X-ray imaging
  • SPECT-MRI single photon emission computed tomography-MRI Resonance imaging
  • PET-CT positron emission tomography-computed tomography
  • DSA-MRI digital subtraction angiography-magnetic resonance imaging
  • the image capture device 130 may be used to capture image data (eg, a first image, a target image) of a target object.
  • Exemplary image acquisition devices may include cameras, optical sensors, radar sensors, structured light scanners, etc., or any combination thereof.
  • the image acquisition device 130 may include a camera (eg, a depth camera, a stereo triangulation camera, etc.), an optical sensor (eg, a red-green-blue-depth (RGB-D) sensor, etc.), and other devices capable of capturing optical data of a target object.
  • the image acquisition device 130 may include a laser imaging device (for example, a phase laser acquisition device, a point laser acquisition device, etc.) that can acquire point cloud data of an object.
  • Point cloud data can include multiple data points, each data point can represent a physical point on the body surface of the object, and one or more feature values of the physical point can be used (for example, related to the position and/or composition of the physical point related feature values) to describe the target object.
  • Point cloud data can be used to reconstruct images of target objects.
  • the image acquisition device 130 may include a device capable of acquiring position data and/or depth data of a target object, such as a structured light scanner, a time-of-flight (TOF) device, an optical triangulation device, a stereo matching device, etc., or any combination.
  • the acquired position data and/or depth data can be used to reconstruct an image of the target object.
  • the image acquisition device 130 can be installed on the robot through a detachable or non-detachable connection.
  • the image acquisition device 130 may be detachably arranged at the end of the mechanical arm of the robot.
  • the image acquisition device 130 can be installed outside the robot through a detachable or non-detachable connection.
  • the image acquisition device 130 may be set at a certain fixed position in the room where the robot is located.
  • the correspondence between the positions of the image acquisition device 130 and the robot 140 can be determined based on the position of the image acquisition device 130, the position of the robot, and the calibration parameters (for example, size, shooting angle) of the image acquisition device 130. For example, a mapping relationship (ie, a first conversion relationship) between the first coordinate system corresponding to the image acquisition device 130 and the second coordinate system corresponding to the robot may be determined.
  • a mapping relationship ie, a first conversion relationship
  • image capture device 130 may include multiple image capture devices.
  • the robot control system 100 may further include a robot 140 .
  • the robot 140 may perform a corresponding operation based on the instruction. For example, based on the movement instruction, the robot 140 may perform a movement operation (eg, translation, rotation, etc.).
  • exemplary robots may include surgical robots, rehabilitation robots, biological robots, telepresence robots, companion robots, disinfection robots, etc., or any combination thereof.
  • the robot 140 may include a multi-degree-of-freedom robotic arm.
  • the robotic arm can include a base joint where the base of the robotic arm is located and an end joint where the flange of the robotic arm is located, wherein the flange of the robotic arm can be fixedly connected to end tools, such as electrode needles, puncture needles, syringes, ablation needles, etc.
  • end tools such as electrode needles, puncture needles, syringes, ablation needles, etc.
  • end tools such as electrode needles, puncture needles, syringes, ablation needles, etc.
  • FIG. 2 is a block diagram of an exemplary processor 102 shown in accordance with some embodiments of the present specification.
  • the processor 102 may include an acquisition module 210, a determination module 220, and a location module 230.
  • the acquiring module 210 can be used to acquire the first image and the second image of the target object.
  • the first image can be captured by an image capture device, and the second image can be captured by a medical imaging device.
  • the first image and the second image can be captured by an image capture device, and the second image can be captured by a medical imaging device.
  • the determining module 220 may be configured to determine at least one target area corresponding to at least one target part of the target object from the first image.
  • the target site can be less affected by physiological movement than other sites. For more content about determining at least one target area, reference may be made to step 304 in FIG. 3 and related descriptions.
  • the positioning module 230 may be configured to determine positioning information of the robot based on at least one target area and the second image. Positioning information may refer to location information of the robot or specific components thereof (eg, the end of a robotic arm for mounting a surgical device). In some embodiments, the positioning module 230 may obtain a first conversion relationship between the first coordinate system corresponding to the image acquisition device and the second coordinate system corresponding to the robot. The positioning module 230 may further determine a second conversion relationship between the first coordinate system and the third coordinate system corresponding to the medical imaging device based on the registration relationship between the at least one target area and the second image. The positioning module 230 may further determine the positioning information of the robot based on the first conversion relationship and the second conversion relationship. For more information about determining the positioning information of the robot, please refer to step 306 in FIG. 3 and related descriptions.
  • Each module in the above robot positioning device can be fully or partially realized by software, hardware and a combination thereof.
  • the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, and can also be stored in the memory of the computer device in the form of software, so that the processor can invoke and execute the corresponding operations of the above-mentioned modules.
  • the above description of the robot control system and its modules is only for the convenience of description, and does not limit this description to the scope of the illustrated embodiments. It can be understood that for those skilled in the art, after understanding the principle of the system, it is possible to combine various modules arbitrarily, or form a subsystem to connect with other modules without departing from this principle.
  • the obtaining module 210, the determining module 220 and the positioning module 230 disclosed in FIG. 2 may be different modules in one system, or one module may realize the functions of the above two modules.
  • each module in the robot control system may share a storage module, and each module may also have its own storage module. Such deformations are within the protection scope of this specification.
  • FIG. 3 is a flowchart of an exemplary robot localization process 300 according to some embodiments of the present specification.
  • the process 300 may be performed by the robot control system 100 .
  • process 300 may be stored in a storage device (eg, memory 104 ) in the form of a set of instructions (eg, an application program).
  • the processor 102 eg, one or more modules shown in FIG. 2
  • the robot In the medical field, robots have a wide range of applications. In order to accurately control the robot to perform operations, the robot needs to be positioned.
  • the commonly used robot positioning technology is marker-based positioning technology. Taking neurosurgery as an example, it is necessary to implant markers in the patient's skull or stick markers on the head, and let the patient carry the markers for medical scans. Further, the corresponding position information of the marker in the image space and the physical space can be determined, so as to complete the positioning of the robot according to the corresponding relationship between the image space and the physical space.
  • the markers usually cause additional harm to the patient, and once the relative displacement between the markers and the patient's head in the preoperative image, it will lead to inaccurate positioning of the robot, which will affect the preoperative planning or operation execution. Therefore, there is a need to provide efficient systems and methods for robot localization.
  • the robot may be positioned by performing the following operations of process 300 .
  • the processor 102 may acquire a first image and a second image of a target object.
  • the first image can be captured by an image capture device, and the second image can be captured by a medical imaging device.
  • target objects may include biological objects and/or non-biological objects.
  • a target object may be an animate or inanimate organic and/or inorganic substance.
  • a target object may include a specific part, organ, and/or tissue of a patient.
  • the target object may be the patient's head or face.
  • the first image may refer to an image captured by an image capture device (for example, the image capture device 130).
  • the first image may include a three-dimensional (3D) image and/or a two-dimensional (2D) image.
  • the first image may include a depth image of the target object, which includes distance information from points on the body surface of the target object to the reference point.
  • the processor 102 may acquire image data of the target object from an image acquisition device (eg, the image acquisition device 130 ), and determine the first image of the target object based on the image data.
  • an image acquisition device eg, the image acquisition device 130
  • the processor 102 may acquire optical data of the target object from the camera, and determine the first image based on the optical data.
  • the processor 102 may acquire point cloud data of the target object from the laser imaging device, and determine the first image based on the point cloud data.
  • the processor 102 may acquire depth data of the target object from the depth camera, and generate a depth image based on the depth data as the first image.
  • processor 102 may acquire the first image directly from an image capture device or a storage device (eg, memory 104).
  • the surgical position of the target object can be determined according to the preoperative planning, the target object can be kept fixed, and the pose of the image acquisition device can be adjusted so that the image The acquisition device photographs the target object from the target shooting angle and/or the target shooting height.
  • the pose of the image acquisition device can be adjusted so that the patient's face is completely within the collection field of view of the image acquisition device, and the image acquisition device shoots the patient's face vertically.
  • the second image may refer to a medical image acquired with a medical imaging device (eg, medical imaging device 120 ).
  • the medical imaging device may be a CT device.
  • the processor 102 may acquire CT image data of the target object through a CT device, and reconstruct a CT image according to the CT image data.
  • the processor 102 may further perform three-dimensional reconstruction on the CT image, so as to obtain a second image.
  • the processor 102 may acquire the second image of the target object directly from the medical imaging device (eg, the medical imaging device 120 ). Alternatively, processor 102 may retrieve the second image of the target object from a storage device (eg, memory 104 ) that stores the second image of the target object.
  • a storage device eg, memory 104
  • the processor 102 may first acquire a first initial image and/or a second initial image, the first initial image is acquired by using an image acquisition, and the second initial image is acquired by using a medical imaging device.
  • the processor 102 may generate the first image and/or the second image by processing the first initial image and/or the second initial image.
  • processor 102 may acquire a full-body depth image and a full-body CT map of the patient.
  • the processor 102 may segment the corresponding part of the patient's face from the whole body depth image as the first image.
  • the processor 102 may perform three-dimensional reconstruction on the whole body CT image, and segment the corresponding part of the patient's face from the three-dimensional reconstructed image as the second image.
  • the processor 102 may perform preprocessing operations (for example, target region segmentation, size adjustment, image resampling, image normalization, etc.) after acquiring the first image and the second image of the target object.
  • the processor 102 may further perform other steps in the process 300 on the preprocessed first image and the preprocessed second image.
  • the execution process of the process 300 is described below by taking the original first image and the second image as examples.
  • the processor 102 may determine at least one target region corresponding to at least one target part of the target object from the first image.
  • target sites may be less affected by physiological motion than other sites.
  • Physiological movements may include eye blinking, breathing movements, heartbeat movements, and the like.
  • the target site may be a static area of the face.
  • Static areas of the face may refer to areas that are less susceptible to changes in facial expression, for example, areas adjacent to facial bony structures.
  • the shape data of the human face under different facial expressions may be collected, the shape data may be statistically analyzed to obtain an area less affected by changes in facial expression, and the area may be determined as a static area of the face.
  • facial static regions may be determined through physiological structure information.
  • the area of the face that is close to the bony structure is defined as the static area of the face.
  • Exemplary facial static regions may include the forehead region, the bridge of the nose region, and the like.
  • the target area may refer to an area corresponding to the target part in the first image.
  • the processor 102 may determine at least one target area corresponding to at least one target part of the target object from the first image by using an image recognition technology (for example, a three-dimensional image recognition model).
  • an image recognition technology for example, a three-dimensional image recognition model.
  • the processor 102 may input the first image into a three-dimensional image recognition model, and the three-dimensional image recognition model may segment at least one target region from the first image.
  • the 3D image recognition model can be trained and generated based on training samples.
  • the training sample may include a sample first image of a sample object and a corresponding sample target area, wherein the sample first image is a training input, and the corresponding sample target area is a training label (label).
  • the processor 102 (or other processing devices) iteratively updates the initial model based on the training samples until certain conditions are met (for example, the loss function is less than a certain threshold, and the number of iterative training reaches a certain number).
  • the first image may be a three-dimensional image (eg, a three-dimensional depth image).
  • the processor 102 may acquire a two-dimensional reference image of the target object captured by an image capture device.
  • the processor 102 may determine at least one reference region corresponding to at least one target part from the two-dimensional reference image. Further, the processor 102 may determine at least one target area from the first image based on at least one reference area. For more details about determining at least one target area, please refer to FIG. 4 and related descriptions.
  • the processor 102 may determine localization information of the robot based on the at least one target area and the second image.
  • Positioning information of the robot may refer to position information of the robot or specific components thereof (eg, the end of a robotic arm for installing surgical devices).
  • the location information of specific components of the robot is referred to as the location information of the robot for short in the following.
  • the positioning information of the robot may include a positional relationship between the robot and a reference object (e.g., a target object, a reference object determined by the user, and/or the system), a coordinate system corresponding to the robot (i.e., a second coordinate system ) and other coordinate systems (for example, the first coordinate system, the third coordinate system), etc.
  • the positioning information may include the positional relationship between the coordinates of the robot and the coordinates of the target object in the same coordinate system.
  • the processor 102 may acquire a first conversion relationship between the first coordinate system corresponding to the image acquisition device and the second coordinate system corresponding to the robot.
  • the processor 102 may further determine a second conversion relationship between the first coordinate system and the third coordinate system corresponding to the medical imaging device based on the registration relationship between the at least one target area and the second image.
  • the processor 102 may further determine the positioning information of the robot based on the first conversion relationship and the second conversion relationship.
  • the processor 102 may determine a third conversion relationship between the second coordinate system corresponding to the robot and the third coordinate system corresponding to the medical imaging device based on the first conversion relationship and the second conversion relationship.
  • the first coordinate system corresponding to the image acquisition device may refer to a coordinate system established based on the image acquisition device. For example, a three-dimensional coordinate system established with the geometric center point of the image acquisition device as the origin.
  • the second coordinate system corresponding to the robot may refer to a coordinate system established based on the robot.
  • the second coordinate system may be the coordinate system of the end of the robot arm or the robot tool coordinate system.
  • the third coordinate system corresponding to the medical imaging device may refer to a coordinate system established based on the medical imaging device.
  • a conversion relationship between two coordinate systems may represent a mapping relationship between positions in the two coordinate systems.
  • the conversion relationship can be expressed as a transformation matrix, through which the first coordinate of any point in one coordinate system can be converted into the corresponding second coordinate in another coordinate system.
  • the conversion relationship between the coordinate system corresponding to the first object and the coordinate system corresponding to the second object may also be referred to as a relative position relationship or a position mapping relationship between the first object and the second object.
  • the first conversion relationship may also be referred to as a relative position relationship or a position mapping relationship between the image acquisition device and the robot.
  • the processor 102 may determine the first conversion relationship through a preset calibration method (for example, a hand-eye calibration algorithm). For example, the processor 102 can construct the intermediate reference object, and according to the first coordinate of the intermediate reference object in the first coordinate system (or the relative positional relationship between the intermediate reference object and the image acquisition device) and the intermediate reference object in the second coordinate system The second coordinate (or the relative positional relationship between the intermediate reference object and the robot) determines the first conversion relationship.
  • the image acquisition device can be installed on the robot, for example, installed at the end of the robot's operating arm.
  • the image acquisition device can also be set at any fixed position in the room where the robot is located, and the processor 102 can determine the position of the robot and the position of the image acquisition device according to the position of the image acquisition device, the position of the robot, and the position of the intermediate reference object. mapping relationship between them.
  • the processor 102 may determine the second transformation between the first coordinate system corresponding to the image acquisition device and the third coordinate system corresponding to the medical imaging device based on the registration relationship between at least one target area and the second image relation.
  • the registration relationship may reflect a correspondence relationship and/or a coordinate transformation relationship between points in at least one target area and points in the second image. Because both the at least one target area and the second image correspond to the same target object, there is a corresponding relationship between the points in the at least one target area and the points in the second image, so that the at least one target area and the second image can be determined by a registration technique. Registration relationship between images. For more details about determining the registration relationship between the at least one target area and the second image, please refer to FIG. 5 and related descriptions.
  • the first image and the second image of the target object may be acquired, at least one target area corresponding to at least one target part of the target object may be determined from the first image, and based on the at least one target area and the second
  • the second image determines the positioning information of the robot.
  • the target area and the second image that is, the medical image
  • the conversion relationship between the robot, the image acquisition device, and the coordinate system of the medical imaging device can be determined, without the need to paste or set additional markers on the target object, avoiding In order to cause additional damage to the target object.
  • using the target area and the second image for registration instead of directly using the first image and the second image for registration can reduce the influence of physiological motion on the registration result, thereby improving the accuracy of the registration result.
  • the positioning operation of the robot based on medical images can improve the accuracy of robot positioning, thereby improving the accuracy of preoperative planning or surgical implementation.
  • process 300 may be accomplished with one or more additional operations not described and/or omitting one or more operations discussed above.
  • the processor 102 may verify the positioning information to ensure the accuracy of the positioning of the robot.
  • the processor 102 may control the robot to plan surgery accordingly. Specifically, based on the positioning information and the surgical plan, the processor 102 can control the robot to move to the target position and perform surgical operations.
  • Fig. 4 is a flowchart of an exemplary process 400 for determining at least one target area according to some embodiments of the present specification.
  • the process 400 may be performed by the robot control system 100 .
  • process 400 may be stored in a storage device (eg, memory 104 ) in the form of a set of instructions (eg, an application program).
  • the processor 102 eg, one or more modules shown in FIG. 2
  • the processor 102 can execute a set of instructions and instruct one or more components of the robot control system 100 to perform the process 400 accordingly.
  • at least one target area described by operation 304 in FIG. 3 may be determined according to process 400 .
  • the processor 102 may acquire a two-dimensional reference image of the target object captured by an image capture device.
  • the first image shown in FIG. 3 may be a three-dimensional image, such as a depth image.
  • the processor 102 may directly acquire the two-dimensional reference image of the target object from the image acquisition device or the storage device.
  • the image acquisition device is a depth camera or a laser acquisition device, which can simultaneously acquire a two-dimensional reference image and a depth image of the target object.
  • the processor 102 may generate a two-dimensional reference image of the target object based on the first image.
  • the processor 102 may use an image transformation algorithm to convert the three-dimensional first image into a two-dimensional reference image.
  • the processor 102 may determine at least one reference region corresponding to at least one target part from the two-dimensional reference image.
  • the reference area may refer to an area corresponding to the target part in the two-dimensional reference image.
  • FIG. 8 is a schematic diagram of an exemplary two-dimensional reference image according to some embodiments of the present specification.
  • the shaded part may be a reference area, which corresponds to static facial areas such as the forehead and the bridge of the nose.
  • the processor 102 may determine at least one feature point related to at least one target part from the two-dimensional reference image.
  • the processor 102 may determine at least one feature point related to at least one target part from the two-dimensional reference image according to a preset feature point extraction algorithm.
  • Exemplary feature point extraction algorithms may include scale-invariant features transform (SIFT) algorithm, accelerated robust features (speeded up robust features, SURF) algorithm, direction gradient histogram (histogram of oriented gradient, HOR) algorithm , Gaussian function difference (difference of gaussian, DOG) algorithm, feature point extraction algorithm based on machine learning model, etc., or any combination thereof.
  • an image feature point extraction model (such as a trained neural network model) can be used to extract feature points from a two-dimensional reference image.
  • the processor 102 may input the two-dimensional reference image into the image feature point extraction model, and the image feature point extraction model may output at least one feature point related to at least one target part in the two-dimensional reference image.
  • the two-dimensional reference image may be a two-dimensional facial image
  • the two-dimensional facial image is input into the face feature point detection model, and at least one feature point related to at least one target part in the two-dimensional facial image may be obtained.
  • the facial feature points recognized in the two-dimensional facial image may include eye feature points, mouth feature points, eyebrow feature points, nose feature points, and the like.
  • the processor 102 may determine at least one reference area corresponding to at least one target part based on at least one feature point.
  • the eye region can be determined according to the feature points of the eye.
  • each of the at least one feature point has a fixed serial number, and at least one reference region can be determined from a two-dimensional reference image according to the fixed serial number of the feature point.
  • the feature points numbered 37 to 42 among the facial feature points are right eye feature points, and the right eye area can be determined according to these feature points.
  • the processor 102 may determine at least one reference region corresponding to at least one target part from the two-dimensional reference image by using an image recognition technology (for example, a two-dimensional image recognition model).
  • an image recognition technology for example, a two-dimensional image recognition model.
  • the processor 102 may two-dimensionally input the two-dimensional reference image into the image recognition model, and the two-dimensional image recognition model may segment at least one reference region from the two-dimensional reference image.
  • the two-dimensional image recognition model can be trained and generated based on training samples.
  • the training sample may include a sample two-dimensional reference image of the sample object and a corresponding sample reference area, wherein the sample two-dimensional reference image is a training input, and the corresponding sample reference area is a training label (label).
  • the processor 102 (or other processing devices) iteratively updates the initial model based on the training samples until certain conditions are met (for example, the loss function is smaller than a certain threshold, and the number of iterative training reaches a certain number of times).
  • the processor 102 may determine at least one target region from the first image based on the at least one reference region.
  • the processor 102 may determine at least one target area corresponding to the at least one reference area from the first image based on the mapping relationship between the two-dimensional reference image and the first image. Wherein, the mapping relationship between the two-dimensional reference image and the first image may be determined according to parameters of the image acquisition device.
  • At least one target area corresponding to at least one target part of the target object in the first image may be determined by using at least one reference area corresponding to at least one target part determined from the two-dimensional reference image.
  • determining the target area based on the two-dimensional reference image can reduce the influence of the depth parameter in the first image on determining the target area, and improve the accuracy of target area determination.
  • only one two-dimensional reference image and one first image can be used in the determination process, reducing the amount of data in the determination process and saving data processing time and resources.
  • process 400 may be accomplished with one or more additional operations not described and/or omitting one or more operations discussed above.
  • multiple two-dimensional reference images and the first image may be used to determine the target area.
  • at least one corresponding feature point may be determined from the first image based on the at least one feature point, and then the target area may be determined based on the at least one corresponding feature point.
  • Fig. 5 is a flowchart of an exemplary process 500 for determining a registration relationship according to some embodiments of the present specification.
  • the process 500 may be performed by the robot control system 100 .
  • process 500 may be stored in a storage device (eg, memory 104 ) in the form of a set of instructions (eg, an application program).
  • the processor 102 eg, one or more modules shown in FIG. 2
  • the registration relationship described by operation 306 in FIG. 3 may be determined according to process 500 .
  • the registration relationship between the at least one target region and the second image may be determined by a registration technique.
  • Exemplary registration techniques may include global registration techniques, local registration techniques, and the like.
  • the global registration technology can be based on the correspondence between the target plane in the at least one target area and the reference plane in the second image, and the local registration can be based on the target point in the at least one target area and the second image.
  • the corresponding relationship of the reference points is registered.
  • the registration relationship between the at least one target region and the second image may be determined through a global registration technique and a local registration technique.
  • the processor 102 may determine at least one reference point from the second image.
  • At least one reference point may constitute at least one reference point group, and each reference point group in the at least one reference point group may include at least three reference points located on the same plane. That is, each set of reference points (eg, at least three reference points) can determine a reference plane from the second image. Step 502 is to determine at least one reference plane from the second image.
  • the number of reference points in each reference point group can be determined according to actual conditions.
  • each reference point group may include at least four reference points lying on the same plane. It should be noted that three points can be coplanar, but the positional relationship of each point in the reference plane in the medical image can be more accurately determined based on the reference point group containing at least four reference points, thereby improving the accuracy of the determined registration relationship. accuracy.
  • the processor 102 may randomly determine at least one reference point from the second image, and determine position information of the at least one reference point based on a third coordinate system corresponding to the medical imaging device. For example, the processor 102 may use a random sampling algorithm to determine four reference points located on the same plane from the second image, and determine the corresponding coordinates of the four reference points in the third coordinate system corresponding to the medical imaging device, wherein the Four reference points can form a reference point group.
  • the processor 102 may determine at least one target point corresponding to at least one reference point from at least one target area.
  • the processor 102 may determine at least one target point corresponding to at least one reference point from at least one target region by using a global registration technique.
  • the processor 102 may determine a positional relationship between reference point pairs in the reference point group.
  • Reference point pairs can include adjacent reference points or non-adjacent reference points.
  • the positional relationship between the reference point pairs may include the distance between the reference points, the relative direction, and the like.
  • the positional relationship between pairs of reference points can be represented by vector information between the reference points.
  • the processor 102 may determine the distance and direction between any two reference points according to the coordinates of at least three reference points in each reference point group, and use vector information to represent the distance and direction.
  • the processor 102 may determine the target point group corresponding to the reference point group from at least one target area.
  • Each target point group may include at least three target points on the same plane. That is, each target point group (eg, at least three target points) can define a target plane from at least one target area.
  • the target point group corresponding to the reference point group can be determined from at least one target area.
  • the processor 102 may determine multiple candidate target point groups from at least one target area, and the number of candidate target points in each candidate target point group is the same as the number of reference points in the reference point group.
  • the processor 102 may further determine a candidate target point group most similar to the reference point group as a target point group.
  • the most similar to the reference point group may refer to the smallest difference between the vector information of the candidate target point group and the vector information of the reference point group.
  • FIG. 6A is a schematic diagram of an exemplary set of reference points 600A for a second image, according to some embodiments of the present specification.
  • the reference point group 600A in the second image may include four points a, b, c, and d, and the four points form the S1 plane.
  • the processor 102 may determine the distance between a-b, a-c, a-d, b-c, b-d, and c-d as the first distance according to the position information of the four points a, b, c, and d.
  • FIG. 6A is a schematic diagram of an exemplary set of reference points 600A for a second image, according to some embodiments of the present specification.
  • the reference point group 600A in the second image may include four points a, b, c, and d, and the four points form the S1 plane.
  • the processor 102 may determine the distance between a-b, a-c, a-d, b-c, b-d, and
  • FIG. 6B is a schematic diagram of a candidate target point group 600B of an exemplary target area according to some embodiments of the present specification.
  • the candidate target point group 600B includes four points a', b', c', and d', and the four points form the S2 plane.
  • the processor 102 can determine a'-b', a'-c', a'-d', b'-c', b' according to the position information of four points a', b', c', d' The distance between -d' and c'-d' is used as the second distance.
  • Processor 102 may determine a deviation between each first distance and a corresponding second distance.
  • the deviation may include a difference between the first distance and the second distance or a ratio of the first distance to the second distance.
  • the processor 102 may determine the difference between a-b and a'-b', the difference between a-c and a'-c', ..., the difference between c-d and c'-d', respectively.
  • the processor 102 may respectively determine the ratio of a-b to a'-b', the ratio of a-c to a'-c', ..., the ratio of c-d to c'-d'.
  • the processor 102 may determine the difference between the candidate target point group and the reference point group based on the deviation between the first distance and the second distance. For example, the processor 102 may sum or average the above differences, and use the sum or average of the differences as the difference between the candidate target point group and the reference point group.
  • at least one target area may include multiple candidate target point groups.
  • the processor 102 may separately determine a difference between each of the plurality of candidate target point groups and the reference point group.
  • the processor 102 may determine the candidate target point group with the smallest difference as the target point group.
  • the target point group is determined according to the distance of each reference point pair in the reference point group and the distance between two candidate target points in the candidate target point group. Since the position information of each reference point and each candidate target point is known, the calculation process of the distance is simple. , the target point group corresponding to the reference group can be effectively determined.
  • the processor 102 may determine a target point corresponding to each reference point in the set of reference points. For example, based on the target point group, the processor 102 may determine the target point corresponding to each reference point through the position correspondence between each target point in the target point group and each reference point in the reference point group.
  • the processor 102 can determine the target point group corresponding to the reference point group in the second image from at least one target area, and then determine the target point corresponding to the reference point.
  • the implementation process of the global registration technology is simple, which can improve the efficiency of determining the target point, and also ensure the accuracy of the target point selection.
  • the processor 102 may determine a target point corresponding to the reference point based on a local registration technique, so as to register the reference point and the target point. For example, the processor 102 may determine the target point corresponding to the reference point based on an iterative closest point algorithm (iterative closest point, ICP algorithm).
  • ICP algorithm iterative closest point algorithm
  • the processor 102 may obtain position information (for example, coordinates, depth information, etc.) of each candidate target point in at least one target area.
  • the processor 102 may determine the target point corresponding to each reference point according to the position information of each candidate target point, the position information of each reference point and a preset iterative closest point algorithm. For example, based on the iterative closest point algorithm, the processor 102 may determine the distance (such as the Euclidean distance ) to the nearest target point.
  • the processor 102 may determine the reference point from the second image, and search for the candidate target point closest to the reference point from at least one target area as the corresponding target point; according to the reference point and the corresponding target point, at least A transformation matrix (such as a rotation matrix and/or translation matrix) between a target region and the second image. At least one target area can be transformed based on the transformation matrix, and a new target point corresponding to the reference point can be determined from the transformed at least one target area.
  • a transformation matrix such as a rotation matrix and/or translation matrix
  • the above process can be iterated until a specific condition is met, for example, the condition can be that the distance between the reference point and the corresponding latest target point is less than a preset threshold, or the number of iterations is equal to the preset threshold, or the reference point and the corresponding latest target point The difference between the distance between the target points and the distance between the reference point and the corresponding last target point is smaller than a preset threshold.
  • the corresponding relationship eg, transformation matrix
  • the processor 102 may also determine an initial registration relationship between at least one target region and the second image based on the global registration technique described above. Further, the processor 102 may also use a local registration technique (for example, an iterative closest point algorithm) to adjust the initial registration relationship, so as to determine the registration relationship.
  • a local registration technique for example, an iterative closest point algorithm
  • the processor 102 may determine an initial registration relationship (ie, an initial corresponding relationship (eg, an initial transformation matrix)) between a target point in the target area and a reference point in the second image determined through the above-mentioned global registration technique. For each target point, the processor 102 may confirm or adjust an initial reference point corresponding to each target point based on a preset iterative closest point algorithm. For example, for each target point, the processor 102 may determine the reference point with the shortest distance (such as Euclidean distance) to the target point from the reference points. If the reference point is different from the initial reference point in the initial registration relationship, the initial registration relationship may be updated, and the reference point closest to the target point is used as a corresponding new reference point.
  • an initial registration relationship ie, an initial corresponding relationship (eg, an initial transformation matrix)
  • a transformation matrix (such as a rotation matrix and/or a translation matrix) between at least one target area and the second image can be determined.
  • At least one target area can be transformed based on the transformation matrix, and a corresponding new reference point can be determined from the second image according to the transformed target point.
  • the above process can be iterated until a specific condition is met, for example, the condition can be that the distance between the transformed target point and the new reference point is less than a preset threshold, or the number of iterations is equal to the preset threshold, or the transformed target point and The difference between the distance between the new reference points and the distance between the last target point and the last reference point is smaller than a preset threshold.
  • the corresponding relationship eg, transformation matrix
  • an initial registration relationship between at least one target area and the second image may be adjusted to determine the registration relationship.
  • the target point may be determined from at least one target area according to the reference point in the second image, so as to determine the registration relationship based on the position information of the reference point and the target point.
  • the processor 102 may determine a registration relationship between the at least one target region and the second image based on the correspondence relationship between the at least one reference point and the at least one target point.
  • the processor 102 may acquire the position information of at least one reference point and the position information of at least one target point, and determine the conversion relationship between the position information of at least one reference point and the position information of at least one target point, as a registration relationship. For example, the processor 102 may determine a transformation matrix between at least one reference point and at least one target point according to the coordinates of at least one reference point and the coordinates of at least one target point. The transformation matrix can represent the transformation relationship between the coordinate system of the reference point and the coordinate system of the target point. Further, the transformation matrix can also represent the registration relationship between at least one target region and the second image.
  • process 500 may be accomplished by one or more additional operations not described and/or by omitting one or more operations discussed above.
  • FIG. 9 is a block diagram of an exemplary processor 102 according to some embodiments of the present specification.
  • the processor 102 may include an acquisition module 910 , a feature point determination module 920 and a pose determination module 930 .
  • the acquiring module 910 may be used to acquire a target image of a target object.
  • the target image may include a three-dimensional image (eg, a depth image) and/or a two-dimensional image of the target object.
  • the target image can be captured using an image capture device. For more information on acquiring the target image, refer to step 1002 in FIG. 10 and related descriptions.
  • the feature point determining module 920 may be used to determine at least one target feature point of the target object from the target image.
  • the target feature points can be used to characterize the feature points of the target object in the image.
  • the feature point determination module 920 may also be configured to determine at least one reference feature point corresponding to at least one target feature point from the reference model of the target object.
  • the reference model can correspond to the target shooting angle.
  • the pose determination module 930 may be configured to determine a first target pose of the image acquisition device in the base coordinate system based on at least one target feature point and at least one reference feature point.
  • the first target pose may enable the image acquisition device to capture the target object from the target shooting angle and/or target shooting distance. For more information on determining the first target pose, please refer to step 1008 in FIG. 10 and related descriptions.
  • the above description of the robot control system and its modules is only for the convenience of description, and does not limit this description to the scope of the illustrated embodiments. It can be understood that for those skilled in the art, after understanding the principle of the system, it is possible to combine various modules arbitrarily, or form a subsystem to connect with other modules without departing from this principle.
  • the acquisition module 910, feature point determination module 920, and pose determination module 930 disclosed in FIG. 9 may be different modules in one system, or one module may realize the functions of the above two modules.
  • each module in the robot control system may share a storage module, and each module may also have its own storage module.
  • the obtaining module 910 and the obtaining module 210 may be the same module. Such deformations are within the protection scope of this specification.
  • the processor 102 may further include a registration module for realizing the registration and registration of the target object and the planning image.
  • Fig. 10 is a flowchart of an exemplary process 1000 for adjusting the pose of an image acquisition device according to some embodiments of the present specification.
  • the process 1000 can be executed by the robot control system 100 .
  • process 1000 may be stored in a storage device (eg, memory 104 ) in the form of a set of instructions (eg, an application program).
  • the processor 102 eg, one or more modules shown in FIG. 9
  • the processor 102 can execute a set of instructions and instruct one or more components of the robot control system 100 to perform the process 1000 accordingly.
  • adjusting the pose of the image acquisition device described in operation 302 in FIG. 3 may be performed according to the process 1000 .
  • the surgical body position of the target object can be determined according to the preoperative planning, the target object can be kept fixed, and the pose of the image acquisition device can be adjusted so that The target part of the target object is completely within the collection field of view of the image collection device.
  • the image acquisition device is usually installed on the robot, and the doctor manually drags the image acquisition device to align it with the target object. Because the doctor does not pay attention to the physical characteristics of the image acquisition device itself during the process of dragging the image acquisition device, it is difficult to quickly and accurately adjust the image acquisition device to the best posture, resulting in low shooting efficiency and reduced captured images. The precision of the data. In addition, doctors dragging the image acquisition device may also reduce the installation accuracy of the image acquisition device and the robot, further reducing the accuracy of the collected image data. Therefore, it is necessary to provide an effective system and method for adjusting the pose of an image acquisition device. In some embodiments, the pose of the image acquisition device can be adjusted by performing the following operations of the process 900 .
  • processor 102 may acquire a target image of a target object.
  • target objects may include biological objects and/or non-biological objects.
  • the target object may be the patient's head or face.
  • the target image may include a three-dimensional image (eg, a depth image) and/or a two-dimensional image of the target object.
  • the target image can be captured using an image capture device.
  • the acquisition manner of the target image may be similar to the acquisition manner of the first image. For more information about acquiring the target image, please refer to FIG. 3 and its related descriptions.
  • the processor 102 may determine at least one target feature point of the target object from the target image.
  • the target feature points can be used to characterize the feature points of the target object in the image.
  • the target object is the patient's head or face
  • the target feature points may be the facial contour points of the target object, as shown in FIG. 14 .
  • the processor 102 may use a feature point extraction algorithm to determine at least one target feature point of the target object from the target image.
  • at least one target feature point can be determined from the target image by using a feature point extraction model.
  • a feature point extraction model For more description about the feature point extraction algorithm, please refer to step 404 .
  • the processor 102 may determine at least one reference feature point corresponding to at least one target feature point from the reference model of the target object.
  • a reference model may refer to a standard model constructed from the characteristics of a target object.
  • the reference model can be a standard face model created according to the features of the human head, or a standard face model downloaded from an open source website.
  • the schematic diagrams of the front view and the side view of the standard human face model can be shown in Fig. 13A and Fig. 13B respectively.
  • the reference model may be stored or presented as a three-dimensional image, and analysis and processing of the reference model may be performed based on the three-dimensional image.
  • the reference model may correspond to the target shooting angle.
  • the target shooting angle may refer to the angle of the target object.
  • the target object is the patient's head or face, and the target shooting angle may be an angle directly facing the face.
  • each target feature point has its corresponding reference feature point.
  • the reference feature point and the corresponding target feature point may correspond to the same physical point on the target object.
  • the processor 102 may use the above-mentioned feature point extraction algorithm for determining at least one target feature point to determine a corresponding reference feature point set from the reference model, and then determine the reference feature point corresponding to the target feature point.
  • the processor 102 may determine at least one reference feature point from a reference model of the target object based on the at least one target feature point.
  • the processor 102 may determine at least one reference feature point corresponding to at least one target feature point from a reference model of the target object based on the structural features of the target object.
  • the processor 102 may use a machine learning model (for example, a mapping model, an active appearance model (active appearance model, AAM), a MediaPipe model, etc.), to determine at least one corresponding to at least one target feature point from the reference model of the target object.
  • a machine learning model for example, a mapping model, an active appearance model (active appearance model, AAM), a MediaPipe model, etc.
  • the processor 102 may input the target image and the reference model into the mapping model, and the mapping model may output the mapping relationship between the target image and the points in the reference model.
  • the processing device 102 may determine at least one reference feature point corresponding to the at least one target feature point based on the mapping relationship.
  • the processor 102 may determine a first target pose of the image acquisition device in the base coordinate system based on at least one target feature point and at least one reference feature point.
  • the base coordinate system can be any coordinate system.
  • the base coordinate system may refer to a coordinate system established based on the base of the robot.
  • the base coordinate system can be established with the center of the bottom surface of the base of the robot as the origin, the bottom surface of the base as the XY plane, and the vertical direction as the Z axis.
  • the first target pose may reflect the adjusted pose of the image acquisition device in the base coordinate system.
  • the first target pose may enable the image acquisition device to capture the target object from the target shooting angle and/or the target shooting distance.
  • the first target pose may be expressed as a conversion relationship between the coordinate system corresponding to the image acquisition device (ie, the updated first coordinate system) and the base coordinate system at the target shooting angle and target shooting distance.
  • the processor 102 may determine an initial pose of the target object relative to the image acquisition device based on at least one target feature point and at least one reference feature point.
  • the processor 102 may determine a second target pose of the target object relative to the image acquisition device based on the initial pose. Further, the processor 102 may determine the first target pose of the image acquisition device in the base coordinate system based on the second target pose.
  • FIG. 11 and its related descriptions For more information about determining the first target pose, please refer to FIG. 11 and its related descriptions.
  • the image acquisition device may be mounted on the robot.
  • the image acquisition device can be mounted on the end of the robotic arm of the robot. Therefore, the processor 102 can control the movement of the robot so that the image acquisition device is adjusted to the first target pose.
  • the processor 102 may determine a third target pose of the robot in the base coordinate system based on the first target pose. The processor 102 may control the robot to adjust to the third target pose, so that the image acquisition device adjusts to the first target pose. For more information about determining the third target pose, please refer to FIG. 12 and its related descriptions.
  • the first target pose of the image acquisition device in the base coordinate system can be determined, so that the image acquisition device can take pictures from the target angle and/or The target shooting distance is used to shoot the target object, so as to realize the automatic positioning of the image acquisition device to the best position. Therefore, the accuracy of the image data acquired by the image acquisition device can be improved, thereby improving the accuracy of subsequent robot positioning and preoperative planning.
  • process 1000 can be accomplished with one or more additional operations not described and/or omitting one or more operations discussed above. For example, before acquiring the target object, it may be determined whether the field of view of the image acquisition device includes the target object. For another example, after the image acquisition device is adjusted to the first target pose, it can acquire image data (for example, the first image) of the target object for registering the target object with the planning image and/or positioning the robot.
  • image data for example, the first image
  • Fig. 11 is a flowchart of an exemplary process 1100 for determining a first target pose according to some embodiments of the present specification.
  • the process 1100 may be performed by the robot control system 100 .
  • process 1100 may be stored in a storage device (eg, memory 104) in the form of a set of instructions (eg, an application program).
  • the processor 102 eg, one or more modules shown in FIG. 9
  • the first target pose described by operation 1008 in FIG. 10 may be determined according to process 1100 .
  • the processor 102 may determine an initial pose of the target object relative to the image acquisition device based on at least one target feature point and at least one reference feature point.
  • the initial pose may be expressed as a conversion relationship between the coordinate system corresponding to the target object and the first coordinate system corresponding to the image acquisition device when capturing the target image.
  • the initial pose can reflect the initial pose of the target object in the first coordinate system when the image acquisition device is at the target shooting angle, and/or the target object can be adjusted to correspond to the reference model in the first coordinate system The angle to be adjusted for the pose of .
  • determining the initial pose of the target object in the first coordinate system corresponding to the image acquisition device can be transformed into solving a PNP (Perspective N Points) problem.
  • the PNP problem can refer to the following object positioning problem: Assuming that the image acquisition device is a pinhole model and has been calibrated, take an image of N spatial points whose coordinates are known in the object coordinate system, and the coordinates of the N image points It is known that the coordinates of the N spatial points in the first coordinate system corresponding to the image acquisition device are determined.
  • the at least one target feature point may be a facial contour point of the target object.
  • the coordinates of the facial contour points in the coordinate system corresponding to the target object can be obtained.
  • the coordinate system corresponding to the target object may be a coordinate system established based on the target object. Taking the face as the target object as an example, the tip of the nose can be used as the origin of the face coordinate system, the plane parallel to the face is the XY plane, and the direction perpendicular to the face is the Z axis to establish the face coordinate system.
  • the initial pose of the target object relative to the image acquisition device before adjusting the pose It can also be called the pose transformation matrix transformed from the coordinate system corresponding to the target object to the first coordinate system It can be determined by formula (1):
  • M is a parameter matrix of the image acquisition device, which may be determined by intrinsic parameters of the image acquisition device.
  • the initial pose of the target object relative to the image acquisition device before adjusting the pose It can also be called the pose transformation matrix that transforms the coordinate system corresponding to the target object into the first coordinate system as It can be determined by formula (2):
  • the processor 102 may determine a second target pose of the target object relative to the image acquisition device based on the initial pose.
  • the second target pose may be expressed as a conversion relationship between the coordinate system corresponding to the target object and the updated first coordinate system corresponding to the adjusted image acquisition device.
  • the second target pose may reflect the pose of the target object in the updated first coordinate system after the image acquisition device adjusts the pose (for example, after adjusting to the target shooting angle and target shooting distance). , and/or in the updated first coordinate system, adjust the shooting distance to the distance that needs to be adjusted for the pose corresponding to the target shooting distance.
  • the target shooting distance may refer to the distance in the height direction between the target object and the camera when the quality of the image data collected by the image collection device meets a preset standard.
  • the processor 102 may acquire the target shooting distance of the image acquisition device.
  • the target shooting distance may be predetermined and stored in a storage device (eg, memory 104 ), and the processor 102 may acquire the target shooting distance in the storage device.
  • a complete target object or other reference objects such as a face model
  • the image acquisition device is moved along the optical axis direction of the image acquisition device, and the image acquisition device is acquired in multiple shots The image data of the target object captured at the distance.
  • the target shooting distance of the image acquisition device can be determined through the marked points.
  • multiple marking points can be set on the target object, and the coordinates of these marking points can be determined by a high-precision image acquisition device as standard coordinates.
  • Use the image acquisition device to shoot these marked points at different shooting distances, and determine the coordinates of these marking points at different shooting distances; compare the coordinates determined at different shooting distances with the standard coordinates; and compare the coordinates with the smallest difference from the standard coordinates.
  • the shooting distance is used as the target shooting distance.
  • the processor 102 may determine a distance transformation matrix based on the shooting distance of the target. As an example only, if the target shooting distance is H, the distance transformation matrix P can be shown as formula (3):
  • the processor 102 may determine the second target pose of the target object in the updated first coordinate system at the target shooting distance based on the distance transformation matrix and the initial pose (that is, the target object relative to the adjusted The target pose of the image acquisition device after the pose).
  • the second target pose can be determined according to formula (4):
  • the initial pose of the target object in the first coordinate system before updating may be the second target pose of the target object in the updated first coordinate system at the target shooting distance.
  • the shooting distance between the target object and the image acquisition device can be adjusted to the target shooting distance, thereby improving the image acquisition device. Accuracy of the captured image data.
  • the processor 102 may determine the first target pose of the image capture device in the base coordinate system based on the second target pose of the target object relative to the image capture device.
  • the image acquisition device can be installed on the robot, so through the connection structure between the image acquisition device and the robot, the fourth conversion relationship between the first coordinate system and the base coordinate system can be determined. Further, the first target pose may be determined based on the fourth conversion relationship and the second target pose.
  • the processor 102 may obtain a first conversion relationship between the first coordinate system and the second coordinate system corresponding to the robot.
  • the first conversion relationship may refer to a mapping relationship between the position of the robot and the position of the image acquisition device. For more information on obtaining the first conversion relationship, refer to FIG. 3 and related descriptions.
  • the processor 102 may also acquire a fifth transformation relationship between the second coordinate system and the base coordinate system.
  • the processor 102 may determine the fifth conversion relationship through a preset calibration method (for example, a hand-eye calibration method). For example, the processor 102 may determine the fifth conversion relationship in a manner similar to that for determining the first conversion relationship.
  • the fifth conversion relationship may be a parameter of the robot, and the processor 102 may acquire it from a controller of the robot. Further, the processor 102 may determine a fourth conversion relationship based on the first conversion relationship and the fifth conversion relationship. As an example only, the fourth conversion relationship can be determined according to formula (5):
  • T may represent the fourth conversion relationship between the first coordinate system corresponding to the image acquisition device and the base coordinate system, may represent the first conversion relationship between the first coordinate system corresponding to the image acquisition device and the second coordinate system corresponding to the robot, It may represent the fifth conversion relationship between the second coordinate system corresponding to the robot and the base coordinate system.
  • the processor 102 may determine the first target pose based on the fourth transformation relationship and the second target pose.
  • the first target pose can be determined according to formula (6):
  • in and may represent the first target pose.
  • the second target pose determines the first target pose of the target object in the base coordinate system, thereby adjusting the image acquisition device to the target shooting angle and target shooting distance, and improving the accuracy of the collected image data.
  • process 1100 may be accomplished with one or more additional operations not described and/or omitting one or more operations discussed above.
  • Fig. 12 is a flowchart of an exemplary process 1200 for adjusting the pose of an image acquisition device according to some embodiments of the present specification.
  • the process 1200 may be performed by the robot control system 100 .
  • process 1200 may be stored in a storage device (eg, memory 104 ) in the form of a set of instructions (eg, an application program).
  • the processor 102 eg, one or more modules shown in FIG. 9
  • the image acquisition device may be mounted on the robot.
  • the image acquisition device can be mounted on the end of the robotic arm of the robot.
  • the processor 102 may determine a third target pose of the robot in the base coordinate system based on the first target pose.
  • the third target pose can reflect the pose of the robot in the base coordinate system when the image acquisition device is in the first target pose, and/or adjust the robot in the base coordinate system so that the image acquisition The distance and angle that the robot needs to move when the device is adjusted to the first target pose.
  • the third target pose may be expressed as a conversion relationship between the robot's corresponding second coordinate system and the base coordinate system when the image acquisition device is in the first target pose.
  • the processor 102 may determine a third target pose of the robot in the base coordinate system based on the first conversion relationship and the first target pose.
  • the third target pose can be determined according to formula (7):
  • the processor 102 may control the robot to adjust to the third target pose, so that the image acquisition device adjusts to the first target pose.
  • FIG. 16A is a schematic diagram of an exemplary image acquisition device before pose adjustment according to some embodiments of the present specification.
  • Fig. 16B is a schematic diagram of an exemplary pose-adjusted image acquisition device according to some embodiments of the present specification.
  • Fig. 16C, Fig. 16E and Fig. 16G are schematic diagrams of image data collected by an exemplary image acquisition device before pose adjustment according to some embodiments of this specification, and Fig. 16D, Fig. 16F and Fig. 16H are according to some embodiments of this specification A schematic diagram of image data collected by an exemplary pose-adjusted image collection device shown in the embodiment.
  • the image acquisition device 1610 is adjusted from the pose in FIG. 16A to the first target pose in FIG. 16B .
  • Fig. 16C and Fig. 16D, Fig. 16E and Fig. 16F, Fig. 16G and Fig. 16H respectively, it can be determined that after the image acquisition device is adjusted to the first target pose, the target object 1620, the target object 1630 and the target object 1640 are in the image data
  • the imaging position of is adjusted to the center of the image data. That is to say, after the image acquisition device is adjusted to the first target pose, the image acquisition device can acquire image data of the target object at the target shooting angle and target shooting height.
  • the pose of the robot in the base coordinate system (that is, the third target pose) can be accurately determined, so that the image acquisition device can be accurately adjusted according to the pose of the robot in the base coordinate system pose. Therefore, the image acquisition device can acquire the image data of the target object at the target shooting angle and the target shooting distance, thereby improving the accuracy of the image data.
  • process 1200 may be accomplished with one or more additional operations not described and/or omitting one or more operations discussed above.
  • 15A-15C are schematic diagrams of exemplary adjustment of the pose of the image acquisition device 1510 according to some embodiments of the present specification.
  • the pose of the robot can be at a random initial position.
  • the patient's face i.e., the target object
  • the image capture device 1510 is used to capture a target image of the target object.
  • the image collection device 1510 is adjusted such that the field of view of the image collection device 1510 includes at least one target feature point.
  • the robot control system 100 (for example, the processor 102 ) can perform the pose adjustment of the image acquisition device as shown in FIGS. 10 to 12 The process is such that the image acquisition device 1510 can acquire the image data of the target image from the target shooting angle and the target shooting distance (as shown in FIG. 15C ).
  • the efficiency of adjusting the pose of the image acquisition device can be improved.
  • Fig. 17 is a schematic diagram of an exemplary adjustment of the pose of an image acquisition device according to some embodiments of the present specification.
  • the corresponding first coordinate system is camera_link
  • the coordinate system corresponding to the target object is face_link
  • the updated first coordinate system of the image acquisition device at the target shooting distance is face_link_view
  • the second coordinate system corresponding to the robot is tool_link
  • the base coordinate system is base_link.
  • the initial pose of the target object in the first coordinate system camera_link can be determined by formula (1) or formula (2) Through the formula (4), it can be determined that the second target pose of the target object in the updated first coordinate system is face_link_view at the target shooting distance
  • the first conversion relationship of the first coordinate system camera_link and the second coordinate system tool_link Get the fifth transformation relationship between the second coordinate system tool_link and the base coordinate system base_link
  • the fourth transformation relationship T between the first coordinate system camera_link and the base coordinate system base_link can be determined according to formula (5).
  • the first target pose can be determined according to formula (6)
  • the inverse operation based on the first conversion relationship and the first target pose can be determined according to formula (7).
  • it can be determined according to formula (7)
  • FIG. 18 is a schematic diagram of an exemplary robot control system 1800 according to some embodiments of the present specification.
  • the robot control system 1800 may include a robot 1810 , an image acquisition device 1820 and a processor 1830 .
  • the image capture device 1820 may be mounted on the robot 1810 (eg, the end of a robotic arm of the robot 1810 ).
  • the processor 1830 may be connected to the robot 1810 and the image acquisition device 1820 respectively. During operation, the processor 1830 may execute the robot positioning process and the image acquisition device pose adjustment process shown in some embodiments of the present application.
  • a computer device is provided.
  • the computer device may be a server, and its internal structure may be as shown in FIG. 19 .
  • the computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus.
  • the processor of the computer device is used to provide calculation and control capabilities.
  • the robot positioning method and the image acquisition device pose adjustment method shown in some embodiments of the present application may be implemented.
  • the memory of the computer device includes a non-volatile storage medium and an internal memory.
  • the non-volatile storage medium stores an operating system and computer programs.
  • the internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium.
  • the communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be realized through WIFI, an operator network, NFC (Near Field Communication) or other technologies.
  • a robot positioning method is realized.
  • the display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen
  • the input device of the computer device may be a touch layer covered on the display screen, or a button, a trackball or a touch pad provided on the casing of the computer device , and can also be an external keyboard, touchpad, or mouse.
  • Figure 19 is only a block diagram of a part of the structure related to the solution of this application, and does not constitute a limitation on the computer equipment on which the solution of this application is applied.
  • the specific computer equipment can be More or fewer components than shown in the figures may be included, or some components may be combined, or have a different arrangement of components.
  • FIG. 20 is a flowchart of an exemplary robot control process 2000 according to some embodiments of the present specification.
  • the process 2000 can be executed by the robot control system 100 .
  • the process 2000 may be stored in a storage device (eg, the memory 104 ) in the form of an instruction set (eg, an application program).
  • processor 102 e.g., one or more modules shown in FIG. 2 and/or one or more modules shown in FIG. 9
  • processor 102 can execute a set of instructions and instruct robot control system 100 accordingly One or more components perform the process 2000 .
  • the processor 102 may capture a target image of a target object using an image capture device.
  • the processor 102 may determine at least one target feature point of the target object from the target image.
  • the processor 102 may determine at least one reference feature point corresponding to at least one target feature point from the reference model of the target object.
  • the reference model can correspond to the target shooting angle.
  • the processor 102 may determine a first target pose of the image acquisition device in the base coordinate system based on at least one target feature point and at least one reference feature point, so that the image capture device can capture the target object from the target shooting angle.
  • the processor 102 may acquire a first image and a second image of the target object.
  • the first image can be captured using the adjusted image capture device, and the second image can be captured using the medical imaging device.
  • the processor 102 may determine at least one target region corresponding to at least one target part of the target object from the first image. At least one target site is less affected by physiological motion than other sites.
  • the processor 102 may determine positioning information of the robot based on the at least one target area and the second image.
  • the pose adjustment of the image acquisition device can be performed first, and then the positioning operation of the robot can be performed.
  • the image acquisition device can acquire the first image at the target shooting angle and target shooting distance, which can improve the accuracy of the first image, thereby improving the accuracy of robot positioning, thereby improving the accuracy of preoperative planning or surgical implementation sex.
  • process 2000 can be accomplished by one or more additional operations not described and/or by omitting one or more operations discussed above.
  • the processor 102 may also implement registration and registration of the target object and the planning image based on the first image, thereby improving the accuracy of the registration scheme.
  • Some embodiments of this specification also provide an electronic device, which includes: at least one storage medium storing computer instructions; at least one processor executing the computer instructions to implement the robot positioning method and image acquisition described in this specification Device pose adjustment method.
  • the electronic device may further include a transmission device and an input and output device, wherein the transmission device and the input and output device may be connected to the processor.
  • Some embodiments of this specification also provide a computer-readable storage medium, which stores computer instructions.
  • the computer When the computer reads the computer instructions, the computer executes the robot positioning method and the image acquisition device pose adjustment method described in this specification. .
  • the relevant descriptions in FIG. 1A to FIG. 17 which will not be repeated here.
  • numbers describing the quantity of components and attributes are used. It should be understood that such numbers used in the description of the embodiments use the modifiers "about”, “approximately” or “substantially” in some examples. grooming. Unless otherwise stated, “about”, “approximately” or “substantially” indicates that the stated figure allows for a variation of ⁇ 20%. Accordingly, in some embodiments, the numerical parameters used in the specification and claims are approximations that can vary depending upon the desired characteristics of individual embodiments. In some embodiments, numerical parameters should take into account the specified significant digits and adopt the general digit reservation method. Although the numerical ranges and parameters used in some embodiments of this specification to confirm the breadth of the range are approximations, in specific embodiments, such numerical values are set as precisely as practicable.

Landscapes

  • Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Robotics (AREA)
  • Mechanical Engineering (AREA)
  • General Health & Medical Sciences (AREA)
  • Theoretical Computer Science (AREA)
  • Surgery (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Medical Informatics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Computation (AREA)
  • Veterinary Medicine (AREA)
  • Physics & Mathematics (AREA)
  • Public Health (AREA)
  • Molecular Biology (AREA)
  • Multimedia (AREA)
  • General Physics & Mathematics (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Artificial Intelligence (AREA)
  • Biomedical Technology (AREA)
  • Computing Systems (AREA)
  • Databases & Information Systems (AREA)
  • Software Systems (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Animal Behavior & Ethology (AREA)
  • Pathology (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Manipulator (AREA)

Abstract

本说明书实施例提供一种机器人定位和位姿调整方法和系统，该方法包括：获取图像采集装置采集的第一图像和医学成像设备采集的第二图像，并基于第一图像和第二图像，确定机器人的定位信息。该方法还包括利用图像采集装置采集目标对象的目标图像，基于目标图像和目标对象的参考模型，确定图像采集装置在基坐标系下的目标位姿，并控制机器人使图像采集装置调整至目标位姿。

Description

机器人定位和位姿调整方法和系统
交叉引用
本申请要求2021年05月10日提交的名称为“机器人定位方法、装置、系统及计算机设备”的中国专利申请202110505732.6的优先权和2021年11月19日提交的名称为“相机位姿调整方法、空间注册方法、系统和存储介质”的中国专利申请202111400891.6的优先权，上述申请的全部内容以引用方式并入本文。
技术领域
本说明书涉及机器人领域，尤其涉及一种机器人定位和位姿调整方法和系统。
背景技术
近几年来，机器人在医疗领域有着较为广泛的应用，比如骨科、神经外科、胸腹介入手术或治疗等等。一般而言，机器人包括多自由度结构的机械臂，该机械臂可以包括机械臂基座所在的基座关节和机械臂法兰盘所在的末端关节。其中，机械臂法兰盘固定连接有末端工具，比如电极针、穿刺针、注射器、消融针等各种手术工具。
在使用机器人时，需要对机器人进行精确地定位和位姿调整，从而准确地完成术前规划和/或手术执行。
发明内容
本说明书实施例之一提供一种机器人定位方法。该方法可以包括获取目标对象的第一图像和第二图像，第一图像利用图像采集装置采集，第二图像利用医学成像设备采集；从第一图像中确定与目标对象的至少一个目标部位对应的至少一个目标区域，至少一个目标部位受生理运动的影响小于其他部位；以及基于至少一个目标区域和第二图像，确定机器人的定位信息。
本说明书实施例之一提供一种图像采集装置位姿调整方法。该方法可以包括利用图像采集装置采集目标对象的目标图像；从目标图像中确定目标对象的至少一个目标特征点；从目标对象的参考模型中确定至少一个目标特征点对应的至少一个参考特征点，参考模型对应目标拍摄角度；基于至少一个目标特征点和至少一个参考特征点，确定图像采集装置在基坐标系下的第一目标位姿。
本说明书实施例之一提供一种机器人定位系统。该系统可以包括存储设备，存储计算机指令；处理器，与存储设备相连接，当执行计算机指令时，处理器使系统执行下述操作：获取目标对象的第一图像和第二图像，第一图像利用图像采集装置采集，第二图像利用医学成像设备采集；从第一图像中确定与目标对象的至少一个目标部位对应的至少一个目标区域， 至少一个目标部位受生理运动的影响小于其他部位；以及基于至少一个目标区域和第二图像，确定机器人的定位信息。
附图说明
本说明书将以示例性实施例的方式进一步说明，这些示例性实施例将通过附图进行详细描述。这些实施例并非限制性的，在这些实施例中，相同的编号表示相同的结构，其中：
图1A是根据本说明书一些实施例所示的示例性机器人控制系统的应用场景示意图；
图1B是根据本说明书一些实施例所示的示例性机器人控制系统的应用场景示意图；
图1C是根据本说明书一些实施例所示的示例性服务器的示意图；
图2是根据本说明书的一些实施例所示的示例性处理器的模块图；
图3是根据本说明书的一些实施例所示的示例性机器人定位流程的流程图；
图4是根据本说明书的一些实施例所示的示例性的确定至少一个目标区域的流程的流程图；
图5是根据本说明书的一些实施例所示的示例性的确定配准关系的流程的流程图；
图6A是根据本说明书的一些实施例所示的示例性第二图像的参考点组的示意图；
图6B是根据本说明书的一些实施例所示的示例性目标区域的候选目标点组的示意图；
图7是根据本说明书的一些实施例所示的示例性面部特征点示意图；
图8是根据本说明书的一些实施例所示的示例性二维面部图像的示意图；
图9是根据本说明书的一些实施例所示的示例性处理器的模块图；
图10是根据本说明书的一些实施例所示的示例性的调整图像采集装置位姿的流程的流程图；
图11是根据本说明书的一些实施例所示的示例性的确定第一目标位姿的流程的流程图；
图12是根据本说明书的一些实施例所示的示例性的调整图像采集装置位姿的流程的流程图；
图13A是根据本说明书的一些实施例所示的示例性的标准人脸模型的正视图；
图13B是根据本说明书的一些实施例所示的示例性的标准人脸模型的侧视图；
图14是根据本说明书的一些实施例所示的示例性面部轮廓点的示意图；
图15A-图15C是根据本说明书的一些实施例所示的示例性调整图像采集装置的位姿的示意图；
图16A是根据本说明书的一些实施例所示的示例性位姿调整前的图像采集装置示意图；
图16B是根据本说明书的一些实施例所示的示例性位姿调整后的图像采集装置示意图；
图16C、图16E和图16G是根据本说明书的一些实施例所示的示例性位姿调整前的图像采集装置采集的图像数据的示意图；
图16D、图16F和图16H是根据本说明书的一些实施例所示的示例性位姿调整后的图像采集装置采集的图像数据的示意图；
图17是根据本说明书的一些实施例所示的示例性调整图像采集装置位姿的示意图；
图18是根据本说明书的一些实施例所示的示例性机器人控制系统的示意图；
图19是根据本说明书的一些实施例所示的示例性计算机设备的示意图；
图20是根据本说明书的一些实施例所示的示例性机器人控制流程的流程图。
具体实施方式
为了更清楚地说明本说明书实施例的技术方案，下面将对实施例描述中所需要使用的附图作简单的介绍。显而易见地，下面描述中的附图仅仅是本说明书的一些示例或实施例，对于本领域的普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图将本说明书应用于其它类似情景。除非从语言环境中显而易见或另做说明，图中相同标号代表相同结构或操作。
应当理解，本文使用的“系统”、“装置”、“单元”和/或“模块”是用于区分不同级别的不同组件、元件、部件、部分或装配的一种方法。然而，如果其他词语可实现相同的目的，则可通过其他表达来替换所述词语。
如本说明书和权利要求书中所示，除非上下文明确提示例外情形，“一”、“一个”、“一种”和/或“该”等词并非特指单数，也可包括复数。一般说来，术语“包括”与“包含”仅提示包括已明确标识的步骤和元素，而这些步骤和元素不构成一个排它性的罗列，方法或者设备也可能包含其它的步骤或元素。
本说明书中使用了流程图用来说明根据本说明书的实施例的系统所执行的操作。应当理解的是，前面或后面操作不一定按照顺序来精确地执行。相反，可以按照倒序或同时处理各个步骤。同时，也可以将其他操作添加到这些过程中，或从这些过程移除某一步或数步操作。
图1A是根据本说明书一些实施例所示的示例性机器人控制系统100的应用场景示意图。
机器人控制系统100可以用于对机器人进行地定位和位姿调整。如图1A所示，在一些实施例中，机器人控制系统100中可以包括服务器110、医学成像设备120和图像采集装置130。机器人控制系统100中的多个组件之间可以通过网络互相连接。例如，服务器110和医学成像设备120可以通过网络连接或通信。又例如，服务器110和图像采集装置130可以 通过网络连接或通信。在一些实施例中，机器人控制系统100中的部件之间的连接是可变的。例如，医学成像设备120可以直接与图像采集装置130连接。
服务器110可以用于处理来自机器人控制系统100的至少一个组件(例如，医学成像设备120、图像采集装置130)或外部数据源(例如，云数据中心)的数据和/或信息。例如，服务器110可以获取图像采集装置130采集的第一图像和医学成像设备120采集的第二图像，并基于第一图像和第二图像，确定机器人的定位信息。又例如，服务器110可以利用图像采集装置130采集目标对象的目标图像，基于目标图像和目标对象的参考模型，确定图像采集装置130在基坐标系下的第一目标位姿。在一些实施例中，服务器110可以是单一服务器或服务器组。该服务器组可以是集中式或分布式的(例如，服务器110可以是分布式系统)。在一些实施例中，服务器110可以是区域的或者远程的。在一些实施例中，服务器110可以在云平台上实施，或者以虚拟方式提供。仅作为示例，云平台可以包括私有云、公共云、混合云、社区云、分布云、内部云、多层云等或其任意组合。
在一些实施例中，服务器110可以包括一个或多个组件。如图1C所示，服务器110可以包括一个或多个(图1C中仅示出一个)处理器102、存储器104、传输设备106以及输入输出设备108。本领域普通技术人员可以理解，图1C所示的结构仅为示意，并不对服务器110的结构造成限制。例如，服务器110还可以包括比图1C中所示更多或者更少的组件，或者具有与图1C所示出的不同配置。
处理器102可以处理从其他设备或系统组成部分中获得的数据和/或信息。处理器102可以基于这些数据、信息和/或处理结果执行程序指令，以执行一个或多个本说明书中描述的功能。在一些实施例中，处理器102可以包含一个或多个子处理设备(例如，单核处理设备或多核多芯处理设备)。仅作为示例，处理器102可以包括微处理器(MCU)、中央处理器(CPU)、专用集成电路(ASIC)、专用指令处理器(ASIP)、图形处理器(GPU)、物理处理器(PPU)、数字信号处理器(DSP)、现场可编程门阵列(FPGA)、可编辑逻辑电路(PLD)、控制器、微控制器单元、精简指令集电脑(RISC)等，或以上任意组合。在一些实施例中，处理器102可以集成或包括于机器人控制系统100的一个或多个其他组件(例如，医学成像设备120、图像采集装置130或其他可能的组件)中。
存储器104可以存储数据、指令和/或任何其他信息。例如，存储器104可以用于存储计算机程序，例如，应用软件的软件程序以及模块，如在本实施例中的定位方法和位姿调整方法对应的计算机程序。处理器102通过运行存储在存储器104内的计算机程序，从而执行各种功能应用以及数据处理，即实现上述的方法。存储器104可以包括高速随机存储器，还可包括非易失性存储器，如一个或者多个磁性存储装置、闪存、或者其他非易失性固态存储 器。在一些实例中，存储器104可进一步包括相对于处理器102远程设置的存储器，这些远程存储器可以通过网络连接至终端。上述网络的实例包括但不限于互联网、企业内部网、局域网、移动通信网及其组合。在一些实施例中，存储器104可以在云平台上实现。
传输设备106可以用于实现通信功能。例如，传输设备106可以用于经由一个网络接收或者发送数据。在一个实例中，传输设备106可以包括一个网络适配器(Network Interface Controller，简称为NIC)，其可通过基站与其他网络设备相连从而可与互联网进行通讯。在一个实例中，传输设备106可以为射频(Radio Frequency，简称为RF)模块，其用于通过无线方式与互联网进行通讯。
输入输出设备108可以用于输入或输出信号、数据或信息。在一些实施例中，输入输出设备108可以使用户与机器人控制系统100进行联系。示例性输入设备可以包括键盘、鼠标、触摸屏和麦克风等，或其任意组合。示例性输出设备可以包括显示装置、扬声器、打印机、投影仪等，或其任意组合。示例性显示装置可以包括液晶显示器(LCD)、基于发光二极管(LED)的显示器、平板显示器、曲面显示器、电视设备、阴极射线管(CRT)等，或其任意组合。
在一些实施例中，服务器110可以设定于任意位置(例如，机器人所在的房间、专门放置服务器的房间等)，只需要确保服务器110与医学成像设备120、图像采集装置130可以正常通信。
医学成像设备120可以用于对检测区域或扫描区域内的目标对象进行扫描，得到目标对象的成像数据。在一些实施例中，对象可以包括生物对象和/或非生物对象。例如，对象可以是有生命或无生命的有机和/或无机物质。
在一些实施例中，医学成像设备120可以是用于疾病诊断或研究目的的非侵入性成像装置。例如，医学成像设备120可以包括单模态扫描仪和/或多模态扫描仪。单模态扫描仪可以包括例如超声波扫描仪、X射线扫描仪、计算机断层扫描(CT)扫描仪、核磁共振成像(MRI)扫描仪、超声检查仪、正电子发射计算机断层扫描(PET)扫描仪、光学相干断层扫描(OCT)扫描仪、超声(US)扫描仪、血管内超声(IVUS)扫描仪、近红外光谱(NIRS)扫描仪、远红外(FIR)扫描仪等或其任意组合。多模态扫描仪可以包括例如X射线成像-核磁共振成像(X射线-MRI)扫描仪、正电子发射断层扫描-X射线成像(PET-X射线)扫描仪、单光子发射计算机断层扫描-核磁共振成像(SPECT-MRI)扫描仪、正电子发射断层扫描-计算机断层摄影(PET-CT)扫描仪、数字减影血管造影-核磁共振成像(DSA-MRI)扫描仪等或其任意组合。上述扫描仪仅用于说明目的，而无意限制本说明书的范围。仅作为示例，医学成像设备120可以包括CT扫描仪。
图像采集装置130可以用于采集目标对象的图像数据(例如，第一图像、目标图像)。示例性图像采集装置可以包括相机、光学传感器、雷达传感器、结构光扫描仪等，或其任意组合。例如，图像采集装置130可以包括相机(例如，深度相机、立体三角测量相机等)、光学传感器(例如，红绿蓝-深度(RGB-D)传感器等)等可以捕获目标对象光学数据的设备。又例如，图像采集装置130可以包括激光成像设备(例如，相激光采集设备、点激光采集设备等)等能够获取对象的点云数据的设备。点云数据可以包括多个数据点，每个数据点可以代表对象体表上的一个物理点，并且可以使用该物理点的一个或多个特征值(例如，与物理点的位置和/或组成相关的特征值)来描述目标对象。点云数据可以用于重建目标对象的图像。再例如，图像采集装置130可以包括能够获取目标对象的位置数据和/或深度数据的设备，例如结构光扫描仪、飞行时间(TOF)设备、光三角测量设备、立体匹配设备等，或其任意组合。获取的位置数据和/或深度数据可以用于重建目标对象的图像。
在一些实施例中，图像采集装置130可以通过可拆卸或不可拆卸的连接方式安装于机器人上。例如，图像采集装置130可以可拆卸地设置于机器人的机械臂末端。在一些实施例中，图像采集装置130可以通过可拆卸或不可拆卸的连接方式安装在机器人以外的位置。例如，图像采集装置130可以设置于机器人所处房间的某一固定位置。
在一些实施例中，基于图像采集装置130的位置、机器人的位置以及图像采集装置130的标定参数(例如，尺寸、拍摄角度)，可以确定图像采集装置130与机器人140的位置之间的对应关系。例如，可以确定图像采集装置130对应的第一坐标系与机器人对应的第二坐标系之间的映射关系(即，第一转换关系)。
应该注意的是，上述描述仅出于说明性目的而提供，并不旨在限制本说明书的范围。对于本领域普通技术人员而言，在本说明书内容的指导下，可做出多种变化和修改。可以以各种方式组合本说明书描述的示例性实施例的特征、结构、方法和其他特征，以获取另外的和/或替代的示例性实施例。例如，图像采集装置130可以包括多个图像采集装置。
在一些实施例中，如图1B所示，机器人控制系统100还可以包括机器人140。
机器人140可以基于指令执行对应操作。例如，基于移动指令，机器人140可以执行移动操作(例如，平移、旋转等)。示例性机器人可以包括手术机器人、康复机器人、生物机器人、远程呈现机器人、随播机器人、消毒机器人等，或其任意组合。
仅作为示例，机器人140可以包括多自由度结构的机械臂。机械臂可以包括机械臂基座所在的基座关节和机械臂法兰盘所在的末端关节，其中，机械臂法兰盘可以固定连接末端工具，例如，电极针、穿刺针、注射器、消融针等各种手术工具。
图2是根据本说明书的一些实施例所示的示例性处理器102的模块图。处理器102可 以包括获取模块210、确定模块220和定位模块230。
获取模块210可以用于获取目标对象的第一图像和第二图像。第一图像可以利用图像采集装置采集，第二图像可以利用医学成像设备采集。关于获取第一图像和第二图像的更多内容可以参考图3的步骤302及其相关描述。
确定模块220可以用于从第一图像中确定与目标对象的至少一个目标部位对应的至少一个目标区域。目标部位受生理运动的影响可以小于其他部位。关于确定至少一个目标区域的更多内容可以参考图3的步骤304及其相关描述。
定位模块230可以用于基于至少一个目标区域和第二图像，确定机器人的定位信息。定位信息可以指机器人或其特定组件(例如，用于安装手术器件的机械臂末端)的位置信息。在一些实施例中，定位模块230可以获取图像采集装置对应的第一坐标系与机器人对应的第二坐标系之间的第一转换关系。定位模块230可以进一步基于至少一个目标区域和第二图像之间的配准关系，确定第一坐标系和医学成像设备对应的第三坐标系的第二转换关系。定位模块230可以进而基于第一转换关系和第二转换关系，确定机器人的定位信息。关于确定机器人的定位信息的更多内容可以参考图3的步骤306及其相关描述。
上述机器人定位装置中的各个模块可全部或部分通过软件、硬件及其组合来实现。上述各模块可以硬件形式内嵌于或独立于计算机设备中的处理器中，也可以以软件形式存储于计算机设备中的存储器中，以便于处理器调用执行以上各个模块对应的操作。
需要注意的是，以上对于机器人控制系统及其模块的描述，仅为描述方便，并不能把本说明书限制在所举实施例范围之内。可以理解，对于本领域的技术人员来说，在了解该系统的原理后，可能在不背离这一原理的情况下，对各个模块进行任意组合，或者构成子系统与其他模块连接。例如，图2中披露的获取模块210、确定模块220和定位模块230可以是一个系统中的不同模块，也可以是一个模块实现上述的两个模块的功能。又例如，机器人控制系统中各个模块可以共用一个存储模块，各个模块也可以分别具有各自的存储模块。诸如此类的变形，均在本说明书的保护范围之内。
图3是根据本说明书的一些实施例所示的示例性机器人定位流程300的流程图。在一些实施例中，流程300可以由机器人控制系统100执行。例如，流程300可以以指令集(例如，应用程序)的形式存储在存储设备(例如，存储器104)中。在一些实施例中，处理器102(例如，图2中所示的一个或多个模块)可以执行指令集并相应指示机器人控制系统100的一个或多个组件执行流程300。
在医疗领域中，机器人有着较为广泛的应用。为了准确地控制机器人执行操作，需要对机器人进行定位。现在常用的机器人定位技术为基于标记物的定位技术。以神经外科手术 为例，需要在患者头骨中植入标记物或在头部粘贴标记物，让患者带着标记物进行医学扫描。进一步地，可以确定标记物在影像空间和物理空间中对应的位置信息，从而根据影像空间与物理空间的对应关系，完成机器人的定位。但标记物通常会对患者造成额外的伤害，且一旦标记物与术前影像中患者的头部发生相对位移，则会导致机器人的定位不准确，进而影响术前规划或手术执行。因此，需要提供有效的系统和方法用于进行机器人定位。在一些实施例中，可以通过执行流程300的以下操作来定位机器人。
在302中，处理器102(例如，获取模块210)可以获取目标对象的第一图像和第二图像。第一图像可以利用图像采集装置采集，第二图像可以利用医学成像设备采集。
在一些实施例中，目标对象可以包括生物对象和/或非生物对象。例如，目标对象可以是有生命或无生命的有机和/或无机物质。又例如，目标对象可以包括患者的特定部分、器官和/或组织。仅作为示例，在神经外科的场景下，目标对象可以为患者的头部或面部。
第一图像可以指利用图像采集装置(例如，图像采集装置130)采集的图像。第一图像可以包括三维(3D)图像和/或二维(2D)图像。在一些实施例中，第一图像可以包括目标对象的深度图像，其中包含目标对象体表上的点到参考点的距离信息。
在一些实施例中，处理器102可以从图像采集装置(例如，图像采集装置130)获取目标对象的图像数据，并基于图像数据确定目标对象的第一图像。例如，图像采集装置为相机时，处理器102可以从相机获取目标对象的光学数据，并基于光学数据确定第一图像。又例如，图像采集装置为激光成像设备时，处理器102可以从激光成像设备获取目标对象的点云数据，并基于点云数据确定第一图像。再例如，图像采集装置为深度相机时，处理器102可以从深度相机获取目标对象的深度数据，并基于深度数据生成深度图像作为第一图像。在一些实施例中，处理器102可以直接从图像采集装置或存储设备(例如，存储器104)获取第一图像。
在一些实施例中，利用图像采集装置采集的目标对象的第一图像前，可以根据术前规划确定目标对象的手术体位，将目标对象保持固定不动，调整图像采集装置的位姿，使得图像采集装置从目标拍摄角度和/或目标拍摄高度拍摄目标对象。例如，可以调整图像采集装置的位姿，以使患者面部完全位于图像采集装置的采集视野范围内，并使图像采集装置垂直正对患者的面部进行拍摄。关于调整图像采集装置位姿的更多内容可以参见图9-17及其相关描述。
第二图像可以指利用医学成像设备(例如，医学成像设备120)采集的医学图像。仅作为示例，医学成像设备可以为CT设备。相应地，处理器102可以通过CT设备获取目标对象的CT影像数据，根据CT影像数据重建CT图像。处理器102可以进一步对CT图像进行三 维重建，从而获取第二图像。
在一些实施例中，处理器102可以直接从医学成像设备(例如，医学成像设备120)获取目标对象的第二图像。或者，处理器102可以从存储目标对象的第二图像的存储设备(例如，存储器104)获取目标对象的第二图像。
在一些实施例中，处理器102可以先获取第一初始图像和/或第二初始图像，第一初始图像利用图像采集，第二初始图像利用医学成像设备采集。处理器102可以通过处理第一初始图像和/或第二初始图像，以生成第一图像和/或第二图像。仅作为示例，处理器102可以获取病人的全身深度图像和全身CT图。处理器102可以从全身深度图像中分割出患者面部对应的部分，以作为第一图像。处理器102可以对全身CT图进行三维重建，并从三维重建后的图中分割出患者面部对应的部分，以作为第二图像。
在一些实施例中，处理器102可以在获取目标对象的第一图像和第二图像后进行预处理操作(例如，目标区域分割、尺寸调整、图像重采样、图像归一化等)。处理器102可以进一步对预处理后的第一图像和预处理后的第二图像执行流程300中的其他步骤。出于示例目的，下文以原始的第一图像和第二图像为例描述流程300的执行过程。
在304中，处理器102(例如，确定模块220)可以从第一图像中确定与目标对象的至少一个目标部位对应的至少一个目标区域。
在一些实施例中，目标部位受生理运动的影响可以小于其他部位。生理运动可以包括眨眼、呼吸运动、心跳运动等。仅作为示例，目标部位可以为面部静态区域。面部静态区域可以指不易受面部表情变化影响的区域，例如，临近面部骨质结构的区域。在一些实施例中，可以采集人脸在不同面部表情下的形状数据，对形状数据进行统计分析以获取受面部表情变化影响较小的区域，并且确定该区域为面部静态区域。在一些实施例中，可以通过生理结构信息确定面部静态区域。例如，将面部距离骨质结构近的区域定为面部静态区域。示例性面部静态区域可以包括额头区域、鼻梁区域等。
目标区域可以指在第一图像中和目标部位对应的区域。在一些实施例中，处理器102可以通过图像识别技术(例如，三维图像识别模型)从第一图像中确定与目标对象的至少一个目标部位对应的至少一个目标区域。仅作为示例，处理器102可以将第一图像输入三维图像识别模型，三维图像识别模型可以将至少一个目标区域从第一图像中分割出来。三维图像识别模型可以基于训练样本训练生成。训练样本可以包括样本对象的样本第一图像和对应的样本目标区域，其中，样本第一图像为训练输入，对应的样本目标区域为训练标签(label)。在一些实施例中，处理器102(或其他处理设备)基于训练样本对初始模型进行迭代更新，直至特定条件被满足(例如，损失函数小于一定阈值，迭代训练次数达到一定次数)。
在一些实施例中，第一图像可以是三维图像(例如，三维深度图像)。处理器102可以获取利用图像采集装置采集的目标对象的二维参考图像。处理器102可以从二维参考图像中确定至少一个目标部位对应的至少一个参考区域。进一步地，处理器102可以基于至少一个参考区域，从所述第一图像中确定至少一个目标区域。关于确定至少一个目标区域的更多内容可以参见图4及其相关描述。
在306中，处理器102(例如，定位模块230)可以基于至少一个目标区域和第二图像，确定机器人的定位信息。
机器人的定位信息可以指机器人或其特定组件(例如，用于安装手术器件的机械臂末端)的位置信息。为方便描述，机器人特定组件的位置信息在后文中被简称为机器人的定位信息。在一些实施例中，机器人的定位信息可以包括表示为机器人和参考物(例如，目标对象、用户和/或系统确定的参考物)的位置关系、机器人对应的坐标系(即，第二坐标系)与其他坐标系(例如，第一坐标系、第三坐标系)的转换关系等。例如，定位信息可以包括同一坐标系下机器人的坐标和目标对象的坐标之间的位置关系。
在一些实施例中，处理器102可以获取图像采集装置对应的第一坐标系与机器人对应的第二坐标系之间的第一转换关系。处理器102可以进一步基于至少一个目标区域和第二图像之间的配准关系，确定第一坐标系和医学成像设备对应的第三坐标系的第二转换关系。处理器102可以进而基于第一转换关系和第二转换关系，确定机器人的定位信息。例如，处理器102可以基于第一转换关系和第二转换关系，确定机器人对应的第二坐标系和医学成像设备对应的第三坐标系之间第三转换关系。
图像采集装置对应的第一坐标系可以指基于图像采集装置建立的坐标系。例如，以图像采集装置的几何中心点为原点建立的三维坐标系。机器人对应的第二坐标系可以指基于机器人建立的坐标系。例如，第二坐标系可以是机械臂末端的坐标系、机器人工具坐标系。医学成像设备对应的第三坐标系可以指基于医学成像设备建立的坐标系。例如，以医学成像设备的机架旋转中心为原点建立的三维坐标系。如本文中使用的，两个坐标系之间的转换关系可以表示两个坐标系中的位置之间的映射关系。例如，转换关系可以表示为变换矩阵，通过该变换矩阵可以将任意点在一个坐标系中的第一坐标转换为在另一坐标系中的对应的第二坐标。在一些实施例中，第一物体对应的坐标系和第二物体对应的坐标系之间的转换关系也可以称为第一物体和第二物体的相对位置关系或位置映射关系。例如，第一转换关系也可以称为图像采集装置和机器人之间的相对位置关系或位置映射关系。
在一些实施例中，处理器102可以通过预设的标定方法(例如，手眼标定算法)确定第一转换关系。例如，处理器102可以构建中间参考物，并且根据中间参考物在第一坐标系 中的第一坐标(或者中间参照物和图像采集装置的相对位置关系)和中间参考物在第二坐标系中的第二坐标(或者中间参照物和机器人的相对位置关系)，确定第一转换关系。其中，图像采集装置可以安装于机器人上，例如，安装于机器人的操作臂末端。或者，图像采集装置还可以设置于机器人所在的房间的任意一个固定位置，处理器102可以根据图像采集装置的位置、机器人的位置以及中间参考物的位置，确定机器人的位置与图像采集装置的位置之间的映射关系。
在一些实施例中，处理器102可以基于至少一个目标区域和第二图像之间的配准关系，确定图像采集装置对应的第一坐标系和医学成像设备对应的第三坐标系的第二转换关系。配准关系可以反映至少一个目标区域中的点和第二图像中的点的对应关系和/或坐标转换关系。因为至少一个目标区域和第二图像均对应同一目标对象，所以至少一个目标区域中的点和第二图像中的点之间存在对应关系，从而可以通过配准技术确定至少一个目标区域和第二图像之间的配准关系。关于确定至少一个目标区域和第二图像之间的配准关系的更多内容可以参见图5及其相关描述。
根据本说明书的一些实施例，可以获取目标对象的第一图像和第二图像，从第一图像中确定与目标对象的至少一个目标部位对应的至少一个目标区域，并且基于至少一个目标区域和第二图像，确定机器人的定位信息。通过目标区域和第二图像(即，医学图像)可以确定机器人、图像采集装置、以及医学成像设备的坐标系之间的转换关系，不需要对目标对象进行额外的标记物的粘贴或者设置，避免了对目标对象造成额外的伤害。其中，使用目标区域和第二图像进行配准，而非直接使用第一图像和第二图像进行配准，可以减少生理运动对配准结果的影响，从而提高配准结果的精度。另外，基于医学图像进行机器人的定位操作，可以提高机器人定位的准确性，进而提高术前规划或手术实施的准确性。
应当注意，以上关于流程300的描述仅出于说明的目的而提供，而不是旨在限制本说明书的范围。对于本领域的普通技术人员，可以在本说明书的指导下进行各种变化和修改。然而，这些变化和修改不脱离本说明书的范围。在一些实施例中，流程300可以通过一个或多个未描述的额外操作和/或省略一个或多个以上讨论的操作来完成。例如，确定机器人的定位信息后，处理器102可以对定位信息验证，以保证机器人定位的准确性。又例如，处理器102可以控制机器人相应地手术规划。具体地，基于定位信息和手术计划，处理器102可以控制机器人移动到目标位置并执行手术操作。
图4是根据本说明书的一些实施例所示的示例性的确定至少一个目标区域的流程400的流程图。在一些实施例中，流程400可以由机器人控制系统100执行。例如，流程400可以以指令集(例如，应用程序)的形式存储在存储设备(例如，存储器104)中。在一些实施 例中，处理器102(例如，图2中所示的一个或多个模块)可以执行指令集并相应指示机器人控制系统100的一个或多个组件执行流程400。在一些实施例中，图3中的操作304描述的至少一个目标区域可以根据流程400确定。
在402中，处理器102(例如，确定模块220)可以获取利用图像采集装置采集的目标对象的二维参考图像。
在一些实施例中，图3所述的第一图像可以是三维图像，例如深度图像。处理器102可以从图像采集装置或存储设备中直接获取目标对象的二维参考图像。例如，图像采集装置为深度相机或激光采集设备，可以同时采集目标对象的二维参考图像和深度图像。在一些实施例中，处理器102可以基于第一图像生成目标对象的二维参考图像。例如，处理器102可以利用图像变换算法将三维的第一图像转化为二维参考图像。
在404中，处理器102(例如，确定模块220)可以从二维参考图像中确定至少一个目标部位对应的至少一个参考区域。
参考区域可以指在二维参考图像中和目标部位对应的区域。仅作为示例，图8是根据本说明书的一些实施例所示的示例性二维参考图像的示意图。如图8所示，阴影部分可以为参考区域，其对应额头、鼻梁等面部静态区域。
在一些实施例中，处理器102可以从二维参考图像中确定与至少一个目标部位有关的至少一个特征点。例如，处理器102可以根据预设的特征点提取算法，从二维参考图像中确定与至少一个目标部位有关的至少一个特征点。示例性特征点提取算法可以包括尺度不变特征变换(scale-invariant features transform，SIFT)算法、加速稳健特征(speeded up robust features，SURF)算法、方向梯度直方图(histogram of oriented gradient，HOR)算法、高斯函数的差分(difference of gaussian，DOG)算法、基于机器学习模型的特征点提取算法等，或其任意组合。
例如，可以利用图像特征点提取模型(如训练好的神经网络模型)对二维参考图像进行特征点提取。处理器102可以将二维参考图像输入至图像特征点提取模型中，图像特征点提取模型可以输出二维参考图像中与至少一个目标部位有关的至少一个特征点。仅作为示例，二维参考图像可以为二维面部图像，将二维面部图像输入至人脸特征点检测模型中，可以得到二维面部图像中与至少一个目标部位有关的至少一个特征点。如图7所示，二维面部图像中识别出的面部特征点可以包括眼睛的特征点、嘴的特征点、眉毛的特征点、鼻子的特征点等。
进一步地，处理器102可以基于至少一个特征点，确定对应至少一个目标部位的至少一个参考区域。例如，可以根据眼睛的特征点确定眼睛区域。在一些实施例中，至少一个特 征点中的每个特征点具有固定序号，可以根据特征点的固定序号，从二维参考图像中确定至少一个参考区域。例如，再次参考图7，面部特征点中序号为37至42的特征点为右眼特征点，可以根据这些特征点确定右眼区域。
在一些实施例中，处理器102可以通过图像识别技术(例如，二维图像识别模型)从二维参考图像中确定至少一个目标部位对应的至少一个参考区域。仅作为示例，处理器102可以将二维参考图像二维输入图像识别模型，二维图像识别模型可以将至少一个参考区域从二维参考图像中分割出来。二维图像识别模型可以基于训练样本训练生成。训练样本可以包括样本对象的样本二维参考图像和对应的样本参考区域，其中，样本二维参考图像为训练输入，对应的样本参考区域为训练标签(label)。在一些实施例中，处理器102(或其他处理设备)基于训练样本对初始模型迭代更新，直至特定条件被满足(例如，损失函数小于一定阈值，迭代训练次数达到一定次数)。
在406中，处理器102(例如，确定模块220)可以基于至少一个参考区域，从第一图像中确定至少一个目标区域。
在一些实施例中，处理器102可以基于二维参考图像和第一图像之间的映射关系，从第一图像中确定与至少一个参考区域对应的至少一个目标区域。其中，二维参考图像和第一图像之间的映射关系可以根据图像采集装置的参数确定。
根据本说明书的一些实施例，可以通过从二维参考图像中确定的至少一个目标部位对应的至少一个参考区域，从而确定第一图像中与目标对象的至少一个目标部位对应的至少一个目标区域。相比于直接从第一图像中确定目标区域，基于二维参考图像确定目标区域可以减少第一图像中深度参数对确定目标区域的影响，提高目标区域确定的准确性。此外，在确定过程中，可以仅使用一张二维参考图像和一张第一图像，降低确定过程中的数据量，节省数据处理的时间和资源。
应当注意，以上关于流程400的描述仅出于说明的目的而提供，而不是旨在限制本说明书的范围。对于本领域的普通技术人员，可以在本说明书的指导下进行各种变化和修改。然而，这些变化和修改不脱离本说明书的范围。在一些实施例中，流程400可以通过一个或多个未描述的额外操作和/或省略一个或多个以上讨论的操作来完成。例如，可以使用多张二维参考图像和第一图像确定目标区域。又例如，可以基于至少一个特征点从第一图像中确定至少一个对应特征点，再基于至少一个对应特征点确定目标区域。
图5是根据本说明书的一些实施例所示的示例性的确定配准关系的流程500的流程图。在一些实施例中，流程500可以由机器人控制系统100执行。例如，流程500可以以指令集(例如，应用程序)的形式存储在存储设备(例如，存储器104)中。在一些实施例中，处理 器102(例如，图2中所示的一个或多个模块)可以执行指令集并相应指示机器人控制系统100的一个或多个组件执行流程500。在一些实施例中，图3中的操作306描述的配准关系可以根据流程500确定。
在一些实施例中，可以通过配准技术确定至少一个目标区域和第二图像之间的配准关系。示例性配准技术可以包括全局配准技术、局部配准技术等。全局配准技术可以为基于至少一个目标区域中的目标平面和第二图像中的参考平面的对应关系进行配准，局部配准可以为基于至少一个目标区域中的目标点和第二图像中的参考点的对应关系进行配准。仅作为示例，可以通过全局配准技术和局部配准技术确定至少一个目标区域和第二图像之间的配准关系。
在502中，处理器102(例如，定位模块230)可以从第二图像中确定至少一个参考点。
在一些实施例中，至少一个参考点可以构成至少一个参考点组，至少一个参考点组中的每个参考点组可以包括位于同一平面上的至少三个参考点。即，每个参考点组(例如，至少三个参考点)可以从第二图像中确定一个参考平面。步骤502也就是从第二图像中确定至少一个参考平面。在一些实施例中，每个参考点组中参考点的数量可以根据实际情况确定。例如，每个参考点组可以包括位于同一平面上的至少四个参考点。需要说明的是，三点即可共面，但通过基于包含至少四个参考点的参考点组可以更准确地确定医学影像中参考平面中各点的位置关系，进而提高确定的配准关系的准确性。
在一些实施例中，处理器102可以从第二图像中随机确定至少一个参考点，并基于医学成像设备对应的第三坐标系，确定至少一个参考点的位置信息。例如，处理器102可以使用随机采样算法从第二图像中确定位于同一平面上的四个参考点，并确定四个参考点在医学成像设备对应的第三坐标系中对应的坐标，其中所述四个参考点可以构成一个参考点组。
在504中，处理器102(例如，定位模块230)可以从至少一个目标区域中确定至少一个参考点对应的至少一个目标点。
在一些实施例中，处理器102可以通过全局配准技术从至少一个目标区域中确定至少一个参考点对应的至少一个目标点。
在一些实施例中，对至少一个参考点组中的每个参考点组，处理器102可以确定参考点组中的参考点对之间的位置关系。参考点对可以包括相邻的参考点，也可以包括不相邻的参考点。参考点对之间的位置关系可以包括参考点之间距离、相对方向等。在一些实施例中，参考点对之间的位置关系可以用参考点之间的向量信息来表示。例如，处理器102可以根据每个参考点组中的至少三个参考点的坐标，确定两两参考点之间的距离和方向，并用向量信息来表示所述距离和方向。
在一些实施例中，基于参考点组的参考点对之间的位置关系，处理器102可以从至少一个目标区域中确定该参考点组对应的目标点组。每个目标点组可以包括位于同一平面上的至少三个目标点。即，每个目标点组(例如，至少三个目标点)可以从至少一个目标区域中确定一个目标平面。在一些实施例中，根据参考点组的向量信息，可以从至少一个目标区域中确定与参考点组对应的目标点组。例如，处理器102可以从至少一个目标区域中确定多个候选目标点组，各个候选目标点组中候选目标点的数量与参考点组中参考点的数量相同。处理器102可以进一步确定与参考点组最相似的候选目标点组，作为目标点组。其中，与参考点组最相似可以指候选目标点组的向量信息和参考点组的向量信息之间的差异最小。
仅作为示例，可以参见图6A和6B。图6A是根据本说明书的一些实施例所示的示例性第二图像的参考点组600A的示意图。如图6A所示，第二图像中参考点组600A可以包括a、b、c、d四个点，四个点形成S1平面。处理器102可以根据a、b、c、d四个点的位置信息，确定a-b、a-c、a-d、b-c、b-d、c-d之间的距离，作为第一距离。图6B是根据本说明书的一些实施例所示的示例性目标区域的候选目标点组600B的示意图。如图6B所示，候选目标点组600B包括a’、b’、c’、d’四个点，四个点形成S2平面。处理器102可以根据a’、b’、c’、d’四个点的位置信息，确定a’-b’、a’-c’、a’-d’、b’-c’、b’-d’、c’-d’之间的距离，作为第二距离。
处理器102可以确定每个第一距离和对应的第二距离之间的偏差。偏差可以包括第一距离和第二距离的差值或者第一距离和第二距离的比值。例如，处理器102可以分别确定a-b与a’-b’的差值、a-c与a’-c’的差值、……、c-d与c’-d’的差值。又例如，处理器102可以分别确定a-b与a’-b’的比值、a-c与a’-c’的比值、……、c-d与c’-d’的比值。进一步地，处理器102可以基于第一距离和第二距离之间的偏差，确定候选目标点组和参考点组的差异。例如，处理器102可以对上述差值进行求和或者求平均值，并将差值的和值或者平均值作为候选目标点组和参考点组的差异。在一些实施例中，至少一个目标区域可以包括多个候选目标点组。处理器102可以分别确定多个候选目标点组中每个候选目标点组和参考点组的差异。在一些实施例中，处理器102可以将差异最小的候选目标点组，确定为目标点组。
根据参考点组中每个参考点对的距离与候选目标点组中两两候选目标点的距离确定目标点组，由于各参考点、各候选目标点的位置信息已知，距离的计算过程简单，可以有效地确定参考组对应的目标点组。
在一些实施例中，基于目标点组，处理器102可以确定参考点组中的每个参考点对应的目标点。例如，基于目标点组，处理器102可以通过目标点组中的每个目标点和参考点组中的每个参考点之间的位置对应关系，确定每个参考点对应的目标点。
通过全局配准技术，处理器102可以从至少一个目标区域中确定第二图像中参考点组 对应的目标点组，进而确定参考点对应的目标点。全局配准技术的实现过程简单，可以提高了确定目标点的效率，同时也确保了目标点选择的准确性。
在一些实施例中，处理器102可以基于局部配准技术，确定参考点对应的目标点，从而将参考点和目标点进行配准。例如，处理器102可以基于迭代最近点算法(iterative closest point，ICP算法)确定参考点对应的目标点。
仅作为示例，处理器102可以获取至少一个目标区域中每个候选目标点的位置信息(例如，坐标、深度信息等)。处理器102可以根据每个候选目标点的位置信息、每个参考点的位置信息和预设的迭代最近点算法，确定每个参考点对应的目标点。例如，处理器102可以基于迭代最近点算法，根据第二图像中的各参考点的位置信息以及至少一个目标区域中的每个候选目标点的位置信息，确定与参考点距离(如欧氏距离)最近的目标点。具体地，处理器102可以从第二图像中确定参考点，从至少一个目标区域中搜索与参考点距离最近的候选目标点作为相应的目标点；根据参考点和对应的目标点，可以确定至少一个目标区域和第二图像之间的转化矩阵(如旋转矩阵和/或平移矩阵)。基于该转化矩阵可以对至少一个目标区域进行变换，并从变换后的至少一个目标区域中确定参考点对应的新的目标点。上述过程可以迭代进行直至特定的条件被满足，例如条件可以是参考点和对应的最新的目标点之间的距离小于预设阈值、或者迭代次数等于预设阈值、或者参考点和对应的最新的目标点之间的距离与参考点和对应的上一个目标点之间的距离的差值小于预设阈值。根据最新一轮迭代中目标点和参考点之间的对应关系(例如，转化矩阵)，可以确定配准关系。
在一些实施例中，处理器102还可以基于上文所述的全局配准技术，确定至少一个目标区域和第二图像之间的初始配准关系。进一步地，处理器102还可以利用局部配准技术(例如，迭代最近点算法)，对初始配准关系进行调整，以确定配准关系。
仅作为示例，处理器102可以根据通过上述全局配准技术确定的目标区域中的目标点和第二图像中的参考点的初始配准关系(即初始对应关系(例如，初始转化矩阵))。对于每个目标点，处理器102可以基于预设的迭代最近点算法，确认或调整每个目标点对应的初始参考点。例如，对于每个目标点，处理器102可以从参考点中确定与目标点距离(如欧氏距离)最近的参考点。若该参考点与初始配准关系中的初始参考点不同，则可以对初始配准关系进行更新，将与目标点距离最近的参考点作为相应的新的参考点。根据目标点和对应的新的参考点，可以确定至少一个目标区域和第二图像之间的转化矩阵(如旋转矩阵和/或平移矩阵)。基于该转化矩阵可以对至少一个目标区域进行变换，并根据变换后的目标点从第二图像中确定对应的新的参考点。上述过程可以迭代进行直至特定的条件被满足，例如条件可以是变换后的目标点和新的参考点之间的距离小于预设阈值、或者迭代次数等于预设阈值、 或者变换后的目标点和新的参考点之间的距离与上一次的目标点和上一次的参考点之间的距离的差值小于预设阈值。根据最新一轮迭代中目标点和参考点之间的对应关系(例如，转化矩阵)，可以调整至少一个目标区域和第二图像之间的初始配准关系，以确定配准关系。
根据本申请的一些实施例，可以根据第二图像中的参考点，从至少一个目标区域中确定目标点，从而基于参考点和目标点的位置信息，确定配准关系。通过使用参考点和目标点进行配准，可以减少计算数据量和处理时间，简化处理过程。此外，通过同时使用全局配准技术和局部配准技术确定至少一个目标区域和第二图像之间的配准关系，可以提高准确性。
在506中，处理器102(例如，定位模块230)可以基于至少一个参考点和至少一个目标点之间的对应关系，确定至少一个目标区域和第二图像之间的配准关系。
在一些实施例中，处理器102可以获取至少一个参考点的位置信息和至少一个目标点的位置信息，并且确定至少一个参考点的位置信息和至少一个目标点的位置信息之间的转换关系，以作为配准关系。例如，处理器102可以根据至少一个参考点的坐标和至少一个目标点的坐标，确定至少一个参考点与至少一个目标点之间的转换矩阵。该转换矩阵可以表征参考点所在坐标系与目标点所在坐标系之间的转换关系，进一步地，该转换矩阵也可以表征至少一个目标区域和第二图像之间的配准关系。
应当注意，以上关于流程500的描述仅出于说明的目的而提供，而不是旨在限制本说明书的范围。对于本领域的普通技术人员，可以在本说明书的指导下进行各种变化和修改。然而，这些变化和修改不脱离本说明书的范围。在一些实施例中，流程500可以通过一个或多个未描述的额外操作和/或省略一个或多个以上讨论的操作来完成。
图9是根据本说明书的一些实施例所示的示例性处理器102的模块图。处理器102可以包括获取模块910、特征点确定模块920和位姿确定模块930。
获取模块910可以用于获取目标对象的目标图像。目标图像可以包括目标对象的三维图像(例如，深度图像)和/或二维图像。目标图像可以利用图像采集装置采集。关于获取目标图像的更多内容可以参考图10的步骤1002及其相关描述。
特征点确定模块920可以用于从目标图像中确定目标对象的至少一个目标特征点。目标特征点可以用于表征图像中目标对象的特征点。在一些实施例中，特征点确定模块920还可以用于从目标对象的参考模型中确定至少一个目标特征点对应的至少一个参考特征点。参考模型可以对应目标拍摄角度。关于获取目标特征点和参考特征点的更多内容可以参考图10的步骤1004和步骤1006及其相关描述。
位姿确定模块930可以用于基于至少一个目标特征点和至少一个参考特征点，确定图像采集装置在基坐标系下的第一目标位姿。在一些实施例中，第一目标位姿可以使图像采集 装置能够从目标拍摄角度和/或目标拍摄距离拍摄目标对象。关于确定第一目标位姿的更多内容可以参考图10的步骤1008及其相关描述。
需要注意的是，以上对于机器人控制系统及其模块的描述，仅为描述方便，并不能把本说明书限制在所举实施例范围之内。可以理解，对于本领域的技术人员来说，在了解该系统的原理后，可能在不背离这一原理的情况下，对各个模块进行任意组合，或者构成子系统与其他模块连接。例如，图9中披露的获取模块910、特征点确定模块920和位姿确定模块930可以是一个系统中的不同模块，也可以是一个模块实现上述的两个模块的功能。又例如，机器人控制系统中各个模块可以共用一个存储模块，各个模块也可以分别具有各自的存储模块。再例如，获取模块910和获取模块210可以是同一个模块。诸如此类的变形，均在本说明书的保护范围之内。
在一些实施例中，处理器102还可以包括注册模块用于实现目标对象与规划图像的注册配准。
图10是根据本说明书的一些实施例所示的示例性的调整图像采集装置位姿的流程1000的流程图。在一些实施例中，流程1000可以由机器人控制系统100执行。例如，流程1000可以以指令集(例如，应用程序)的形式存储在存储设备(例如，存储器104)中。在一些实施例中，处理器102(例如，图9中所示的一个或多个模块)可以执行指令集并相应指示机器人控制系统100的一个或多个组件执行流程1000。在一些实施例中，图3中的操作302描述的调整图像采集装置位姿可以根据流程1000进行。
在一些实施例中，利用图像采集装置采集的目标对象的第一图像前，可以根据术前规划确定目标对象的手术体位，将目标对象保持固定不动，调整图像采集装置的位姿，以使目标对象的目标部位完全位于图像采集装置的采集视野范围之内。
目前，通常将图像采集装置安装在机器人上，由医生手动拖拽图像采集装置使其对准目标对象。由于医生在拖拽图像采集装置的过程中不会注意图像采集装置成像时本身的物理特性，因此很难快速准确地将图像采集装置调整到最佳位姿，造成拍摄效率低下、降低采集的图像数据的精度。此外，医生拖拽图像采集装置也可能降低图像采集装置与机器人的安装精度，进一步降低采集的图像数据的精度。因此，需要提供有效的系统和方法用于调整图像采集装置的位姿。在一些实施例中，可以通过执行流程900的以下操作来调整图像采集装置的位姿。
在1002中，处理器102(例如，获取模块910)可以获取目标对象的目标图像。
在一些实施例中，目标对象可以包括生物对象和/或非生物对象。仅作为示例，在神经外科的场景下，目标对象可以为患者的头部或面部。目标图像可以包括目标对象的三维图像 (例如，深度图像)和/或二维图像。目标图像可以利用图像采集装置采集。目标图像的获取方式可以和第一图像的获取方式相似。关于获取目标图像的更多内容可以参见图3及其相关描述。
在1004中，处理器102(例如，特征点确定模块920)可以从目标图像中确定目标对象的至少一个目标特征点。
目标特征点可以用于表征图像中目标对象的特征点。仅作为示例，如果目标对象为患者的头部或面部，目标特征点可以为目标对象的面部轮廓点，如图14所示。在一些实施例中，处理器102可以使用特征点提取算法从目标图像中确定目标对象的至少一个目标特征点。例如，可以利用特征点提取模型从目标图像中确定至少一个目标特征点。关于特征点提取算法的更多描述可以参考步骤404。
在1006中，处理器102(例如，特征点确定模块920)可以从目标对象的参考模型中确定至少一个目标特征点对应的至少一个参考特征点。
参考模型可以指根据目标对象的特征构建的标准模型。仅作为示例，若目标对象为患者的头部或面部，参考模型可以是根据人类头部特征创建标准人脸模型，也可以是从开源网站中下载标准人脸模型。标准人脸模型的正视图、侧视图示意图可以分别如图13A和图13B所示。在一些实施例中，参考模型可以以三维图像的方式来存储或呈现，对参考模型的分析和处理可以基于该三维图像来执行。
在一些实施例中，参考模型可以对应目标拍摄角度。例如，目标拍摄角度可以指正对目标对象的角度。仅作为示例，目标对象为患者的头部或脸部，目标拍摄角度可以是正对人脸的角度。
在一些实施例中，每个目标特征点都有其对应的参考特征点。参考特征点和对应的目标特征点可以对应目标对象上的同一个物理点。处理器102可以利用上文所述的用于确定至少一个目标特征点的特征点提取算法，从参考模型中确定对应的参考特征点集，并从中确定与目标特征点对应的参考特征点。在一些实施例中，处理器102可以基于至少一个目标特征点，从目标对象的参考模型中确定至少一个参考特征点。例如，处理器102可以基于目标对象的结构特征，从目标对象的参考模型中确定至少一个目标特征点对应的至少一个参考特征点。又例如，处理器102可以使用机器学习模型(例如，映射模型、主动外观模型(active appearance model，AAM)、MediaPipe模型等)，从目标对象的参考模型中确定至少一个目标特征点对应的至少一个参考特征点。仅作为示例，处理器102可以将目标图像和参考模型输入映射模型，映射模型可以输出目标图像和参考模型中的点的映射关系。处理设备102可以基于该映射关系确定与至少一个目标特征点对应的至少一个参考特征点。
在1008中，处理器102(例如，位姿确定模块930)可以基于至少一个目标特征点和至少一个参考特征点，确定图像采集装置在基坐标系下的第一目标位姿。
基坐标系可以是任何坐标系。在一些实施例中，基坐标系可以指基于机器人的基座建立的坐标系。例如，可以以机器人的基座底面中心为原点，基座底面为XY平面，竖直方向为Z轴建立基坐标系。
在一些实施例中，第一目标位姿可以反映调整后的图像采集装置在基坐标系下的位姿。在一些实施例中，第一目标位姿可以使图像采集装置能够从目标拍摄角度和/或目标拍摄距离拍摄目标对象。例如，第一目标位姿可以表示为在目标拍摄角度和目标拍摄距离下图像采集装置对应的坐标系(即，更新后的第一坐标系)和基坐标系的转换关系。在一些实施例中，处理器102可以基于至少一个目标特征点和至少一个参考特征点，确定目标对象相对于图像采集装置的初始位姿。处理器102可以基于初始位姿，确定目标对象相对于图像采集装置第二目标位姿。进一步地，处理器102可以基于第二目标位姿，确定图像采集装置在基坐标系下的第一目标位姿。关于确定第一目标位姿的更多内容可以参见图11及其相关描述。
在一些实施例中，图像采集装置可以安装于机器人上。例如，图像采集装置可以安装于机器人的机械臂末端。因此，处理器102可以控制机器人运动，使得图像采集装置调整至第一目标位姿。在一些实施例中，处理器102可以基于第一目标位姿，确定机器人在基坐标系下的第三目标位姿。处理器102可以控制机器人调整至第三目标位姿，使得图像采集装置调整至第一目标位姿。关于确定第三目标位姿的更多内容可以参见图12及其相关描述。
根据本说明书的一些实施例，可以基于至少一个目标特征点和至少一个参考特征点，确定图像采集装置在基坐标系下的第一目标位姿，使得图像采集装置能够从目标拍摄角度和/或目标拍摄距离拍摄目标对象，从而实现将图像采集装置自动摆位至最佳位置。因此，可以提高图像采集装置获取的图像数据的精度，从而提高后续机器人定位、术前规划的精度。
应当注意，以上关于流程1000的描述仅出于说明的目的而提供，而不是旨在限制本说明书的范围。对于本领域的普通技术人员，可以在本说明书的指导下进行各种变化和修改。然而，这些变化和修改不脱离本说明书的范围。在一些实施例中，流程1000可以通过一个或多个未描述的额外操作和/或省略一个或多个以上讨论的操作来完成。例如，在获取目标对象前，可以确定图像采集装置的视野是否包括目标对象。又例如，图像采集装置调整至第一目标位姿后，可以获取目标对象的图像数据(例如，第一图像)，用于实现目标对象与规划图像的注册配准和/或对机器人进行定位。
图11是根据本说明书的一些实施例所示的示例性的确定第一目标位姿的流程1100的流程图。在一些实施例中，流程1100可以由机器人控制系统100执行。例如，流程1100可 以以指令集(例如，应用程序)的形式存储在存储设备(例如，存储器104)中。在一些实施例中，处理器102(例如，图9中所示的一个或多个模块)可以执行指令集并相应指示机器人控制系统100的一个或多个组件执行流程1100。在一些实施例中，图10中的操作1008描述的第一目标位姿可以根据流程1100确定。
在1102中，处理器102(例如，位姿确定模块930)可以基于至少一个目标特征点和至少一个参考特征点，确定目标对象相对于图像采集装置的初始位姿。
在一些实施例中，初始位姿可以表示为目标对象对应的坐标系和采集目标图像时的图像采集装置对应的第一坐标系之间的转换关系。在一些实施例中，初始位姿可以反映当图像采集装置在目标拍摄角度时目标对象在第一坐标系中的初始位姿，和/或在第一坐标系中将目标对象调整至参考模型对应的位姿所需调整的角度。
在一些实施例中，可以将确定目标对象在图像采集装置对应的第一坐标系中的初始位姿转化为求解PNP(Perspective N Points)问题。PNP问题可以指如下的物体定位问题：假定图像采集装置为小孔模型且已标定好，拍摄一幅在物体坐标系下坐标已知的N个空间点的像，且这N个图像点的坐标已知，确定这N个空间点在图像采集装置对应的第一坐标系下的坐标。
仅作为示例，至少一个目标特征点可以为目标对象的面部轮廓点。在确定目标对象的面部轮廓点后，可以获取面部轮廓点在目标对象对应的坐标系下的坐标。目标对象对应的坐标系可以是基于目标对象建立的坐标系。以人脸为目标对象为例，鼻尖可以作为人脸坐标系的原点，平行于脸部的平面为XY平面，垂直于脸部的方向为Z轴建立人脸坐标系。例如，若目标图像为二维图像，可以记确定的n个面部轮廓点对应的像素坐标为A i(x i，y i)(i＝1、2、3、…、n)，记n个面部轮廓点对应的参考模型中的参考特征点在目标对象对应的坐标系下的空间坐标为B j(X j，Y j，Z j)(j＝1、2、3、…、n)。目标对象相对于调整位姿前的图像采集装置的初始位姿
Figure PCTCN2022092003-appb-000001
也可以称为从目标对象对应的坐标系变换到第一坐标系的位姿变换矩阵
Figure PCTCN2022092003-appb-000002
可以通过公式(1)确定：
Figure PCTCN2022092003-appb-000003
其中，M为图像采集装置的参数矩阵，可以由图像采集装置的固有参数确定。
又例如，若目标图像为三维图像，可以记确定的n个面部轮廓点在目标对象对应的坐标系下的像素坐标为A i(x i，y i，z i)(i＝1、2、3、…、n)，记n个面部轮廓点对应的参考模型中的参考特征点在目标对象对应的坐标系下的空间坐标为B j(X j，Y j，Z j)(j＝1、2、3、…、n)。目标对象相对于调整位姿前的图像采集装置的初始位姿
Figure PCTCN2022092003-appb-000004
也可以称为目标对象对应的 坐标系变换到第一坐标系的位姿变换矩阵为
Figure PCTCN2022092003-appb-000005
可以通过公式(2)确定：
Figure PCTCN2022092003-appb-000006
通过上述步骤，可以基于至少一个目标特征点、至少一个参考特征点和图像采集装置的参数矩阵，确定在第一坐标系中将目标对象的位姿调整至参考模型的位姿时所需要调整的角度，即通过坐标变换确定目标对象相对于调整位姿前的图像采集装置的初始位姿。
在1104中，处理器102(例如，位姿确定模块930)可以基于初始位姿，确定目标对象相对于图像采集装置的第二目标位姿。
第二目标位姿可以表示为目标对象对应的坐标系和调整后的图像采集装置对应的更新后的第一坐标系和之间的转换关系。在一些实施例中，第二目标位姿可以反映当图像采集装置调整位姿后(例如，调整至目标拍摄角度和目标拍摄距离后)，目标对象在更新后的第一坐标系中的位姿，和/或在更新后的第一坐标系中将拍摄距离调整至目标拍摄距离对应的位姿所需调整的距离。
目标拍摄距离可以指图像采集装置采集的图像数据的质量符合预设标准时目标对象和相机之间的在高度方向上的距离。在一些实施例中，处理器102可以获取图像采集装置的目标拍摄距离。例如，目标拍摄距离可以预先确定并存储在存储设备(例如，存储器104)中，处理器102可以获取存储设备中的目标拍摄距离。仅作为示例，可以在图像采集装置的视野中呈现完整的目标对象(或其他参考对象如人脸模型)，沿着图像采集装置的光轴方向移动图像采集装置，获取图像采集装置在多个拍摄距离下拍摄的目标对象的图像数据。根据多个拍摄距离中每个拍摄距离下的图像数据的精度和质量(例如，清晰度)，确定目标对象与图像采集装置的之间的最佳拍摄距离，并且将最佳拍摄距离作为目标拍摄距离。在一些实施例中，可以通过标记点确定图像采集装置的目标拍摄距离。例如，可以在目标对象上设置多个标记点，并且通过高精度的图像采集设备确定这些标记点的坐标，作为标准坐标。使用图像采集装置在不同拍摄距离拍摄这些标记点，分别确定在不同拍摄距离下这些标记点的坐标；将不同拍摄距离下确定的坐标和标准坐标对比；并且将与标准坐标差异最小的坐标对应的拍摄距离作为目标拍摄距离。
在一些实施例中，处理器102可以基于目标拍摄距离，确定距离变换矩阵。仅作为示例，若目标拍摄距离为H，距离变换矩阵P可以如公式(3)所示：
Figure PCTCN2022092003-appb-000007
在一些实施例中，处理器102可以基于距离变换矩阵以及初始位姿，确定在目标拍摄距离下目标对象在更新后的第一坐标系中的第二目标位姿(也就是目标对象相对于调整位姿后的图像采集装置的目标位姿)。仅作为示例，第二目标位姿可以根据公式(4)确定：
Figure PCTCN2022092003-appb-000008
其中，
Figure PCTCN2022092003-appb-000009
可以为目标对象在更新前的第一坐标系中的初始位姿，
Figure PCTCN2022092003-appb-000010
可以为在目标拍摄距离下目标对象在更新后的第一坐标系中的第二目标位姿。
通过确定在目标拍摄距离下目标对象在更新后的第一坐标系中的第二目标位姿，可以将目标对象与图像采集装置之间的拍摄距离调整到目标拍摄距离，从而能够提高图像采集装置采集的图像数据的精度。
在1106中，处理器102(例如，位姿确定模块930)可以基于目标对象相对于图像采集装置的第二目标位姿，确定图像采集装置在基坐标系下的第一目标位姿。
在一些实施例中，图像采集装置可以安装于机器人上，因此通过图像采集装置和机器人的连接结构，可以确定第一坐标系和基坐标系的第四转换关系。进一步地，可以基于第四转换关系和第二目标位姿，确定第一目标位姿。
仅作为示例，处理器102可以获取第一坐标系和机器人对应的第二坐标系的第一转换关系。第一转换关系可以指机器人的位置与图像采集装置的位置之间的映射关系。关于获取第一转换关系的更多内容可以参见图3及其相关描述。处理器102还可以获取第二坐标系和基坐标系的第五转换关系。在一些实施例中，处理器102可以通过预设的标定方法(例如手眼标定方法)确定第五转换关系。例如，处理器102可以采用类似于确定第一转换关系的方式确定第五转换关系。在一些实施例中，第五转换关系可以是机器人的参数，处理器102可以从机器人的控制器中获取。进一步地，处理器102可以基于第一转换关系和第五转换关系，确定第四转换关系。仅作为示例，第四转换关系可以根据公式(5)确定：
Figure PCTCN2022092003-appb-000011
其中，T可以表示图像采集装置对应的第一坐标系和基坐标系之间的第四转换关系，
Figure PCTCN2022092003-appb-000012
可以表示图像采集装置对应的第一坐标系和机器人对应的第二坐标系之间的第一转换关系，
Figure PCTCN2022092003-appb-000013
可以表示机器人对应的第二坐标系和基坐标系之间的第五转换关系。
在一些实施例中，处理器102可以基于第四转换关系和第二目标位姿，确定第一目标位姿。仅作为示例，第一目标位姿可以根据公式(6)确定：
Figure PCTCN2022092003-appb-000014
其中，
Figure PCTCN2022092003-appb-000015
和
Figure PCTCN2022092003-appb-000016
可以表示第一目标位姿。
根据本说明书的一些实施例，通过确定图像采集装置对应的第一坐标系和基坐标系之间的第四转换关系，并基于第四转换关系和目标对象在更新后的第一坐标系中的第二目标位姿确定目标对象在基坐标系中的第一目标位姿，从而将图像采集装置调整至目标拍摄角度和目标拍摄距离，提高采集的图像数据的精度。
应当注意，以上关于流程1100的描述仅出于说明的目的而提供，而不是旨在限制本说明书的范围。对于本领域的普通技术人员，可以在本说明书的指导下进行各种变化和修改。然而，这些变化和修改不脱离本说明书的范围。在一些实施例中，流程1100可以通过一个或多个未描述的额外操作和/或省略一个或多个以上讨论的操作来完成。
图12是根据本说明书的一些实施例所示的示例性的调整图像采集装置位姿的流程1200的流程图。在一些实施例中，流程1200可以由机器人控制系统100执行。例如，流程1200可以以指令集(例如，应用程序)的形式存储在存储设备(例如，存储器104)中。在一些实施例中，处理器102(例如，图9中所示的一个或多个模块)可以执行指令集并相应指示机器人控制系统100的一个或多个组件执行流程1200。
在一些实施例中，图像采集装置可以安装于机器人上。例如，图像采集装置可以安装于机器人的机械臂末端。在图10描述的步骤1008后可以执行流程1200，通过控制机器人运动使得图像采集装置调整至第一目标位姿。
在1202中，处理器102(例如，位姿确定模块930)可以基于第一目标位姿，确定机器人在基坐标系下的第三目标位姿。
在一些实施例中，第三目标位姿可以反映当图像采集装置在第一目标位姿时，机器人在基坐标系中的位姿，和/或在基坐标系系中将机器人调整使得图像采集装置调整到第一目标位姿时机器人所需移动的距离和角度。在一些实施例中，第三目标位姿可以表示为当图像采集装置在第一目标位姿时，机器人对应的第二坐标系和基坐标系之间的转换关系。
在一些实施例中，处理器102可以基于第一转换关系和第一目标位姿，确定机器人在基坐标系下的第三目标位姿。仅作为示例，第三目标位姿可以根据公式(7)确定：
Figure PCTCN2022092003-appb-000017
其中，
Figure PCTCN2022092003-appb-000018
可以表示第三目标位姿。
在1204中，处理器102(例如，位姿确定模块930)可以控制机器人调整至第三目标位姿，使得图像采集装置调整至第一目标位姿。
仅作为示例，参见图16A至16H，图16A是根据本说明书的一些实施例所示的示例性位姿调整前的图像采集装置示意图。图16B是根据本说明书的一些实施例所示的示例性位姿 调整后的图像采集装置示意图。图16C、图16E和图16G是根据本说明书的一些实施例所示的示例性位姿调整前的图像采集装置采集的图像数据的示意图，图16D、图16F和图16H是根据本说明书的一些实施例所示的示例性位姿调整后的图像采集装置采集的图像数据的示意图。结合图16A和图16B，通过控制机器人调整至第三目标位姿，图像采集装置1610从图16A的位姿调整到了图16B中的第一目标位姿。分别对比图16C和图16D、图16E和图16F、图16G和图16H，可以确定在图像采集装置调整到第一目标位姿后，目标对象1620、目标对象1630和目标对象1640在图像数据中的成像位置调整至图像数据的中心。也就是说，在图像采集装置调整到第一目标位姿后，图像采集装置可以在目标拍摄角度和目标拍摄高度下采集目标对象的图像数据。
根据本说明书的一些实施例，能够准确的确定机器人在基坐标系中的位姿(即，第三目标位姿)，从而能够根据机器人在基坐标系中的位姿来准确地调整图像采集装置的位姿。由此可以使图像采集装置在目标拍摄角度和目标拍摄距离获取目标对象的图像数据，提高了图像数据的精度。
应当注意，以上关于流程1200的描述仅出于说明的目的而提供，而不是旨在限制本说明书的范围。对于本领域的普通技术人员，可以在本说明书的指导下进行各种变化和修改。然而，这些变化和修改不脱离本说明书的范围。在一些实施例中，流程1200可以通过一个或多个未描述的额外操作和/或省略一个或多个以上讨论的操作来完成。
图15A-图15C是根据本说明书的一些实施例所示的示例性调整图像采集装置1510的位姿的示意图。
在图像采集装置1510完成安装后，机器人的位姿可以处于随机初始位置。如图15A所示，图像采集装置1510的视野中没有呈现患者的面部(即，目标对象)，因此需要先局部调整机器人的位姿使得图像采集装置1510的视野中出现患者的完整面部或者部分面部。仅作为示例，在获取目标对象的目标图像前，可以确定图像采集装置1510的视野中是否包括至少一个目标特征点。响应于确定图像采集装置1510的视野中包括至少一个目标特征点，利用图像采集装置1510采集目标对象的目标图像。响应于确定图像采集装置1510的视野中不包括至少一个目标特征点，调整图像采集装置1510，使得图像采集装置1510的视野中包括至少一个目标特征点。
当图像采集装置1510的视野中包括至少一个目标特征点时(如图15B所示)，机器人控制系统100(例如，处理器102)可以执行如图10至图12中的图像采集装置位姿调整流程，使得图像采集装置1510可以从目标拍摄角度和目标拍摄距离获取目标图像的图像数据(如图15C所示)。
通过在获取目标图像之前，调整图像采集装置，使得图像采集装置的视野中包括至少一部分目标对象，可以提高调整图像采集装置位姿的效率。
图17是根据本说明书的一些实施例所示的示例性调整图像采集装置位姿的示意图。
如图17所示，图像采集装置采集目标图像时对应的第一坐标系为camera_link，目标对象对应的坐标系为face_link，图像采集装置在目标拍摄距离下的更新后的第一坐标系为face_link_view，机器人对应的第二坐标系为tool_link，以及基坐标系为base_link。
基于至少一个目标特征点和至少一个参考特征点，通过公式(1)或公式(2)可以确定目标对象在第一坐标系camera_link中的初始位姿
Figure PCTCN2022092003-appb-000019
通过公式(4)，可以确定在目标拍摄距离下目标对象在更新后的第一坐标系为face_link_view的第二目标位姿
Figure PCTCN2022092003-appb-000020
通过获取第一坐标系camera_link和第二坐标系tool_link的第一转换关系
Figure PCTCN2022092003-appb-000021
获取第二坐标系tool_link和基坐标系base_link的第五转换关系
Figure PCTCN2022092003-appb-000022
可以根据公式(5)可以确定第一坐标系camera_link和基坐标系base_link的第四转换关系T。基于第四转换关系T和第二目标位姿
Figure PCTCN2022092003-appb-000023
可以根据公式(6)确定第一目标位姿
Figure PCTCN2022092003-appb-000024
在一些实施例中，基于第一转换关系的逆运算
Figure PCTCN2022092003-appb-000025
和第一目标位姿
Figure PCTCN2022092003-appb-000026
可以根据公式(7)确定机器人在基坐标系下的第三目标位姿。仅作为示例，可以根据公式(7)确定
Figure PCTCN2022092003-appb-000027
图18是根据本说明书的一些实施例所示的示例性机器人控制系统1800的示意图。
如图18所示，机器人控制系统1800可以包括机器人1810、图像采集装置1820和处理器1830。图像采集装置1820可以安装于机器人1810(例如，机器人1810的机械臂末端)上。处理器1830可以分别连接机器人1810和图像采集装置1820。处理器1830运行时可以执行本申请中一些实施例所示的机器人定位流程和图像采集装置位姿调整流程。
在一个实施例中，提供了一种计算机设备，该计算机设备可以是服务器，其内部结构图可以如图19所示。该计算机设备包括通过系统总线连接的处理器、存储器、通信接口、显示屏和输入装置。其中，该计算机设备的处理器用于提供计算和控制能力。例如，可以执行本申请中一些实施例所示的机器人定位方法和图像采集装置位姿调整方法。该计算机设备的存储器包括非易失性存储介质、内存储器。该非易失性存储介质存储有操作系统和计算机程序。该内存储器为非易失性存储介质中的操作系统和计算机程序的运行提供环境。该计算机设备的通信接口用于与外部的终端进行有线或无线方式的通信，无线方式可通过WIFI、运营商网络、NFC(近场通信)或其他技术实现。该计算机程序被处理器执行时以实现一种机器人定位方法。该计算机设备的显示屏可以是液晶显示屏或者电子墨水显示屏，该计算机设备的输入装置可以是显示屏上覆盖的触摸层，也可以是计算机设备外壳上设置的按键、轨迹球 或触控板，还可以是外接的键盘、触控板或鼠标等。
本领域技术人员可以理解，图19中示出的结构，仅仅是与本申请方案相关的部分结构的框图，并不构成对本申请方案所应用于其上的计算机设备的限定，具体的计算机设备可以包括比图中所示更多或更少的部件，或者组合某些部件，或者具有不同的部件布置。
图20是根据本说明书的一些实施例所示的示例性机器人控制流程2000的流程图。在一些实施例中，流程2000可以由机器人控制系统100执行。例如，流程2000可以以指令集(例如，应用程序)的形式存储在存储设备(例如，存储器104)中。在一些实施例中，处理器102(例如，图2中所示的一个或多个模块和/或图9中所示的一个或多个模块)可以执行指令集并相应指示机器人控制系统100的一个或多个组件执行流程2000。
在2002中，处理器102可以利用图像采集装置采集目标对象的目标图像。
在2004中，处理器102可以从目标图像中确定目标对象的至少一个目标特征点。
在2006中，处理器102可以从目标对象的参考模型中确定至少一个目标特征点对应的至少一个参考特征点。参考模型可以对应目标拍摄角度。
在2008中，处理器102可以基于至少一个目标特征点和至少一个参考特征点，确定图像采集装置在基坐标系下的第一目标位姿，使得图像采集装置能够从目标拍摄角度拍摄目标对象。
在2010中，处理器102可以获取目标对象的第一图像和第二图像。第一图像可以利用调整后的图像采集装置采集，第二图像可以利用医学成像设备采集。
在2012中，处理器102可以从第一图像中确定与目标对象的至少一个目标部位对应的至少一个目标区域。至少一个目标部位受生理运动的影响小于其他部位。
在2014中，处理器102可以基于至少一个目标区域和第二图像，确定机器人的定位信息。
根据本说明书的一些实施例，可以先对图像采集装置进行位姿调整，再对机器人进行定位操作。经过位姿调整后的图像采集装置可以在目标拍摄角度和目标拍摄距离下获取第一图像，可以提高第一图像的精度，从而提高机器人定位的准确性，进而提高术前规划或手术实施的准确性。
应当注意，以上关于流程2000的描述仅出于说明的目的而提供，而不是旨在限制本说明书的范围。对于本领域的普通技术人员，可以在本说明书的指导下进行各种变化和修改。然而，这些变化和修改不脱离本说明书的范围。在一些实施例中，流程2000可以通过一个或多个未描述的额外操作和/或省略一个或多个以上讨论的操作来完成。例如，处理器102还可以基于第一图像，实现目标对象与规划图像的注册配准，从而提高注册方案的精度。
本说明书一些实施例中，(1)基于至少一个目标特征点和至少一个参考特征点，确定图像采集装置在基坐标系下的第一目标位姿，使得图像采集装置能够从目标拍摄角度和/或目标拍摄距离拍摄目标对象，可以实现将图像采集装置自动摆位至最佳位置，从而提高图像采集装置获取的图像数据的精度，提高注册方案的精度；(2)通过调整后的图像采集装置获取第一图像，可以提高第一图像的精度，从而提高机器人定位的准确性；(3)通过第一图像中的目标区域和第二图像确定机器人、图像采集装置、以及医学成像设备的坐标系之间的转换关系，不需要对目标对象进行额外的标记物的粘贴或者设置，避免了对目标对象造成额外的伤害；(4)基于医学图像进行机器人的定位操作，可以提高机器人定位的准确性，进而提高术前规划或手术实施的准确性。
本说明一些实施例还提供了一种电子装置，该电子装置包括：至少一个存储介质，存储计算机指令；至少一个处理器，执行该计算机指令，以实现本说明书所述的机器人定位方法和图像采集装置位姿调整方法。该电子装置还可以包括传输设备以及输入输出设备，其中，传输设备和输入输出设备可以和处理器连接。有关更多技术细节可参见图1A至图17的相关描述，在此不再赘述。
本说明一些实施例还提供了一种计算机可读存储介质，该存储介质存储计算机指令，当计算机读取该计算机指令时，计算机执行本说明书所述的机器人定位方法和图像采集装置位姿调整方法。有关更多技术细节可参见图1A至图17的相关描述，在此不再赘述。
上文已对基本概念做了描述，显然，对于本领域技术人员来说，上述详细披露仅仅作为示例，而并不构成对本说明书的限定。虽然此处并没有明确说明，本领域技术人员可能会对本说明书进行各种修改、改进和修正。该类修改、改进和修正在本说明书中被建议，所以该类修改、改进、修正仍属于本说明书示范实施例的精神和范围。
同时，本说明书使用了特定词语来描述本说明书的实施例。如“一个实施例”、“一实施例”、和/或“一些实施例”意指与本说明书至少一个实施例相关的某一特征、结构或特点。因此，应强调并注意的是，本说明书中在不同位置两次或多次提及的“一实施例”或“一个实施例”或“一个替代性实施例”并不一定是指同一实施例。此外，本说明书的一个或多个实施例中的某些特征、结构或特点可以进行适当的组合。
此外，除非权利要求中明确说明，本说明书所述处理元素和序列的顺序、数字字母的使用、或其他名称的使用，并非用于限定本说明书流程和方法的顺序。尽管上述披露中通过各种示例讨论了一些目前认为有用的发明实施例，但应当理解的是，该类细节仅起到说明的目的，附加的权利要求并不仅限于披露的实施例，相反，权利要求旨在覆盖所有符合本说明书实施例实质和范围的修正和等价组合。例如，虽然以上所描述的系统组件可以通过硬件设 备实现，但是也可以只通过软件的解决方案得以实现，如在现有的服务器或移动设备上安装所描述的系统。
同理，应当注意的是，为了简化本说明书披露的表述，从而帮助对一个或多个发明实施例的理解，前文对本说明书实施例的描述中，有时会将多种特征归并至一个实施例、附图或对其的描述中。但是，这种披露方法并不意味着本说明书对象所需要的特征比权利要求中提及的特征多。实际上，实施例的特征要少于上述披露的单个实施例的全部特征。
一些实施例中使用了描述成分、属性数量的数字，应当理解的是，此类用于实施例描述的数字，在一些示例中使用了修饰词“大约”、“近似”或“大体上”来修饰。除非另外说明，“大约”、“近似”或“大体上”表明所述数字允许有±20％的变化。相应地，在一些实施例中，说明书和权利要求中使用的数值参数均为近似值，该近似值根据个别实施例所需特点可以发生改变。在一些实施例中，数值参数应考虑规定的有效数位并采用一般位数保留的方法。尽管本说明书一些实施例中用于确认其范围广度的数值域和参数为近似值，在具体实施例中，此类数值的设定在可行范围内尽可能精确。
针对本说明书引用的每个专利、专利申请、专利申请公开物和其他材料，如文章、书籍、说明书、出版物、文档等，特此将其全部内容并入本说明书作为参考。与本说明书内容不一致或产生冲突的申请历史文件除外，对本说明书权利要求最广范围有限制的文件(当前或之后附加于本说明书中的)也除外。需要说明的是，如果本说明书附属材料中的描述、定义、和/或术语的使用与本说明书所述内容有不一致或冲突的地方，以本说明书的描述、定义和/或术语的使用为准。
最后，应当理解的是，本说明书中所述实施例仅用以说明本说明书实施例的原则。其他的变形也可能属于本说明书的范围。因此，作为示例而非限制，本说明书实施例的替代配置可视为与本说明书的教导一致。相应地，本说明书的实施例不仅限于本说明书明确介绍和描述的实施例。

Claims (20)

  1. 一种机器人定位方法，其特征在于，所述方法包括：
    获取目标对象的第一图像和第二图像，所述第一图像利用图像采集装置采集，所述第二图像利用医学成像设备采集；
    从所述第一图像中确定与所述目标对象的至少一个目标部位对应的至少一个目标区域，所述至少一个目标部位受生理运动的影响小于其他部位；
    基于所述至少一个目标区域和所述第二图像，确定所述机器人的定位信息。
  2. 如权利要求1所述的方法，其特征在于，所述从所述第一图像中确定与所述目标对象的至少一个目标部位对应的至少一个目标区域包括：
    获取利用所述图像采集装置采集的所述目标对象的二维参考图像；
    从所述二维参考图像中确定所述至少一个目标部位对应的至少一个参考区域；以及
    基于所述至少一个参考区域，从所述第一图像中确定所述至少一个目标区域。
  3. 如权利要求2所述的方法，其特征在于，所述从所述二维参考图像中确定所述至少一个目标部位对应的至少一个参考区域包括：
    从所述二维参考图像中确定与所述至少一个目标部位有关的至少一个特征点；以及
    基于所述至少一个特征点，确定所述至少一个参考区域。
  4. 如权利要求1所述的方法，其特征在于，所述基于所述至少一个目标区域和所述第二图像，确定所述机器人的定位信息包括：
    获取所述图像采集装置对应的第一坐标系与机器人对应的第二坐标系之间的第一转换关系；
    基于所述至少一个目标区域和所述第二图像之间的配准关系，确定所述第一坐标系和所述医学成像设备对应的第三坐标系的第二转换关系；以及
    基于所述第一转换关系和所述第二转换关系，确定所述机器人的所述定位信息。
  5. 如权利要求4所述的方法，其特征在于，所述方法进一步包括：
    从所述第二图像中确定至少一个参考点；
    从所述至少一个目标区域中确定所述至少一个参考点对应的至少一个目标点；以及
    基于所述至少一个参考点和所述至少一个目标点之间的对应关系，确定所述至少一个目标区域和所述第二图像之间的所述配准关系。
  6. 如权利要求5所述的方法，其特征在于，所述至少一个参考点构成至少一个参考点组，所述至少一个参考点组中的每个参考点组包括位于同一平面上的至少三个参考点，
    所述从所述至少一个目标区域中确定所述至少一个参考点对应的至少一个目标点包括：
    对所述至少一个参考点组中的每个参考点组，
    确定所述参考点组中的参考点对之间的位置关系；
    基于所述位置关系，从所述至少一个目标区域中确定所述参考点组对应的目标点组，所述目标点组包括位于同一平面上的至少三个目标点；以及
    基于所述目标点组，确定所述参考点组中的每个参考点对应的目标点。
  7. 如权利要求5所述的方法，其特征在于，所述基于所述至少一个参考点和所述至少一个目标点之间的对应关系，确定所述至少一个目标区域和所述第二图像之间的配准关系包括：
    基于所述至少一个参考点和所述至少一个目标点之间的所述对应关系，确定所述至少一个目标区域和所述第二图像之间的初始配准关系；以及
    利用迭代最近点算法，对所述初始配准关系进行调整，以确定所述配准关系。
  8. 如权利要求1所述的方法，其特征在于，所述图像采集装置安装于所述机器人上，以及
    在利用所述图像采集装置采集所述第一图像之前，所述方法进一步包括：
    获取所述目标对象的第三图像和参考模型，所述第三图像利用具有初始位姿的所述图像采集装置采集，所述参考模型对应目标拍摄角度；
    基于所述第三图像和所述参考模型，确定所述图像采集装置在基坐标系下的目标位姿；以及
    控制所述机器人，使所述图像采集装置调整至所述目标位姿。
  9. 如权利要求8所述的方法，所述基于所述第三图像和所述参考模型，确定所述图像采集装置在基坐标系下的目标位姿包括：
    从所述第三图像中确定所述目标对象的至少一个目标特征点；
    从所述参考模型中确定所述至少一个目标特征点对应的至少一个参考特征点；以及
    基于所述至少一个目标特征点和所述至少一个参考特征点，确定所述图像采集装置在所述基坐标系下的所述目标位姿。
  10. 一种图像采集装置位姿调整方法，其特征在于，所述方法包括：
    利用图像采集装置采集目标对象的目标图像；
    从所述目标图像中确定所述目标对象的至少一个目标特征点；
    从所述目标对象的参考模型中确定所述至少一个目标特征点对应的至少一个参考特征点，所述参考模型对应目标拍摄角度；
    基于所述至少一个目标特征点和所述至少一个参考特征点，确定所述图像采集装置在基坐标系下的第一目标位姿。
  11. 如权利要求10所述的方法，其特征在于，所述基于所述至少一个目标特征点和所述至少一个参考特征点，确定所述图像采集装置在基坐标系下的第一目标位姿包括：
    基于所述至少一个目标特征点和所述至少一个参考特征点，确定所述目标对象相对于所述图像采集装置的初始位姿；
    基于所述初始位姿，确定所述目标对象相对于所述图像采集装置的第二目标位姿；以及
    基于所述第二目标位姿，确定所述第一目标位姿。
  12. 如权利要求11所述的方法，其特征在于，所述基于所述第二目标位姿，确定所述第一目标位姿包括：
    确定所述第一坐标系和所述基坐标系的转换关系；以及
    基于所述第一坐标系和所述基坐标系的转换关系和所述第二目标位姿，确定所述第一目标位姿。
  13. 如权利要求12所述的方法，其特征在于，所述图像采集装置安装于机器人上，所述确定所述第一坐标系和所述基坐标系的转换关系包括：
    获取所述第一坐标系和所述机器人对应的第二坐标系的转换关系；
    获取所述第二坐标系和所述基坐标系的转换关系；以及
    基于所述第一坐标系和所述第二坐标系的转换关系以及所述第二坐标系和所述基坐标系的转换关系，确定所述第一坐标系和所述基坐标系的转换关系。
  14. 如权利要求10所述的方法，其特征在于，所述图像采集装置安装于机器人上，所述方法进一步包括：
    基于所述第一目标位姿，确定所述机器人在所述基坐标系下的第三目标位姿；以及
    控制所述机器人调整至所述第三目标位姿，使得所述图像采集装置调整至所述第一目标位姿。
  15. 如权利要求14所述的方法，其特征在于，所述方法进一步包括：
    获取所述目标对象的第一图像和第二图像，所述第一图像利用所述调整后的图像采集装置采集，所述第二图像利用医学成像设备采集；
    从所述第一图像中确定与所述目标对象的至少一个目标部位对应的至少一个目标区域，所述至少一个目标部位受生理运动的影响小于其他部位；
    基于所述至少一个目标区域和所述第二图像，确定所述机器人的定位信息。
  16. 一种机器人定位系统，包括：
    存储设备，存储计算机指令；
    处理器，与所述存储设备相连接，当执行所述计算机指令时，所述处理器使所述系统执行下述操作：
    获取目标对象的第一图像和第二图像，所述第一图像利用图像采集装置采集，所述第二图像利用医学成像设备采集；
    从所述第一图像中确定与所述目标对象的至少一个目标部位对应的至少一个目标区域，所述至少一个目标部位受生理运动的影响小于其他部位；
    基于所述至少一个目标区域和所述第二图像，确定所述机器人的定位信息。
  17. 如权利要求16所述的系统，其特征在于，所述从所述第一图像中确定与所述目标对象的至少一个目标部位对应的至少一个目标区域包括：
    获取利用所述图像采集装置采集的所述目标对象的二维参考图像；
    从所述二维参考图像中确定所述至少一个目标部位对应的至少一个参考区域；以及
    基于所述至少一个参考区域，从所述第一图像中确定所述至少一个目标区域。
  18. 如权利要求16所述的系统，其特征在于，所述基于所述至少一个目标区域和所述第二图像，确定所述机器人的定位信息包括：
    获取所述图像采集装置对应的第一坐标系与机器人对应的第二坐标系之间的第一转换关系；
    基于所述至少一个目标区域和所述第二图像之间的配准关系，确定所述第一坐标系和所 述医学成像设备对应的第三坐标系的第二转换关系；以及
    基于所述第一转换关系和所述第二转换关系，确定所述机器人的所述定位信息。
  19. 如权利要求16所述的系统，其特征在于，所述图像采集装置安装于所述机器人上，以及
    在利用所述图像采集装置采集所述第一图像之前，所述处理器使所述系统执行下述操作：
    获取所述目标对象的第三图像和参考模型，所述第三图像利用具有初始位姿的所述图像采集装置采集，所述参考模型对应目标拍摄角度；
    基于所述第三图像和所述参考模型，确定所述图像采集装置在基坐标系下的目标位姿；以及
    控制所述机器人，使所述图像采集装置调整至所述目标位姿。
  20. 如权利要求19所述的系统，所述基于所述第三图像和所述参考模型，确定所述图像采集装置在基坐标系下的目标位姿包括：
    从所述第三图像中确定所述目标对象的至少一个目标特征点；
    从所述参考模型中确定所述至少一个目标特征点对应的至少一个参考特征点；以及
    基于所述至少一个目标特征点和所述至少一个参考特征点，确定所述图像采集装置在所述基坐标系下的所述目标位姿。
PCT/CN2022/092003 2021-05-10 2022-05-10 机器人定位和位姿调整方法和系统 WO2022237787A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP22806750.0A EP4321121A1 (en) 2021-05-10 2022-05-10 Robot positioning and pose adjustment method and system
US18/506,980 US20240075631A1 (en) 2021-05-10 2023-11-10 Methods and systems for positioning robots and adjusting postures

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
CN202110505732.6 2021-05-10
CN202110505732.6A CN113397704B (zh) 2021-05-10 2021-05-10 机器人定位方法、装置、系统及计算机设备
CN202111400891.6A CN114098980B (zh) 2021-11-19 2021-11-19 相机位姿调整方法、空间注册方法、系统和存储介质
CN202111400891.6 2021-11-19

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US18/506,980 Continuation US20240075631A1 (en) 2021-05-10 2023-11-10 Methods and systems for positioning robots and adjusting postures

Publications (1)

Publication Number Publication Date
WO2022237787A1 true WO2022237787A1 (zh) 2022-11-17

Family

ID=84028865

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/092003 WO2022237787A1 (zh) 2021-05-10 2022-05-10 机器人定位和位姿调整方法和系统

Country Status (3)

Country Link
US (1) US20240075631A1 (zh)
EP (1) EP4321121A1 (zh)
WO (1) WO2022237787A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117109435A (zh) * 2023-08-21 2023-11-24 北京纳通医用机器人科技有限公司 一种机器人对准方法、装置、设备及存储介质

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2005028468A (ja) * 2003-07-08 2005-02-03 National Institute Of Advanced Industrial & Technology ロボットの視覚座標系位置姿勢同定方法、座標変換方法および装置
CN110215284A (zh) * 2019-06-06 2019-09-10 上海木木聚枞机器人科技有限公司 一种可视化系统和方法
CN110377015A (zh) * 2018-04-13 2019-10-25 北京三快在线科技有限公司 机器人定位方法和机器人定位装置
US20200242339A1 (en) * 2019-01-25 2020-07-30 Biosense Webster (Israel) Ltd. Registration of frames of reference
CN111862180A (zh) * 2020-07-24 2020-10-30 三一重工股份有限公司 一种相机组位姿获取方法、装置、存储介质及电子设备
CN113397704A (zh) * 2021-05-10 2021-09-17 武汉联影智融医疗科技有限公司 机器人定位方法、装置、系统及计算机设备
CN114098980A (zh) * 2021-11-19 2022-03-01 武汉联影智融医疗科技有限公司 相机位姿调整方法、空间注册方法、系统和存储介质

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2005028468A (ja) * 2003-07-08 2005-02-03 National Institute Of Advanced Industrial & Technology ロボットの視覚座標系位置姿勢同定方法、座標変換方法および装置
CN110377015A (zh) * 2018-04-13 2019-10-25 北京三快在线科技有限公司 机器人定位方法和机器人定位装置
US20200242339A1 (en) * 2019-01-25 2020-07-30 Biosense Webster (Israel) Ltd. Registration of frames of reference
CN110215284A (zh) * 2019-06-06 2019-09-10 上海木木聚枞机器人科技有限公司 一种可视化系统和方法
CN111862180A (zh) * 2020-07-24 2020-10-30 三一重工股份有限公司 一种相机组位姿获取方法、装置、存储介质及电子设备
CN113397704A (zh) * 2021-05-10 2021-09-17 武汉联影智融医疗科技有限公司 机器人定位方法、装置、系统及计算机设备
CN114098980A (zh) * 2021-11-19 2022-03-01 武汉联影智融医疗科技有限公司 相机位姿调整方法、空间注册方法、系统和存储介质

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117109435A (zh) * 2023-08-21 2023-11-24 北京纳通医用机器人科技有限公司 一种机器人对准方法、装置、设备及存储介质
CN117109435B (zh) * 2023-08-21 2024-05-28 北京纳通医用机器人科技有限公司 一种机器人对准方法、装置、设备及存储介质

Also Published As

Publication number Publication date
US20240075631A1 (en) 2024-03-07
EP4321121A1 (en) 2024-02-14

Similar Documents

Publication Publication Date Title
KR102013866B1 (ko) 실제수술영상을 이용한 카메라 위치 산출 방법 및 장치
US11253171B2 (en) System and method for patient positioning
US9978141B2 (en) System and method for fused image based navigation with late marker placement
US11576645B2 (en) Systems and methods for scanning a patient in an imaging system
US9480456B2 (en) Image processing apparatus that simultaneously displays two regions of interest on a body mark, processing method thereof and storage medium
EP3411818B1 (en) System for image processing to generate three-dimensional (3d) view of an anatomical portion
US20200268251A1 (en) System and method for patient positioning
WO2018018981A1 (zh) 用于对三维表面的图像进行重建的方法、装置和系统
US11672505B2 (en) Correcting probe induced deformation in an ultrasound fusing imaging system
JP2016508406A (ja) 複数の三次元ビューからの大きな対象のセグメンテーション
US10657661B2 (en) Information processing apparatus, imaging system, information processing method, and program causing computer to execute information processing
US10078906B2 (en) Device and method for image registration, and non-transitory recording medium
JP6833533B2 (ja) 超音波診断装置および超音波診断支援プログラム
CN113647967A (zh) 一种医学扫描设备的控制方法、装置及系统
US20240075631A1 (en) Methods and systems for positioning robots and adjusting postures
Meng et al. An automatic markerless registration method for neurosurgical robotics based on an optical camera
CN116883471B (zh) 面向胸腹部经皮穿刺的线结构光无接触点云配准方法
KR20160057024A (ko) 마커리스 ３차원 객체추적 장치 및 그 방법
JP2021532903A (ja) 医療装置を撮像するためのコンセンサス面の決定
US20230200775A1 (en) Ultrasonic imaging system
CN114098985A (zh) 患者与患者医学影像的空间匹配方法、装置、设备及介质
WO2024095134A1 (en) Registration and navigation in cranial procedures
CN112766258A (zh) 图像的分割方法、系统、电子设备及计算机可读存储介质
WO2023051899A1 (en) System and method for identifying a region of interest
CN115120345A (zh) 导航定位方法、装置、计算机设备和存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22806750

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 2022806750

Country of ref document: EP

ENP Entry into the national phase

Ref document number: 2022806750

Country of ref document: EP

Effective date: 20231110

NENP Non-entry into the national phase

Ref country code: DE