JP7379373B2 - 3D visualization camera and integrated robot platform - Google Patents

Description

The present invention relates to a stereoscopic visualization camera and an integrated robot platform.

Surgery is an art. Great artists create works of art that far exceed the abilities of ordinary people. The artist uses a brush to transform a container of paint into a vivid image that evokes strong and unique emotions in the viewer. Artists transform ordinary words written on paper into dramatic and majestic performances. An artist grabs an instrument and lets it emit beautiful music. Similarly, surgeons take seemingly ordinary scalpels, tweezers, and probes and create life-changing biological miracles.

Like artists, surgeons have their own methods and preferences. Aspiring artists are taught the basics of the craft. Beginners often follow a prescribed method. As you gain experience, confidence, and knowledge, you develop a unique artistic reflection of yourself and your personal environment. Similarly, medical students are taught the basics of surgical procedures. These methods are rigorously tested. As students progress through training and practice, they develop what is derived from the basics (still within medical standards) based on how they believe the surgery should be best completed. For example, consider the same medical procedure performed by different renowned surgeons. The order of events, pace, staffing, instrumentation, and use of imaging equipment will vary for each surgeon based on preference. Even the size and shape of the incision can be specific to the surgeon.

The artist's uniqueness and skill of the surgeon makes the surgeon weary of surgical instruments that alter or alter the surgeon's methods. The instruments are an extension of the surgeon and should operate simultaneously and/or in harmonious synchronization. Surgical instruments that dictate the course of a procedure or change a surgeon's rhythm are often discarded or modified to accommodate.

In one example, consider microsurgical visualization, where a particular surgical procedure involves patient structures that are too small to be easily visualized by the naked human eye. These microsurgical procedures require magnification to appropriately view the microstructures. Surgeons generally desire visualization instruments that are natural extensions of their eyes. In fact, early efforts in microsurgical visualization involved attaching magnifying glasses to head-mounted optical eyepieces (called surgical loupes). The first one was developed in 1876. Much improved versions of surgical loupes (some including optical zooms and integrated light sources) are still used by surgeons today. FIG. 1 shows a diagram of one surgical loupe 100 having a light source 102 and a magnifying glass 104. FIG. One reason surgical loupes have remained so powerful for 150 years is that they are literally an extension of the surgeon's eyes.

Despite their long lifespan, surgical loupes are not perfect. A loupe with a magnifying glass and a light source, such as loupe 100 of FIG. 1, has a much greater weight. Placing even a small amount of weight in front of the surgeon's face can increase discomfort and fatigue, especially during long surgeries. Surgical loupe 100 also includes a cable 106 connected to a remote power source. The cable effectively functions as a chain, thereby limiting the surgeon's mobility during the surgical performance.

Another microsurgical visualization instrument is a surgeon's microscope, also called a surgical microscope. Widespread commercial development of surgical microscopes began in the 1950's with the purpose of replacing surgical loupes. A surgical microscope includes an optical path, a lens, and a focusing element that provides high magnification compared to surgical loupes. The large array of optical elements (and resulting weight) meant that the surgical microscope had to be removed from the surgeon. Although this removal gives the surgeon more space to maneuver, the bulk of the surgical microscope consumes considerable maneuvering space above the patient, thereby reducing the size of the surgical stage.

FIG. 2 shows a diagram of a surgical microscope 200 according to the prior art. As can be imagined, the size and presence of the surgical microscope in the surgical area made it susceptible to shaking. To provide stability and rigidity to scope head 201, the microscope is connected to relatively large boom arms 202 and 204 or other similar support structures. Large boom arms 202 and 204 consume additional surgical space and reduce surgeon and staff mobility. In total, the surgical microscope 200 shown in FIG. 2 can weigh as much as 350 kilograms ("kg").

To view the target surgical site using surgical microscope 200, the surgeon looks directly through eyepiece 206. To reduce stress on the surgeon's back, the eyepiece 206 is generally positioned along the surgeon's natural line of sight with height adjustment using the arm 202. However, the surgeon does not perform surgery by looking only at the target surgical site. The eyepiece 206 needs to be positioned so that the surgeon is within an arm length of the working distance to the patient. Such precise positioning is extremely important to ensure that the surgical microscope 200 becomes an extension rather than an obstacle to the surgeon, especially when used for extended periods of time.

Like any complex instrument, it takes tens to hundreds of hours for a surgeon to feel comfortable using a surgical microscope. As shown in FIG. 2, the design of surgical microscope 200 requires an approximately 90 degree optical path from the surgeon to the target surgical site. For example, a completely perpendicular optical path from the target surgical site to the scope head 201 is required. This means that in any microsurgical procedure it is necessary to position the scope head 201 directly above the patient. In addition, the surgeon must view the eyepiece 206 generally horizontally (or tilted somewhat downwardly). This is because the surgeon's natural tilt directs the eye toward both hands at the surgical site. Some surgeons also desire to move their hands closer to the operative site for more precise control of hand movements. Unfortunately, surgical microscope 200 does not provide the surgeon with this flexibility. Instead, the surgical microscope 200 inexorably requires the surgeon to place both eyes in the eyepieces 206 and hold the head at arm's length while performing the surgery, all while consuming valuable surgical space above the patient. It is decided that it is. Because the scope head 201 blocks the surgeon's view, the surgeon cannot even simply look down at the patient.

To make matters worse, some surgical microscopes 200 include a second pair of eyepieces 208 for a co-operator (eg, an assistant surgeon, nurse, or other clinical staff). A second pair of eyepieces 208 is typically positioned perpendicularly from the first eyepiece 206 . The closeness of eyepieces 206 and 208 dictates that the assistant must stand (or sit) close to the surgeon, further restricting movement. This can be uncomfortable for some surgeons who prefer to operate with some space. Despite its magnifying benefits, surgical microscope 200 is not a natural extension of the surgeon. Instead, it's the overbearing director in the operating room.

The present disclosure relates to a stereoscopic viewing robot system that includes a stereoscopic visualization camera and a robotic arm. An example stereoscopic robotic system is configured to obtain stereoscopic images of a target surgical site while allowing an operator to position a stereoscopic visualization camera using a robotic arm. As disclosed herein, the robotic arm provides structural stability that allows the stereoscopic visualization camera to record high-resolution images without jitter or other artifacts that may result from unintentional camera movement. Contains electromechanically operated joints. The robotic arm also provides structural flexibility that allows the operator to position the stereoscopic visualization camera in different positions and/or orientations to obtain desired views of the target surgical site. Thus, one example stereoscopic robotic system allows surgeons to comfortably complete life-changing microsurgery wherever it suits them.

The stereoscopic robotic system of the present disclosure can be positioned in any number of orientations relative to the surgical field that best suit the needs of the surgeon or patient, rather than the physical and mechanical limitations of the visualization device. The stereoscopic robotic system is configured to provide powered joint movement assistance that allows a surgeon or other operator to effortlessly position a stereoscopic visualization camera. In some embodiments, the present stereoscopic robotic system is configured to provide power-assisted movement of the robotic arm based on forces detected from an operator positioning the stereoscopic camera. The stereoscopic robotic system may also allow the operator to select visual lock to the target surgical site while allowing the operator to change the orientation and/or position of the stereoscopic visualization camera. Additionally or alternatively, the stereoscopic robotic system is configured with one or more boundaries that prevent the stereoscopic visualization camera and/or the robotic arm from contacting the patient, surgical staff, and/or surgical equipment. . In short, the present stereoscopic robotic system acts as an extension of the surgeon's eye while generally giving the surgeon the freedom to perform microsurgical procedures without restriction or hindrance.

Any aspect of the subject matter described herein may be useful alone or in combination with one or more other aspects described herein. Without limiting the above description, in a first aspect of the present disclosure, a robotic imaging device includes a base section configured to be connected to a secure structure or cart, and a robotic arm connected to the base section. a first end including a coupling interface, a second end including a coupling interface, and a robotic arm having a plurality of joints and links connecting the first end to the second end. Each joint includes a motor configured to rotate the joint about an axis and a joint sensor configured to transmit a respective position of the joint. The robotic imaging device also includes a stereo camera connected to the robot arm at a coupling interface. The stereoscopic camera is configured to record left and right images of the target surgical site to generate a stereoscopic image stream of the target surgical site. The robotic imaging device further includes a sensor positioned at the coupling interface and configured to detect translational and rotational forces applied to the stereoscopic camera by an operator and to transmit output data indicative of the translational and rotational forces. . The robotic imaging device includes one or more instructions that specify the direction, speed, and duration of rotation of each joint of the robotic arm based on at least the current position of the robotic arm and the detected translational and rotational forces. and/or a memory for storing at least one algorithm defined by the data structure. Additionally, the robotic imaging device includes at least one processor communicatively coupled to the sensor and the robotic arm. receiving output data indicative of translational and rotational forces from the sensor; and using at least one algorithm in the memory to determine a movement sequence of the robot arm based on the current position of the robot arm and the output data from the sensor. at least one processor configured to: determine; The at least one processor is also configured to rotate at least one of the joints of the robot arm based on the determined movement sequence via one or more motor control signals provided to the at least one joint. be done. Rotation of the at least one joint provides powered movement of the robotic arm based on detected translational forces and detected rotational forces applied to the stereoscopic camera by an operator.

According to a second aspect of the present disclosure, which may be used in conjunction with any other aspect recited herein, unless otherwise noted, at least one processor comprises a plurality of joints. The robot arm is configured to identify the current position of the robot arm based on output data from the joint sensor.

According to a third aspect of the present disclosure, which may be used in conjunction with any other aspect listed herein, the sensor includes a six degree of freedom haptic sensor. It includes at least one of a device and a torque sensor.

According to a fourth aspect of the disclosure, which may be used in conjunction with any other aspect listed herein, unless otherwise noted, the stereoscopic camera is capable of powered movement. at least one control arm having a release button to select the sensor; and determining a movement sequence using output data from.

According to a fifth aspect of the present disclosure, which may be used in conjunction with any other aspect recited herein, unless otherwise noted, the apparatus is adapted to a coupling interface of a robotic arm. The apparatus further includes a coupling plate having a first end configured to connect and a second end including a second coupling interface configured to connect to a stereoscopic camera. The coupling plate includes at least one joint including a joint sensor configured to transmit respective positions of the joint and motor controllable by the at least one processor according to a movement sequence.

According to a sixth aspect of the present disclosure, which may be used in conjunction with any other aspect listed herein, unless otherwise noted, the sensor is connected to a coupling interface or a second coupling placed in the interface.

According to a seventh aspect of the present disclosure, which may be used in conjunction with any other aspect recited herein, except as otherwise noted, the coupling plate aligns the stereoscopic camera with a horizontal orientation. A second joint is included to allow manual rotation by an operator between vertical orientations. The second joint includes a joint sensor configured to transmit a position of the second joint.

According to an eighth aspect of the present disclosure, which may be used in conjunction with any other aspect listed herein, unless otherwise noted, a stereoscopic camera is attached to a robot arm at a coupling interface. including a housing including a bottom surface configured to connect.

According to a ninth aspect of the disclosure, which may be used in conjunction with any other aspect recited herein, except as otherwise noted, a robotic imaging device includes a robotic arm; The arm has a first end connected to the secure structure, a second end including a coupling interface, and a plurality of joints and links connecting the first end to the second end, each The joint includes a plurality of joints and links including a motor configured to rotate the joint about an axis and a joint sensor configured to transmit a respective position of the joint. The robotic imaging apparatus also includes an imaging device connected to the robotic arm at the coupling interface, the imaging device positioned at the coupling interface with the imaging device configured to record an image of the target surgical site; and a sensor configured to detect force and/or torque applied to the imaging device by an operator and transmit force and/or torque output data indicative of the force and/or torque. The robotic imaging device further includes at least one processor communicatively coupled to the sensor and the robotic arm. At least one processor receives force and/or torque output data from the sensor, converts the force and/or torque output data into translational vectors and rotational vectors, and uses kinematics to control the robot arm. determining a movement sequence of the robot arm based on the current position and the translation and rotation vectors, the movement sequence specifying a rotational direction, velocity, and duration of movement of at least some of the joints of the robot arm; , and rotating at least one of the joints of the robot arm based on the determined movement sequence via one or more motor control signals provided to the at least one joint. configured.

According to a tenth aspect of the present disclosure, which may be used in conjunction with any other aspect listed herein, except as otherwise noted, the processor is configured to: configured to determine at least one scaling factor based on at least one of a current position of the arm or a characteristic position of the robot arm; and apply the scaling factor to at least one joint velocity of the movement sequence. .

According to an eleventh aspect of the present disclosure, which may be used in conjunction with any other aspect recited herein, unless otherwise noted, at least one scaling factor from a virtual boundary of the robot arm or imaging device. The at least one scaling factor decreases to a value of "0" as the virtual boundary is approached.

According to a twelfth aspect of the present disclosure, which may be used in conjunction with any other aspect recited herein, unless otherwise noted, the virtual boundary is a patient, a medical device, or Corresponding to at least one member of the operating room staff.

According to a thirteenth aspect of the present disclosure, which may be used in conjunction with any other aspect listed herein, except as otherwise noted, the processor is configured to move the at least one scaling factor. The display device is configured to cause the display device to display an icon indicating that the sequence has been applied.

According to a fourteenth aspect of the present disclosure, which may be used in conjunction with any other aspect recited herein, except as otherwise noted, the processor is configured to The apparatus is configured to determine at least one scaling factor based on a joint angle between limits and to apply the scaling factor to at least one joint velocity of the movement sequence.

According to a fifteenth aspect of the present disclosure, which may be used in conjunction with any other aspect listed herein, except as otherwise noted, the processor provides force and/or torque output data. and force application compensation to the force and/or torque output data to account for the offset between the location of the sensor and the location of the imaging device at which the force and/or torque is applied by the operator. and configured to compensate.

According to a sixteenth aspect of the present disclosure, which may be used in conjunction with any other aspect recited herein, except as otherwise noted, the processor comprises a plurality of joints of a robot arm. The system is configured to identify or identify joint singularities to control hysteresis and backlash, and to determine movement sequences based on kinematics while avoiding robot arm movement through joint singularities. Ru.

According to a seventeenth aspect of the present disclosure, which may be used in conjunction with any other aspect recited herein, unless otherwise noted, the robotic imaging device comprises: The apparatus further includes a coupling plate having a first end configured to connect to the interface and a second end including a second coupling interface configured to connect to the stereoscopic camera. The coupling plate includes at least one joint including a joint sensor configured to transmit a position of each of the joints and a motor controllable by the at least one processor according to a movement sequence. The sensor is located at the coupling interface or the second coupling interface.

Unless otherwise noted, according to an eighteenth aspect of the disclosure, which may be used in conjunction with any other aspect recited herein, the robotic arm includes at least four joints; The coupling plate includes at least two joints.

According to a nineteenth aspect of the disclosure, which may be used in conjunction with any other aspect recited herein, unless otherwise noted, the processor performs a movement specified by a movement sequence. The robot arm is configured to rotate at least one of the joints of the robot arm by sending one or more command signals to the joint's respective motors indicating a direction, speed, and duration of rotation.

According to a twentieth aspect of the present disclosure, which may be used in conjunction with any other aspect recited herein, except as otherwise noted, the processor causes the robot arm to move during the movement sequence. While moving, the robot arm is configured to compare images recorded by the imaging device to ensure that the robot arm is moving as determined during the movement sequence.

According to a twenty-first aspect of the disclosure, which may be used in conjunction with any other aspect recited herein, unless otherwise noted, the kinematics is an inverse kinematics or a Jacobian motion. including at least one of the following:

According to twenty-two aspects of the present disclosure, the structures and features shown and described in connection with FIGS. It may be used in combination with any of the structures and features shown and described in connection with the plurality.

Accordingly, in view of the above aspects and the disclosure herein, an advantage of the present disclosure is to provide a stereoscopic robotic system that provides seamless coordination between a stereoscopic camera and a robotic arm.

Another advantage of the present disclosure is to provide a stereoscopic robotic system that uses a robotic arm to increase the focal length, working distance, and/or magnification of a stereoscopic robotic camera.

Yet another advantage of the present disclosure is to provide a stereoscopic robotic system that provides powered movement of a robotic arm based on forces/torques applied to a stereoscopic camera by an operator.

The advantages discussed herein may be found in one or some, and perhaps not all, of the embodiments disclosed herein. Additional features and advantages are described herein and will be apparent from the detailed description and figures below.

FIG. 1 shows a diagram of a pair of prior art surgical loupes. FIG. 2 shows a diagram of a surgical microscope according to the prior art. FIG. 3 shows an illustration of a perspective view of a stereoscopic visualization camera according to an example embodiment of the present disclosure. FIG. 4 shows an illustration of a perspective view of a stereoscopic visualization camera according to an example embodiment of the present disclosure. FIG. 5 depicts a diagram of a microsurgical environment including the stereoscopic visualization camera of FIGS. 3 and 4, according to an example embodiment of the present disclosure. FIG. 6 depicts a diagram of a microsurgical environment including the stereoscopic visualization camera of FIGS. 3 and 4, according to an example embodiment of the present disclosure. FIG. 7 shows a diagram illustrating optical elements within the example stereoscopic visualization camera of FIGS. 3-6, according to an example embodiment of the present disclosure. FIG. 8 shows a diagram illustrating optical elements within the example stereoscopic visualization camera of FIGS. 3-6, according to an example embodiment of the present disclosure. FIG. 9 shows a diagram of deflection elements of the example stereoscopic visualization camera of FIGS. 7 and 8, according to example embodiments of the present disclosure. FIG. 10 shows an illustration of an example right optical image sensor and a left optical image sensor of the example stereoscopic visualization camera of FIGS. 7 and 8, according to an example embodiment of the present disclosure. FIG. 11 shows a diagram of an example carrier of an optical element of the example stereoscopic visualization camera of FIGS. 7 and 8, according to an example embodiment of the present disclosure. FIG. 12 shows a diagram of an example carrier of an optical element of the example stereoscopic visualization camera of FIGS. 7 and 8, according to an example embodiment of the present disclosure. FIG. 13 illustrates a view of the example flexure of the example stereoscopic visualization camera of FIGS. 7 and 8, according to example embodiments of the present disclosure. FIG. 14 depicts a diagram of an example stereo visualization camera module that acquires and processes image data, according to example embodiments of the present disclosure. FIG. 15 shows a diagram of internal components of the module of FIG. 14 according to an example embodiment of the disclosure. FIG. 16 depicts a diagram of the information processor module of FIGS. 14 and 15, according to an example embodiment of the present disclosure. FIG. 17 illustrates an example display monitor according to an example embodiment of the present disclosure. FIG. 18 shows a diagram illustrating pseudo parallax between the right and left optical paths. FIG. 19 shows a diagram illustrating pseudo parallax between the right and left optical paths. FIG. 20 shows a diagram illustrating pseudo parallax between the right and left optical paths. FIG. 21 shows a diagram illustrating pseudo parallax between the right and left optical paths. FIG. 22 is a diagram showing the out-of-focus situation related to the positions of the two parallel lenses in each of the right optical path and the left optical path. FIG. 23 shows a diagram illustrating how pseudo-parallax causes digital graphics and/or images to lose accuracy when fused into a stereoscopic image. FIG. 24 shows a diagram illustrating how pseudo-parallax causes digital graphics and/or images to lose accuracy when fused into a stereoscopic image. FIG. 25 depicts a flowchart illustrating an example procedure for reducing or eliminating pseudo-parallax, according to example embodiments of the present disclosure. FIG. 26 depicts a flowchart illustrating an example procedure for reducing or eliminating pseudo-parallax, according to example embodiments of the present disclosure. FIG. 27 shows a diagram illustrating how zoom iteration points are adjusted relative to a pixel grid of an optical image sensor, according to example embodiments of the present disclosure. FIG. 28 depicts a diagram illustrating a template matching program for finding zoom iteration points, according to example embodiments of the present disclosure. FIG. 29 depicts a diagram illustrating a template matching program for finding zoom iteration points, according to example embodiments of the present disclosure. FIG. 30 depicts a diagram illustrating a template matching program for finding zoom iteration points, according to example embodiments of the present disclosure. FIG. 31 depicts a diagram illustrating a template matching program that finds zoom iteration points, according to example embodiments of the present disclosure. FIG. 32 depicts a diagram illustrating a template matching program that finds zoom iteration points, according to example embodiments of the present disclosure. FIG. 33 depicts a side view of the microsurgical environment of FIG. 5 in accordance with an example embodiment of the present disclosure. FIG. 34 illustrates an embodiment of the example robot arm of FIG. 5 in accordance with an example embodiment of the present disclosure. FIG. 35 shows a diagram of the robotic arm of FIGS. 33 and 34 connected to a cart, according to an example embodiment of the present disclosure. FIG. 36 shows a view of the robot arm of FIGS. 33 and 34 mounted to a ceiling plate, according to an example embodiment of the present disclosure. FIG. 37 illustrates an embodiment of a coupling plate of a robot arm, according to an example embodiment of the present disclosure. FIG. 38 shows views of the coupling plate in different rotational positions, according to example embodiments of the present disclosure. FIG. 39 shows views of the coupling plate in different rotational positions, according to example embodiments of the present disclosure. FIG. 40 shows views of the coupling plate in different rotational positions, according to example embodiments of the present disclosure. FIG. 41 illustrates an embodiment of the stereoscopic robot platform of FIGS. 3-40, according to example embodiments of the present disclosure. FIG. 42 illustrates an example procedure or routine for calibrating the stereoscopic visualization camera of FIGS. 3-33, according to example embodiments of the present disclosure. FIG. 43 illustrates an embodiment of the example stereoscopic visualization camera of FIGS. 3-33 and 42 that moves the object plane in discrete steps, according to an example embodiment of the present disclosure. FIG. 44 depicts a graph illustrating a routine executable by a processor for determining the projection center of the stereoscopic visualization cameras of FIGS. 3-33 and 42, according to example embodiments of the present disclosure. FIG. 45 shows a plan view of an optical schematic illustrating how the interpupillary distance of the stereoscopic visualization cameras of FIGS. 3-33 is measured and calibrated, according to example embodiments of the present disclosure. FIG. 46 depicts a top view of an optical schematic illustrating how the optical axis of the stereoscopic visualization cameras of FIGS. 3-33 may be measured and calibrated, according to example embodiments of the present disclosure. FIG. 47 shows a diagram of a calibrated stereoscopic visualization camera with well-characterized optical parameters, according to an example embodiment of the present disclosure. FIG. 48 illustrates an example procedure or routine for calibrating the robotic arm of FIGS. 5 and 33-41, according to example embodiments of the present disclosure. FIG. 49 shows a diagram illustrating how a stereoscopic visualization camera and/or robot arm is calibrated to robot space, according to an example embodiment of the present disclosure. FIG. 50 depicts a diagram illustrating horizontal and vertical boundary plans to limit movement of a stereoscopic visualization camera and/or robot arm, according to an example embodiment of the disclosure. FIG. 51 illustrates an example of how a rotating joint velocity of a robot arm and/or a coupling plate may be scaled based on distance to a boundary, according to example embodiments of the present disclosure. FIG. 52 depicts an illustration of an example procedure for fusing images from alternative modality visualizations with stereoscopic images, according to example embodiments of the present disclosure. FIG. 53 shows a diagram illustrating a live cross-sectional fusion visualization produced by the stereoscopic visualization camera and/or robotic arm combination of FIGS. 3-52, according to an example embodiment of the present disclosure. FIG. 54 depicts a diagram illustrating a live cross-sectional fusion visualization produced by the stereoscopic visualization camera and/or robotic arm combination of FIGS. 3-52, according to an example embodiment of the present disclosure. FIG. 55 shows a diagram illustrating a live cross-sectional fusion visualization produced by the stereoscopic visualization camera and/or robotic arm combination of FIGS. 3-52, according to an example embodiment of the present disclosure. FIG. 56 shows a diagram illustrating a live cross-sectional fusion visualization produced by the stereoscopic visualization camera and/or robotic arm combination of FIGS. 3-52, according to an example embodiment of the present disclosure. FIG. 57 shows a diagram illustrating a live cross-sectional fusion visualization produced by the stereoscopic visualization camera and/or robotic arm combination of FIGS. 3-52, according to an example embodiment of the present disclosure. FIG. 58 depicts a diagram illustrating a live cross-sectional fusion visualization produced by the stereoscopic visualization camera and/or robotic arm combination of FIGS. 3-52, according to an example embodiment of the present disclosure. FIG. 59 shows a diagram illustrating a live cross-sectional fusion visualization produced by the stereoscopic visualization camera and/or robotic arm combination of FIGS. 3-52, according to an example embodiment of the present disclosure. FIG. 60 depicts a diagram illustrating a live cross-sectional fusion visualization produced by the stereoscopic visualization camera and/or robotic arm combination of FIGS. 3-52, according to example embodiments of the present disclosure. FIG. 61 depicts a diagram illustrating a live cross-sectional fusion visualization produced by the stereoscopic visualization camera and/or robotic arm combination of FIGS. 3-52, according to example embodiments of the present disclosure. FIG. 62 depicts a diagram illustrating a procedure for providing assisted driving of the stereoscopic visualization cameras of FIGS. 3-52, according to example embodiments of the present disclosure. FIG. 63 depicts an illustration of an example procedure for moving the example visualization camera of FIGS. 3-52 using an input device, according to an example embodiment of the present disclosure. FIG. 64 depicts a diagram illustrating an algorithm, routine, or procedure for providing target locking of a stereoscopic visualization camera, according to example embodiments of the present disclosure. FIG. 65 illustrates a diagram illustrating a virtual sphere of the target lock feature of FIG. 64, according to an example embodiment of the disclosure.

TECHNICAL FIELD This disclosure generally relates to stereoscopic visualization cameras and platforms. A stereoscopic visualization camera is sometimes referred to as a digital stereomicroscope (“DSM”). One example camera and platform is much smaller and lighter than prior art microscopes (such as surgical loupes 100 in FIG. 1 and surgical microscope 200 in FIG. 2), and the microscope optical components are housed in a self-contained head unit that is easy to operate. configured to integrate elements and video sensors. An example camera is configured to transmit stereoscopic video signals to one or more television monitors, projectors, holographic devices, smart glasses, virtual reality devices, or other visual display devices within the surgical environment.

A monitor or other visual display device may be positioned within the surgical environment so that it is easily within the surgeon's line of sight while performing surgery on a patient. This flexibility allows the surgeon to position the display monitor based on personal preference or habit. Additionally, the flexibility and slim profile of the stereoscopic visualization camera disclosed herein reduces the area consumed above the patient. In general, a stereoscopic visualization camera and monitor (e.g., a stereoscopic visualization platform) allows the surgeon and surgical team to move relative to the patient without being dictated or restricted compared to the surgical microscope 200 described above. Allowing complex microsurgical procedures to be performed. Accordingly, one example stereoscopic visualization platform operates as an extension of the surgeon's eye to enable the surgeon to perform microsurgical masterpieces without dealing with the stresses, limitations, and constraints posed by previously known visualization systems. Make it.

The disclosure herein generally refers to microsurgery. An example stereoscopic visualization camera may be used for, for example, cranial surgery, neurosurgery, neurosurgery, spinal surgery, ophthalmology surgery, corneal transplantation, orthopedic surgery, ear, nose, and throat surgery, dental surgery, plastic surgery, and reconstructive surgery, or can be used for virtually any microsurgical procedure, including general surgery.

This disclosure also refers to target sites, scenes, or fields of view. As used herein, a target region or field of view includes the object (or portion of an object) that is being recorded or otherwise imaged by an example stereoscopic visualization camera. Generally, the target region, scene, or field of view is a working distance away from and aligned with the example stereoscopic visualization camera's main objective lens assembly. The target site may include the patient's anatomy, bone, muscle, skin, or a combination thereof. In these cases, the target site may be three-dimensional by having a depth component that corresponds to the progression of the patient's anatomy. The target site can also include one or more templates used to calibrate or verify an example stereoscopic visualization camera. The template can be two-dimensional, such as a graphic design on paper (or a plastic sheet), or three-dimensional, such as an approximation of the patient's anatomy in specific areas.

Throughout, reference is also made to the x, y, z, and tilt directions. The z-direction is along the axis from one example stereoscopic visualization camera to the target site and is commonly referred to as depth. The x and y directions lie in a plane incident on the z direction and constitute the plane of the target site. The x-direction is along an axis 90° from the y-direction axis. Movement along the x-direction and/or y-direction is referred to as in-plane movement, and may include movement of an example stereoscopic visualization camera, movement of an optical element within an example stereoscopic visualization camera, and/or movement of a target region. It can be pointed out.

A tilt direction corresponds to movement along Euler angles (eg, yaw, pitch, and roll axes) with respect to the x, y, and/or z directions. For example, a perfectly aligned lens has approximately 0° relative to the x, y, and/or z directions. In other words, the surface of the lens is 90° or perpendicular to the light along the z direction. Additionally, the edges of the lens (if the lens has a rectangular shape) are parallel along the x and y directions. The lens and/or optical image sensor may be tilted through yaw, pitch, and/or rule motion. For example, the lens and/or the optical image sensor may be tilted along the pitch axis relative to the z-direction to face upward or downward. Light along the z-direction contacts the surface of the lens (pitched upward or downward) at a non-orthogonal angle. By tilting the lens and/or the optical image sensor along the yaw, pitch, or roll axes, focus or ZRP can be adjusted, for example.

I. Example Stereo Visualization Camera FIGS. 3 and 4 illustrate illustrations of perspective views of a stereo visualization camera 300 according to example embodiments of the present disclosure. An example camera 300 includes a housing 302 configured to enclose optical elements, a lens motor (eg, an actuator), and signal processing circuitry. Camera 300 has a width (along the x-axis) of 15 centimeters (cm) to 28 cm, preferably about 22 cm. Additionally, camera 300 has a length (along the y-axis) of 15 cm to 32 cm, preferably about 25 cm. Furthermore, the camera 300 has a height (along the z-axis) of 10 cm to 20 cm, preferably about 15 cm. The camera 300 weighs between 3 kg and 7 kg, preferably about 3.5 kg.

Camera 300 also includes control arms 304a and 304b (eg, operating handles) that are configured to control magnification level, focus, and other microscope features. Control arms 304a and 304b may include respective control mechanisms 305a and 305b that activate or select particular features. For example, control arms 304a and 304b select the fluorescence mode, adjust the amount/type of light projected onto the target area, and control display output signals (e.g., selection of 1080p or 4K and/or stereoscopic viewing). Control mechanisms 305a and 305b may be included. Additionally, control mechanism 305a and/or 305b may be used to initiate and/or perform a calibration procedure and/or move a robotic arm connected to stereoscopic visualization camera 300. In some cases, controls 305a and 305b may include the same buttons and/or features. In other cases, control mechanisms 305a and 305b may include different features. Further, control arms 304a and 304b can also be configured as grips that allow an operator to position stereoscopic visualization camera 300.

Each control arm 304 is connected to the housing 302 via a rotatable post 306, as shown in FIG. This connection allows control arm 304 to rotate relative to housing 302. This rotation provides flexibility for the surgeon to position control arm 304 as desired, further enhancing the adaptability of stereoscopic visualization camera 300 in synchronization with surgical performance.

Although the example camera 300 shown in FIGS. 3 and 4 includes two control arms 304a and 304b, it is understood that the camera 300 may include only one control arm or no control arms. I want to be understood. If stereoscopic visualization camera 300 does not include a control arm, a control mechanism may be integrated into housing 302 and/or provided via a remote control mechanism.

FIG. 4 shows a bottom-to-top perspective view of the rear side of a stereoscopic visualization camera 300 according to an example embodiment of the present disclosure. Stereo visualization camera 300 includes a mounting bracket 402 configured to connect to a support. As explained in more detail in FIGS. 5 and 6, the support may include an arm with one or more joints that provide great maneuverability. The arm may be connected to a movable cart or fixed to a wall or ceiling.

Stereo visualization camera 300 also includes a power port 404 configured to receive a power adapter. Power may be received from an AC outlet and/or the cart's battery. In some cases, stereoscopic visualization camera 300 may include an internal battery to facilitate cordless operation. In these cases, power port 404 may be used to charge the battery. In an alternative embodiment, power port 404 may be integrated into mounting bracket 402 such that stereoscopic visualization camera 300 receives power via wire (or other conductive wiring material) within the support.

FIG. 4 also shows that stereoscopic visualization camera 300 may include a data port 406. Example data ports 406 include, for example, an Ethernet interface, a high-definition multimedia interface ("HDMI") interface, a universal serial bus ("USB") interface, a serial digital interface ("SDI"), etc. , digital optical interfaces, RS-232 serial communication interfaces, and the like. Data port 406 is configured to provide a communication connection between stereoscopic visualization camera 300 and a cord wired to one or more computing devices, servers, recording devices, and/or display devices. The communication connection may transmit stereoscopic or two-dimensional video signals for further processing, storage, and/or display. Data port 406 may also enable control signals to be sent to stereoscopic visualization camera 300. For example, an operator at a connected computer (e.g., a laptop computer, a desktop computer, and/or a tablet computer) can send control signals to the stereoscopic visualization camera 300 to direct operations, perform calibrations, or change output display settings. can be sent to.

In some embodiments, data port 406 may replace (and/or supplement) a wireless interface. For example, stereoscopic visualization camera 300 may transmit stereoscopic display signals to one or more display devices via Wi-Fi. The use of a wireless interface in combination with an internal battery allows the stereoscopic visualization camera 300 to be wire-free, thereby further improving maneuverability within the surgical environment.

The stereoscopic visualization camera 300 shown in FIG. 4 also includes a front working distance main objective 408 of the main objective lens assembly. An example lens 408 is at the beginning of the optical path within stereoscopic visualization camera 300. Light from a light source within stereoscopic visualization camera 300 is directed through lens 408 to the target site. Additionally, light reflected from the target site is received by lens 408 and passed to downstream optical elements.

II. Exemplary Operability of a Stereo Visualization Camera FIGS. 5 and 6 show diagrams of a stereo visualization camera 300 used within a microsurgical environment 500, according to example embodiments of the present disclosure. As shown, the small footprint and maneuverability of the stereoscopic visualization camera 300 (especially when used in conjunction with a multi-degree-of-freedom arm) allows for flexible positioning relative to the patient 502. The portion of patient 502 in the view of stereoscopic visualization camera 300 includes target region 503 . The surgeon 504 can position the stereoscopic visualization camera 300 in nearly any orientation while leaving more than enough surgical space above the patient 502 (lying supine). Accordingly, the stereoscopic visualization camera 300 minimally invasively (or non-invasively) enables the surgeon 504 to perform life-changing microsurgical procedures without interference or obstruction.

In FIG. 5, stereoscopic visualization camera 300 is connected to mechanical arm 506 via mounting bracket 402. Arm 506 may include one or more rotatable or extendable joints with electromechanical brakes to facilitate easy repositioning of stereoscopic visualization camera 300. To move stereoscopic visualization camera 300, surgeon 504 or assistant 508 activates a brake release on one or more joints of arm 506. After the stereoscopic visualization camera 300 is moved to the desired position, a brake may be engaged to lock the joint of the arm 506 in place.

A major feature of the stereoscopic visualization camera 300 is that it does not include an eyepiece. This means that there is no need to align the stereoscopic visualization camera 300 with the surgeon's 504 eyes. This freedom allows the stereoscopic visualization camera 300 to be positioned and oriented in a desired position that was not practical or possible with previously known surgical microscopes. In other words, the surgeon 504 can perform microsurgery using the optimal view to perform the procedure, rather than being limited to just the appropriate view determined by the eyepiece of the surgical microscope.

Returning to FIG. 5, the stereoscopic visualization camera 300 is connected via a mechanical arm 506 to a cart 510 (collectively a stereoscopic visualization platform or a stereoscopic robotic platform 516) having display monitors 512 and 514. In the illustrated configuration, stereoscopic visualization platform 516 is self-contained and may be moved to any desired location within microsurgical environment 500, including between operating rooms. The integration platform 516 allows the stereoscopic visualization camera 300 to be moved and used on demand without the necessary time to configure the system by connecting display monitors 512 and 514.

Display monitors 512 and 514 can include any type of display, including high-definition televisions, ultra-high-definition televisions, smart glasses, projectors, one or more computer screens, laptop computers, tablet computers, and/or smartphones. may be included. Display monitors 512 and 514, like stereoscopic visualization camera 300, may be connected to a mechanical arm to allow flexible positioning. In some cases, display monitors 512 and 514 may include touch screens to allow an operator to send commands to stereoscopic visualization camera 300 and/or adjust display settings.

In some embodiments, cart 516 may include computer 520. In these embodiments, computer 520 may control a robotic mechanical arm connected to stereoscopic visualization camera 300. Additionally or alternatively, computer 520 may process video (or stereoscopic video) signals (eg, image streams or frame streams) from stereoscopic visualization camera 300 for display on display monitors 512 and 514. For example, computer 520 may combine or interleave the left and right video signals from stereoscopic visualization camera 300 to create a stereoscopic signal that displays a stereoscopic image of the target site. Computer 520 can also be used to store video signals and/or stereoscopic video signals in video files (stored in memory) so that the surgeon's procedures can be documented and played back. Additionally, computer 520 can also send control signals to stereoscopic visualization camera 300 to select settings and/or perform calibrations.

In some embodiments, the microsurgical environment 500 of FIG. 5 includes an ophthalmic surgical procedure. In this embodiment, mechanical arm 506 may be programmed to perform a circumferential sweep of the patient's eye. Such a sweep allows the surgeon to examine the peripheral retina during vitreoretinal procedures. In contrast, when using a traditional light microscope, the only way a surgeon can see the peripheral retina is by pushing the side of the eye into the field of view using a technique known as scleral compression. It is.

FIG. 6 shows a diagram of a microsurgical environment 500 with a patient 502 in a seated position for posterior access skull base neurosurgery. In the illustrated embodiment, stereoscopic visualization camera 300 is placed in a horizontal position facing the back of the patient's 502 head. Mechanical arm 506 includes a joint that allows stereoscopic visualization camera 300 to be positioned as shown. Additionally, cart 510 includes a monitor 512 that may be aligned with the surgeon's natural viewing direction.

The lack of an eyepiece allows the stereoscopic visualization camera 300 to be positioned horizontally and below the surgeon's 504 eye level view. Additionally, the relative light weight and flexibility allows the stereoscopic visualization camera 300 to be positioned in ways unimaginable with other known surgical microscopes. The stereoscopic visualization camera 300 thereby provides a microsurgical view to the patient 502 and/or surgeon 504 in any desired position and/or orientation.

Although FIGS. 5 and 6 illustrate two example embodiments of positioning the stereoscopic visualization camera 300, the stereoscopic visualization camera 300 can be positioned in any number of positions depending on the number of degrees of freedom of the mechanical arm 506. I want you to understand that. In some embodiments, it is entirely possible to position the stereoscopic visualization camera 300 so that it is facing upwards (eg, upside down).

III. Comparison of an Example Stereo Visualization Platform with Known Surgical Microscopes When comparing the stereo visualization camera 300 of FIGS. 3-6 with the surgical microscope 200 of FIG. 2, the differences are readily apparent. The inclusion of the eyepiece 206 in the surgical microscope requires the surgeon to always direct his eyes toward the eyepiece in a fixed position relative to the scope head 201 and the patient. Additionally, the bulk and weight of surgical microscopes allows them to be positioned only in a generally perpendicular orientation relative to the patient. In contrast, the example stereoscopic visualization camera 300 does not include an eyepiece and can be positioned in any orientation or position relative to the patient, allowing the surgeon to move freely during the procedure. I can do it.

To allow other clinical staff to view the microsurgical target area, surgical microscope 200 requires the addition of a second eyepiece 208. Generally, most known surgical microscopes 200 do not allow the addition of a third eyepiece. In contrast, the example stereoscopic visualization camera 300 can be communicatively coupled to an unlimited number of display monitors. Although FIGS. 5 and 6 previously showed display monitors 512 and 514 connected to cart 510, the operating room may be surrounded by display monitors showing all microsurgical views recorded by stereoscopic visualization camera 300. can. Thus, instead of limiting the view to one or two people (or requiring sharing of eyepieces), the entire surgical team can see a magnified view of the target surgical site. Additionally, people in other rooms, such as training rooms and observation rooms, can be presented with the same magnified view that is displayed to the surgeon.

Compared to the stereoscopic visualization camera 300, the two-eyepiece surgical microscope 200 is more susceptible to collisions or unintentional movement. During surgery, the scope head 201 is subjected to constant forces and periodic collisions as the surgeon positions his head over the eyepieces 206 and 208 in order to view through the eyepieces. The addition of the second eyepiece 208 doubles the power from the second angle. Together, constant force and periodic bumps by the surgeon can move scope head 201, thereby requiring repositioning of scope head 201. This repositioning delays the surgical procedure and makes the surgeon uncomfortable.

The example stereoscopic visualization camera 300 does not include an eyepiece and is not intended to receive contact from the surgeon once locked in place. This corresponds to a significant reduction in the possibility that the stereoscopic visualization camera 300 will move or collide unintentionally during the surgeon's procedure.

To facilitate the second eyepiece 208, the surgical microscope 200 must be equipped with a beam splitter 210, which may include a glass lens and mirror housed within a precision metal tube. The use of beam splitter 210 reduces the light received by the first eyepiece because some of the light is reflected to the second eyepiece 208. Additionally, the addition of second eyepiece 208 and beam splitter 210 increases the weight and bulk of scope head 201.

In contrast to surgical microscope 200, stereoscopic visualization camera 300 includes only a sensor optical path, thereby reducing weight and bulk. In addition, there is no need for a beam splitter to redirect some of the light, so the optical sensor receives all of the incident light. This means that the images received by the optical sensors of the example stereoscopic visualization camera 300 are as bright and clear as possible.

Some models of surgical microscopes may be able to be fitted with a video camera. For example, the surgical microscope 200 of FIG. 2 includes a planar viewing video camera 212 connected to the optical path via a beam splitter 214. Video camera 212 may be monoscopic or stereoscopic, such as a Leica® TrueVision® 3D Visualization System Ophthalmic Camera. Video camera 212 records images received from beam splitter 214 for display on a display monitor. The addition of video camera 212 and beam splitter 214 further adds weight to scope head 201. In addition, beam splitter 214 consumes additional light directed to eyepiece 206 and/or 208.

Each beam splitter 210 and 214 partially splits the incoming light into three paths and removes the light from the surgeon's view. The surgeon's eye has limited low light sensitivity such that the light from the surgical site presented to the surgeon must be sufficient to enable the surgeon to perform the procedure. However, surgeons are not always able to increase the intensity of the light applied to the target area of the patient, especially in ophthalmic procedures. The patient's eyes have a limited high light sensitivity, beyond which phototoxicity occurs. Therefore, there are limits to the number and proportion of beam splitters and the amount of light that can be split from the first eyepiece 206 to enable use of the auxiliary devices 208 and 212.

The example stereoscopic visualization camera 300 of FIGS. 3-6 does not include a beam splitter so that the optical imaging sensor receives all of the light from the main objective lens assembly. This allows the use of sensors with low light sensitivity, or even optical sensors with sensitivities outside the wavelengths of visible light used, since post-processing can make them bright enough to be visible (and tunable) for display on a monitor. becomes possible.

Furthermore, the optical elements that define the optical path are contained within the stereoscopic visualization camera 300 itself and thus can be controlled through the camera. This control allows the placement and adjustment of optical elements to be optimized for a three-dimensional stereoscopic display rather than a microscope eyepiece. This configuration of the camera allows control to be provided electronically from the camera control mechanism or from a remote computer. In addition, the control may include one or more controls mounted on camera 300 configured to adjust the optical elements to maintain focus while zooming or to adjust for optical defects and/or pseudo-parallax. Can be delivered automatically through multiple programs. In contrast, the optical elements of surgical microscope 200 are external to video camera 212 and are typically controlled only via operator input, which is optimized for viewing the target site through eyepiece 206. Ru.

In a final comparison, surgical microscope 200 includes an XY panning device 220 that moves the field of view or target area. The XY pan device 220 is typically a large, heavy, and expensive electromechanical module because it must firmly support and move the surgical scope head 201. Additionally, moving scope head 201 repositions the surgeon to a new position of eyepiece 206.

In contrast, one example stereoscopic visualization camera 300 includes memory containing instructions that, when executed, cause a processor to selectively coordinate pixel data of an optical sensor to enable X-Y panning over a wide pixel grid. include. Additionally, an example stereoscopic visualization camera 300 may include a small motor or actuator that controls the main objective lens optics to deflect the working distance to the target site without moving the camera 300.

IV. Optical Elements of an Example Stereo Visualization Camera FIGS. 7 and 8 illustrate diagrams illustrating optical elements within an example stereo visualization camera 300 of FIGS. 3-6, according to example embodiments of the present disclosure. It may seem relatively simple to obtain left and right views of the target site in order to construct a stereoscopic image. However, without careful design and compensation, many stereoscopic images have registration problems between left and right views. When viewed over long periods of time, alignment problems can create confusion in the observer's brain as a result of the difference between the left and right views. This confusion can cause headaches, fatigue, dizziness, and even nausea.

An example stereoscopic visualization camera 300 is constructed by having a right optical path and a left optical path with independent control and/or adjustment of some optical elements while other left and right optical elements are fixed to a common carrier. , reduce (or eliminate) alignment problems. In example embodiments, several left and right zoom lenses may be fixed to a common carrier to ensure that the left and right magnifications are approximately the same. However, the front lens or the rear lens can be independently adjusted radially, rotationally, axially, and/or tilted, resulting in small differences in zoom exposure, such as movement of the zoom repetition point, and visual defects. , and/or compensate for pseudo parallax. The compensation provided by the adjustable lens results in a substantially perfectly aligned optical path throughout the entire zoom magnification range.

Additionally or alternatively, alignment issues may be reduced (or eliminated) using pixel readout and/or rendering techniques. For example, the right image (recorded by the right optical sensor) may be adjusted upward or downward relative to the left image (recorded by the left optical sensor) to correct for vertical misalignment between the images. Similarly, the right image may also be adjusted to the left or right relative to the left image to correct for horizontal misalignment between the images.

7 and 8 below illustrate an example configuration and position of optical elements that provide aligned optical paths that are substantially free of artifacts, pseudo-parallax, and distortion. As discussed below, certain optical elements may be moved during calibration and/or use to further align the optical path and eliminate any residual distortion, spurious parallax, and/or defects. In the illustrated embodiment, the optical elements are positioned in two parallel paths to produce a left view and a right view. Alternative embodiments may include optical paths that are folded, deflected, or otherwise non-parallel.

The path shown is approximately such that the left and right views displayed on a stereoscopic display are similar to the convergence angle of an adult human eye viewing an object approximately 4 feet away. Corresponds to the human visual system which appears to be separated by a distance that produces a convergence angle of 6 degrees, thereby producing stereopsis. In some embodiments, image data generated from the left and right views are combined together at display monitors 512 and 514 to generate a stereoscopic image of the target site or scene. Alternative embodiments include other stereoscopic displays in which the left view is presented only to the viewer's left eye and the corresponding right view is presented only to the right eye. In an exemplary embodiment used to adjust and verify proper alignment and calibration, both views are displayed superimposed on both eyes.

Stereo views are better than flat views because they more closely mimic the human visual system. The stereoscopic view provides depth perception, distance perception, and relative size perception to provide the surgeon with a more realistic view of the target surgical site. In procedures such as retinal surgery, stereoscopic views are essential because surgical movements and forces are too small for the surgeon to feel. Providing a stereoscopic view helps the surgeon's brain magnify the sense of touch when the brain detects small movements while perceiving depth.

FIG. 7 shows a side view of an example stereoscopic visualization camera 300 having a housing 302 that is transparent and exposes optical elements. FIG. 8 shows a diagram illustrating the optical path provided by the optical element shown in FIG. As shown in FIG. 8, the optical path includes a right optical path and a left optical path. The light path in FIG. 8 is shown from a perspective facing forward and looking down on the stereoscopic visualization camera 300. From this figure, the left optical path is visible on the right side of FIG. 8, while the right optical path is shown on the left side.

The optical elements shown in FIG. 7 are part of the left optical path. It should be appreciated that the right optical path in FIG. 7 is generally identical to the left optical path with respect to relative location and configuration of optical elements. As mentioned above, the interpupillary distance between the centers of the optical paths is 58 mm to 70 mm, which can be scaled from 10 mm to 25 mm. Each optical element comprises a lens having a particular diameter (eg, 2 mm to 29 mm). The distance between the optical elements themselves is therefore between 1 mm and 23 mm, preferably about 10 mm.

The example stereoscopic visualization camera 300 is configured to capture images of a target region 700 (also referred to as a scene or field of view (“FOV” or target surgical region). The target region 700 is a location of a patient's anatomy. Target site 700 may also include a laboratory biological sample, a calibration slide/template, etc. Images from target site 700 are received at stereoscopic visualization camera 300 via main objective lens assembly 702; Main objective lens assembly 702 includes a front working distance lens 408 (shown in FIG. 4) and a rear working distance lens 704.

A. Example Main Objective Lens Assembly The example main objective lens assembly 702 may include any type of refractive or reflective assembly. FIG. 7 shows objective lens assembly 702 as an achromatic refractive assembly in which front working distance lens 408 is stationary and rear working distance lens 704 is movable along the z-axis. Front working distance lens 408 may include a plano-convex (“PCX”) lens and/or a meniscus lens. Rear working distance lens 704 may include an achromatic lens. In examples where main objective lens assembly 702 includes an achromatic refractive assembly, front working distance lens 408 may include a hemispherical lens and/or a meniscus lens. Additionally, the rear working distance lens 704 may include an achromatic doublet lens, an achromatic doublet group of lenses, and/or an achromatic triplet lens.

The magnification of the main objective lens assembly 702 is 6x to 20x. In some cases, the magnification of main objective lens assembly 702 may vary slightly based on working distance. For example, main objective lens assembly 702 may have a magnification of 8.9x at a 200 mm working distance and 8.75x at a 450 mm working distance.

The example rear working distance lens 704 is configured to be movable relative to the front working distance lens 408 to thereby change the spacing between the rear working distance lens 704 and the front working distance lens 408. The spacing between lenses 408 and 704 determines the overall front focal length of main objective lens assembly 702 and thus the location of the focal plane. In some embodiments, the focal length is the distance between lenses 408 and 704 plus half the thickness of front working distance lens 408.

The front working distance lens 408 and the rear working distance lens 704 are configured together to provide an infinite conjugate image to provide optimal focus to the downstream optical image sensor. In other words, an object placed exactly in the focal plane of the target site 700 will project its image to an infinite distance, thereby being combined infinitely at the working distance provided. Generally, an object appears in focus for a certain distance along the optical path from the focal plane. However, beyond a certain threshold distance, objects begin to appear blurry or out of focus.

FIG. 7 shows working distance 706, which is the distance between the outer surface of front working distance lens 408 and the focal plane of target region 700. The working distance 706 may correspond to the viewing angle; the longer the working distance, the wider the field of view or the larger the viewable area. Working distance 706 accordingly sets the plane of the target region or scene in focus. In the illustrated example, working distance 706 is adjustable from 200 mm to 450 mm by moving rear working distance lens 704. In one example, the field of view is adjustable from 20 mm x 14 mm to 200 mm x 140 mm using an upstream zoom lens when the working distance is 450 mm.

The main objective lens assembly 702 shown in FIGS. 7 and 8 provides images of the target region 700 in both the left and right optical paths. This means that the width of lenses 408 and 704 should be at least twice the width of the left and right optical paths. In an alternative embodiment, the main objective lens assembly 702 may include separate left and right front working distance lenses 408 and separate left and right rear working distance lenses 704. The width of each pair of separate working distance lenses can be 1/4 to 1/2 the width of lenses 408 and 704 shown in FIGS. 7 and 8. Additionally, each rear working distance lens 704 may be independently adjustable.

In some embodiments, main objective lens assembly 702 may be replaceable. For example, different main objective lens assemblies may be added to change the working distance range, magnification, numerical aperture, and/or refractive/reflective type. In these embodiments, the stereoscopic visualization camera 300 may change the positioning of downstream optical elements, the characteristics of the optical image sensor, and/or the parameters of the underlying image processing during main objective lens assembly installation. An operator may specify which main objective lens assembly is installed on stereoscopic visualization camera 300 using one of the controls 305 and/or the user input device of FIG. 3.

B. Example Illumination Sources To illuminate target region 700, example stereoscopic visualization camera 300 includes one or more illumination sources. 7 and 8 illustrate three illumination sources, including a visible light source 708a, a near-infrared ("NIR") light source 708b, and a near-ultraviolet ("NUV") light source 708c. In other examples, stereoscopic visualization camera 300 may include additional or fewer light sources (or may include no light sources). For example, NIR and NUV light sources can be omitted. An example light source 708 is configured to generate light that is projected onto the target scene 700. The generated light interacts with and is reflected from the target scene, and some of the light is reflected to the main objective lens assembly 702. Other examples may include ambient light from an external light source or environment.

An example visible light source 708a is configured to output light in the human visible portion of the light spectrum in addition to some light having wavelengths outside the visible range. NIR light source 708b is configured to output light primarily at wavelengths slightly beyond the red portion of the visible spectrum, also referred to as "near infrared." NUV light source 708c is configured to output light at wavelengths primarily in the blue portion of the visible spectrum, also referred to as "near ultraviolet light." The light spectrum output by the light source 708 is controlled by each controller described below. The brightness of the light emitted by light source 708 may be controlled by the switching rate and/or the applied voltage waveform.

7 and 8 show that a visible light source 708a and a NIR light source 708b are provided directly to the target site 700 through the main objective lens assembly 702. As shown in FIG. 8, visible light from visible light source 708a propagates along visible path 710a. Additionally, NIR light from NIR light source 708b propagates along NIR path 710b. Although light sources 708a and 708b are shown as being behind main objective assembly 702 (with respect to target site 700), in other examples light sources 708a and 708b may be provided in front of main objective assembly 702. It is possible. In one embodiment, light sources 708a and 708b may be provided external to housing 302 facing target site 700. In yet other embodiments, light source 708 may be provided separately from stereoscopic visualization camera 300 using, for example, a Koeher lighting setup and/or a darkfield lighting setup.

In contrast to light sources 708a and 708b, NUV light from NUV light source 708c is reflected by deflection element 712 (eg, a beam splitter) to main objective lens assembly 702 using an epi-illumination setup. Deflection element 712 may be coated or otherwise configured to reflect only light above the NUV wavelength range, thereby filtering NUV light. NUV light from NUV light source 708c propagates along NUV path 710c.

In some embodiments, NIR light source 708b and NUV light source 708c may be used in conjunction with an excitation filter to further filter light that is not blocked by the filter (eg, filter 740). Filters may be placed before the main objective assembly 702 and/or after the main objective assembly and before the light sources 708b and 708c. The light from NUV light source 708b and NIR light source 708c, after filtering, includes wavelengths that excite fluorescence at fluorescent site 914 (shown in FIG. 9) of the anatomical object. Additionally, the light from NUV light source 708c and NIR light source 708b, after filtering, may include wavelengths that are not in the same range as that emitted by fluorescent moieties 914.

Projecting light from light source 708 through the main objective lens assembly provides the advantage of varying the illuminated field of view based on working distance 706 and/or focal plane. As the light passes through the main objective lens assembly 702, the angle at which the light is projected varies based on the working distance 706 and corresponds to the viewing angle. This arrangement accordingly ensures that the field of view is suitably illuminated by the light source 708, regardless of the working distance or magnification.

C. Example Deflection Element The example deflection element 712 shown in FIGS. 7 and 8 is configured to transmit light of a particular wavelength from the NUV light source 708c through the main objective lens assembly 702 and to the target site 700. Deflection element 712 is also configured to reflect light received from target site 700 to downstream optical elements, including front lens set 714, for zooming and recording. In some embodiments, deflection element 712 may filter light received from target site 700 through main objective lens assembly 702 such that light of a particular wavelength reaches front lens set 714.

Deflection element 712 may include any type of mirror or lens that reflects light in a specified direction. In one example, deflection element 712 includes a dichroic mirror or filter that has different reflection and transmission properties at different wavelengths. The stereoscopic visualization camera 300 of FIGS. 7 and 8 includes one deflection element 712 that provides light in both the right and left optical paths. In other examples, camera 300 may include separate deflection elements in each of the right and left optical paths. Additionally, a separate deflection element may be provided to NUV light source 708c.

FIG. 9 shows a diagram of the deflection element 712 of FIGS. 7 and 8, according to an example embodiment of the present disclosure. For simplicity, main objective lens assembly 702 is not shown. In this example, deflection element 712 includes two parallel surfaces 902 and 904 that transmit and reflect light of specific wavelengths. Parallel surfaces 902 and 904 are set at a 45° angle with respect to the left and right optical paths (represented as path 906). The 45° angle is selected because this angle causes the reflected light to propagate at a 90° angle from the transmitted light, thereby providing optimal separation without causing the separated light to be detected downstream at the front lens set 714. , selected. In other embodiments, the angle of deflection element 712 can be between 10 degrees and 80 degrees without unintentionally propagating unwanted wavelengths of light.

An example NUV light source 708c is positioned behind deflection element 712 (with respect to target site 700). Light from light source 708c propagates along path 908 and contacts deflection element 712. NUV light around the main wavelength range of NUV light source 708c passes through deflection element 712 and reaches target site 700 along path 910. Light from NUV light source 708c having wavelengths above (and below) the primary wavelength range of NUV light source 708c is reflected along path 912 to an optical sink or unused area of housing 302.

Once the NUV light reaches the target site 700, it is absorbed by one or more fluorescent sites 914 on the anatomical object. In some cases, the anatomical object may be injected with a contrast agent configured to absorb NUV light and emit light at a different dominant wavelength. In other cases, anatomical objects may naturally absorb NUV light and emit light at a different dominant wavelength. At least some of the light reflected or emitted by fluorescent site 914 propagates along path 916 until it contacts deflection element 712. Most of the light is reflected from surface 904 and travels along path 906 to front lens set 714 . A portion of the light containing NUV light around the main wavelength range of NUV light source 708c is transmitted through deflection element 712 and sent along path 918 to an optical sink or unused area of housing 302. Accordingly, the deflection element 712 shown in FIG. Make it.

It should be appreciated that the reflection and transmission properties of deflection element 712 can be modified to suit other optical spectrum requirements. In some cases, the housing 302 may include slots that allow the deflection element 712 and/or the NUV light source 708c to be replaced based on desired light reflection and transmission properties. A first path within the deflection element 712 between paths 908 and 910 and a second path within the deflection element 712 between paths 916 and 918 are each tilted so that the light is directed between the air and the deflection element 712. It should also be understood that it schematically represents the refraction of light as it travels to and from the interior of the . The angles shown are not intended to represent actual reflection angles.

D. Example Zoom Lens The example stereoscopic visualization camera 300 of FIGS. 7 and 8 includes one or more zoom lenses that change the focal length and viewing angle of the target region 700 to provide zoom magnification. In the illustrated example, the zoom lens includes a front lens set 714, a zoom lens assembly 716, and a lens barrel set 718. It should be appreciated that in other embodiments, front lens set 714 and/or lens barrel set 718 may be omitted. Alternatively, a zoom lens may include additional lenses to provide additional magnification and/or image resolution.

Front lens set 714 includes a right front lens 720 for the right optical path and a left front lens 722 for the left optical path. Lenses 720 and 722 may each include a positive converging lens that directs light from deflection element 712 to each lens in zoom lens assembly 716. The lateral positions of lenses 720 and 722 accordingly define the beam from main objective lens assembly 702 and deflection element 712 that propagates to zoom lens assembly 716.

One or both of lenses 720 and 722 may be radially adjustable to align with the optical axes of the left and right optical paths. In other words, one or both of lenses 720 and 722 may move left and right and/or up and down in the plane incident on the optical path. In some embodiments, one or more of lenses 720 and 722 may be rotated or tilted to reduce or eliminate image optical defects and/or pseudo-parallax. Movement of either or both lenses 720 and 722 during zooming may cause the zoom repetition point (“ZRP”) of each optical path to appear to the user to remain stationary. In addition to radial movement, one or both of front lenses 720 and 722 may move axially (along each optical path) to match the magnification of the optical paths.

One example zoom lens assembly 716 forms an afocal zoom system that changes the size of the field of view (eg, a linear field of view) by changing the size of the light rays that propagate to the lens barrel set 718. Zoom lens assembly 716 includes a front zoom lens set 724 having a front right zoom lens 726 and a front left zoom lens 728. Zoom lens assembly 716 also includes a rear zoom lens set 730 having a right rear zoom lens 732 and a left rear zoom lens 734. Front zoom lenses 726 and 728 may be positive converging lenses, while rear zoom lenses 732 and 734 include negative diverging lenses.

The size of each image beam in the left and right optical paths is determined based on the distance between front zoom lenses 726 and 728, rear zoom lenses 732 and 734, and lens barrel set 718. Generally, the size of the optical paths decreases as the rear zoom lenses 732 and 734 move towards the lens barrel set 718 (along each optical path), thereby decreasing the magnification. In addition, front zoom lenses 726 and 728 also move towards (or away from) lens barrel set 718 (equivalently in a parabolic arc) as rear zoom lenses 732 and 734 move towards lens barrel set 718. ), the focal plane position may be maintained over the target region 700, thereby maintaining focus.

Front zoom lenses 726 and 728 may be contained within a first carrier (e.g., front zoom set 724), while rear zoom lenses 732 and 724 may be contained within a second carrier (e.g., rear zoom set 724). set 730). Each carrier 724 and 730 may move along the optical path on a track (or rail) such that the left and right magnifications change simultaneously. In this embodiment, any slight magnification difference between the left and right optical paths may be corrected by moving the right front lens 720 and/or the left front lens 722. Additionally or alternatively, right lens barrel 736 and/or left lens barrel 738 of lens barrel set 718 may move axially.

In an alternative embodiment, the right front zoom lens 726 may move axially independently of the left front zoom lens 728. Additionally, right rear zoom lens 732 may move axially independently of left rear zoom lens 734. The separate movements allow zoom lens assembly 716 to correct for small magnification differences, especially when front lens set 714 and lens barrel set 718 are stationary along the optical path. Further, in some embodiments, the right front zoom lens 726 and/or the left front zoom lens 728 may be radially and/or rotationally adjustable (and/or tiltable), thereby allowing the ZRP maintain the apparent position of within the optical path. Additionally or alternatively, right rear zoom lens 732 and/or left rear zoom lens 734 may be waveform-wise and/or rotationally adjustable (and/or tiltable), thereby adjusting the apparent position of the ZRP. in the optical path.

An example lens barrel set 718 includes a right lens barrel 736 and a left lens barrel 738, which are part of an afocal zoom system in addition to the zoom lens assembly 716. Lenses 736 and 738 may include positive converging lenses configured to straighten or focus the light rays from zoom lens assembly 716. In other words, lenses 736 and 738 focus the infinitely coupled output of zoom lens assembly 716.

In some examples, lens barrel set 718 is radially and axially fixed within housing 302. In other examples, lens barrel set 718 may be axially movable along the optical path, thereby providing increased magnification. Additionally or alternatively, each of lenses 736 and 738 may be radially and/or rotationally adjustable (and/or tiltable), such that, for example, front lens set 714, front zoom lens The optical characteristic difference (magnification or natural glass shift) between the left and right lenses of the set 724 and/or the rear zoom lens set 730 is corrected.

An example front lens set 714, zoom lens assembly 716, and lens barrel set 718 are configured together to achieve an optical zoom of 5X to about 20X, preferably in a zoom lens with diffraction-limited resolution. In some embodiments, front lens set 714, zoom lens assembly 716, and lens barrel set 718 may provide a higher zoom range (e.g., 25X to 100X) if image quality can be compromised. . In these embodiments, the stereoscopic visualization camera 300 may output a message to the operator indicating that the selected optical range is outside the optical range and will suffer a reduction in image quality.

In some embodiments, the lenses of front lens set 714, zoom lens assembly 716, lens barrel set 718, and/or main objective lens assembly 702 each use materials that balance the optical distortion parameters of each other. can be constructed as a doublet from multiple optical sub-elements. Doublet construction reduces chromatic and optical aberrations. For example, front working distance lens 408 and rear working distance lens 702 may each be constructed as a doublet. In another example, front lenses 720 and 722, front zoom lenses 726 and 728, rear zoom lenses 732 and 734, and lens barrels 736 and 738 may each include a doublet lens.

In further embodiments, the lenses of front lens set 714, zoom lens assembly 716, lens barrel set 718, and/or main objective lens assembly 702 may be adjusted differently and/or have different characteristics. , thereby providing two parallel optical paths with different functions. For example, the right lens in zoom lens assembly 716 may be selected to provide 5X to 10X of optical zoom in the right optical path, while the left lens in zoom lens assembly 716 may be selected to provide 15X to 20X in the left optical path. selected to provide optical zoom. Such a configuration may allow two different magnifications to be displayed simultaneously and/or on the same screen, although in planar view.

E. Example Filters The example stereoscopic visualization camera 300 of FIGS. 7 and 8 includes one or more optical filters 740 (or filter assemblies) that selectively transmit desired wavelengths of light. FIG. 8 shows that one filter 740 can be applied to the left and right optical paths. In other examples, each optical path may have a separate filter. The inclusion of separate filters allows, for example, different wavelengths of light from the left and right optical paths to be filtered simultaneously, thereby allowing, for example, a fluorescence image to be displayed in conjunction with a visible light image.

FIG. 7 shows that filter 740 includes wheels that rotate about an axis of rotation. In the illustrated embodiment, filter 740 can house three different optical filter pairs. However, in other embodiments, filter 740 may include additional or fewer filter pairs. Generally, the light received at filter 740 from target site 700 includes a broad spectrum of wavelengths. The lenses of main objective lens assembly 702, front lens set 714, zoom lens assembly 716, and lens barrel set 718 are configured to transmit a relatively broad band of light, including wavelengths of interest and undesirable wavelengths to the operator. configured. Additionally, downstream optical image sensors are sensitive to specific wavelengths. The example filter 740 accordingly transmits and blocks certain portions of the light spectrum to achieve different desired characteristics.

As a wheel, filter 740 comprises a mechanical device that can change position approximately 4 times per second. In other embodiments, filter 740 may include a digital micromirror that can change transition path direction at a video frame rate, such as 60 times per second. In these other embodiments, each of the left and right optical paths includes a micromirror. The left and right micromirrors may have synchronized switching or simultaneous switching.

In some embodiments, filter 740 may be synchronized with light source 708 to achieve "time interleaved" multispectral imaging. For example, filter 740 may include an infrared blocking filter, a near-infrared bandpass filter, and a near-ultraviolet blocking filter. Different filter types are selected to operate in conjunction with the different spectra of light source 708 and the characteristics and transmission characteristics of deflection element 712 to transmit light of a particular desired wavelength a predetermined number of times.

In one mode, filter 740 and light source 708 are configured to provide a visible light mode. In this mode, the visible light source 708 a forces light from the visible region down onto the target site 700 , some of which is reflected into the main objective lens assembly 702 . Reflected light may include some light beyond the visible spectrum that can affect optical image sensors. Visible light is reflected by deflection element 712 and transmitted through front lens set 714, zoom lens assembly 716, and lens barrel set 718. In this example, filter 740 applies an infrared blocking filter or a near ultraviolet blocking filter to the optical path so that only light within the visible spectrum passes through final optics set 742 and optical image sensor 744. configured to remove.

In another mode, filter 740 and light source 708 are configured to provide only narrow wavelength fluorescence to optical sensor 744. In this mode, NUV light source 708c transmits light from the blue region of the spectrum to target site 700. The deflection element 712 transmits the desired light in the blue region while reflecting the undesired light. The blue light interacts with the target site 700 such that fluorescence is emitted. In some examples, delta-aminolevulinic acid ("5ala") and/or protoporphyrin IX is applied to target site 700 to cause it to fluoresce when it receives indigo light. Main objective lens assembly 702 receives reflected blue light and some visible light, as well as fluorescent light. The blue light exits from the left and right optical paths and passes through the deflection element 712. Therefore, only visible light and fluorescence pass through front lens set 714, zoom lens assembly 716, and lens barrel set 718. In this example, filter 740 is configured to apply a near-UV blocking filter to the optical path to remove light outside the desired fluorescence spectrum, including visible light and any residual NUV blue light. Therefore, only narrow wavelength fluorescence reaches the optical image sensor 744, allowing the fluorescence to be more easily detected and distinguished based on relative intensity.

In yet another mode, filter 740 and light source 708 are configured to provide indocyanine green (“ICG”) fluorescence to optical sensor 744. In this mode, the NIV light source 708b sends light in the far-red region of the visible spectrum (also considered near-infrared) to the target site 700. Additionally, visible light source 708a sends visible light to target scene 700. The visible and far-red light is absorbed by the material bearing the ICG at the target site, and the ICG then emits highly stimulated fluorescence in the far-red region. Main objective lens assembly 702 receives reflected NIR and visible light as well as fluorescent light. Light is reflected by deflection element 712 to front lens set 714, zoom lens assembly 716, and lens barrel set 718. In this example, filter 740 is configured to apply a near-infrared bandpass filter to the optical path to remove light outside the desired fluorescence spectrum, including at least some visible and NIR light. Therefore, only the fluorescence in the far-red region reaches the optical image sensor 744, allowing the fluorescence to be more easily detected and distinguished based on relative intensity.

Table 1 above summarizes the different possible combinations of light sources and filters that allow light of a particular desired wavelength to reach the optical sensor 744. It should be appreciated that other types of filters and/or light sources may be used to further increase the different types of light received at image sensor 744. For example, a bandpass filter configured to transmit a narrow wavelength of light corresponding to a particular biological dye or contrast agent applied to target area 700 may be used. In some examples, filter 740 may include a concatenated filter or two or more filters, thereby allowing light from two different ranges to be filtered. For example, the first filter 740 may apply an infrared blocking filter and a near ultraviolet blocking filter so that only visible light in a desired wavelength range is transmitted to the optical sensor 744.

In other embodiments, separate filters 740 may be used for the left and right optical paths. For example, the right filter may include an infrared blocking filter, while the left filter includes a near infrared transmitting filter. Such a configuration allows display of target site 700 at visible wavelengths simultaneously with IGC green fluorescence wavelengths. In another example, the right filter may include an infrared blocking filter, while the left filter includes a near ultraviolet blocking filter. In this configuration, target site 700 can be displayed in visible light simultaneously with 5ALA fluorescence. In these other embodiments, the right and left image streams may still be combined into a stereoscopic view that provides a fluorescent view of specific anatomical structures combined with a view of the target site 700 in visible light.

F. Example Final Optical Element Set The example stereoscopic visualization camera 300 of FIGS. 7 and 8 includes a final optical element set 742 that focuses the light received from the filter 740 onto an optical image sensor 744. Final optical element set 742 includes right final optical element 745 and left final optical element 747, each of which may include a positive converging lens. In addition to focusing light, optical elements 745 and 747 may be configured to correct for small aberrations in the left and right optical paths before the light reaches optical image sensor 744. In some examples, lenses 745 and 747 may be radially and/or axially movable, thereby increasing the magnification and /or correct focusing aberrations. In one example, the left final optical element 747 may move in the radial direction, while the right final optical element 745 is fixed to eliminate ZRP movement during magnification changes.

G. Example Image Sensor The example stereoscopic visualization camera 300 of FIGS. 7 and 8 includes an image sensor 744 that captures and/or records incident light received from a final set of optical elements 742. Image sensors 744 include a right optical image sensor 746 that captures and/or records light propagating along a right optical path and a left optical image sensor 748 that captures and/or records light propagating along a left optical path. Each of left optical image sensor 746 and right optical image sensor 748 may include, for example, a complementary metal oxide semiconductor (“CMOS”) sensing element, an N-type metal oxide semiconductor (“NMOS”), and/or a semiconductor charge-coupled device. (“CCD”) includes a sensing element. In some embodiments, left optical sensor 746 and right optical sensor 748 are the same and/or have the same characteristics. In other embodiments, left optical sensor 746 and right optical sensor 748 include different sensing elements and/or characteristics that provide different functions. For example, right optical image sensor 746 (using a first color filter array) may be configured to be more sensitive to blue fluorescence, while left optical image sensor 748 (using a second color filter array) may be configured to be more sensitive to blue fluorescence. ) is configured to be more sensitive to visible light.

FIG. 10 illustrates an example of a right optical image sensor 746 and a left optical image sensor 748 of image sensor 744, according to example embodiments of the present disclosure. The right optical image sensor 746 includes a first two-dimensional grid or matrix 1002 of photosensitive elements (eg, pixels). Additionally, left optical image sensor 748 includes a second two-dimensional pixel grid 1004 of photosensitive elements. Each pixel includes a filter that allows only certain wavelengths of light to pass through and thus contact the photodetector below. Different colored filters are spread across sensors 746 and 748 to provide full wavelength light detection across the grid. Photodetectors may be sensitive to visible light and additional ranges above and below the visible spectrum.

The photosensitive elements of grids 1002 and 1004 are configured to record light of various wavelengths as a representation of target region 700 within the field of view. Light incident on the photosensitive element causes a charge to accumulate. The charge is read out to determine the amount of light being received at the sensing element. Additionally, the range of wavelengths of the received light is known because the filter characteristics of the sensing element are known to be within manufacturing tolerances. A representation of target region 700 is directed to the photosensitive element such that grids 1002 and 1004 of each optical image sensor 746 and 748 spatially sample target region 700. Spatial sampling resolution is a parameter that affects image quality and parity.

The number of pixels shown in pixel grids 1002 and 1004 in FIG. 10 does not represent the actual number of pixels in optical image sensors 746 and 748. Instead, the sensor typically has a resolution of 1280 x 720 pixels and 8500 x 4500 pixels, preferably about 2048 x 1560 pixels. However, not all pixels of grids 1002 and 1004 are selected for image transmission. Instead, a subset or pixel set of grids 1002 and 1004 is selected for transmission. For example, in FIG. 10, pixel set 1006 is selected from pixel grid 1002 for transmission as the right image, and pixel set 1008 is selected from pixel grid 1004 for transmission as the left image. As shown, pixel set 1006 need not be co-located with pixel set 1008 in relation to each pixel grid 1002 and 1004. Separate control of pixel sets 1006 and 1008 allows left and right images to be aligned and/or correct for image defects such as moving ZRP and/or pseudo-parallax.

Selection of pixel sets from the pixel grid may correct image defects/pseudo-parallax and/or select a portion of the pixel grid that more accurately aligns the right and left optical images. In other words, the pixel set may be moved or adjusted (in real time) relative to the pixel grid to improve image quality by reducing or eliminating pseudo-parallax. Alternatively, one or both of the left and right views of the stereo image can be moved vertically in the image processing pipeline (eg, during rendering of the view for display) to achieve the same effect. Rotational misalignment of the sensor can also be corrected vertically. The pixel set may also be moved across the pixel grid during use to provide the appearance of panning the field of view. In one example, a pixel set for a window of 1920x1080 pixels may be selected from a pixel grid having 2048x1560 pixels. The position of the pixel window or set may be controlled by software/firmware and may be moved during setup and/or use. Accordingly, the resolution of optical image sensors 746 and 748 is specified based on the number of pixels in the length and width of a pixel set or window.

1. Color Sensing with an Example Image Sensor As mentioned above, optical sensing elements 746 and 748 include pixels with different filters that detect light of a particular color. For example, some pixels are covered with filters that primarily transmit red light, some are covered with filters that primarily transmit green light, and some are covered with filters that primarily transmit blue light. In some embodiments, a Bayer pattern is applied to pixel grids 1002 and 1004. However, it should be understood that other embodiments may use different color patterns optimized for particular wavelengths of light. For example, the green filter in each sensing region can be replaced with a broadband filter or a near-infrared filter, thereby broadening the sensing spectrum.

The Bayer pattern is implemented by grouping two rows by two columns of pixels and covering each with a red filter, one with a blue filter, and two with a green filter in a checkerboard pattern. Therefore, the red and blue resolutions are each one-fourth of the total sensing area of interest, while the green resolution is one-half of the total sensing area of interest.

Green color may be assigned to half of the sensing area to cause optical image sensors 746 and 748 to act as luminance sensors and mimic the human visual system. In addition, red and blue mimic the chrominance sensors of the human visual system, but are less important than green sensing. Once the amount of red, green, and blue in a particular region is determined, the amount of red, green, and blue can be determined by averaging the red, green, and blue values, as discussed below in conjunction with the debayer program 1580a of FIG. , other colors within the visible spectrum are identified.

In some embodiments, optical image sensors 746 and 748 may detect color using laminated components rather than filters. For example, the sensing elements may include red, green, and blue sensing components stacked vertically within the pixel area. In some other examples, the prism splits the incident light into its components one or more times (usually At least twice, a three-component color known as a "three-chip" is produced). Other sensor types use different patterns, such as replacing one of the green filters with a broadband filter or a near-infrared filter, thereby extending the detection potential of the digital surgical microscope.

2. Detecting Light Outside the Visible Range Using an Example Image Sensor The example sensing element filters of optical image sensors 746 and 748 are also configured to transmit near-ultraviolet light within the range that the sensing elements can detect. be done. This allows optical image sensors 746 and 748 to detect at least some light outside the visible range. Such sensitivity can reduce image quality in the visible part of the spectrum by "washing out" the image, lowering contrast on many types of screens, and negatively impacting color quality. As a result, filter 740 may block near-infrared wavelengths while allowing visible wavelengths to pass to optical image sensors 746 and 748 using an infrared blocking filter.

However, such near-infrared sensitivity may be desirable. For example, a fluorescent agent such as ICG can be introduced into target site 700. When excited or activated by visible or other wavelengths or light, ICG emits fluorescence in the near-infrared range. As mentioned above, NIR light source 708b provides NIR light and visible light source 708a provides visible light to excite the drug using ICG. The excitation light is further along the red spectrum, which may be transmitted to filter 740 using a near-infrared bandpass or highpass filter. Light from the red spectrum is then detected by optical image sensors 746 and 748. By combining the spectral characteristics of filter 740 with the expected behavior of light source 708 and fluorescent agent, drug and biological structures, such as drug-containing blood, at target site 700 are distinguished from other non-drug-containing structures. Can be done.

Note that in this example, NIR light source 708b has a different dominant wavelength than the near-infrared filter in filter 740. In particular, NIR light source 708b has a dominant wavelength of approximately 780 nanometers ("nm"), around which most of the light output spectrum lies. In contrast, the near-infrared filter of filter 740 transmits light at wavelengths in the range of approximately 810 nm to 910 nm. The light from NIR light source 708b and the light transmitted through filter 740 are both at "near infrared" wavelengths. However, the light wavelengths are separated such that the example stereoscopic visualization camera 300 can stimulate using the light source 708 and detect using the optical image sensor 744 while filtering the stimulating light. This configuration accordingly allows the use of fluorescent agents.

In another embodiment, the agent can be excited in the blue, violet, and near ultraviolet regions and fluoresce light in the red region. An example of such an agent is porphyrin accumulation within malignant gliomas caused by the introduction of 5ALA. In this example, blue light needs to be filtered out while transmitting the rest of the spectrum. Near UV blocking filters are used in this situation. Similar to the "near infrared" case described above, the NUV light source 708c has a different dominant wavelength than the near ultraviolet blocking filter in the filter 740.

H. Example Lens Carrier In Section IV(D) above, at least some of the lenses of front lens set 714, zoom lens assembly 716, and/or lens barrel set 718 are mounted on one or more carriers along a rail. I mentioned that it can be moved. For example, front zoom lens set 724 may include a carrier that axially moves front zoom lenses 726 and 728 together.

11 and 12 illustrate diagrams of an example carrier according to example embodiments of the present disclosure. In FIG. 11, the carrier 724 includes a right front zoom lens 726 and a left front zoom lens 728 within the support structure 1102. Carrier 724 includes a rail holder 1104 configured to movably connect to rail 1106. Force “F” is applied to actuation section 1108 to move carrier 724 along rail 1106. Force "F" may be applied by a lead screw or other linear actuation device. As shown in FIG. 11, force "F" is applied to the offset of carrier 724. The friction between rail ₁₁₀₆ and carrier 724 creates a moment M _y that causes support structure 1102 to move slightly about the Y axis shown in FIG. 11 . This slight movement may cause the front right zoom lens 726 and the front left zoom lens 728 to shift slightly in opposite directions, creating pseudo-parallax, which is a parallax error between views of a stereoscopic image.

FIG. 12 shows another example carrier 724. In this example, force “F” is applied symmetrically to central structure 1202 connected to rail holder 1104 and support structure 1102. Force “F” creates a moment M _x that causes carrier 724 to rotate or move slightly about the X-axis shown _in FIG. 12 . The rotational movement shifts the front right zoom lens 726 and the front left zoom lens 728 by the same degree of movement in the same direction, thereby reducing (or eliminating) the occurrence of pseudo parallax.

Although FIGS. 11 and 12 show lenses 726 and 728 in one carrier, in other embodiments lenses 726 and 728 can be in each carrier. In these examples, each lens is on a separate track or rail. Separate lead screws may be provided for each lens to provide independent axial movement along each optical path.

I. Example Bends In Section IV(D) above, at least some of the lenses of front lens set 714, zoom lens assembly 716, and/or lens barrel set 718 are radially moved, rotated, and/or I mentioned that it can be tilted. Additionally or alternatively, optical image sensors 746 and 748 may move axially and/or tilt with respect to each incident optical path. Axial movement and/or tilt movement may be provided by one or more bends. In some examples, the flexures may be coupled such that a first flexure provides movement in a first direction and a separate flexure provides independent movement in a second direction. In another example, the first flexure provides tilt along the pitch axis and the separate flexure provides tilt along the yaw axis.

FIG. 13 shows a diagram of an example double bend 1300, according to example embodiments of the present disclosure. The flexure 1300 shown in FIG. 13 is for optical image sensor 744 and independently moves right optical image sensor 746 and left optical image sensor 748 along each optical axis for final focus. configured to allow The flexure 1300 includes a support beam 1301 for connecting to the housing 302 of the example stereoscopic visualization camera 300 and providing a rigid base for operation. Flexure 1300 also includes a beam 1302 for each channel (eg, sensors 746 and 748) that is rigid in all directions except the direction of motion 1310. Beam 1302 is connected to a flexible hinge 1303 that allows beam 1302 to move in a direction of motion 1310, in this example a parallelogram translation.

Actuator device 1304 bends beam 1302 a desired distance in a desired direction. Actuator device 1304 includes a push screw 1306 and a pull screw 1308 in each channel that apply opposing forces to beam 1302 to move flexible hinge 1303. Beam 1302 may be moved inward by, for example, turning push screw 1306 to push beam 1302. The flexure 1300 shown in FIG. 13 is configured to axially move the right optical image sensor 746 and the left optical image sensor 748 independently along their respective optical axes.

After the beam 1302 has flexed to the desired position, it engages a locking mechanism to prevent further movement, thereby creating a rigid column. The locking mechanism includes a set screw 1306 and a respective concentric pull screw 1308 that, when tightened, generates a large opposing force that creates a rigid column of beam 1302.

Although optical image sensors 746 and 748 are shown connected to the same flexure 1300, in other examples the sensors may be connected to separate flexures. For example, returning to FIG. 8, right optical image sensor 746 is connected to flexure 750 and left optical image sensor 748 is connected to flexure 752. Through the use of separate flexures 750 and 752, optical image sensors 746 and 748 can be adjusted separately to, for example, align left and right optical views and/or reduce or eliminate pseudo-parallax.

Additionally, although FIG. 13 shows image sensors 746 and 748 connected to flexure 1300, in other examples front lens set 714, zoom lens assembly 716, lens barrel set 718, and/or final optical element The lenses of set 742 may alternatively be connected to alternative or additional bends. In some cases, each left and right lens of front lens set 714, zoom lens assembly 716, lens barrel set 718, and/or final optical element set 742 is connected to a separate flexure 1300 to provide an independent radius. Directional, rotational, and/or tilt adjustments may be provided.

Flexure 1300 may provide motion resolution of less than 1 μm. As a result of very fine motion coordination, images from the left and right optical paths can have registration accuracy of several pixels or even one pixel on a 4K display monitor. Such accuracy can be viewed on each display 512, 514 by overlapping the left and right views and observing both views with both eyes rather than stereoscopically.

In some embodiments, the flexure 1300 may include a flexure as disclosed in U.S. Pat. , which patent is hereby incorporated by reference in its entirety. In yet other embodiments, the lenses of front lens set 714, zoom lens assembly 716, lens barrel set 718, and/or final optical element set 742 may be radially stationary. Alternatively, deflection elements (eg, mirrors) with adjustable deflection directions in the optical paths may be used to steer the right and/or left optical paths to adjust alignment and/or pseudo-parallax. Additionally or alternatively, a tilt/shift lens may be provided in the optical path. For example, the tilt of the optical axis may be controlled using an adjustable wedge lens. In further embodiments, the lenses of front lens set 714, zoom lens assembly 716, lens barrel set 718, and/or final optical element set 742 include dynamic lenses with parameters that can be changed electronically. may be included. For example, the lens may include a Varioptic liquid crystal lens manufactured by Invenios France SAS.

V. Processor of an Example Stereo Visualization Camera The example stereo visualization camera 300 is configured to record image data from the right and left optical paths and output the image data to monitors 512 and/or 514 for display as stereo images. It is composed of FIG. 14 shows a diagram of modules of an example stereoscopic visualization camera 300 that acquires and processes image data, according to example embodiments of the present disclosure. It is to be understood that modules refer to operations, methods, algorithms, routines, and/or steps performed by particular hardware, controllers, processors, drivers, and/or interfaces. In other embodiments, modules may be combined, further divided, and/or deleted. Additionally, one or more of the modules (or portions of modules) may be provided external to stereoscopic visualization camera 300, such as on a remote server, computer, and/or distributed computing environment.

In the illustrated embodiment of FIG. 14, components 408, 702-750, and 1300 in FIGS. 7-13 are collectively referred to as optical element 1402. Optical elements 1402 (particularly optical image sensors 746 and 748) are communicatively coupled to image capture module 1404 and motor and illumination module 1406. Image capture module 1404 is communicatively coupled to information processor module 1408, which may be communicatively coupled to externally located user input device 1410 and one or more display monitors 512 and/or 514.

An example image capture module 1404 is configured to receive image data from optical image sensors 746 and 748. Additionally, image capture module 1404 may define pixel sets 1006 and 1008 within each pixel grid 1002 and 1004. Image capture module 1404 can also specify image storage characteristics such as frame rate and exposure time.

An example motor and illumination module 1406 controls one or more motors (or actuators) to change one or more radial positions, axial positions, and/or tilt positions of optical element 1402. It is composed of For example, a motor or actuator may turn a drive screw to move carrier 724 along track 1106, as shown in FIGS. 11 and 12. The motor or actuator may also turn push screw 1306 and/or pull screw 1308 of flexure 1300 in FIG. 13 to adjust the radial, axial, or tilt position of the lens and/or optical image sensor. can. Motor and lighting module 1406 may also include a drive to control light source 708.

An example information processor module 1408 is configured to process image data for display. For example, information processor module 1408 may provide color correction to the image data, filter defects from the image data, and/or render the image data for stereoscopic display. Information processor module 1408 also performs one or more calibration routines by providing instructions to image capture module 1404 and/or motor and illumination module 1406 to perform specified adjustments to the optical elements. , the stereoscopic visualization camera 300 may also be calibrated. Information processor module 1408 may further determine and provide instructions to image capture module 1404 and/or motor and illumination module 1406 in real time to improve image alignment and/or reduce pseudo-parallax.

An example user input device 1410 may include a computer that provides instructions to change the operation of stereoscopic visualization camera 300. User input device 1410 may also include controls for selecting parameters and/or characteristics of stereoscopic visualization camera 300. In one embodiment, user input device 1410 includes control arm 304 of FIG. User input device 1410 may be hardwired to information processor module 1408. Additionally or alternatively, user input device 1410 is communicatively coupled to information processor module 1408 wirelessly or optically.

Example display monitors 512 and 514 include, for example, a television and/or computer monitor configured to provide a three-dimensional viewing experience. For example, the display monitor may include an LG® 55LW5600 television. Alternatively, display monitors 512 and 514 may include laptop screens, tablet screens, smartphone screens, smart glasses, projectors, holographic displays, etc.

The following sections describe the image capture module 1404, motor and lighting module 1406, and information processor module 1408 in more detail.

A. Example Image Capture Module FIG. 15 shows a diagram of an image capture module 1404 according to an example embodiment of the disclosure. An example image capture module 1404 includes an image sensor controller 1502 that includes a processor 1504, a memory 1506, and a communication interface 1508. Processor 1504, memory 1506, and communication interface 1508 may be communicatively coupled together via image sensor controller bus 1512.

Processor 1504 is programmable with one or more programs 1510 permanently stored in memory 1506. Program 1510 includes machine-readable instructions that, when executed, cause processor 1504 to perform one or more steps, routines, algorithms, etc. In some embodiments, program 1510 may be transmitted to memory 1506 from information processor module 1408 and/or from user input device 1410. In other examples, program 1510 may be sent directly to processor 1504 from information processor module 1408 and/or from user input device 1410.

An example image sensor controller 1502 is communicatively coupled to a right optical image sensor 746 and a left optical image sensor 748 of optical element 1402. Image sensor controller 1502 is configured to provide power to optical image sensors 746 and 748 in addition to timing control data and/or programming data. Additionally, image sensor controller 1502 is configured to receive images and/or diagnostic data from optical image sensors 746 and 748.

Each optical image sensor 746 and 748 includes programmable registers that control certain parameters and/or characteristics. One or more of the registers may specify the location of pixel sets 1006 and 1008 within each pixel grid 1002 and 1004 of FIG. The registers may store starting position values relative to the origin or edge points of pixel grids 1002 and 1004. The register can also specify the width and height of pixel sets 1006 and 1008 to define a rectangular area of interest. Image sensor controller 1502 is configured to read pixel data for pixels within designated pixel sets 1006 and 1008. In some embodiments, the registers of optical image sensors 746 and 748 may facilitate specifying pixel sets of other shapes, such as circles, ellipses, triangles, etc. Additionally or alternatively, registers in optical image sensors 746 and 748 may allow multiple pixel sets to be designated simultaneously to pixel grids 1002 and 1004, respectively.

The photosensitive portions of the pixels of pixel grids 1002 and 1004 are controlled by embedded circuitry that specifies different photosensitive modes. The modes include reset mode, integration mode, and readout mode. During the reset mode, the charge storage components of the pixel are reset to a known voltage level. During integration mode, the pixels are switched to the "on" state. Light reaching an element of the sensing area or pixel causes charge to accumulate in a charge storage component (eg, a capacitor). The amount of charge stored corresponds to the amount of light incident on the sensing element during the integration mode. During the readout mode, the amount of charge is converted to a digital value and read from the optical image sensors 746 and 748 via embedded circuitry and sent to the image sensor controller 1502. To read out every pixel, the charge storage components of each pixel within a given area are sequentially connected to a readout circuit by a switching internal circuit, and the readout circuit performs the conversion of the charge from analog values to digital values. . In some embodiments, pixel analog data is converted to 12-bit digital data. However, it should be understood that the resolution may be higher or lower based on noise tolerance, settling time, frame rate, and data transmission rate. Digital pixel data for each pixel may be stored in a register.

The example processor 1504 of the image sensor controller 1502 of FIG. ). Processor 1504 forms a right image from the pixel data received from right optical image sensor 746. Additionally, processor 1504 forms a left image from the pixel data received from left optical image sensor 748. Alternatively, processor 1504 forms only a portion (eg, one or several rows) of each of the left and right images before transmitting the data downstream. In some embodiments, processor 1504 uses register locations to locate each pixel within the image.

After the right and left images are created, processor 1504 synchronizes the right and left images. Processor 1504 then sends both the right and left images to communication interface 1508 which processes the images into a format for transmission to information processor module 1408 via communication channel 1514. . In some embodiments, communication channel 1514 is compliant with USB 2.0 or 3.0 standards and may include copper wire or fiber optic cable. The communication channel 1514 may be capable of transmitting up to about 60 or more pairs of left and right images (with a stereoscopic resolution of 1920×1080 and a data conversion resolution of 12 bits) per second. Power can be provided from the information processor module 1408 to the image capture module 1404 through the use of copper wire USB.

The following sections further describe features provided by processor 1504 of image sensor controller 1502 that executes particular programs 1510 to acquire and/or process image data from optical image sensors 746 and 748.

1. Example Exposure The example processor 1504 may control or program the amount of time that the optical image sensors 746 and 748 are in the integration mode described above. Integration mode occurs over a period of time called the exposure time. Processor 1504 may set the exposure time by writing values to exposure registers of optical image sensors 746 and 748. Additionally or alternatively, processor 1504 may send instructions to optical image sensors 746 and 748 to signal the start and end of the exposure time. Exposure times can be programmed from a few milliseconds ("ms") to a few seconds. Preferably, the exposure time is approximately the inverse of the frame rate.

In some embodiments, processor 1504 may apply a rolling shutter method to optical image sensors 746 and 748 to read pixel data. Under this method, the exposure time for a given row of pixels in pixel sets 1006 and 1008 begins immediately after the pixels in that row are read out and then reset. After a short period of time, the next row (usually physically closest to the row just set) is read out, reset accordingly, and its exposure time is resumed. Sequential reading of each pixel row continues until the last or bottom row of pixel sets 1006 and 1008 is read and reset. Processor 1504 then returns to the top row of pixel sets 1006 and 1008 and reads the pixel data for the next image.

In another embodiment, processor 1504 applies a global shutter method. Under this method, processor 1504 performs reads and resets similar to the rolling shutter method. However, in this method, the integration is performed on all pixels in pixel sets 1006 and 1008 simultaneously. The global shutter method has the advantage of reducing defects in the image compared to the rolling shutter method because all pixels are exposed at the same time. By way of comparison, in the rolling shutter method there is a slight time delay between the exposure of lines of pixel sets. Small defects may occur during the time between the top and bottom lines where small changes in target region 700 may occur between line exposures, especially between readouts.

2. Example Dynamic Range Example processor 1504 may execute one or more programs 1510 to detect light outside the dynamic range of optical image sensors 746 and 748. Typically, extremely bright light completely fills the charge storage area of the pixel, thereby causing a loss of image information regarding the exact brightness level. Similarly, very low light or no light cannot affect any meaningful charge within the pixel, which also leads to loss of image information. The image created from this pixel data accordingly does not accurately reflect the light intensity at the target site 700.

To detect light outside the dynamic range, processor 1504 may run one of several high dynamic range ("HDR") programs 1510, including, for example, a multi-exposure program, a multi-slope pixel integration program, and a multi-sensor image fusion program. one can be carried out. In one example, the multi-exposure program may utilize HDR features integrated or embedded in optical image sensors 746 and 748. Under this method, pixel sets 1006 and 1008 are in integration mode for a normal exposure time. The lines of pixel sets 1006 and 1008 are read out and stored in memory of optical image sensors 746 and 748 and/or memory 1506 of image sensor controller 1502. After readout is performed by processor 1504, each line in pixel sets 1006 and 1008 is turned on again for a second exposure time that is less than the normal exposure time. Processor 1504 reads each line of pixels after the second exposure time and combines this pixel data with pixel data from the same line's normal exposure time. Processor 1504 applies tone mapping to select (or combine) pixel data from normal length exposure times and pixel data from short length exposure times, and transmits the generated pixel data downstream. mapping to a range compatible with processing and display. When using a multi-exposure program, processor 1504 can expand the dynamic range of optical image sensors 746 and 748 and compress the resulting range of pixel data for display.

Processor 1510 may operate similar programs even in relatively dim light. However, instead of the second exposure time being less than the normal time, the second exposure time is longer than the normal time, thereby providing the pixel with more time to accumulate charge. Processor 1510 may use tone mapping to adjust pixel data readout to compensate for longer exposure times.

3. Example Frame Rate Example processor 1510 may control or specify the frame rate of optical image sensors 746 and 748. In some embodiments, optical image sensors 746 and 748 include onboard timing circuitry and programmable control registers that specify the number of times per second that each pixel in pixel sets 1006 and 1008 cycles through the imaging modes described above. A frame or image is formed each time a set of pixels progresses through the three modes. The frame rate is the number of times per second that the pixels in pixel sets 1006 and 1008 are integrated, read out, and reset.

Processor 1510 may synchronize optical image sensors 746 and 748 so that readouts occur at the appropriate times. In other examples, processor 1510 is asynchronous with optical image sensors 746 and 748. In these other examples, optical image sensors 746 and 748 may store pixel data after local readout to a temporary memory or queue. Pixel data may then be read periodically by processor 1510 for right and left image synchronization.

Processing frames or images in time sequence (eg, creating an image stream) provides the illusion of motion that is conveyed as video. One example processor 1510 is configured to program a frame rate that provides a smooth video appearance to a viewer. A frame rate that is too low will make any motion choppy or uneven. Video quality above the maximum threshold frame rate is not discernible to the observer. An example processor 1510 is configured to generate approximately 20-70 frames per second, preferably 50-60 frames per second for typical surgical visualization.

4. Example Sensor Synchronization The example processor 1504 of FIG. 15 is configured to control the synchronization of optical image sensors 746 and 748. Processor 1504 may, for example, simultaneously provide power to optical image sensors 746 and 748. Processor 1504 may then provide clock signals to both optical image sensors 746 and 748. The clock signal allows optical image sensors 746 and 748 to operate independently, but synchronized and/or simultaneously in a free running mode. Therefore, optical image sensors 746 and 748 record pixel data substantially simultaneously. An example processor 1504 receives pixel data from optical image sensors 746 and 748, constructs at least a portion of the image and/or frame, and configures the image and/or frame (or some of them). Typically, the lag between optical image sensors 746 and 748 is less than 200 microseconds. In other embodiments, processor 1504 may use a synchronization pin to simultaneously activate optical image sensors 746 and 748, for example, after each reset mode.

B. Example Motor and Lighting Module The example stereoscopic visualization camera 300 of FIG. 15 controls one or more motors or actuators that move the lens of optical element 1402 and/or control the illumination output from light source 708. A motor/lighting module 1406 is included. One example motor and lighting module 1406 includes a motor and lighting controller 1520 that includes a processor 1522, a memory 1524, and a communication interface 1526 communicatively coupled together via a communication bus 1528. Memory 1524 stores one or more programs 1530 that are executable on processor 1522 and that perform control, adjustment, and/or calibration of lenses and/or light sources 708 of optical element 1402. In some embodiments, program 1530 may be transmitted to memory 1524 from information processor module 1408 and/or user input device 1410.

Communication interface 1526 is communicatively coupled to communication interface 1508 of image capture module 1404 and communication interface 1532 of information processor module 1408. Communication interface 1526 is configured to receive command messages, timing signals, status messages, etc. from image capture module 1404 and information processor module 1408. For example, the processor 1504 of the image capture module 1404 may send timing signals to the processor 1522 to synchronize the timing of the lighting control and exposure times of the optical image sensors 746 and 748. In another example, information processing module 1408 may send a command message instructing a particular light source 708 to activate and/or move a particular lens of optical element 1402. The commands may be responsive to input received from an operator via user input device 1410, for example. Additionally or alternatively, the commands may be responsive to calibration routines and/or real-time adjustments to reduce or eliminate image misregistration and/or defects, such as pseudo-parallax.

An example motor and illumination module 1406 includes a drive that provides power to control a motor that adjusts the axial and/or radial position of the lens of the optical element 1402 and/or the light output from the light source 708. In particular, the motor and lighting module 1406 includes a NUV light driver 1534 that sends NUV signals to NUV light source 708c, a NIR light driver 1536 that sends NIR signals to NIR light source 708b, and a visible light signal to visible light source 708a. A visible light driving device 1538 is included.

In addition, motor and lighting module 1406 includes a filter motor drive 1540 that sends filter motor signals to filter motor 1542 that control filter 740 of FIGS. 7 and 8. The motor and illumination module 1406 includes a rear zoom lens motor drive 1544 that sends rear zoom lens motor signals to a rear zoom lens motor 1546 and a front zoom lens motor that sends front zoom lens motor signals to a front zoom lens motor 1550 . A drive 1548 and a rear working distance lens motor drive 1552 that sends working distance lens motor signals to a working distance lens motor 1554 . Motor and lighting module 1406 may also include a motor and/or actuator to move and/or tilt deflection element 712.

Rear zoom lens motor 1546 is configured to rotate a drive screw that moves carrier 730 axially along a track or rail. Front zoom lens motor 1550 is configured to rotate a drive screw that moves carrier 724 axially along track 1106 shown in FIGS. 11 and 12. Working distance lens motor 1554 is configured to rotate a drive screw that moves rear working distance lens 702 axially along a track or rail.

Drives 1536, 1538, and 1540 may include any type of lighting drive, transformer, and/or ballast. Drives 1536 , 1538 , and 1540 are configured to output pulse width modulated (“PWM”) signals that control the intensity of light output by light source 708 . In some embodiments, processor 1522 may control the timing of drives 1536, 1538, and 1540 to correspond to the timing of applying a particular filter using filter motor drive 1540.

Example drives 1540, 1544, 1548, and 1552 may include, for example, a stepper motor drive and/or a DC motor drive. Similarly, motors 1542, 1546, 1550, and/or 1554 may include stepper motors, DC motors, or other electrical, magnetic, thermal, hydraulic, or pneumatic actuators. Motors 1542, 1546, 1550, and/or 1554 may include, for example, rotary encoders, slotted optical switches (e.g., photointerrupters), and/or linear encoders for feedback reporting and control, and may include shaft and/or axle may report the angular position of. Alternative embodiments may include voice coil motors, piezoelectric motors, linear motors, and equivalents thereof with suitable drives.

To control drives 1534, 1536, 1538, 1540, 1544, 1548, and 1552, processor 1522 is configured to use a program 1530 that converts command messages into digital and/or analog signals. Processor 1522 transmits digital and/or analog signals to appropriate drives, which output analog power signals, such as PWM signals, corresponding to the received signals. The analog power signal provides power to the appropriate motor or actuator to rotate (or otherwise move) the motor or actuator by the desired amount.

Processor 1522 may receive feedback from drives 1534, 1536, 1538, 1540, 1544, 1548, and 1552, motors 1542, 1546, 1550, and/or 1554, and/or light source 708. The feedback corresponds to the lighting level or the lighting output, for example. For motors, the feedback corresponds to the position and/or amount of movement of the motor (or other actuator). Processor 1522 uses program 1530 to translate the received signal into digital feedback to determine, for example, the radial position, tilt position, and/or axial position of the lens based on the angular position of the corresponding motor or actuator shaft. Identify. Processor 1522 may then send a message with the position information to information processor module 1408 to track the position of the lens of optical element 1402 for display to a user and/or for calibration.

In some embodiments, motor and illumination module 1406 may include additional drives to change the axial position, tilt position, and/or radial position of individual lenses within optical element 1402. For example, motor and illumination module 1406 may include a drive that controls a motor that operates flexures 750 and 752 of optical image sensors 746 and 748 for tilt and/or radial/axial adjustment. In addition, the motor and illumination module 1406 individually connects front lenses 720 and 722, front zoom lenses 726 and 728, rear zoom lenses 732 and 734, lens barrels 736 and 738, and/or final optical elements 745 and 747. It may include a drive for controlling a motor (or actuator) to tilt and/or adjust axially along the x-axis or y-axis and/or axially. Independent adjustment of the lenses and/or sensors allows the motor and lighting controller 1520 to remove image defects and/or align left and right images, for example.

The following sections describe how processor 1552 executes one or more programs 1530 to deflect working distance, zoom, filter position, lens position, and/or light output.

1. Example Working Distance The example processor 1522 of the motor and lighting module 1406 of FIG. 15 is configured to adjust the working distance of the stereoscopic visualization camera 300. The working distance is set by adjusting the distance between the rear working distance lens 704 and the front working distance lens 408. Processor 1522 adjusts the distance by moving rear working distance lens 704 relative to front working distance lens 408. In particular, processor 1522 sends a signal to rear working distance lens motor drive 1552 that activates working distance lens motor 1554 for a predetermined period of time proportional to the amount to move rear working distance lens 704. Working distance lens motor 1554 drives a lead screw through a screw attached to a sliding track that holds rear working distance lens 704. Working distance lens motor 1554 moves lens 704 a desired distance, thereby adjusting the working distance. Working distance lens motor 1554 may provide a feedback signal to processor 1522, which determines whether rear working distance lens 704 has moved the desired amount. If the movement is less than or more than desired, processor 1522 may send instructions to further refine the position of rear working distance lens 704. In some embodiments, information processor module 1408 may determine feedback control of rear working distance lens 704.

To determine the position of rear working distance lens 704, processor 1522 may operate one or more calibration programs 1530. For example, when activated, processor 1522 drives a lead screw to working distance lens motor 1554 to move rear working distance lens 704 along a track or rail until it triggers a limit switch at one end of the range of motion. You can order it to move. Processor 1522 may indicate this stop position as the zero point of the encoder of motor 1554. Knowing the current position of the rear working distance lens 704 and the corresponding encoder value allows the processor 1522 to determine the number of shaft rotations to move the rear working distance lens 704 to the desired position. The shaft rotation speed is sent in an analog signal to the working distance lens motor 1554 (via drive 1552), which moves the lens 704 to the specified position accordingly.

2. Example Zoom The example processor 1522 of FIG. 15 is configured to execute one or more programs 1530 to change the zoom level of the stereoscopic visualization camera 300. As discussed above, zooming (e.g., changing magnification) is accomplished by changing the position of front zoom set 724 and rear zoom set 730 with respect to each other and with respect to front lens set 714 and lens barrel set 718. be done. Similar to the calibration procedure described above for rear working distance lens 704, processor 1522 may calibrate the position of sets 724 and 730 along the track or rail. In particular, processor 1522 sends commands to rear zoom lens motor 1546 and front zoom lens motor 1550 to move sets 724 and 730 (e.g., carriers) along a rail (or rails) to a stop position at a limit switch. do. Processor 1522 receives encoder feedback from motors 1546 and 1550 to identify encoder values associated with set 724 and 730 stop positions. Processor 1522 then determines how much to activate motors 1546 and 1550 to achieve the desired position of sets 724 and 730 along the rail, either by zeroing the encoder value or using the known encoder value at the stop position. You can decide whether to

In addition to calibrating the stop positions, processor 1522 may execute a program 1530 that defines the positions of sets 724 and 730 to achieve the desired zoom level. For example, during a calibration procedure, a known pattern of distance settings and a set of desired zoom values may be stored as a program 1530 (or lookup table). The calibration procedure involves placing a template at the target site 700 and instructing the processor 522 to move the sets 724 and 730 until a particular indicated marker or character is of a particular size in the left and right images or frames. may include. For example, the calibration routine determines the positions of sets 724 and 730 on the corresponding rails when the letter "E" on the template at target site 700 is displayed in the left and right images as having a height of 10 pixels. can be specified.

In some embodiments, information processor module 1408 may perform visual analysis and send instructions to processor 1522 regarding desired movement of sets 724 and 730 to zoom in or out. Additionally, information processor 1408 may send instructions to move the focal plane so that target region 700 at a desired zoom level is in focus. The instructions may include, for example, instructions to move rear working distance lens 704 and/or instructions to move sets 724 and 730 together and/or individually. In some alternative embodiments, processor 1522 may receive calibration parameters for the rail positions of front zoom set 724 and rear zoom set 730 at a particular zoom level from user input device 1410 or another computer.

An example processor 1522 and/or information processor module 1408 may send instructions such that the image remains in focus when the magnification is changed. Processor 1522 may, for example, use program 1530 and/or look-up tables to determine how a particular lens should move along the optical axis to maintain focus on target region 700. The program 1530 and/or the lookup table may specify the magnification level and/or the set point on the rail and the corresponding lens adjustment required to avoid moving the focal plane.

Table 2 below shows an example program 1530 or lookup table that processor 1522 may use to maintain focus while changing magnification. The positions of the front zoom lens set 724 and the rear zoom lens set 730 are normalized based on the length of the rail to the stop position of each set 724 and 730. To decrease magnification, the rear zoom lens set moves toward lens barrel set 718, thereby increasing its position along the rail. The front zoom lens set 724 also moves. However, that movement does not necessarily have to be equal to the movement of rear zoom lens set 730. Instead, movement of front zoom lens set 724 accounts for changing the distance between sets 724 and 730 to maintain focal plane position to maintain focus while changing magnification. For example, to decrease the magnification level from 10X to 9X, processor 1522 commands rear zoom lens set 730 to move along the rail from position 10 to position 11. Additionally, processor 1522 commands front zoom lens set 724 to move along the rail (or the same rail as set 730) from position 5 to position 4. Not only were sets 724 and 730 moved to change magnification, but they were also moved relative to each other to maintain focus.

It should be appreciated that Table 2 provides an example of how sets 724 and 730 may be moved. In other examples, Table 2 may include additional rows to describe more precise magnification and/or locations of sets 724 and 730. Additionally or alternatively, Table 2 may include a row of rear working distance lenses 704. For example, rear working distance lens 704 may be moved as an alternative or addition to front zoom lens set 724 to maintain focus. Additionally, Table 2 may include rows specifying the position of rear working distance lens 704 and sets 724 and 730 that maintain focus while changing working distance.

The values in Table 2 may be determined through calibration and/or may be received from a remote computer or user input device 1410. During calibration, information processor module 1408 may run calibration program 1560 that advances through different magnifications and/or working distances. A processor 1562 in the information processor module 1408 may perform image processing on the image itself or may receive pixel data to achieve a desired magnification using, for example, a template with predetermined shapes and/or characters. be able to judge when Processor 1562 determines whether the received image is in focus. In response to determining that the image is out of focus, processor 1562 sends instructions to processor 1522 to adjust front zoom lens set 724 and/or rear working distance lens set 704. Adjustments may include repeated movements forward and backward along the optical path until the image is determined to be in focus by processor 1562. In determining that an image is in focus, processor 1562 may, for example, look for an image with minimal light ambiguity and/or determine the difference in light values between adjacent pixel regions (the greater the difference, the more An image analysis may be performed that analyzes pixel data for (corresponding to a more focused image). After determining that the image is in focus at the desired working distance and magnification, processor 1562 and/or processor 1522 then determines the position and corresponding magnification level of sets 724 and 730 and/or rear working distance lens 704. can be recorded.

3. Example Filter Position The example processor 1522 of the motor and lighting module 1406 of FIG. 15 is configured to move the filter 740 to the left and right optical paths based on received instructions. In some examples, filter 740 may include a mirror array. In these examples, processor 1522 sends commands to filter motor drive 1540 to operate one or more motors 1542 to change the position of the mirror. In some cases, driver 1540 may send a charge along one or more paths to filter 740 to switch a particular mirror element to an on or off position. In these examples, filter type selection is generally based dually on which mirror to activate.

In other examples, filter 740 may include wheels with different types of filters, such as infrared blocking filters, near-infrared bandpass filters, and near-UV blocking filters. In these examples, the wheel is rotated by filter motor 1542. Processor 1522 determines wheel stop positions corresponding to different filter partitions. Processor 1522 also determines the rotary encoder value corresponding to each stop position.

Processor 1522 may operate calibration program 1530 and/or processor 1562 may operate calibration program 1560 to determine the stop position. For example, processor 1522 may slowly rotate filter wheel 740 and processor 1562 determines when the light received at a pixel changes (using image analysis or reading pixel data from image capture module 1404). . A change in light value at a pixel indicates a change in the filter type being applied to the light path). In some cases, processor 1522 may change which light source 708 is activated, creating further differentiation in pixels when different filter types are applied.

4. Example Light Control and Filters As previously disclosed, processor 1522 may control light source 708 in conjunction with filter 740 to direct desired wavelengths of light to optical image sensors 746 and 748. In some examples, processor 1522 may control or synchronize the timing between activation of one or more of light sources 708 and activation of one or more of filters 740. To synchronize timing, program 1530 may specify a delay time to activate a particular filter. Processor 1522 uses program 1530 to determine when to send a signal to activate filter 740, for example, relative to sending a signal to turn on light source 708. Scheduled timing ensures that the appropriate filter 740 is applied when the specified light source 708 is activated. Such a configuration allows features highlighted by one light source 708 (such as fluorescence) to be displayed on top of or in conjunction with features displayed under a second light source 708, such as white light or ambient light. I can do it.

In some cases, the light source 708 may be switched as fast as the light filter 740 may be changed, thereby allowing images recorded with different lights to be displayed together on top of each other. For example, veins or other anatomical structures that fluoresce (due to administration of a dye or contrast agent) may be displayed on top of the image under ambient illumination. In this example, the veins are highlighted against background anatomical features shown in visible light. In this case, the processor 1562 and/or the graphics processing unit 1564 (e.g., a video card or graphics card) of the information processor module 1408 may process one or more images recorded during the application of a filter by applying a subsequent filter. It can be combined with or superimposed on the image recorded in it.

In some embodiments, processor 1522 may activate multiple light sources 708 simultaneously. Light sources 708 may be activated simultaneously or sequentially to "interleave" different wavelengths of light, allowing the appropriate pixels in optical image sensors 746 and 748 to be used to extract different information. Activating the light sources simultaneously can be useful for dark field illumination. For example, some applications use UV light to stimulate fluorescence at target site 700. However, UV light is perceived by operators as very dim. Accordingly, the processor 1522 periodically activates the visible light source 1538 to allow the surgeon to observe the field of view without overwhelming pixels that are sensitive to UV light but may also detect some visible light. , may add some visible light to the field of view. In another example, alternating the multiple light sources 708 avoids washout of pixels of optical image sensors 746 and 748 that have overlapping sensitivities at the edges of the range in some cases.

5. Light Intensity Control Processor 1522 in the example of FIG. 15 is configured to execute one or more programs 1530 to change the intensity of the illumination level provided by light source 708. It should be appreciated that the depth of field depends on the illumination level at the target area 700. Generally, higher illumination provides greater depth of field. Processor 1522 is configured to ensure that the appropriate amount of illumination is provided for the desired depth of field without washing out or overheating the field of view.

Visible light source 708a is driven by visible light driver 1538 and outputs light in the human visible portion of the spectrum and some light outside that region. The NIR light source 708b is driven by the NIR light driving device 1536, and mainly outputs light of a wavelength called near-infrared rays. The NUV light source 708c is driven by the NUV light driving device 1534, and mainly outputs light having a wavelength in the blue portion of the visible spectrum, called near ultraviolet light. Each optical driver 1534 , 1536 , and 1538 is controlled by commands provided by processor 1522 . Control of each output spectrum of light source 708 is accomplished by a PWM signal, with the control voltage or current being switched between a minimum (eg, off) value and a maximum (eg, on) value. The brightness of the light output from light source 708 is controlled by varying the switching rate and the percentage of time that the voltage or current is at its maximum level per cycle of the PWM signal.

In some examples, processor 1522 controls the output of light source 708 based on the size of the field of view or zoom level. Processor 1522 may execute a program 1530 that specifies a particular light sensitivity setting where light intensity is a function of zoom. Program 1530 may include, for example, a lookup table that correlates zoom levels to light intensity values. Processor 1522 uses program 1530 to select a PWM signal for light source 708 based on the selected magnification level. In some examples, processor 1522 may reduce the light intensity as magnification is increased to maintain the amount of light provided to the field of view per unit area.

C. Example Information Processor Module An example information processor module 1408 within stereoscopic visualization camera 300 of FIG. 15 is configured to analyze and process images/frames received from image capture module 1404 for display. In addition, information processor module 1408 is configured to interface with different devices and translate control instructions into messages to image capture module 1404 and/or motor and lighting module 1406. Information processor module 1408 may also provide an interface for manual calibration and/or manage automatic calibration of optical element 1402.

As shown in FIG. 15, information processor module 1408 is communicatively and/or electrically coupled to image capture module 1404 and motor and lighting module 1406. For example, communication channel 1514 may include communication channels 1566 and 1568 as well as a USB 2.0 or USB 3.0 connection. Accordingly, information processor module 1408 regulates and provides power to modules 1404 and 1406. In some embodiments, information processor module 1408 converts 110 volt alternating current ("AC") power from a wall outlet to 5 volts, 10 volts, 12 volts, and/or 24 volts direct current for modules 1404 and 1406. Convert to a current (“DC”) supply. Additionally or alternatively, information processor module 1408 receives power from a battery within housing 302 of stereoscopic visualization camera 300 and/or from a battery in cart 510.

The example information processor module 1408 includes a communication interface 1532 for bi-directional communication with the image capture module 1404 and the motor and lighting module 1406. Information processor module 1408 also includes a processor 1562 configured to execute one or more programs 1560 to process images/frames received from image capture module 1404. Program 1560 may be stored in memory 1570. In addition, processor 1562 may perform a calibration of optical element 1402 and/or adjust optical element 1402 to align the right and left images and/or remove visual results.

The example information processor module 1408 includes a graphics processing unit 1564 to process images and/or frames for rendered three-dimensional stereoscopic display. FIG. 16 shows a diagram of a graphics processing unit 1564 according to an example embodiment of the disclosure. In operation, processor 1562 receives images and/or frames from image capture module 1404. An unpack routine 1602 converts or otherwise changes the image/frame from a format conducive to transmission over communication channel 1514 to a format conducive to image processing. For example, images and/or frames may be transmitted across communication channel 1514 in multiple messages. An example unpack routine 1602 combines data from multiple messages to reassemble a frame/image. In some embodiments, unpack routine 1602 may place frames and/or images in a queue until requested by graphics processing unit 1564. In other examples, processor 1562 may completely receive and unpack each of the left and right image/frame pairs before transmitting.

An example graphics processing unit 1564 uses one or more programs 1580 (shown in FIG. 15) to prepare images for rendering. Examples of the program 1580 are shown in FIGS. 15 and 16. Program 1580 may be executed by the processor of graphics processing unit 1564. Alternatively, each program 1580 shown in FIG. 16 may be executed by a separate graphics processor, microcontroller, and/or application specific integrated circuit (“ASIC”). For example, debayer program 1580a smooths or averages pixel values over neighboring pixels to compensate for the Bayer pattern applied to pixel grids 1002 and 1004 of left and right optical image sensors 746 and 748 in FIGS. 7 and 8. It is configured as follows. Graphics processing unit 1564 may also include color correction and/or white balance adjustment programs 1580b, 1580c, and 1580d. Graphics processing unit 1564 also includes a renderer program 1580e that prepares color corrected images/frames for display on display monitors 512 and 514. Graphics processing unit 1564 may further interact with and/or include a peripheral input unit interface 1574 that may provide other images and/or graphics for presentation along with the stereoscopic representation of target region 700. combined, fused, or otherwise configured to include. Further details of program 1580 and information processor module 1408 are discussed more generally below.

The example information processor module 1408 may execute one or more programs 1562 to check and improve the latency of the stereoscopic visualization camera 300. Latency refers to the amount of time it takes for an event to occur at target site 700 and for that same event to be displayed on display monitors 512 and 514. Low latency provides the feeling that the stereoscopic visualization camera 300 is an extension of the surgeon's eye, while high latency tends to divert attention from the microsurgical procedure. An example processor 1562 determines how much time elapses between images being read from optical image sensors 746 and 748 before a combined stereo image based on the read images is transmitted for display. can be tracked. Detection of high latency may cause processor 1562 to reduce queue time, increase frame rate, and/or skip some color correction steps.

1. Example User Input The example processor 1562 of the information processor module 1408 of FIG. 15 is configured to convert user input instructions into messages to the motor and lighting module 1406 and/or the image capture module 1402. User input instructions may include requests to change optical aspects of stereoscopic visualization camera 300, including magnification level, working distance, focal plane (e.g., focus) height, light source 708, and/or filter type of filter 740. . User input instructions include requests to perform calibration, including displaying images in focus and/or displaying image alignment and/or displaying ZRP aligned between left and right images. can also be included. User input instructions may further include adjustments to parameters of stereoscopic visualization camera 300 such as frame rate, exposure time, color correction, image resolution, and the like.

User input instructions may be received from user input device 1410, which may include control mechanism 305 of control arm 304 of FIG. 3 and/or a remote control mechanism. User input device 1410 may also include a computer, tablet computer, or the like. In some embodiments, instructions are received via network interface 1572 and/or peripheral input unit interface 1574. In other embodiments, the instructions may be received from a wired connection and/or an RF interface.

An example processor 1562 includes a program 1560 that determines instruction types and how user input is to be processed. In one example, a user may press a button on control 305 to change the magnification level. The operator continues to press the button until the stereo visualization camera 300 reaches the desired magnification level. In these examples, the user input instructions include, for example, information indicating that the magnification level should be increased. At each command received (or each time period in which a signal indicating a command is received), processor 1562 sends a control command to motor and lighting processor 1406 indicating a change in magnification. Processor 1522 determines the amount to move zoom lens sets 724 and 730 from program 1530, for example using Table 2. Processor 1522 accordingly sends signals or messages to rear zoom lens motor drive 1544 and/or front zoom lens motor drive 1548 to cause rear zoom lens motor 1546 and/or front zoom lens motor 1550 to The rear zoom lens set 730 and/or the front zoom lens set 724 are moved by the amount specified by 1562 to achieve the desired magnification level.

It should be appreciated that in the example above, stereoscopic visualization camera 300 provides changes based on user input, but also makes automatic adjustments to maintain focus and/or high image quality. For example, instead of simply changing the magnification level, processor 1522 also determines how zoom lens sets 724 and 730 should be moved to maintain focus, thereby allowing the operator to manually perform this task. Save from what you need to do. Additionally, processor 1562 may adjust and/or align the ZRP in the left and right images when the magnification level is changed in real time. This may be done, for example, by selecting or changing the positions of pixel sets 1006 and 1008 with respect to pixel grids 1002 and 1004 in FIG. 10.

In another example, processor 1562 may receive instructions from user input device 1410 to change the frame rate. Processor 1562 sends the message to processor 1504 of image capture module 1404. Processor 1504 then writes an indication of the new frame rate to the registers of right image sensor 746 and left image sensor 748. Processor 1504 may also update internal registers with the new frame rate to change the pace at which pixels are read out.

In yet another example, processor 1562 may receive an instruction from user input device 1410 to initiate a ZRP calibration routine. In response, processor 1562 may execute a program 1560 that specifies how the calibration should operate. Program 1560 may include, for example, routines to verify image quality, as well as progressions or iterations of magnification levels and/or working distances. The routine may specify that focus should be verified in addition to ZRP at each magnification level. The routine may also specify how zoom lens sets 724 and 730 and/or rear working distance lens 704 should be adjusted to achieve a focused image. The routine may further specify how the ZRP of the right and left images should be centered at the magnification level. Program 1560 may store the positions of pixel sets 1006 and 1008 and the corresponding magnification levels, as well as the position of zoom lens sets 724 and/or 730 and/or rear working distance lens 704, once image quality is verified ( lookup table). Therefore, if the same magnification level is later requested, processor 1562 uses the lookup table to determine the position and image of zoom lens set 724 and/or 730 and/or rear working distance lens 704 to motor and illumination module 1406. Specify the location of pixel sets 1006 and 1008 to acquisition module 1404. In some calibration routines, at least some of the lenses of optical element 1402 may be adjusted and/or tilted in a radial/rotational direction to center the ZRP and/or align the left and right images. I want you to understand.

2. Example Interfaces To facilitate communication between stereoscopic visualization camera 300 and external devices, example information processor module 1408 includes a network interface 1572 and a peripheral input unit interface 1574. An example network interface 1572 communicatively couples a remote device to the information processor module 1408 to store recorded video, work distance, zoom level, focus, calibration, or other information of the stereoscopic visualization camera 300, for example. configured to allow control of the characteristics of the In some embodiments, a remote device may provide values or parameters to calibrate the lookup table, or more generally may provide a program 1530 with calibration parameters. Network interface 1572 may include an Ethernet interface, a local area network interface, and/or a Wi-Fi interface.

An example peripheral input unit interface 1574 is configured to communicatively couple to one or more peripherals 1576 and facilitate integrating peripheral data, such as patient physiological data, with stereoscopic image data. Peripheral input unit interface 1574 may include a Bluetooth interface, a USB interface, an HDMI interface, SDI, and the like. In some embodiments, peripheral input unit interface 1574 may be coupled to network interface 1572.

Peripherals 1576 may include, for example, data or video storage units, patient physiological sensors, medical imaging devices, infusion pumps, dialysis machines, tablet computers, and/or the like. Ambient data may include image data from a dedicated two-dimensional infrared-only camera, diagnostic images from a user's laptop computer, and/or data from systems such as the Alcon Constellation® system and the WaveTec Optiwave Refractive Analysis (ORA®) system. It may include images from an ophthalmic device or patient diagnostic text.

An example peripheral input unit interface 1574 is configured to convert and/or format data from peripheral 1576 into a digital form suitable for use with stereoscopic images. Once in digital form, graphics processing unit 1564 integrates the peripheral data with other system data and/or stereoscopic images/frames. The data is rendered with stereoscopic images for display on display monitors 512 and/or 514.

Processor 1562 may control the integration setup to configure the inclusion of peripheral data into the stereo image. In one example, processor 1562 may cause graphics processing unit 1564 to display configuration panels on display monitors 512 and/or 514. The configuration panel may allow an operator to connect peripheral device 1576 to interface 1574 and processor 1562 and subsequently establish communication with device 1576. Processor 1564 may then read what data is available or allow the operator to select a data directory location using a configuration panel. Peripheral data within the directory location is displayed in the configuration panel. The configuration panel may also provide the operator with the option of overlaying the peripheral data with the stereoscopic image data or displaying it as a separate picture.

Selection of peripheral data (and overlay format) causes processor 1562 to read and transmit the data to graphics processing unit 1564. Graphics processing unit 1564 may be used to display overlay graphics (such as merging preoperative images or graphics with real-time stereoscopic images), "picture-in-picture", and/or for presentation as a subwindow next to or above the main stereoscopic image window. Apply peripheral data to stereoscopic image data.

3. Example Debayer Program The example debayer program 1580a of FIG. 16 is configured to generate images and/or frames having red, green, and blue values for every pixel value. As mentioned above, the pixels of the right optical image sensor 746 and the left optical image sensor 748 have filters that transmit light in the red wavelength range, the blue wavelength range, or the green wavelength range. Therefore, each pixel contains only a portion of the light data. Accordingly, each image and/or frame received at the information processor module 1408 from the image capture module 1404 has pixels that include either red, blue, or green pixel data.

The example debayer program 1580a is configured to average red, blue, and green pixel data of adjacent and/or neighboring pixels to determine more complete color data for each pixel. In one example, a pixel with red data and a pixel with blue data are placed between two pixels with green data. The green pixel data of the two pixels are averaged and assigned to a pixel with red data and a pixel with blue data. In some cases, the averaged green data may be weighted based on the distance of pixels with red data and pixels with blue data from each green pixel. After calibration, pixels that originally had only red or blue data now contain green data. Thus, after debayer program 1580a is executed by graphics processing unit 1564, each pixel includes pixel data for amounts of red light, blue light, and green light. Pixel data of different colors are blended to determine a resulting color in the color spectrum, which may be used by renderer program 1580e and/or display monitors 512 and 514 for display. In some examples, debayer program 1580a may determine a result color and store data or an identifier indicative of the color.

4. Example Color Correction Example color correction programs 1580b, 1580c, and 1580d are configured to adjust pixel color data. Sensor color correction program 1580b is configured to account for or adjust for variability in color sensing of optical image sensors 746 and 748. User color correction program 1580c is configured to adjust pixel color data based on operator perception and feedback. Additionally, display color correction program 1580d is configured to adjust pixel color data based on display monitor type.

To correct colors for sensor variability, example color correction program 1580b specifies a calibration routine executable by graphics processing unit 1564 and/or processor 1562. Sensor calibration was performed by X-Rite, Inc. The method includes placing a calibrated color chart at the target site 700, such as the ColorChecker® Digital SG by ColorChecker® Digital SG. Processor 1562 and/or graphics processing unit 1564 execute program 1580b, which includes sending instructions to image capture module 1404 to record left and right images of the color chart. Pixel data from the left and right images (after processing by debayer program 1580a) may be compared to pixel data associated with a color chart that may be stored in memory 1570 from a peripheral unit 1576 and/or a remote computer via network interface 1572. Processor 1562 and/or graphics processing unit 1564 identify differences between pixel data. The differences are stored in memory 1570 as calibration data or parameters. Sensor color correction program 1580b applies the calibration parameters to subsequent left and right images.

In some examples, the difference is the best fit of color correction data that program 1580b can apply globally to all pixels of optical image sensors 746 and 748 to produce colors as close to the color chart as possible. can be averaged over a region of pixels to find . Additionally or alternatively, program 1580b may process user input instructions received from user input device 1410 to correct colors. The instructions may include local and/or global changes to red, blue, and green pixel data based on operator preferences.

The example sensor color correction program 1580b is also configured to correct white balance. In general, white light should produce red, green, and blue pixels with equal values. However, differences between pixels can arise from the color temperature of the light used during imaging, unique aspects of each pixel's filter and sensing elements, and the spectral filtering parameters of deflection element 712 of FIGS. 7 and 8, for example. . An example sensor color correction program 1580b is configured to specify a calibration routine that corrects for light imbalance.

To perform white balancing, processor 1562 (according to instructions from program 1580b) may display instructions on display monitor 512 and/or 514 for the operator to place a neutral card at target site 700. Processor 1562 may then instruct image capture module 1404 to record one or more images of the neutral card. After processing by the unpack routine 1602 and the debayer program 1580a, the program 1580b locally and/or Determine global white balance calibration weight values. White balance calibration weight values are stored in memory 1570. In operation, graphics processing unit 1564 uses program 1580b to apply white balance calibration parameters to provide white balance.

In some examples, program 1580b determines white balance calibration parameters for right optical image sensor 746 and left optical image sensor 748 individually. In these examples, program 1580b may store separate calibration parameters for the left and right images. Otherwise, the sensor color correction program 1580b determines the weights between the right and left views such that the color pixel data is substantially the same on the right optical image sensor 746 and the left optical image sensor 748. The determined weights may be applied to white balance calibration parameters for continued use during operation of stereoscopic visualization camera 300.

In some embodiments, the sensor color correction program 1580b of FIG. 16 specifies that white balance calibration parameters should be provided as digital gains to the pixels of the right optical image sensor 746 and the left optical image sensor 748. For example, processor 1504 of image capture module 1404 applies digital gain to pixel data read from each pixel. In other embodiments, the white balance calibration parameter should be applied as an analog gain of each pixel's color sensing element.

An example sensor color correction program 1580b may perform white balance and/or color correction when different light sources 708 and/or filter types 740 are activated. As a result, memory 1570 may store different calibration parameters based on which light source 708 is selected. Additionally, sensor color correction program 1580b may perform white balance and/or color correction for different types of ambient light. An operator may specify the characteristics and/or type of the external light source using user input device 1410. This calibration allows the stereoscopic visualization camera 300 to provide color correction and/or white balance in different lighting environments.

The example program 1580b is configured to perform calibration for each optical image sensor 746 and 748 separately. Thus, during operation, program 1580b applies different calibration parameters to the left and right images. However, in some examples, calibration may be performed on only one sensor 746 or 748, and the calibration parameters are used for the other sensor.

An example user color correction program 1580c is configured to request operator-provided feedback regarding image quality parameters such as brightness, contrast, gamma, hue, and/or saturation. Feedback may be received as instructions from user input device 1410. Adjustments made by the user are stored in memory 1570 as user calibration parameters. These parameters are subsequently applied to the left and right optical images by user color correction program 1580c after color correction of optical image lines 746 and 748.

The example display color correction program 1580d of FIG. 16 is configured to correct the color of an image with respect to a display monitor using, for example, a Datacolor® Spyder color checker. Program 1580d, like program 1580b, instructs image capture module 1404 to record an image of the displayed color temperature in target scene 700. Display color correction program 1580d operates routines to adjust pixel data to match the expected display output stored in lookup tables in memory 1570. The adjusted pixel data may store display calibration parameters in memory 1570. In some examples, a camera or other imaging sensor may be connected to peripheral input unit interface 1574, which provides images or other feedback regarding the colors recorded from display monitors 512 and 514; Image or other feedback is used to adjust the pixel data.

5. Example Stereo Image Display The example renderer program 1580e of graphics processing unit 1564 of FIG. 16 is configured to prepare left and right images and/or frames for three-dimensional stereo display. After the left and right image pixel data is color corrected by programs 1580b, 1580c, and 1580d, renderer program 1580e formats the left and right eye data suitable for stereoscopic display and sends them to one of display monitors 512 or 514. , is configured to place the final rendered into the output buffer.

Generally, renderer program 1580e receives a right image and/or frame and a left image and/or frame. Renderer program 1580e combines the right and left images and/or frames into one frame. In some embodiments, program 1580e operates in top-down mode and condenses the left image data to half the height. Next, program 1580e places the condensed left image data in the top half of the combined frame. Similarly, program 1580e condenses the right image data to half the height and places the condensed right image data in the bottom half of the combined frame.

In other embodiments, the renderer program 1580e operates in side-by-side mode, where each left and right image is condensed to half the width, with left image data provided to the left half of the image, while right image data is provided to the right half of the image. are combined into one image so that it is provided. In a further alternative embodiment, renderer program 1580e operates in a line interleaved mode, with every other line in the left and right frames being discarded. The left and right frames are joined together to form a complete stereoscopic image.

An example renderer program 1580e is configured to render the combined left and right images separately to each connected display monitor. For example, if both display monitors 512 and 514 are connected, renderer program 1580e renders a first combined stereoscopic image toward display monitor 512 and a second combined stereoscopic image toward display monitor 514. . Renderer program 1580e formats the first and second combined stereoscopic images to be compatible with the display monitor and/or screen type and/or screen size.

In some embodiments, renderer program 1580e selects an image processing mode based on how the display monitor should display stereoscopic data. Proper interpretation of the stereoscopic image data by the operator's brain requires that the left eye data of the stereoscopic image be communicated to the operator's left eye and the right eye data of the stereoscopic image be communicated to the operator's right eye. Generally, a display monitor provides a first polarization for left-eye data and a second, opposite polarization for right-eye data. Therefore, the combined stereoscopic image must match the polarization of the display monitor.

FIG. 17 illustrates an example display monitor 512 according to example embodiments of the present disclosure. Display monitor 512 may be, for example, an LG® 55LW5600 three-dimensional television with screen 1702. An example display monitor 512 uses a polarizing film on the screen 1702 such that all odd rows 1704 have a first polarization and all even rows 1706 have the opposite polarization. For compatibility with the display monitor 512 shown in FIG. 17, the renderer program 1580e should select a row interleave mode such that left and right image data are in alternating lines. In some cases, renderer program 1580e may request (or otherwise receive) the display characteristics of display monitor 512 before preparing the stereoscopic image.

To view the stereo image displayed on screen 1702, surgeon 504 (recall the surgeon from FIG. 5) wears glasses 1712 that include a left lens 1714 that includes a first polarization that matches the first polarization of row 1704 Installing. Additionally, glasses 1712 include a right lens 1716 that includes a second polarization that matches the second polarization of row 1706. Thus, the left lens 1714 blocks most of the light from the right image data while only transmitting most of the light from the left image data from the left row 1704. In addition, right lens 1716 blocks most of the left image data from the left image data while only transmitting most of the light from the right image data from right row 1706 . The amount of light from the "false" view that reaches each eye is known as "crosstalk" and is generally kept low enough to allow comfortable viewing. Thus, the surgeon 504 views the right image data recorded by the right optical image sensor 746 with his right eye while viewing the left image data recorded by the left optical image sensor 748 with his left eye. The surgeon's brain fuses the two views together to create a three-dimensional distance and/or depth perception. Furthermore, the use of such a display monitor is advantageous for observing the accuracy of the stereoscopic visualization camera 300. If the surgeon or operator does not wear glasses, both left and right views are visible with both eyes. If a flat target is placed in the focal plane, the two images are theoretically aligned. If misalignment is detected, a recalibration procedure can be initiated by processor 1562.

An example renderer program 1580e is configured to render left and right views for circularly polarized light. However, in other embodiments, the renderer program 1580e may provide stereoscopic images that are compatible with linear polarization. Regardless of what type of polarization is used, example processor 1562 may execute program 1560 to verify or check the polarization of the stereo image output by renderer program 1580e. To check polarization, processor 1562 and/or peripheral input unit interface 1574 insert diagnostic data into the left image and/or right image. For example, processor 1562 and/or peripheral input unit interface 1574 may overlay "left" text on the left image and "right" text on the right image. Processor 1562 and/or peripheral input unit interface 1574 may prompt the operator to close one eye at a time while wearing glasses 1712 to ensure that the left view is received by the left eye and the right view is received by the right eye. may display prompts for instructions. The operator may provide confirmation via user input device 1410 indicating whether the polarization is accurate. If the polarization is not accurate, the example renderer program 1580e is configured to reverse the insertion of the left and right images into the combined stereoscopic image.

In yet other embodiments, the example renderer program 1580e is configured to provide sequential projections of frames instead of creating a combined stereoscopic image. Here, the renderer program 1580e renders the left image and/or frame sequentially interleaved with the right image and/or frame. Therefore, the left and right images are alternately presented to the surgeon 504. In these other embodiments, screen 1702 is not polarized. Alternatively, the left and right lenses of glasses 1712 may be electronically or optically synchronized with each portion of the frame sequence, thereby providing the user with corresponding left and right views to discern depth.

In some examples, the renderer program 1580e may provide some of the right and left images for display on separate display monitors or in separate windows on one display monitor. Such a configuration may be particularly advantageous if the lenses in the left and right optical paths of optical element 1402 are independently adjustable. In one example, the right optical path may be set to a first magnification level, while the left optical path is set to a second magnification level. Accordingly, example renderer program 1580e may display the image stream from the left view on display monitor 512 and the image stream from the right view on display monitor 514. In some cases, the left view may be displayed in a first window of display monitor 512, while the right view is displayed in a second window of the same display monitor 512 (eg, picture-in-picture). Therefore, although not stereoscopic, the simultaneous display of left and right images provides useful information to the surgeon.

In another example, light source 708 and filter 740 may be quickly switched to produce alternating images with visible light and fluorescence. An example renderer program 1580e may combine the left and right views under different light sources to provide a stereoscopic display to highlight veins with dye while displaying the background in visible light, for example.

In yet another example, digital zoom may be applied to the right and/or left optical image sensor 746 or 748. Digital zoom generally affects the perceived resolution of the image and depends on factors such as display resolution and viewer preference. For example, processor 1504 of image capture module 1404 may apply digital zoom by creating synchronized interspersed interpolation pixels between digital zoom pixels. Processor 1504 may operate a program 1510 that adjusts the selection and interpolation pixels of optical image sensors 746 and 748. Processor 1504 transmits the right and left images with digital zoom applied to information processor module 1408 for subsequent rendering and display.

In some embodiments, processor 1504 should record digitally zoomed images between images without digitally zoomed to provide a picture-in-picture (or separate window) display of the digitally zoomed region of interest of target region 700. instructions are received from processor 1562. Processor 1504 accordingly applies digital zoom to every other readout from pixel grids 1002 and 1004. This allows the renderer program 1580e to simultaneously display the highest resolution stereoscopic image in addition to the digitally zoomed stereoscopic image. Alternatively, the image to be digitally zoomed is copied from the current image, scaled and placed in the appropriate position superimposed on top of the current image during the rendering phase. This alternative configuration avoids the "alternating" recording requirement.

6. Example Calibration The example information processor module 1408 of FIGS. 14-16 may be configured to execute one or more calibration programs 1560 to calibrate, for example, working distance and/or magnification. For example, processor 1562 performs a calibration step to provide instructions for mapping the working distance (measured in millimeters) from main objective lens assembly 702 to target site 700 to a known motor position of working distance lens motor 1554. may be sent to the motor and lighting module 1406. Processor 1562 performs calibration by sequentially moving the object plane in discrete steps along the optical axis and refocusing the left and right images while recording encoder counts and working distances. In some examples, the working distance may be measured by an external device that transmits the measured working distance value to the processor 1562 via the peripheral input unit interface 1574 and/or the interface to the user input device 1410. do. Processor 1562 may store the position of rear working distance lens 704 (based on the position of working distance lens motor 1554) and the corresponding working distance.

Example processor 1562 may also execute program 1560 to perform magnification calibration. Processor 1562 may select a magnification level and set optical element 1402 using motor and illumination module 1406. Processor 1562 may record the position of optical element 1402 or the corresponding motor position for each magnification level. The magnification level may be determined by measuring the height within the image of an object of known size. For example, processor 1562 may measure the object as having a height of 10 pixels and use a lookup table to determine that 10 pixels height corresponds to a 5X magnification.

In order to match the stereo perspective of two different imaging modalities, it is often desirable to model them as if they were simple pinhole cameras. The perspective of a 3D computer model, such as an MRI brain tumor, can be viewed from user-adjustable directions and distances (eg, as if the images were recorded by a synthetic stereo camera). Adjustability can be used to match the perspective of the live surgical image, so the perspective of the live surgical image must be known. An example processor 1562 may calibrate one or more of these pinhole camera model parameters, such as, for example, the center of projection (“COP”) of right optical image sensor 746 and left optical image sensor 748. To identify the projection center, processor 1562 determines the focus distance from the projection center to the object plane. First, processor 1562 sets optical element 1402 to a magnification level. Processor 1562 then records measurements of the height of the image at three different distances along the optical axis, including the object plane, a distance d less than the object plane distance, and a distance d greater than the object plane distance. Processor 1562 uses algebraic formulas for similar triangles at the two most extreme positions to determine the focus distance to the projection center. Processor 1562 may determine focus distances at other magnifications using the same method or by determining the ratio between the magnifications used for calibration. The processor may use the projection center to center an image of a desired fused object, such as an MRI tumor model, on the live stereoscopic surgical image. Additionally or alternatively, existing camera calibration procedures, such as OpenCV calibrateCamera, may be used to find the above parameters and additional camera information, such as a distortion model of optical element 1402.

The example processor 1562 may further calibrate the left and right optical axes. Processor 1562 determines the interpupillary distance between the left and right optical axes for calibration. To determine interpupillary distance, example processor 1562 records left and right images, and pixel sets 1006 and 1008 are centered on pixel grids 1002 and 1004. Processor 1562 determines the location of the ZRP (and/or distance to the displaced object) of the left and right images, which is indicative of image misregistration and degree of parallax. Additionally, processor 1562 scales the disparity and/or distance based on the magnification level. Processor 1562 then determines the interpupillary distance using triangulation calculations, taking into account the degree of parallax and/or the scaled distance to objects in the display. Processor 1562 then relates the interpupillary distance to the optical axis at the specified magnification level as a calibration point.

VI. Image Registration and Pseudo-Parallax Adjustment Embodiments Similar to human vision, stereoscopic images include right and left views that converge at a point of interest. The right and left views are recorded at slightly different angles from the point of interest, thereby creating a parallax between the two views. Items in the scene before or after the point of interest exhibit parallax such that the distance or depth of the item from the viewer can be inferred. The accuracy of the perceived distance depends, for example, on the clarity of the viewer's vision. Most humans exhibit some level of imperfection in their visual acuity, resulting in some inaccuracy between right and left views. However, it is still possible to achieve three-dimensionality, and the brain fuses the views with some level of precision.

If the left and right images are recorded by a camera rather than viewed by a human, the parallax between the combined images on the display screen creates stereopsis, providing the appearance of a three-dimensional stereo image on a two-dimensional display. . Parallax errors may affect the quality of three-dimensional stereoscopic images. The inaccuracy of the observed parallax compared to the theoretically preferred parallax is known as spurious parallax. Unlike humans, cameras do not have brains that automatically compensate for inaccuracies.

If pseudo-parallax becomes large, three-dimensional stereoscopic images can become unviewable to the point of inducing dizziness, headaches, and nausea. There are many factors that can affect parallax in a microscope and/or camera. For example, the optical channels of the left and right views may not be exactly equal. The optical channels may have mismatched focus, magnification, and/or misalignment of points of interest. These problems may have varying severity at different magnifications and/or working distances, thereby reducing efforts to correct through calibration.

Known surgical microscopes, such as surgical microscope 200 of FIG. 2, are configured to provide a suitable view through eyepiece 206. In many cases, the image quality of the optical elements of known surgical microscopes is not sufficient for stereoscopic cameras. The reason for this is that surgical microscope manufacturers assume that the primary viewing is through the eyepiece. Any camera accessory (such as camera 212) is either monoscopic and not subject to pseudo-parallax, or stereoscopic at a lower image resolution where pseudo-parallax is not as apparent.

International standards such as ISO10936-1:2000, Optics and optical instruments-Operation microscopes-Part 1: Requirements and test methods specify specifications for the image quality of surgical microscopes. Created to provide limits. Specification limits are generally set for viewing through the eyepiece of a surgical microscope and do not take into account three-dimensional stereoscopic viewing. For example, regarding pseudo-parallax, ISO 10936-1:2000 specifies that the vertical axis difference between the left and right views should be less than 15 minutes. Small angular deviations of axis are often quantified in minutes corresponding to 1/60th of a degree or seconds corresponding to 1/60th of a minute. The 15 minute specification limit corresponds to a 3% difference between the left and right views for a typical surgical microscope with a working distance of 250 mm and a field of view of 35 mm (with a viewing angle of 8°).

A 3% difference is acceptable for eyepiece viewing where the surgeon's brain can resolve small degrees of error. However, this 3% difference creates a noticeable difference between the left and right views when viewed stereoscopically on a display monitor. For example, if the left and right views are viewed together, a 3% difference produces an image that appears disconnected and difficult to view over long periods of time.

Another problem is that known surgical microscopes may meet the 15 minute specification limit at only one or a few magnification levels, and/or only individual optical elements may meet certain specification limits. For example, individual lenses are manufactured to meet specific criteria. However, when individual optical elements are combined in the optical path, small deviations from specification can be amplified rather than canceled out. This may be particularly noticeable when more than five optical elements are used in the optical path, including a common main objective. In addition, it is very difficult to perfectly match optical elements on parallel channels. At most, during manufacturing, the optical elements of a surgical microscope are calibrated to meet the 15 minute specification limit at only one or a few specific magnification levels. Therefore, even though surgical microscopes are supposed to meet ISO 10936-1:2000 specifications, the errors between calibration points can be large.

Additionally, the ISO 10936-1:2000 specification allows for greater tolerances when additional components are added. For example, adding a second eyepiece (eg, eyepiece 208) increases the pseudo-parallax by two minutes. Again, the error may be acceptable when viewed through the eyepieces 206 and 208, but the image misregistration becomes more noticeable when viewed stereoscopically through the camera.

In comparison to known surgical microscopes, the example stereoscopic visualization camera 300 disclosed herein is configured to automatically adjust at least some of the optical elements 1402 to reduce or eliminate pseudo-parallax. be done. Incorporating optical elements into the stereoscopic visualization camera 300 allows fine adjustments to be made automatically (sometimes in real time) toward a three-dimensional stereoscopic display. In some embodiments, an example stereoscopic visualization camera 300 may provide an accuracy of 20 seconds to 40 seconds, which is a 97% reduction in optical error compared to the 15 minute accuracy of known surgical microscopes. Close to.

The improved accuracy allows the example stereoscopic visualization camera 300 to provide features that are not possible with known stereoscopic microscopes. For example, many new microsurgical procedures rely on accurate measurements at the in-vivo surgical site for optimal sizing, positioning, matching, orientation, and diagnosis. This includes determining the size of the blood vessels, the angle of placement of the toric intraocular lens ("IOL"), matching the vasculature from the pre-operative image to the live view, determining the depth of the tumor below the artery, etc. Accordingly, the example stereoscopic visualization camera 300 uses, for example, graphical overlays or image analysis to determine the size of anatomical structures and allow precise measurements to be made.

Known surgical microscopes require the surgeon to place an object of known size (such as a micro-rule) within the field of view. The surgeon compares the size of the object to surrounding anatomical structures to determine its approximate size. However, this procedure is relatively slow because the surgeon must place the object in the appropriate position and then remove the object after measurements are taken. In addition, the measurements provide only an approximation since the sizes are based on the surgeon's subjective comparisons and measurements. Some known stereoscopic cameras provide a graphical overlay to specify size. However, if pseudo-parallax exists between the left and right views, the accuracy of these overlays decreases.

A. ZRP as a cause of pseudo parallax
ZRP inaccuracy provides a large source of error between the left and right images resulting in pseudo parallax. A ZRP or zoom repeat point refers to a point in the field of view that remains the same as the magnification level changes. Figures 18 and 19 show examples of ZRP in left and right fields of view at different magnification levels. In particular, FIG. 18 shows the left field of view 1800 at a low magnification level and the left field of view 1850 at a high magnification level. Additionally, FIG. 19 shows the right field of view 1900 at a low magnification level and the right field of view 1950 at a high magnification level.

Note that FIGS. 18 and 19 show crosshairs 1802 and 1902 that provide exemplary reference points in this disclosure. The crosshairs 1802 include a first crosshair 1802a positioned along the y direction or y-axis and a second crosshair 1802b positioned along the x direction or x-axis. Additionally, the crosshairs 1902 include a first crosshair 1902a positioned along the y direction or y-axis and a second crosshair 1902b positioned along the x direction or x-axis. In an actual implementation, the example stereoscopic visualization camera 300 typically by default does not include or add crosshairs to the optical path unless requested by an operator.

Ideally, the ZRP should be located at the center or origin. For example, the ZRP should be centered on crosshairs 1802 and 1902. However, inaccuracies in optical elements 1402 and/or slight misalignments between optical elements 1402 cause the ZRP to be positioned away from the center of crosshairs 1802 and 1902. The degree of pseudo-parallax corresponds to the degree to which each ZRP of the left and right views is arranged apart from the center, in addition to the ZRP that is misaligned between the left and right views. Furthermore, inaccuracies in optical element 1402 may cause the ZRP to vary slightly with magnification changes, thereby further increasing the degree of pseudo-parallax.

FIG. 18 shows three crescent-shaped objects 1804, 1806, and 1808 within fields of view 1800 and 1850 of target site 700 of FIG. It should be appreciated that fields of view 1800 and 1850 are linear fields of view with respect to optical image sensors 746 and 748. Objects 1804, 1806, and 1808 were placed within the field of view 1800 to illustrate how pseudo-parallax results from misalignment of the left and right images. Object 1804 is positioned along crosshair 1802a and above crosshair 1802b. Object 1806 is positioned along crosshair 1802b to the left of crosshair 1802a. Object 1808 is positioned slightly below crosshair 1802b and to the right of crosshair 1802a. ZRP 1810 in left field of view 1800 is positioned in a notch in object 1808.

Left field of view 1800 is changed to left field of view 1850 by increasing the magnification level (eg, zooming) using zoom lens assembly 716 of example stereoscopic visualization camera 300. As the magnification is increased, objects 1804, 1806, and 1808 appear to enlarge or grow, as shown in field of view 1850. In the illustrated example, field of view 1850 is approximately 3X the magnification level of field of view 1800.

Compared to low magnification field 1800, objects 1804, 1806, and 1808 in high magnification field 1850 have approximately 3X increased size while moving 3X away from each other with respect to ZRP 1810. Additionally, the positions of objects 1804, 1806, and 1808 have moved relative to crosshair 1802. Object 1804 is now shifted to the left of crosshair 1802a and slightly away from crosshair 1802b. Additionally, object 1806 is now shifted further to the left of crosshair 1802a and slightly above crosshair 1802b. Generally, object 1808 is placed in the same (or approximately the same) position with respect to crosshair 1802 and ZRP 1810 is placed in exactly the same (or approximately the same) position with respect to crosshair 1802 and object 1806. In other words, as the magnification increases, objects 1804, 1806, and 1808 (and anything else within field of view 1850) appear to move outward and away from ZRP 1810.

Objects 1804, 1806, and 1808 are shown in right fields of view 1900 and 1950 shown in FIG. However, the location of the ZRP is different. In particular, in right fields of view 1900 and 1950, ZRP 1910 is positioned above crosshair 1902b and to the left of crosshair 1902a. Therefore, ZRP 1910 is placed at a different position than ZRP 1810 in left fields of view 1800 and 1850. In the illustrated example, it is assumed that the left and right optical paths are preferably aligned at a first magnification level. Thus, objects 1804, 1806, and 1808 shown in right field of view 1900 in the same position as the same objects 1804, 1806, and 1808 in left field of view 1800. Since the left and right views are aligned, there is no pseudo-parallax.

However, in high magnification field of view 1950, objects 1804, 1806, and 1808 magnify and move away from ZRP 1910. Given the position of ZRP 1910, object 1804 moves or shifts to the right and object 1806 moves or shifts downward. Additionally, object 1808 moves downward and to the right compared to its position within field of view 1900.

FIG. 20 shows a pixel diagram comparing the high magnification left field 1850 to the high magnification right field. Grid 2000 represents object 1804 on pixel grid 1004 of left optical image sensor 748 with the positions of objects 1804(R), 1806(R), and 1808(R) on pixel grid 1002 of left optical image sensor 746 superimposed. (L), 1806(L), and 1808(L). FIG. 20 clearly shows that objects 1804, 1806, and 1808 are in different positions in left field of view 1850 and right field of view 1950. For example, object 1804(R) is placed to the right of crosshair 1902a and above crosshair 1902b, while the same object 1804(L) is placed to the left of crosshair 1802a and further above crosshair 1802b. .

The positional difference of objects 1804, 1806, and 1808 corresponds to pseudo-parallax, which is caused by a defect in the optical alignment of optical element 1402 that produces ZRPs 1810 and 1910 at different positions. Assuming no distortion or other imaging errors, the pseudo-parallax shown in FIG. 20 is generally the same for all points in the image. When viewed through the eyepiece of a surgical microscope (such as microscope 200 in FIG. 2), the difference in position of objects 1804, 1806, and 1808 may not be noticeable. However, when viewed in stereo on display monitors 512 and 514, the differences are readily apparent and can cause headaches, nausea, and/or dizziness.

FIG. 21 shows a diagram showing pseudo parallax regarding left and right ZRP. The illustration includes a pixel grid 2100 that includes an overlay of left pixel grid 1002 and left pixel grid 1004 of FIG. In this illustrated example, the left ZRP 2102 of the left optical path is positioned at +4 along the x-axis and 0 along the y-axis. Additionally, the right ZRP 2104 of the right optical path is positioned at −1 along the x-axis and 0 along the y-axis. The origin 2106 is shown at the intersection of the x and y axes.

In this example, object 2108 is aligned with respect to the left and right images at a first low magnification. As the magnification increased by 3X, object 2108 increased in size and moved away from ZRPs 2102 and 2104. Contour object 2110 shows the theoretical position of object 2108 at a second higher magnification based on ZRPs 2102 and 2104 being aligned to origin 2106. In particular, the cutout of object 2108 at the first magnification level is at position +2 along the x-axis. When using 3X magnification, the cutout is moved 3X along the x-axis such that the cutout is positioned at +6 along the x-axis at the higher magnification level. Additionally, because ZRPs 2102 and 2104 are theoretically aligned to origin 2106, object 2110 is aligned between the left and right views (shown in FIG. 21 as one object given the overlay).

However, in this example, misalignment of left ZRP 2102 and right ZRP 2104 causes object 2110 to be misaligned between the left and right views at higher magnification. For the right optical path, the right ZRP 2104 is positioned −1 along the x-axis so that it is 3 pixels away from the notch of the object 2108 at low magnification. When magnified by 3X, this difference becomes 9 pixels, shown as object 2110(R). Similarly, left ZRP 2102 is located at +4 pixels along the x-axis. At 3X magnification, object 2108 moves from 2 pixels apart to 6 pixels apart, which is shown as object 2110 (L) at -2 along the x-axis.

The positional difference between object 2110(L) and object 2110(R) corresponds to pseudo-parallax between the left and right views at higher magnification. When the left and right views are combined to generate the displayed stereoscopic image, the position of object 2110 is misaligned with each row when renderer program 1850e uses row interleaved mode. Misalignment is detrimental to the production of stereoscopic vision and can produce images that appear blurry or confusing to the operator.

B. Other causes of pseudo parallax Although ZRP misalignment between the left and right optical paths is a major cause of pseudo parallax, other error causes also exist. For example, pseudo-parallax may result from unequal magnification changes between the left and right optical paths. Magnification differences between parallel optical paths may result from slight variations in the optical properties or properties of the lenses of optical element 1402. Furthermore, slight differences may result from positioning if left and right front zoom lenses 726 and 728, respectively, and left and right rear zoom lenses 736 and 738, respectively, of FIGS. 7 and 8 are independently controlled. .

Referring again to FIGS. 18 and 19, the magnification change difference produces objects of different sizes and different spacing between objects in the left and right optical paths. For example, if the left optical path has a higher magnification change, objects 1804, 1806, and 1808 will appear larger compared to objects 1804, 1806, and 1808 in the right field of view 1950 of FIG. do. Even if ZRPs 1810 and 1910 are aligned, the difference in position of objects 1804, 1806, and 1808 creates pseudo-parallax.

Another source of pseudo-parallax arises from unequal focus of the left and right optical paths. In general, any focus difference between the left and right views can create a perceived reduction in image quality and create potential confusion as to whether the left or right view should predominate. obtain. If the focus difference is significant, it can create an out-of-focus (“OOF”) situation. The OOF situation is particularly noticeable in stereoscopic images where left and right views are shown in the same image. Additionally, OOF situations are not easily correctable because refocusing an out-of-focus optical path typically brings other optical paths out of focus. Generally, it is necessary to identify the point at which both optical paths are focused, which may involve changing the position of the left and right lenses along the optical path and/or adjusting the working distance from the target site 700.

FIG. 22 shows a diagram illustrating how an OOF situation arises. The diagram relates perceived resolution (eg, focus) to lens position relative to optimal resolution section 2202. In this example, left rear zoom lens 734 is at position L1, while right rear zoom lens 732 is at position R1. At positions L1 and R1, the rear zoom lenses 732 and 734 are within the optimal resolution 2202 such that the left and right optical paths have matched focus levels. However, there is a difference in the positions of L1 and R1, which correspond to the distance ΔP. Later, the working distance 706 is changed so that one point is out of focus. In this example, both rear zoom lenses 732 and 734 move the same distance relative to positions L2 and R2 so that the distance ΔP remains the same. However, due to the position change, the resolution changes significantly such that the left rear zoom lens 734 has higher resolution (eg, better focus) than the right rear zoom lens 732. The resolution ΔR corresponds to an OOF situation, which causes pseudo-parallax from focus misalignment between the left and right optical paths.

Yet another source of pseudo-parallax may arise from an imaging object that is moving at target region 700. Pseudo-parallax results from small synchronization errors between the exposures of the right optical image sensor 746 and the left optical image sensor 748. If the left and right views are not recorded simultaneously, the object will appear displaced or misaligned between the two views. The combined stereo image shows the same object in two different positions in left and right views.

Furthermore, another source of pseudo-parallax involves ZRP points that move during magnification. In the example described above in Section IV(A), it is assumed that the ZRPs of the left and right views do not move in the x or y direction. However, the ZRP may shift during magnification if the zoom lenses 726, 728, 732, and/or 734 do not move strictly parallel to the optical path or axis (eg, in the z direction). As described above with reference to FIG. 11, carrier 724 may shift or rotate slightly when a force is applied to actuation section 1108. This rotation may cause the left and right ZRPs to move slightly if the magnification level is changed.

In one example, during a magnification change, carrier 730 moves in one direction, while carrier 724 moves in the same direction for part of the magnification change and in the opposite direction for the remainder of the magnification change for focus adjustment. If the axis of movement of the carrier 724 is tilted or rotated slightly from the optical axis, the ZRP of the left optical path and/or the right optical path will shift in one direction over the first portion and subsequently over the second portion of the magnification change. Shift in the opposite direction. Additionally, because the forces are applied unequally, the front right zoom lens 726 and the front left zoom lens 728 may experience varying degrees of ZRP shift between the left and right optical paths. Together, the ZRP position changes cause optical path misalignment, thereby creating pseudo-parallax.

C. Reducing pseudo-parallax facilitates the incorporation of digital graphics and images with stereoscopic views As surgical microscopes become more digital, designers are increasing the ability to incorporate graphics, images, and/or other digital effects into live view. Adds features that can be superimposed on images. For example, guidance overlays, fusion of stereoscopic magnetic resonance imaging ("MRI") images, and/or external data may be combined with images recorded by the camera and even images displayed within the eyepiece itself. Pseudo-parallax reduces the accuracy of overlay with the underlying stereo image. Surgeons typically require that tumors visualized, for example, via MRI, often in three dimensions, be positioned as precisely as possible within a fused live surgical stereoscopic view. Otherwise, preoperative tumor images provide little information to the surgeon, thereby impairing surgical performance.

For example, the surgical guide may be aligned with the right view image while being misaligned with the left view. A misaligned surgical guide between the two views is readily apparent to the operator. In another example, the surgical guide may be registered with the left and right views separately in the information processor module 1408 before the graphics processing unit 1564 creates the combined stereoscopic image. However, misalignment between the left and right views creates misalignment between the guides, thereby reducing the effectiveness of the guide and creating confusion and delays during microsurgical procedures.

No. 9,552,660, entitled "IMAGING SYSTEM AND METHODS DISPLAYING A FUSED MULTIDIMENSIONAL RECONSTRUCTED IMAGE," which is incorporated herein by reference, describes how is , it is disclosed how the images can be visually fused with stereoscopic images. FIGS. 23 and 24 show diagrams illustrating how pseudo-parallax can cause digital graphics and/or images to lose accuracy when fused into track images. FIG. 24 shows a front view of a patient's eye 2402, and FIG. 23 shows a cross-sectional view of the eye along plane AA of FIG. In FIG. 23, information processor module 1408 is instructed to determine a caudal distance d from focal plane 2302 to an object of interest 2304 on, for example, the posterior capsule of eye 2402. Information processor module 1408 operates program 1560 that specifies, for example, that distance d is determined by triangulating image data from the left and right views of eye 2402. View 2306 is shown from the perspective of left optical image sensor 748 and view 2308 is shown from the perspective of right optical image sensor 746. Left view 2306 and right view 2308 are assumed to coincide with the anterior center 2310 of eye 2402. Additionally, left view 2306 and right view 2308 are two-dimensional views of object 2304 projected onto focal plane 2302 as a theoretical right projection 2312 and a theoretical left projection 2314. In this example, processor 1562 calculates the distance d to object of interest 2304 by calculating the intersection of the extrapolation of theoretical right projection 2312 and the extrapolation of theoretical left projection 2314 using a triangulation routine. Identify.

However, in this example, there is a pseudo-parallax that causes the actual left projection 2316 to be placed a distance P to the left of the theoretical left projection 2314, as shown in FIGS. 23 and 24. Processor 1562 uses actual left projection 2316 and right projection 2312 and uses a triangulation routine to determine the distance to intersection 2320 of the extrapolation of right projection 2312 and the extrapolation of actual left projection 2316. The distance to intersection 2320 is equal to distance d plus error distance e. Accordingly, pseudo-parallax results in incorrect distance calculations using data taken from stereoscopic images. As shown in FIGS. 23 and 24, even a small amount of pseudo parallax can produce a large error. In the context of fused images, incorrect distances lead to inaccurate placement of the three-dimensional tumor visualization that is fused with the stereo image. Inaccurate placement can delay the surgery, interfere with the surgeon's work, or cause the entire visualization system to be ignored. To make matters worse, surgeons may rely on inaccurate placement of tumor images and make mistakes during microsurgical procedures.

D. An example stereoscopic visualization camera reduces or eliminates pseudo-parallax The example stereoscopic visualization camera 300 of FIGS. configured to reduce or eliminate In some examples, stereoscopic visualization camera 300 reduces pseudo-parallax by aligning the ZRPs of the left and right optical paths to the respective centers of pixel sets 1006 and 1008 of left optical image sensor 746 and right optical image sensor 748. Or lose it. Additionally or alternatively, stereoscopic visualization camera 300 may align the optical paths of the left and right images. It should be appreciated that stereoscopic visualization camera 300 may perform operations to reduce pseudo-parallax during calibration. Additionally, stereoscopic visualization camera 300 may reduce detected pseudo-parallax in real time during use.

25 and 26 depict a flowchart illustrating an example procedure 2500 for reducing or eliminating pseudo-parallax, according to example embodiments of the present disclosure. Although procedure 2500 is described with reference to the flowcharts shown in FIGS. 25 and 26, it is understood that many other methods of performing the steps associated with procedure 2500 may be used. For example, the order of many of the blocks may be changed, certain blocks may be combined with other blocks, and many of the blocks described are optional. Additionally, the operations described in step 2500 may be performed within a plurality of devices including, for example, optical element 1402, image capture module 1404, motor and lighting module 1406, and/or information processor module 1408 of example stereoscopic visualization camera 300. It can be executed with For example, procedure 2500 may be performed by one of the programs 1560 of information processor module 1408.

The example methodology 2500 begins when stereoscopic visualization camera 300 receives a command to align left and right optical paths (block 2502). Instructions may be received from user input device 1410 in response to an operator requesting that stereoscopic visualization camera 300 perform a calibration routine. In other examples, the instructions may be received from the information processor module 1408 after the left and right images are determined to be misaligned. The information processor module 1408 determines whether the images are aligned by running a program 1560 that overlays the left and right images and identifies differences in pixel values where large differences over large areas of pixels indicate misaligned images. It can be concluded that there is no such thing. In some examples, program 1560 may compare pixel data of left and right images without performing an overlay function, for example, left pixel data may be subtracted from right pixel data to determine the severity of misalignment. do.

After receiving the instruction to reduce pseudo-parallax, the example stereoscopic visualization camera 300 finds a ZRP in one of the left optical path or the right optical path. For purposes of illustration, procedure 2500 includes first identifying the ZRP of the left optical path. However, in other embodiments, procedure 2500 may first identify the ZRP of the right optical path. To identify the left ZRP, stereoscopic visualization camera 300 moves at least one zoom lens (e.g., left front zoom lens 728 and/or left rear zoom lens 734) to a first position along the z-axis of the left optical path. Move to magnification level (block 2504). If front zoom lenses 726 and 728 are connected to the same carrier 724 and rear zoom lenses 732 and 734 are connected to the same carrier 730, movement of the left lens also moves the right lens. However, during this section of procedure 2500, movement of only the left lens is considered.

At the first magnification level, the stereoscopic visualization camera 300 moves the left zoom lens along the z direction (block 2506). The movement may include, for example, back and forth movement around the first magnification level. For example, if the first magnification level is 5X, the movement may be 4X to 6X. Movement can also include movement in one direction, such as from 5X to 4X. During this movement, stereoscopic visualization camera 300 may adjust one or more other lenses to maintain focus on target region 700. At block 2508, during movement of the left zoom lens, stereoscopic visualization camera 300 records a stream or sequence of images and/or frames 2509 of target region 700 using, for example, left optical image sensor 748. Image 2509 is recorded using an oversized pixel set 1008 that is calibrated to include the origin of pixel grid 1004 and the potential location of the left ZRP.

A processor 1562 of the example information processor module 1408 analyzes the image stream to find portions of the area that do not move in the x or y direction between images (block 2510). The portion of the area may include one or a few pixels and corresponds to the left ZRP. As mentioned above, during a magnification change, the object moves away from or towards the ZRP. Only objects in the ZRP remain in a constant position with respect to the field of view when the magnification is changed. Processor 1562 may calculate a delta between image streams for each pixel using the pixel data. The area with the smallest delta across the image stream corresponds to the left ZRP.

The example processor 1562 of the information processor module 1408 then determines the coordinates of a portion of the area that does not move between image streams with respect to the pixel grid 1004 (eg, locates the left ZRP) (block 2512). In another example, processor 1562 of information processor module 1408 determines the distance between the origin and the portion of the area that corresponds to the left ZRP. The distance is used to locate the left ZRP on the pixel grid 1004. Once the left ZRP is located, the processor 1562 of the information processor module 1408 locates the pixel set (e.g., pixel set 1008) (block 2514). At this point, the left ZRP is centered within the left optical path.

In some examples, blocks 2504-2514 may be performed iteratively by reselecting the pixel set until the left ZRP is within the origin pixel and pseudo-parallax is minimized. After the pixel grid is identified, processor 1562 of information processor module 1408 stores at least one of the pixel set coordinates and/or left ZRP coordinates in memory 1570 as a calibration point (block 2516). Processor 1562 of information processor module 1408 may associate the calibration point with the first magnification level such that the same pixel set is selected when stereoscopic visualization camera 300 returns to the first magnification level.

FIG. 27 shows a diagram illustrating how the left ZRP is adjusted with respect to the pixel grid of the left optical image sensor 748. First, an initial (eg, oversized) pixel set 2702 centered at the origin 2704 is selected. Pixel set 2702 is large enough to record potential ZRPs in the image stream. In this illustrated example, left ZRP 2706 is located to the upper right of origin 2704. Based on the position of left ZRP 2706, processor 1562 of information processor module 1408 identifies a pixel set 2708 such that left ZRP 2706 is centered or located in pixel set 2708.

After the left ZRP is identified and aligned with the origin of the pixel set in FIG. 25, the example procedure 2500 aligns the left and right images in FIG. To register the images, the example processor 1562 compares pixel data from the left and right images recorded after the left ZRP is registered with the origin. In some embodiments, processor 1562 overlays the left and right images and identifies differences using, for example, subtraction and/or template methods. Processor 1562 selects or identifies a set of pixels in the right optical path such that the resulting right image is aligned or coincident with the left image (block 2519).

The example processor 1562 identifies the right ZRP in the illustrated embodiment. The steps are similar to those discussed in blocks 2504-2512 for the left ZRP. For example, at block 2518, the stereoscopic visualization camera 300 moves the right zoom lens to a first magnification level. In some embodiments, the magnification level of the right lens is different than the magnification level used to identify the left ZRP. The example processor 1562 of the information processor module 1408 then moves the right zoom lens back and forth through the magnification level and receives a stream of images 2521 from the right optical image sensor 746 during the movement (blocks 2520 and 2522). The example processor 1562 of the information processor module 1408 identifies the right ZRP from the right stream of images by finding portions of the area that do not move between images (block 2524). Processor 1562 then determines the coordinates of the right ZRP and/or the distance between the center of aligned pixel set 1006 and the right ZRP (block 2526).

The processor 1562 then moves the at least one lens in the right optical path in at least one of the x direction, y direction, and/or tilt direction using, for example, the right ZRP distance or coordinates; The motor and illumination module 1406 is commanded to align the right ZRP with the center of the aligned pixel set 1006 (block 2528). In other words, the right ZRP moves to coincide with the center of the aligned pixel set 1006. In some examples, the right front lens 720, the right lens barrel 736, the right final optical element 745, and/or the right image sensor 746 move in the x direction, the y direction, and/or the tilt direction with respect to the z direction of the right optical path. (e.g. using a bend). The degree of movement is proportional to the distance of the right ZRP from the center of pixel set 1006. In some embodiments, processor 1562 digitally changes attributes of right front lens 720, right lens barrel 736, and/or right final optical element 745 to have the same effect as lens movement. Processor 1562 repeats steps 2520-2528 and/or uses subsequent right images to verify that the right ZRP is aligned with the center of pixel set 1006 and/or aligns the right ZRP with the center of the pixel set. Additional lens movements needed to align may be iteratively determined.

The example processor 1562 stores the coordinates of the right pixel set and/or right ZRP in memory 1570 as a calibration point (block 2530). Processor 1562 may also store the position of the right lens moved to align the right ZRP at a calibration point. In some examples, the right optical path calibration point is stored along with the left optical path calibration point in conjunction with the first magnification level. Accordingly, when the stereoscopic visualization camera 300 is subsequently set to the first magnification level, the processor 1562 converts the data within the calibration points to the radius of the optical image sensors 746 and 748 and/or the one or more optical elements 1402. Applies to directional positioning.

In some examples, procedure 2500 may be repeated at different magnification levels and/or working distances. Therefore, processor 1562 determines whether ZRP calibration is required at another magnification level or working distance (block 2532). If another magnification level is to be selected, procedure 2500 returns to block 2504 in FIG. However, if another magnification level is not needed, the example procedure ends.

Each calibration point may be stored in a lookup table. Each row in the table may correspond to a different magnification level and/or working distance. Columns in the lookup table may provide coordinates for the left ZRP, right ZRP, left pixel set, and/or right pixel set. In addition, one or more columns may be used to determine the relative position of the lens of optical element 1402 (e.g., radial position, rotational position, tilt position and/or axial position).

Accordingly, the procedure 2500 generates a right ZRP and a left ZRP in addition to the pixel grid of each optical image sensor 746 and 748 and the views of the target region to be aligned with each other in the three-dimensional stereo image. In some cases, the left and right images and the corresponding ZRP have accuracy and registration within one pixel. Such accuracy may be observable on the display 514 or 514 by overlapping the left and right views (eg, images from the left and right optical paths) and observing both views with both eyes rather than stereoscopically.

It should be appreciated that in some examples, the right pixel set is first selected such that the right ZRP is aligned or coincident with the origin of the pixel set. The left and right optical images may then be aligned by moving one or more right and/or left lenses of optical element 1402. An alternative procedure still provides right and left ZRPs that are centered and aligned with each other and with respect to optical image sensors 746 and 748.

The procedure 2500 ultimately reduces pseudo-parallax in the stereoscopic visualization camera 300 throughout the entire optical magnification range by ensuring that the left and right ZRPs remain aligned and the left and right images remain aligned. Or lose it. In other words, the dual optics of left and right optical image sensors 746 and 748 are aligned such that the parallax at the center of the image between the left and right optical paths is approximately zero at the focal plane. Furthermore, because the ZRP of each optical path is assigned to the center of each pixel set, the example stereoscopic visualization camera 300 is parfocal over a range of magnifications and concentric over a range of magnifications and working distances. Therefore, only changing the magnification maintains focus of the target region 700 on both optical image sensors 746 and 748 while aligning to the same center point.

The above procedure 2500 may be performed in calibration before a surgical procedure is performed and/or as requested by an operator. The example procedure 2500 may also be performed prior to image registration with preoperative microsurgical images and/or surgical guidance graphics. Further, the example procedure 2500 may be performed automatically in real time during operation of the stereoscopic visualization camera 300.

1. Example Template Matching In some embodiments, the example processor 1562 of the information processor module 1408 uses the program 1560 in conjunction with one or more templates to locate the right ZRP and/or the left ZRP. It is configured as follows. FIG. 28 shows a diagram illustrating how processor 1562 uses target template 2802 to locate the left ZRP. In this example, FIG. 28 shows a first left image that includes a template 2802 aligned with the origin 2804 or the center of the left pixel grid 1004 of the left image sensor 748. Template 2802 may be aligned by moving stereoscopic visualization camera 300 to the appropriate position. Alternatively, template 2802 may be moved at target site 700 until registered. In other examples, template 2802 may include another pattern that does not require alignment with the center of pixel grid 1004. For example, the template may include a graphical waveform pattern, a graphical respirator pattern, a view of a patient's surgical site, and/or visually distinguishable features that have some degree of aperiodicity in both the x and y directions. may include a grid with a The template is configured to prevent a periodic subset of images from being perfectly aligned to a larger image at multiple locations, and to ensure that a periodic subset of images is not perfectly aligned to a larger image at multiple locations. alignment makes such templates unsuitable for matching. Template images suitable for template matching are known as "template matchable" template images.

The template 2802 shown in FIG. 28 is imaged at a first magnification level. Left ZRP 2806 is shown with respect to template 2802. ZRP 2806 has coordinates L _x , L _y with respect to origin 2804 . However, at this point, processor 1562 has not yet identified left ZRP 2806.

To find ZRP 2806, processor 1562 magnifies the left zoom lenses (e.g., left front zoom lens 728 and/or left rear zoom lens 734) from a first magnification level to a second magnification level, particularly in this example. , change from 1X to 2X. FIG. 29 shows a view of the second left image containing target 2802 on pixel grid 1004 with the magnification level doubled. From the first magnification level to the second magnification level, a portion of the target 2802 increases in size and expands uniformly away from the left ZRP 2806, which remains stationary with respect to the first and second images. Additionally, the distance between the origin 2804 of pixel grid 1004 and left ZRP 2806 remains the same.

The example processor 1562 synchronizes the digital template image 3000 from the second image shown in FIG. To create the digital template image, processor 1562 copies the second image shown in FIG. 29 and scales the copied image by scaling back and forth from a first magnification to a second magnification. For example, if the magnification change from the first image to the second image is 2x, the second image is scaled by 1/2. FIG. 30 shows an illustration of a digital template image 3000 that includes template 2802. Template 2802 in digital template image 3000 of FIG. 30 is scaled to the same size as template 2802 in the first left image shown in FIG.

The example processor 1562 uses the digital template image 3000 to find the left ZRP 2806. FIG. 31 shows a diagram showing a digital template image 3000 superimposed on a first left image (or a subsequent left image recorded at a first magnification level) recorded in a pixel grid 1004. Combining the digital template image 3000 with the first left image produces a resulting view, as shown in FIG. 31. First, the digital template image 3000 is centered at the origin 2804 of the pixel grid 1004.

The example processor 1562 compares the digital template image 3000 to the underlying template 2802 to determine whether it is aligned or matched. The example processor 1562 then moves the digital template image 3000 horizontally or vertically by one or more pixels and performs another comparison. Processor 1562 iteratively moves digital template image 3000 and compiles a matrix of metrics at each location regarding how closely digital template image 3000 matches underlying template 2802. Processor 1562 selects the position within the matrix that corresponds to the best matching metric. In some examples, processor 1562 uses the OpenCV® Template Match function.

FIG. 32 shows a diagram with a digital template image 3000 aligned with template 2802. The distance that digital template image 3000 has moved to achieve optimal matching is indicated as Δx and Δy. Knowing that the digital template image 3000 was synthesized at a scale of M1/M2 (the first magnification level divided by the second magnification level), the processor 1562 calculates the following equations (1) and (2). to identify the coordinates (L _x ,L _y ) of the left ZRP 2806.
Lx=Δx/(M1/M2) - Formula (1)
Ly=Δy/(M1/M2)-Formula (2)

After the coordinates (L _x ,L _y ) of the left ZRP 2806 are determined, the example processor 1562 has the origin aligned or coincident with the left ZRP 2806, as described above in conjunction with procedure 2500 of FIGS. 25 and 26. Select or identify a pixel subset. In some embodiments, processor 1562 may iteratively use template matching to converge on a more accurate ZRP location and/or pixel subset. Furthermore, although the above example considered finding the left ZRP, the same template matching procedure can be used to find the right ZRP.

In some embodiments, the template matching program 1560 may be used to align left and right images. In these embodiments, left and right images are recorded at the magnification level. Both images may include target template 2802 of FIG. 28, for example. A portion of the right image is selected and superimposed on the left image. Portions of the right image are then shifted horizontally and/or vertically in front of and behind the left image by one or more pixels. The example processor 1562 performs a comparison at each location of the portion of the right image to determine how closely there is a match with the left image. Once the optimal location is identified, a set of pixels 1006 of the right pixel grid 1002 is identified such that the right image generally matches the left image. The position of pixel set 1006 may be determined based on how much the portion of the right image is moved to match the left image. In particular, processor 1562 uses the amount of movement in the x, y, and/or tilt directions to identify corresponding coordinates of right pixel set 1006.

2. Exemplary Left and Right Image Alignment In some embodiments, the example processor 1562 of the information processor module 1408 of FIGS. 14-16 displays an overlay of left and right images on the display monitors 512 and/or 514. Processor 1562 is configured to receive user feedback to align the left and right images. In this example, each pixel data of the left and right images is precisely mapped to each pixel of display monitor 512 using, for example, graphics processing unit 1564. The display of the superimposed left and right images makes any spurious parallax easily apparent to the operator. In general, in the absence of pseudo-parallax, the left and right images should be approximately exactly aligned.

If the operator detects pseudo-parallax, the operator may actuate control mechanism 305 or user input device 1410 to move either the left or right image into alignment with the other of the left or right images. Instructions from control mechanism 305 may cause processor 1562 to adjust the positions of the left and right pixel sets accordingly in real time so that subsequent images are displayed on display monitor 512 to reflect operator input. In other examples, the instructions may cause processor 1562 to change one or more positions of optical element 1402 via radial adjustment, rotational adjustment, axial adjustment, or tilt. The operator continues to provide input via control mechanism 305 and/or user input device 1410 until the left and right images are aligned. Upon receiving the confirmation command, processor 1562 stores calibration points in a lookup table that reflect image alignment at the set magnification level.

Additionally or alternatively, the template matching method described above may be used to provide image alignment while focusing on a flat target generally orthogonal to the stereoscopic optical axis of the stereoscopic visualization camera 300. Additionally, template matching methods may be used to align the left and right views in real time whenever a "template matchable" scene is in view of the left and right optical paths. In one example, the template image is copied from, for example, a subset of the left view and centered at or near the center of the view. Sampling from the center of the focused image ensures that a similar view of the target region 700 is presented in the other view (the right view in this example). In the case of an out-of-focus image, it is not the case in this embodiment that this alignment method is performed only after a successful autofocus operation. The selected template is then matched in the current view (or a copy thereof) of the other view (the right view in this example), and only the y-axis is taken from the result. When the view is vertically aligned, the y value of the template match is at or near zero pixels. A non-zero y value indicates a vertical misalignment between the two views, and a correction using the same value of y is applied to select the pixel readout set of the first view, or a negative value of y The correction using is applied to the pixel readout set of the other view. Alternatively, corrections can be applied to other parts of the visualization pipeline, or can be split between the pixel readout set and the pipeline.

In some examples, the operator may also manually align the right ZRP with the origin of pixel grid 1002. For example, after locating the right ZRP, processor 1562 (and/or peripheral input unit interface 1574 or graphics processing unit 1564) causes the right ZRP to be graphically highlighted in the right image displayed by display monitor 512. Processor 1562 may also display a graphic indicating the origin of pixel grid 1002. The operator uses control mechanism 305 and/or user input device 1410 to advance the right ZRP to the origin. Processor 1562 uses instructions from control mechanism 305 and/or user input device 1410 to move one or more of optical elements 1402 accordingly. Processor 1562 may provide a right image stream in real time, in addition to graphically displaying the current position and origin of the right ZRP to provide the operator with updated feedback regarding positioning. The operator continues to provide input via control mechanism 305 and/or user input device 1410 until the right ZRP is aligned. Upon receiving the verification command, processor 1562 stores a calibration point in a look-up table that reflects the position of optical element 1402 at the set magnification level.

3. Registration Error Comparison The example stereoscopic visualization camera 300 produces less registration error between left and right images compared to known digital surgical microscopes having stereoscopic cameras. The analysis described below compares the example stereoscopic visualization camera 300 with the pseudo vision produced by ZRP misregistration of a known digital surgical microscope with a camera. First, both cameras are set to a first magnification level and the focal plane is positioned at a first location of the patient's eye. Identify each camera-to-eye working distance (“WD”) using equation (3) below.
WD=(IPD/2)/tan(α)-Equation (3)

In this formula, IPD corresponds to the interpupillary distance and is approximately 23 mm. Additionally, α is, for example, half the angle between right optical image sensor 746 and left optical image sensor 748, which is 2.50° in this example. The focusing angle is twice this angle, in this example 5°. The resulting working distance is 263.39 mm.

The camera is zoomed in to a second magnification level and triangulated at a second position of the patient's eye. In this example, the second location is at the same physical distance from the camera as the first location, but is presented at a second magnification level. Magnification changes create horizontal pseudo-parallax due to misalignment of one or both of the ZRPs to the center of the sensor pixel grid. In known camera systems, the pseudo parallax is specified to be, for example, 3 minutes, which corresponds to 0.05°. In equation (3) above, a 0.05° value is added to α, which produces a working distance of 258.22 mm. The working distance difference is 5.17 mm (263.39 mm - 258.22 mm), which corresponds to the error of known digital surgical microscopes with camera attachments.

In contrast, the example stereoscopic visualization camera 300 is capable of automatically aligning the ZRP to within one pixel of the center of a pixel set or grid. If the viewing angle is 5° and recorded using a 4K image sensor used with a 4K display monitor, one pixel accuracy corresponds to 0.00125° (5°/4000) or 4.5 seconds. Using equation (3) above, add 0.00125° to α, resulting in a working distance of 263.25 mm. The working distance difference of the stereoscopic visualization camera 300 is 0.14 mm (263.39 mm-263.25 mm). The example stereoscopic visualization camera 300 reduces alignment error by 97.5% when compared to the 5.17 mm error of known digital surgical microscopes.

In some embodiments, stereoscopic visualization camera 300 may be more accurate with higher resolution. In the above example, the resolution is approximately 4.5 seconds with a 5° field of view. For an 8K ultra-high definition system (with 8000 pixels in each of 4000 rows) with a 2° field of view, the resolution of the stereoscopic visualization camera 300 is approximately 1 second. This means that the ZRP of the left and right views can be aligned down to 1 pixel or 1 second. This is much more precise than known digital microsurgical systems, which have pseudo-parallaxes on the order of minutes.

4. Reducing Other Sources of Pseudo-Parallax The above example considers how an example stereoscopic visualization camera 300 reduces pseudo-parallax as a result of ZRP and/or misalignment of the left and right images themselves. The stereoscopic visualization camera 300 can also be configured to reduce other sources of pseudo-parallax. For example, stereoscopic visualization camera 300 may reduce pseudo-parallax due to motion by simultaneously clocking left and right optical image sensors 746 and 748 to record images at approximately the same instant.

The stereoscopic visualization camera 300 as an example can also reduce pseudo parallax caused by different magnifications between the left and right optical paths. For example, stereoscopic visualization camera 300 may set the magnification level based on the left optical path. The stereoscopic visualization camera 300 may then automatically adjust the magnification of the right image to match the left. For example, processor 1562 may use the image data to calculate control parameters, eg, by measuring the number of pixels between particular features common to the left and right images. Processor 1562 may then equalize the magnification levels of the left and right images by digital scaling, inserting interpolated pixels, and/or removing irrelevant pixels. The example processor 1562 and/or graphics processing unit 1564 may re-render the right image so that the scale factor matches the left image. Additionally or alternatively, stereoscopic visualization camera 300 may include independent adjustment of left and right optical elements 1402. Processor 1562 may separately control left and right optical elements 1402 to achieve the same magnification. In some examples, processor 1562 may first set a left magnification level, for example, and then separately adjust right optical element 1402 to achieve the same magnification level.

The example stereoscopic visualization camera 300 may further reduce pseudo parallax due to different focus. In one example, processor 1562 may execute a program 1560 that determines the best focus of an angular light path at a given magnification and/or working distance. Processor 1562 first performs focusing of optical element 1402 to the best resolution point. Processor 1562 may then check the OOF situation at the appropriate non-object plane location and match the focus of the left and right images. The processor 1562 then rechecks the focus at best resolution and repeatedly adjusts the focus until both left and right optical elements 1402 are sufficiently equally focused both on the object plane and away from the object back surface.

An example processor 1562 may measure and verify optimal focus by monitoring signals related to focus of one or both left and right images. For example, left and right image "sharpness" signals are generated by graphics processing unit 1564 simultaneously and/or synchronously. The signal changes as focus changes and may be determined from an image analysis program, an edge detection analysis program, a Fourier transform bandwidth program of pattern intensity, and/or a modulation transfer function ("MTF") measurement program. Processor 1562 adjusts the focus of optical element 1402 while monitoring the maximum signal indicative of a sharp image.

To optimize the OOF situation, processor 1562 may monitor the sharpness signals of both the left and right images. If the focus moves from the object plane, for example, the signal associated with the left image increases while the signal associated with the right image decreases, processor 1562 determines that optical element 1402 is moving out of focus. It is configured as follows. However, if the signals associated with both left and right images are relatively high and approximately equal, processor 1562 is configured to determine that optical element 1402 is properly positioned for focus.

5. Advantages of Low Pseudo-Parallax The example stereoscopic visualization camera 300 has several advantages over known digital surgical microscopes as a result of the low pseudo-parallax between the left and right images. For example, nearly perfectly aligned left and right images create a nearly perfect stereoscopic view for the surgeon, thereby reducing eye fatigue. This allows the stereoscopic visualization camera 300 to be used as an extension of the surgeon's eye rather than a cumbersome instrument.

In another example, precisely aligned left and right images allow accurate measurements of the surgical site to be digitally captured. For example, the size of the lens capsule in a patient's eye may be measured so that an appropriately sized IOL can be determined and accurately implanted. Alternatively, the motion of a moving blood vessel may be measured so that an infrared fluorescence overlay can be precisely placed on the fused image. Here, the actual movement speed is generally not of concern to the surgeon, but is important for the placement and real-time registration of the overlay images. Properly matched scale, alignment, and perspective of the superimposed images are all important to providing accurately usable combined live stereoscopic and alternating mode images.

In some examples, processor 1562 may enable an operator to draw measured parameters on display monitor 512. Processor 1562 receives the coordinates drawn on the screen and converts the coordinates into a stereoscopic image accordingly. Processor 1562 may determine measurements by scaling a ruler drawn on display monitor 512 to the magnification level shown in the stereo image. Measurements performed by processor 1562 include point-to-point measurements of two or three locations displayed on a stereoscopic display, point-to-surface measurements, surface characterization measurements, volume identification measurements, velocity verification measurements, coordinate transformations, equipment and/or Or there is organization tracking, etc.

VII. Exemplary Stereoscopic Visualization Camera Robotic System As discussed in conjunction with FIGS. can be connected to. The example robotic arm 506 is configured to allow an operator to position and/or direct the stereoscopic visualization camera 300 above and/or next to the patient during one or more procedures. Accordingly, the robotic arm 506 allows the operator to move the stereoscopic visualization camera 300 to a desired field of view ("FOV") of the target surgical site. The surgeon typically positions and/or orients the camera to a similar FOV to the surgeon's own FOV, allowing for easier visual orientation and correspondence between the image displayed on the screen and the surgeon's FOV. I like things. An example robotic arm 506 disclosed herein provides structural flexibility and assisted control to enable positioning that is coincident or consistent with the surgeon's FOV without obstructing the surgeon's own FOV.

In contrast to the stereoscopic robotic platform 516 disclosed herein, known stereoscopic microscope holding devices include a simple mechanism that is manually moved by an operator. These devices include multiple rotating joints equipped with electromechanical brakes that allow manual repositioning. Furthermore, some known holding devices have motorized joints so that the operator can change the view easily and without interfering with the procedure. Powered joints have varying levels of complexity, for example, from simple XY positioning to devices with multiple independent rotational joints operating connected rigid arms. During most procedures, it is desirable to quickly and easily obtain views from various directions. However, known stereomicroscope holding devices have one or more problems.

Known stereoscopic microscope holding devices have limited positional, directional, and/or orientation accuracy that is generally limited by the surgeon's manual ability to manipulate the microscope to view the desired side of the image. Retention devices with multiple joints can be particularly cumbersome to operate because device operation typically involves movement of all the joints at once. The operator often watches how the arm moves. After the arm is positioned at the desired location, the operator checks whether the FOV of the imaging device is aligned at the desired location. In many cases, it is necessary to adjust the focus of the device even if the device is properly aligned. Additional known stereoscopic microscope holding devices are unable to provide a consistent FOV or focal plane with respect to other objects at the target surgical site, because as the patient moves or shifts during the procedure, This is because the device does not have arm position memory or the memory is inaccurate.

Known stereomicroscope holding devices generally have a positioning system whose control is independent of microscope parameters such as object plane focal length, magnification, and illumination. In these devices, positioning, e.g. zoom adjustment, must be performed manually. In one example, an operator may reach a lens limit when focusing or changing working distance. The operator must manually reposition the holding device and then refocus the stereomicroscope.

Known stereomicroscope holding devices are intended solely for observation of the surgical site. Known devices do not identify the location of tissue within the FOV or the distance from tissue within the FOV to another object outside the FOV. Known devices also do not provide comparison of tissue with other objects within a live surgical site to form alternative viewing modalities, such as combining MRI images with a live view. Instead, views from known devices are displayed separately and are not aligned from other medical images or templates.

Furthermore, known stereomicroscope holding devices have parameters that may not be accurate, as they do not place much emphasis on precision, except for observation. The requirements in ISO standard 10936-1:2000 (E) "Optics and optical instruments-Operation microscopes-Part 1: Requirements and test methods" are mostly for ordinary people. The operator uses the eyepiece to obtain a reasonable stereoscopic optical image. It was derived in order to achieve this. The operator's brain combines multiple views to create an image in their head to achieve stereopsis. Views generally are not otherwise combined or compared together. As long as the operator sees acceptable images and does not experience deleterious effects such as headaches, the operator's needs have been met. The same is true for stereomicroscope holding devices, which are allowed to have some instability, arm sag, and imprecise movement control. However, when high-resolution digital cameras are used in conjunction with known devices, structural inaccuracies are easily observable and can compromise their usefulness, especially in the case of microsurgical procedures.

As mentioned above, known stereo microscope holding devices can sag due to the weight of the camera. Generally, known robot positioning systems are calibrated to identify compliance or inaccuracies within the system itself alone. The stereo microscope holding device does not take into account any inaccuracies between the camera or camera mount and the holding device. Sag is generally compensated for by the operator manually positioning the camera while viewing the image on the display. In systems that provide motorized movement, for example, if the center of gravity ("CG") of the camera is repositioned to the opposite side of the axis of rotation of the arm joint, the direction of the restoring torque moment about the axis is reversed; A drooping change occurs. Subsequently, any compliance or sag in the mechanism that is compensated by the operator by adjusting the camera position, direction, and/or orientation increases the position, direction, and/or orientation error. In some cases, for example when the camera moves through a robot singularity, moment reversals occur quickly and the resulting camera image errors quickly shift excessively. Such errors, for example, limit the ability of known stereomicroscope holding devices to accurately follow or track tissue or equipment within a site.

Known stereomicroscope holding devices include features for spatially locating and tracking surgical crises and subsequently providing a representative display thereof on a monitor. However, these known systems require a conspicuously placed stereolocation camera or triangulation device and instrument with a conspicuous reference device. Adding devices increases complexity, cost, and operational nuisance.

An example stereoscopic robot platform 516 disclosed herein includes an example stereoscopic visualization camera 300 connected to a mechanical or robotic arm 506. 5 and 6 illustrate an example stereoscopic robot platform 516. The stereo images recorded by camera 300 are displayed via one or more display monitors 512, 514. Robotic arm 506 is mechanically connected to cart 510, which may also support one or more display monitors 512, 514. Robotic arms may include, for example, articulated robotic arms that are generally human-like in size, nature, function, and operation.

FIG. 33 illustrates a side view of the microsurgical environment 500 of FIG. 5, according to an example embodiment of the present disclosure. In the illustrated example, display monitor 512 may be connected to cart 510 via a mechanical arm 3302 that has one or more joints to allow flexible positioning. In some embodiments, mechanical arm 3302 may be long enough to extend across the patient and provide a relatively close view of the surgeon during surgery.

FIG. 33 also shows a side view of stereoscopic robotic platform 516, including stereoscopic visualization camera 300 and robotic arm 506. Camera 300 is mechanically coupled to robot arm 506 via coupling plate 3304. In some embodiments, coupling plate 3304 may include one or more joints that provide an additional degree of positioning and/or orientation of camera 300. In some embodiments, the coupling plate 3304 needs to be manually moved or rotated by the operator. For example, coupling plate 3304 may allow camera 300 to have an optical axis along the z-axis (i.e., pointing down toward the patient) and an optical axis along the x- or y-axis (i.e., pointing toward the patient). may have a joint that allows for quick positioning between the

An example coupling plate 3304 may include a sensor 3306 configured to detect force and/or torque applied by an operator to move camera 300. In some embodiments, an operator may position camera 300 by grasping control arms 304a and 304b (shown in FIG. 3). After the operator grasps the control arms 304a and 304b, the user may position and/or direct the camera 300 with assistance from the robotic arm 306. Sensor 3306 detects the force vector or torque angle provided by the operator. An example platform 516 disclosed herein uses the sensed force/torque to move the robot arm 500 to provide assisted movement of the camera 300 in response to the force/torque provided by the operator. Determine which joints in the throat should be rotated (and how quickly the joints should be rotated). A sensor 3306 may be positioned at the interface between coupling plate 3304 and camera 300 to detect force and/or torque applied by an operator via control arm 304.

In some embodiments, the sensor 3306 may include, for example, a six degree of freedom tactile sensing module. In these embodiments, sensor 3306 may detect translational force or movement in the x, y, and z axes. Sensor 3306 can also separately detect rotational force or motion about the yaw, pitch, and roll axes. The decoupling of translational and rotational forces may allow stereoscopic robot platform 516 to more easily calculate forward and/or inverse kinematics for controlling robot arm 506.

Because the robot arm 506 may not be movable by the user alone, an example sensor 3306 may be configured to detect force. Instead, the sensor 3306 detects translational and rotational forces applied by the user, which are used in stereoscopic vision to determine which joints to rotate to provide assisted movement control of the robot arm 506. Used by robot platform 516. In other examples, the robotic arm 506 may allow movement by an operator without assistance or at least initial assistance. In these other examples, the sensor 3306 detects motion exerted by a user, which causes the stereoscopic robotic platform to sequentially rotate one or more joints, thereby providing assisted locomotion. 516. The time from initial detection of the motion or force that causes the motion to when the stereoscopic robotic platform 516 rotates the joint may be less than 200 milliseconds (“ms”), less than 100 ms, less than 50 ms, or even less than just 10 ms; The user is unaware of the initial time of unassisted movement of the robot arm 506.

An example sensor 3306 may output digital data indicative of rotational force/motion and digital data indicative of translational force/motion. In this example, the digital data may have 8-bit, 16-bit, 32-bit, or 64-bit resolution for the detected force/motion in each axis. Alternatively, sensor 3306 may transmit an analog signal proportional to sensed force and/or motion. An example sensor 3306 may transmit data at a regular sampling rate, such as 1 ms, 5 ms, 10 ms, 20 ms, 50 ms, 100 ms, etc., for example. Alternatively, sensor 3306 may provide a substantially continuous stream of force/motion data.

In some embodiments, an example sensor 3306 may instead be located between one or more of control arms 304a and 304b or between control arms 304a and 304b and housing 302. In examples where each of the control arms 304a and 304b includes a sensor 3306, the example stereoscopic robot platform 516 may receive two sets of translational forces or movements and rotational forces or rotational movements. In these examples, stereoscopic robot platform 516 may average the values from sensors 3306.

In the illustrated embodiment, a first end of the robot arm 506 is mounted on a cart 510, while a second opposite end of the robot arm is mechanically attached to a stereoscopic visualization camera 300 (e.g., a robot end effector). connected. FIG. 33 shows a robotic arm holding the stereoscopic visualization camera 300 in an extended position, such as positioning the stereoscopic visualization camera 300 above the surgical site while maintaining the remainder of the platform 516 out of the surgeon's way. 506 is shown. The cart 510 is configured to securely hold the stereoscopic robot platform 516 and is weighted and balanced to prevent it from tipping over in a defined operating position.

The example stereoscopic robot platform 516 is configured to provide the following advantages.

1. Enhanced visualization. Communication between robotic arm 506 and stereoscopic visualization camera 300 allows platform 516 to direct and maneuver camera 300 to quickly and more accurately visualize the surgical site. For example, the robotic arm 506 can move the camera 300 along the optical axis to extend the range of focus and zoom beyond just being contained within the camera. The relatively small size of platform 516 provides Heads-Up Surgery® with a greater variety of surgical procedures and orientations, thereby improving surgical efficiency and surgeon ergonomics.

2. Enhanced dimensional performance. An example stereoscopic visualization camera 300, which has the capability of accurately measuring every point in a stereoscopic image, is configured to communicate measurement information to the robotic arm 506. The robotic arm 506 then has precise position, direction, and/or orientation capabilities to accurately determine intra-image and inter-image dimensions with respect to a coordinate system common to the stereoscopic robotic platform 516 and the patient's anatomy. It is registered to camera 300 so that it can be converted.

3. The quality and accuracy of stereo images from stereo visualization camera 300 allows them to be combined with images or diagnostic data from external sources of various modalities to construct a fused image. Such fused images can be used by surgeons to perform procedures more accurately and efficiently and achieve better patient outcomes.

4. The stereoscopic visualization camera 300, the robotic arm 506, and/or the image and motion processor (eg, processor 1408 of FIG. 14) can be programmed for advantageous treatment applications. For example, a particular visualization site location, orientation, and/or orientation can be saved and returned to the procedure later. For example, precise travel paths can be programmed to follow specific lengths or lines of tissue. In other examples, pre-programmed waypoints can be set so that the operator can change the position and/or orientation of the robotic arm 506 based on what steps are being performed during the medical procedure. do it like this.

5. The stereoscopic robotic platform 516 provides surgery that is inherently guided by the use and analysis of precise image location information. Such guidance may be communicated to other devices, such as another robotic system, that performs at least a portion of the surgical procedure. Components of stereoscopic robot platform 516 that share functionality with components of such other devices may be integrated together into a package to achieve efficiency, accuracy, and cost of performance.

A. Robot Arm Embodiment FIG. 34 illustrates an example robot arm 506 embodiment according to example embodiments of the present disclosure. In some embodiments, robotic arm 506 is similar to or includes model UR5 from Universal Robots S/A. The exterior surface of robotic arm 506 includes aluminum and plastic materials, making it compatible for use in operating rooms and easily cleaned.

Although robotic arm 506 is described herein as electromechanical, in other examples, robotic arm 506 can be mechanical, hydraulic, or pneumatic. In some embodiments, the robotic arm 506 may have a mixed actuation mechanism to hold and operate the camera 300 using, for example, a vacuum chuck with a control valve. Additionally, although the robotic arm 506 is described below as including a particular number of joints and links, the robotic arm 506 may include any number of joints, any length of links, and/or any number of joints and links. It should be understood that this may include any type of joint or sensor.

The robotic arm 506 is positioned and the joints are oriented to provide an unrestricted view of the surgical field while providing the operator with a 3D stereoscopic representation of any surgical procedure on a patient, as described herein. It will be done. Movement of the robot arm 506 in non-critical movements is provided to be fast enough to be convenient for the operator, but still safe. Movement of the robotic arm 506 is controlled to be fine and precise during the procedure. Additionally, the movement of the robotic arm is controlled to be smooth and predictable throughout the full range of motion required for the surgical procedure. As described herein, movement of the robotic arm 506 can be controlled by remote control or via manual manipulation of the arm itself. In some embodiments, the robotic arm 506 is configured to be positionable with minimal force (eg, via assisted guidance features), eg, using only one pinky finger.

In some embodiments, robotic arm 506 may include mechanical or electronic locking brakes at the joints. Once set by the operator, the objective or "pose" of the camera 300, generally the location and orientation, may engage the brakes. Robotic arm 506 may include a lock or unlock switch or other input device to avoid unwanted manual or unintentional movement. When locked, the example robotic arm provides sufficient stability to allow the stereoscopic visualization camera 300 to provide stable and clear images. Robotic arm 506 may additionally or alternatively include one or more damping devices to absorb or dampen vibrations following movement of stereoscopic visualization camera 300 to a new pose. The damping device may include, for example, a fluid-filled linear or rotational restraint, a rubber-based anti-vibration mounting restraint, and/or a tuned mass spring restraint. Alternatively or additionally, arm 506 may include electromechanical damping, such as through the use of a proportional-integral-derivative (“PID”) servo system.

An example robotic arm 506 may have one or more links configured to a storage position back for transfer and storage. The retracted position allows the robotic arm to be transported and retracted with a compact footprint, yet allows for deployment with the long reach required in some surgical procedures. Cables, such as wiring for the stereoscopic visualization camera 300, are provided along the robotic arm 506 to avoid interference with the surgical procedure.

In the embodiment shown in FIG. 34, the robot arm 506 includes six joints labeled R1, R2, R3, R4, R5, and R6. In other embodiments, robotic arm 506 may include fewer or additional joints. Additionally, in some embodiments, at least some of the joints R1-R6 have +/-360° rotational movement capability. The rotational motion may be provided by an electromechanical subsystem that includes, for each joint, an electric motor configured to drive the mechanical rotational joint through one or more anti-backlash joint gearboxes. Each of joints R1-R6 may include one or more rotation sensors that detect joint position. Additionally, each joint may include a slip clutch and/or an electromechanical brake.

Each of joints R1-R6 may have an overall repeatability of motion of approximately +/-1/10 of a millimeter ("mm") (with camera 300 attached). The joint may have a variable rotation speed that can be controlled from 0.5° to 180° per second. This together translates into a camera movement of 1 mm/sec to 1 m/sec. In some embodiments, stereoscopic robotic platform 516 may have one or more velocity governors of joints R1-R6 in place during a surgical procedure. Each of joints R1-R6 may be electrically connected to a power source and/or a command line within a controller of robot arm 506. Power and command signal wires may be routed within the joints and links. Additionally, one or more of the joints may include a restraining device, such as an O-ring, for connecting to the link. The suppression device may, for example, reduce or absorb vibrations in the robot arm 506, vibrations from the cart 510, and/or vibrations applied via the stereoscopic visualization camera 300.

Joint R1 includes a base joint that is mechanically coupled to a flange 3402, which is fixed to a stationary structure 3404. Flange 3402 may include any type of mechanical connector. Stationary structure 3404 may include, for example, cart 510 of FIG. 5, a wall, a ceiling, a table, etc. Joint R1 is configured to rotate about a first axis 3410, which may include the z-axis.

Joint R1 is connected to joint R2 via link 3430. An example link 3430 includes a cylindrical body or other tubular structure configured to provide structural support to a downstream section of robotic arm 506. Link 3430 is configured to provide a rotationally fixed connection with joint R2, allowing joint R2 to rotate while link 3430 is held in place by its connection to joint R. Joint R2 may include, for example, a shoulder joint configured to rotate about axis 3412. An example axis 3412 is configured to be perpendicular (or substantially perpendicular) to axis 3410. Axis 3412 is configured to lie in the xy plane, given the rotation of joint R1 about the z-axis.

Joint R2 is mechanically coupled to joint R3 via link 3432. Link 3432 is configured to have a longer length than link 3430 and is configured to provide structural support to a downstream portion of robotic arm 506. Joint R3 may include, for example, an elbow joint. Joint R3, together with joint R2, provides extendable positioning and/or orientation of robot arm 506. Joint R3 is configured to rotate about an axis 3414 that is perpendicular or orthogonal to axis 3410 and parallel to axis 3412.

Joint R3 is connected to joint R4 via link 3434, which provides structural support to the downstream portion of robot arm 506. An example joint R4 can be, for example, a first wrist joint configured to provide rotation about an axis 3416, which can be orthogonal to axes 3412 and 3414. Joint R4 is mechanically connected to joint R5 via link 3436. Joint R5 may be a second wrist joint configured to provide rotation about axis 3418 orthogonal to axis 3416. Joint R5 is mechanically connected to joint R6 via link 3438. Joint R6 may be a third wrist joint configured to rotate about an axis 3420 that is perpendicular to axis 3418. Wrist joints R4-R6 together provide precision flexibility in positioning the stereoscopic visualization camera 300 described herein.

The example robot arm 506 includes a connector 3450. An example connector 3450 is connected to joint R6 via link 3440. In some embodiments, example link 3440 may include a sleeve that allows joint R6 to rotate connector 3450. As discussed herein, connector 3450 may be configured to mechanically couple directly to coupling plate 3304 or stereoscopic visualization camera 300 if a coupling plate is not used. Connector 3450 may include one or more screws that secure robot arm 506 to coupling plate 3304 and/or stereoscopic visualization camera 300.

In some embodiments, the example robotic arm 506 shown may have a maximum reach of 85 mm with an orientation roughly similar to a human arm. Arm 506 may have a payload of 5 kilograms. Additionally, arm 506 may be configured as a "cooperative" device that allows for safe operation in the vicinity of a human. For example, the maximum force that robot arm 506 can exert on an external surface is controlled. If a part of the robot arm accidentally comes into contact with another object, the collision will be detected and the movement will stop instantly. For example, during an emergency shutdown situation such as a power outage, joints R1-R6 can be backdriven or manually rotated such that an operator grabs a portion of the robotic system and swings it out of the way. For example, a slip crunch in the joint limits the maximum torque that the joint motor can rotationally apply to arm 506 during operation. When powered off, the joint's slip clutch slips when manually operated, allowing the operator to quickly move the robot arm 506 out of the way.

35-40 illustrate example configurations of a robotic arm 506 and stereoscopic visualization camera 300, according to example embodiments of the present disclosure. FIG. 35 shows a view of robotic arm 506 connected to cart 510 via flange 3402. In this example, stereoscopic visualization camera 300 is connected directly to connector 3540. In this embodiment, connector 3540 and/or stereoscopic visualization camera 300 may include sensor 3306 of FIG. 33 that senses translational and/or rotational forces/motions applied to stereoscopic visualization camera 300 by an operator. If connector 3540 includes sensor 3306, force/movement output data may be sent to controller via robot arm 506. For example, if sensor 3306 is placed on stereoscopic visualization camera 300, the output data may be sent to a separate controller along with control data. In some embodiments, the controller may be provided within the cart 510 or separately on the server.

FIG. 36 shows an embodiment in which a robot arm 506 is mounted to a ceiling plate 3404 via a flange 3402. The robotic arm may be suspended from the ceiling of the operating room to reduce floor space crowding. Robotic arm 506 is positioned above, including the joints, and can traverse out of the way of the surgeon and operating room staff from the area where surgical activities are performed and provides a clear view of display monitors 512 and 514. while still providing functional positioning and/or orientation of camera 300.

FIG. 37 shows an embodiment of a bonding plate 3304. In the illustrated example, first end 3702 of coupling plate 3304 is connected to connector 3450 of robot arm 506. A second end 3704 of coupling plate 3304 is connected to stereoscopic visualization camera 300. The example coupling plate 3304 is configured to provide additional degrees of freedom to move the stereoscopic visualization camera 300. The coupling plate 3304 also extends the maximum reach of the robot arm 506. Bonding plate 3304 can have a length of 10 cm to 100 cm.

Bonding plate 3304 may include one or more joints. In the illustrated example, coupling plate 3304 includes joints R7, R8, and R9. One example joint is a mechanical joint that provides rotation about each axis. Joints R7-R9 may include rotatable latching mechanisms that are movable after an operator actuates a release button or lever. Each joint R7-R9 may have its own release button, or one button may release each of joints R7-R9.

Joints R7-R9 may be connected together via respective links. Additionally, link 3718 provides a connection to connector 3450 of robot arm 506. Joint R7 is configured to rotate about axis 3710, while joint R8 is configured to rotate about axis 3712, and joint R9 is configured to rotate about axis 3714. Axes 3710 and 3714 are parallel to each other and perpendicular to axis 3712. Joints R7 and R9 may be configured to provide +/-360° rotation. In other examples, joints R7 and R9 may provide +/-90°, +/-180° rotation, or +/-270° rotation about the respective axes 3710 and 3714. Joint R8 may provide +/-90° rotation about axis 3712. In some examples, joint R8 may only be set to +90°, 0°, and -90°.

In some embodiments, joints R7-R9 may include motors that provide continuous movement. Joints R7-R9 may also include control devices such as switches or position sensors that communicate or provide rotational position or indicative data. In this manner, joints R7-R9 may be similar to joints R1-R6 of robot arm 506, providing position sensing for assisted movement and feedback control. Power and control for joints R7-R9 may be provided via wires routed through robot arm 506, a power/wire connector within connector 3450, and/or wires external to robot arm 506.

FIG. 37 shows an example in which stereoscopic visualization camera 300 is positioned horizontally such that optical axis 3720 is provided along the z-axis. Horizontal orientation may be used for imaging a lying patient. In contrast, FIG. 38 shows joint R8 rotated 90° to position camera 300 in a vertical orientation such that optical axis 3720 is provided along the x-axis or the y-axis orthogonal to the x-axis. An embodiment is shown. Vertical orientation may be used for imaging seated patients. It should be appreciated that joint R8 allows stereoscopic visualization camera 300 to be quickly reoriented between horizontal and vertical positions based on the procedure.

In the examples shown in FIGS. 36 and 37, the example sensor 3306 may be connected to, for example, the connector 3450 of the robot arm (connection to the coupling plate 3304) and/or the first end 3702 of the coupling plate (connection to the connector 3450). ). Alternatively or additionally, the example sensor 3306 may be connected to the second end 3704 of the bonding plate (in connection with the camera 300) and/or to the camera 300 in connection with the second end 3704 of the bonding plate 3304, for example. can be placed.

39 and 40 show the stereoscopic visualization camera 300 in a horizontal orientation and rotated +90° about the axis 3714 of joint R9. FIG. 40 shows an example stereoscopic visualization camera 300 oriented horizontally and rotated −90° about axis 3714 of joint R9.

As shown in FIGS. 34-40, an example robotic arm 506 is configured to provide support for the stereoscopic visualization camera 300 and to enable precise positioning and/or orientation and aiming of the optical axis of the camera. It is composed of Because the stereoscopic visualization camera 300 does not have an eyepiece and does not need to be aimed at the surgeon's eyes, it can achieve many positions and/or orientations desirable for its imaging that were previously impractical. The surgeon can perform the procedure in a view that is not optimal due to the surgeon's orientation to the eyepiece, but rather the optimal view for the procedure.

The example robotic arm 506, when used in conjunction with the stereoscopic visualization camera 300, allows the surgeon to see corners and other locations that are not easily visible. Robotic arm 506 also allows the patient to be placed in different positions, including supine, prone, sitting, semi-sitting, and the like. Robotic arm 506 thus allows the patient to be placed in the best position for a particular procedure. An example robotic arm 506 can be placed in the most unobstructed position when used in conjunction with the stereoscopic visualization camera 300. Thus, arm 506 and camera 300 are conveniently placed and oriented out of the way, while providing the surgeon with many possibilities for viewing location and orientation.

The arrangement of the links and joints of the robot arm 506 and/or coupling plate 3304 is typically motorized with six (or nine) degrees of freedom, with the link and joint configuration not unique to the camera pose. To enable a camera 300 to be positioned as desired. As discussed in more detail below, the joints and links of arm 506 and/or plate 3304 may be manually repositioned and/or reoriented without changing the pose or FOV of camera 300. This configuration allows, for example, to move the elbow joint out of the occluded line of sight without changing the view of the surgical site through camera 300. Additionally, the control system can identify the location and orientation of camera 300 and calculate and display alternative positions and/or orientations of robotic arm 506 to avoid occlusion of personnel or displays, for example. The use of various positions and/or orientations of the coupling plate 3304, along with the image processor's ability to flip, rotate, or otherwise reorient the displayed image, allows for even more positions and/or orientations of the robot arm 506. becomes possible.

Robot arm 506 and/or coupling plate 3304 are generally positioned and joints are positioned such that joint singularities are avoided with any general movement. Avoiding joint singularities provides better robot control of hysteresis and backlash. Additionally, the length and configuration of the links and joints of the robot arm 506 and/or coupling plate 3304 provide smooth movement along substantially any desired motion path. For example, repositioning and/or reorienting the robotic arm 506 may change the direction of the camera 300 view of a target point within the surgical site without changing focus, thereby allowing the surgeon to view the same point from a different direction/orientation. Make the target point visible. In another example, the robotic arm 506 can change the working distance to the target point without changing focus by translating the camera 300 along the line of sight toward or away from the target point. It is. Many similar motion paths can be achieved as desired using the robot arm 506 and/or coupling plate 3304 in conjunction with the stereoscopic visualization camera 300 of the stereoscopic robot platform 516.

B. Robot Control Embodiment The example robot arm 506 and/or coupling plate 3304 of FIGS. 34-40 may be controlled by one or more controllers. FIG. 41 illustrates an embodiment of the stereoscopic robot platform 516 of FIGS. 3-40, according to example embodiments of the present disclosure. An example stereoscopic robot platform 516 includes a stereoscopic visualization camera 300 and corresponding image capture module 1404 and motor and lighting module 1406 described in conjunction with FIGS. 14 and 15.

In the illustrated embodiment, the stereoscopic robot platform 516 includes a server or processor 4102 that is located remotely from the stereoscopic visualization camera 300. Processor 4102 is configured with one or more software programs defined, for example, by instructions stored in memory 1570 that, when executed by processor 4102, cause processor 4102 to perform the operations described herein. May include laptop computers, workstations, desktop computers, tablet computers, smartphones, etc. In this example, the example processor 4102 includes (or operates as described in connection with) the information processor module 1408, the image sensor controller 1502, and/or the motor and lighting controller 1520 of FIGS. 14-16. configured to run).

In some examples, at least some of the operations of image sensor controller 1502 and/or motor and lighting controller 1520 may be shared with image capture module 1404 and motor and lighting module 1406, respectively. For example, processor 4102 may generate commands to change focus, magnification, and/or working distance in a first portion of motor and lighting controller 1520 and a second portion of motor and lighting controller 1520 in motor and lighting module 1406. The drive devices 1534 to 1552 are controlled through the section . Additionally or alternatively, a first portion of the information processor module 1408 operably disposed in the processor 4102 may capture individual left/right images and/or stereoscopic images from a second portion of the information processor module 1408 within the image capture module 1404. is configured to receive data from the A first portion of the information processor module 1408 includes 1 Images may be configured to be processed for display on one or more display monitors 512 and/or 514.

Processor 4102 is electrically and/or communicatively coupled to image capture module 1404 and motor and lighting module 1406 of stereoscopic visualization camera 300 via wiring harness 4102 . In some embodiments, the wire harness 4102 may be within or routed through the robot arm. In other embodiments, the wire harness 4102 may be internal or routed through the robot arm. In yet other embodiments, image capture module 1404 and motor and lighting module 1406 may wirelessly communicate with processor 4102 via Bluetooth®, for example.

The example processor 4102 is also electrically and/or communicatively coupled to the sensor 3306 via the wiring harness 4102. Processor 4102 is configured to receive rotational output data and/or translational output data from sensor 3306, for example. The data may include digital data and/or analog signals. In some embodiments, processor 4102 receives a substantially continuous stream of output data from sensor 3306 indicative of detected force and/or motion. In other examples, processor 4102 receives output data at regular sampling intervals. In yet another example, processor 4102 periodically sends a request message requesting output data.

In the illustrated example, the processor 4102 includes a display monitor 512, input devices 1410a, 1410b, and other devices/systems 4104 (e.g., an X-ray machine, a computed tomography ("CT") machine, a magnetic resonance imaging ("MRI") ) a medical imaging device such as a machine, a camera, a workstation storing images or surgical guidance, etc.). Input device 1410a may include a touch screen device and input device 1410b may include a footswitch. Touch screen input device 1410a may be integrated with display monitor 512 and/or may be provided as a separate device in cart 510 of FIG. 5, for example. An example display monitor 512 is configured to display one or more user interfaces that include stereoscopic video (or separate two-dimensional left and right videos) of the target surgical site recorded by stereoscopic visualization camera 300.

Touch screen input device 1410a is configured to provide one or more user interfaces for receiving user input related to controlling stereoscopic visualization camera 300, coupling plate 3304, and/or robotic arm 506. Input device 1401a allows an operator to specify, set, or otherwise provide instructions to control working distance, focus, magnification, source and level of illumination, filters, and/or digital zoom of stereoscopic visualization camera 300. may include one or more graphical control buttons, sliders, etc. configured to do so. Input device 1410a may also include one or more surgical guidance graphics/text, videos, and/or images that allow the operator to focus and/or otherwise overlay the displayed stereoscopic video displayed on display monitor 512. It may also include multiple control buttons. Input device 1410a may also include a user interface configured to allow operator input or creation of surgical procedure visualization templates. Input device 1410a may include options to control operational parameters such as speed, motion, deployment/retraction, calibration, target lock, store view position, and/or change camera 300 orientation or input new orientation. One or more control buttons may further be included to control arm 506 and/or coupling plate 3304. User interface controls for robot arm 506 and/or coupling plate 3304 may include controls for moving camera 300, which are translated into commands for individual joints R1-R9. Additionally or alternatively, user interface controls for robot arm 506 and/or coupling plate 3304 may include controls for individually moving each of joints R1-R9. Input received via input device 1410a is sent to processor 4102 for processing.

An example footswitch input device 1410 may include, for example, a hood pedal configured to receive input to control the position of the stereoscopic visualization camera 300, the coupling plate 3304, and/or the robotic arm 506. For example, footplate input device 1410b may include controls to move camera 300 along the x-axis, y-axis, and/or z-axis. Footplate input device 1410b may also include controls to store the position of camera 300 and/or return a previously stored position. Footplate input device 1410b may further include controls to change focus, zoom, magnification, etc. of camera 300.

In other embodiments, the stereoscopic robot platform 516 may include additional and/or alternative input devices 1410, such as a joystick, mouse, or other similar 2D or 3D manual input device. Input device 1410 is configured to provide input similar to an XY panning device, with additional degrees of freedom resulting in flexibility of system movement. Input devices with 3D functionality, such as a 3D mouse or a controller with six degrees of freedom, are suitable for flexible and convenient input commands. The main advantage of these user-controlled devices is that surgical images can be easily viewed while motion is occurring. Additionally, the surgeon can see what is happening throughout the surgery and around the nearby site, for example, to avoid camera 300 hitting surgical staff and/or nearby equipment.

Optionally, input device 1410 may include a head-, eye-, or glasses-mounted tracking device, a voice recognition device, and/or a gesture input device. These types of input devices 1410 facilitate "hands-free" operability so that the operator does not have to touch anything with sterile gloves. Gesture recognition controls may be used in which surgical specific hand movements are recognized and translated into control signals to camera 300, coupling plate 3304, and/or robotic arm 506. Similar functionality is provided by voice recognition devices, where the microphone detects commands from an operator, such as "move camera to the left", recognizes the utterance as a command, and directs it to the appropriate camera and/or robot control. Convert to signal. Alternative embodiments include an eye tracking device configured to locate the operator's eyes relative to the 3D display and adjust the view depending on where the operator is looking in the displayed scene. .

Other embodiments include positioning the operator's head in the frame of reference (e.g., via a trackable target or set of targets mounted on the operator's 3D glasses) and a footswitch that activates "head tracking." including devices configured to track. One example tracking input device is configured to store the starting position of the operator's head during actuation and then continuously detect the head position at some short time interval. The tracking input device, in conjunction with the processor 4102, may calculate a motion delta vector between the current position and the starting position and convert the vector into a corresponding robot arm or camera lens movement. For example, tracking input device 1410 and processor 4102 may convert left/right head movements into robot arm movements such that an image on the screen moves left/right. Tracking input device 1410 and processor 4102 also convert up/down head movements into robot arm or camera lens movements such that the image on the screen moves up/down, and the image on the screen zooms in/out. Forward/occipital head movements can also be translated into robot arm or camera lens movements, as shown in FIG. Other movement transformations are also possible, for example, by converting head rotation into a “target locking” movement of the robot arm 506. As described herein, the target lock maintains the focus of the robot platform 516 at the same point within the scene or FOV within some tolerance and moves the robot arm in a direction similar to operator head movement. 506 (and thus the view).

Before a particular surgical procedure, a surgical plan is created that establishes the desired path for equipment and visualization. In some embodiments, input device 1410 is configured to follow such predetermined path with substantially no further input from the operator. Thus, the operator can continue to operate while the view of the surgical site changes automatically in a pre-planned manner. In some embodiments, the surgical plan may include a set of preplanned waypoints that correspond to camera position, magnification, focus, etc. The operator may actuate input device 1410 to advance through waypoints (to cause processor 4102 to move robotic arm 506, coupling plate 3304, and/or camera 300 as planned) as the surgical procedure progresses.

In the illustrated embodiment, example sensor 3306 is an input device. Sensor 3306 is configured to detect operator movement or force on stereoscopic visualization camera 300 and convert the detected force/movement into rotational and/or translational data. The sensor 3306 is a motion anticipator, such as a six degree of freedom tactile sensing module or an optical sensor (e.g., a force/torque sensor) that allows the robot arm 506 to electromechanically respond to gentle operator pushes on the camera 300. May include an input device. The optical sensor converts the applied force and/or torque into an electrical signal, thereby detecting the desired force/torque input by the operator and converting it into a provided motion request in the sensed linear and/or rotational direction. may include an electro-optical device configured to enable. In other embodiments, other sensor types may be used for sensor 3306. For example, sensor 3306 may include a strain gauge or piezoelectric device configured to sense tactile requests from an operator.

In embodiments, the surgeon holds one or more of the control arms 304 or presses a release button (which may be located on one or both of the control arms 304). Upon actuation of the release button, the camera 300 sends a message to the processor 4102 indicating that the operator desires to enter "assisted mobility" mode. Processor 4102 configures robotic arm 506 and/or coupling plate 3304 to allow the surgeon to gently maneuver camera 300 in a desired direction. During this movement, the processor 4102 "power steers" the camera 300 to the robot arm 506 and/or coupling plate to safely support its weight, determine which joints to activate, and which joints to brake. Automatically determines force in a coordinated manner to achieve the surgeon's desired movement.

In the illustrated example, the stereoscopic robotic platform 516 of FIG. 41 includes a robotic arm controller 4106 configured to control the robotic arm 506 and/or the coupling plate 3304. Robot arm controller 4106 is configured to convert one or more messages or instructions from processor 4102 into one or more messages and/or signals that cause rotation of any one of joints R1-R9. May include processors, servers, microcontrollers, workstations, etc. Robot arm controller 4106 is also configured to receive sensor information, such as joint position and/or velocity, from robot arm 506 and/or coupling plate 3304 and convert it into one or more messages to processor 4102. be done.

In some embodiments, robotic arm controller 4106 is configured as a standalone module disposed between processor 4102 and robotic arm 506. In other embodiments, robotic arm controller 4106 may be included within robotic arm 506. In yet other embodiments, robotic arm controller 4106 may be included with processor 4102.

An example robot arm controller 4106 includes one or more instructions stored in memory 4120 that are executable by a robot processor 4122. The instructions may be configured into one or more software programs, algorithms, and/or routines. Memory 4120 may include any type of volatile or non-volatile memory. An example robot processor 4122 is communicatively coupled to processor 4102 and configured to receive one or more messages related to operation of robot arm 506 and/or coupling plate 3304. The example robot processor 4120 is also configured to send one or more messages to the processor 4102 indicating the positions and/or velocities of joints R1-R9. The one or more messages may also indicate that the joint has reached a movement stop or that movement is prevented.

The example processor 4120 is configured to determine which joints R1-R9 are powered such that all motions of all joints are adjusted to collectively produce the desired imaging motion in the camera 300. . In the "move camera to the left" example, there may be complex movements of several joints that make the surgical image of the camera appear to be a simple and smooth translation to the left from the surgeon's relative perspective. In the "Move Camera Left" example, depending on how the camera 300 is connected to the robot arm 506 through the coupling plate 3304, the control signals to a particular joint can vary dramatically depending on position/orientation. Please note that.

Memory 4120 may include one or more instructions that specify how joints R1-R9 move based on the known positions of the joints. Robotic arm controller 4106 is configured to execute one or more instructions that determine how commanded camera movements are translated into joint movements. In one example, robotic arm controller 4106 may receive a message from processor 4102 indicating that stereoscopic visualization camera 300 should move downward along the z-axis and sideways in the xy plane. In other words, the processor 4102 transmits an input indicating the input received via the input device 1410 regarding the desired movement of the camera 300. An example robot arm controller 4106 is configured to convert movement vectors in three-dimensional coordinates into joint position movement information that achieves a desired position/orientation. The robot arm controller 4106 determines the movement delta vector by identifying or taking into account the current positions of the links and joints of the robot arm 506 and/or the coupling plate 3304 (and/or the position/orientation of the camera 300) in conjunction with the desired movement. can be determined. In addition, robot arm controller 4106 performs one or more checks to ensure that the desired movement causes camera 300 to move in one or more three dimensions defined in the same coordinate system as arm 506 and coupling plate 3304. It can be guaranteed not to enter or approach the restricted area specified by the boundary. The area near the boundary may specify a reduced scaling factor provided by the robot arm controller 4106 when a movement signal is sent to the joint, and the reduced scaling factor may be applied when the robot arm 506 approaches the boundary. Move the joint more slowly and avoid moving it beyond its boundaries.

After the bounds check is performed, the robot arm controller 4106 uses the translation delta and the current position/orientation of each joint R1-R9 to rotate one or more of the joints to direct the robot arm 506 to the camera 300. Determining an optimal or near-optimal movement sequence to move to a location. Robot arm controller 4106 may, for example, use an optimization routine to determine the minimum amount of joint movement necessary to satisfy the movement delta vector. After the amount of joint movement is determined, the example robot arm controller 4106 may send one or more messages to the motor controller 4124 indicating the amount and speed of rotation taking into account any scaling factors. configured. Robot arm controller 4106 may send a series of messages that move robot arm 506 and/or coupling plate 3304 in a defined or coordinated sequence. The series of messages can also cause joint velocities to change, for example, when the robot arm 506 approaches a virtual or physical boundary.

One example motor controller 4124 is configured to translate or convert the received messages into analog signals, such as pulse width modulated (“PWM”) signals that rotate one or more of joints R1-R9. The motor controller 4124 may, for example, select the input lines to the appropriate joint motor, the pulse duration used to control the amount of time the motor rotates, and the frequency, duty cycle, and/or amplitude of the pulses determined by the rotational speed. used for control. Motor controller 4124 may also provide power to the joint motors and corresponding joint sensors.

In some embodiments, the robot arm controller 4106, in conjunction with the motor controller 4124, is configured to receive or read joint sensor position information and determine the location and orientation of the robot joint and camera 300 through kinematics. Each joint R1-R9 may include at least one sensor that detects and transmits data indicative of joint position, joint rotational speed, and/or joint rotational direction. In some embodiments, the sensor only transmits position information and the velocity/direction is determined by the robot arm controller 4106 based on the difference in position information over time. Robotic arm controller 4106 may send sensor data to processor 4102 to determine movement information.

Robot arm controller 4106 receives movement instructions from processor 4102 and determines which motors and joints to activate, speed and distance, and direction through Jacobian, forward kinematics, and/or inverse kinematics. Robot arm controller 4106 then sends appropriate command signals to motor power amplifiers in motor controller 4124 to drive the joint motors in robot arm 506.

An example robotic arm 506 receives appropriate motor power signals and moves accordingly. Sensors and brakes on arm 506 are responsive to various motion and feedback information from robot arm controller 4106. In some embodiments, the robotic arm 506 is mechanically and communicatively connected to a coupling plate 3304 that transmits coupler status and orientation information to the robotic arm controller 4106.

In some embodiments, the example robotic arm 506 of FIG. 41 includes a coupler controller 4130. The example combiner controller 4130 is configured to bypass the robot processor 4106 and relay control information between the processor 4102 and the combiner plate 3304. Coupler controller 4130 may receive messages from processor 4102 and rotate joints R7-R9 on coupling plate 3304 in response. Coupler controller 4130 may also receive sensor information regarding joint position and/or velocity and send one or more messages to processor 4102 indicating the joint position and/or velocity. In these embodiments, processor 4102 may send messages to control robotic arm 506 and separate messages for coupling plate 3304.

In some embodiments, robot arm controller 4106 is configured to determine how joints R7-R9 should move. However, if coupling plate 3304 is not communicatively coupled directly to robot arm 506, robot processor 4106 may send movement signals to coupler controller 4130 via processor 4102. If at least some operators of robot processor 4106 are co-located with processor 4102, combiner controller 4130 sends movement commands or signals to processor 4102 in conjunction with robot arm 506 receiving movement commands or signals from processor 4102. 4102.

In the embodiment shown in FIG. 41, the example stereoscopic visualization camera 300, processor 4102, coupling plate 3304, robot arm 506, robot arm controller 4106, and/or input device 1410 receive power via an input power module 4140. Receive. One example module 4140 includes an isolation transformer to avoid disturbance of system performance due to power supply (such as power from a wall outlet) and/or power line anomalies. In some cases, the power source can include a battery power source.

The stereoscopic visualization platform 516 may also include an emergency shut-off switch 4142 configured to immediately shut off power. Switch 4142 may only cut off power to robot arm 506 and/or coupling plate 3304. Processor 4106 may detect activation of emergency stop switch 4142 and engage the joint brake to prevent robot arm 506 from falling. In some cases, robotic arm 506 is configured to activate joint brakes after detecting power loss. In some embodiments, joints R1-R6 of the robot arm 506 slip if a force above a threshold is applied, allowing an operator to move the arm out of the way with or without power in an emergency. It is constructed so that it can be moved quickly.

In some embodiments, the example processor 4102 is configured to display one or more graphical representations of the robotic arm 506, the coupling plate 3304, and/or the stereoscopic visualization camera 300. Processor 4102 may cause graphical representations to be displayed on one or more user interfaces that provide control of robotic arm 506, coupling plate 3304, and/or camera 300. Processor 4102 may position and orient the graphical representation to reflect the current position of robotic arm 506, coupling plate 3304, and/or camera. Processor 4102 uses, for example, feedback messages from robot arm controller 4106 to determine which joints in the graphical representation to rotate, thereby changing the orientation and/or position of the display device. In some cases, processor 4102 is configured to receive user input via the graphical representation, such as by an operator moving a link, joint, or camera 300 to a desired position in the graphical representation. If stereo visualization camera 300 moves, processor 4102 may send new coordinates corresponding to the location to which the camera moved. If the joint or link moves, processor 4102 may send a message to robotic arm controller 4106 indicating the rotation of the joint and/or the position of the link.

In some embodiments, the processor 4102 operates in conjunction with the robot arm controller 4106 to perform operations based on or in cooperation with the movement of the robot arm 506 and/or the coupling plate 3304. and operates to adjust one or more lenses of the camera. For example, as the robotic arm 506 moves toward the surgical site, the processor 4102 operates in conjunction with the robotic arm controller 4106 to move one or more of the lenses of the stereoscopic visualization camera 300 to maintain focus. to change the working distance or focus. Processor 4102 operates in conjunction with robot arm controller 4106 to determine, for example, that movement of robot arm 506 will reduce the working distance. EPU processor 4102 operates in conjunction with robot arm controller 4106 to determine a new position of the lens based on the new working distance established by robot arm 506 movement. This may include moving one or more lenses to adjust focus. In some embodiments, processor 4102 may instruct camera 300 to run a calibration routine with respect to the new position of robotic arm 506 to eliminate pseudo-parallax, for example.

In some cases, the operator is changing the position of one or more lenses of the stereoscopic visualization camera 300 and a lens movement limit may be reached. The position of the lens is sent from camera 300 to processor 4102, which determines that the limit has been reached. After detecting that the limit has been reached, processor 4102 moves robot arm 506 (via controller 4106) based on input from the operator, thereby extending the command from lens movement to arm movement. A desired magnification or target area can be reached. Thus, the processor 4102 working in conjunction with the robot arm controller 4106 allows the operator to use only one user interface rather than changing between the robot arm interface and the camera. Processor 4102 and/or controller 4106 checking the desired movement against any predetermined movement limits to ensure that movement does not cause camera 300 or robotic arm 506 to enter restricted patient or operator space. I want you to understand. If a limit violation is detected, the processor 4102, in conjunction with the robot arm controller 4106, may display an alert to the operator indicating the limit and the reason why the robot arm 506 has stopped (touch screen input device 1410a and/or display). (via a user interface displayed on monitor 512).

C. Robotic Arm and Stationary Camera Calibration Embodiment As mentioned above, the example stereoscopic visualization camera 300 is configured to provide high-resolution stereoscopic video images of a target surgical site at various magnifications. As part of the stereoscopic visualization platform 516, the stereoscopic visualization camera 300 operates in conjunction with the robot arm 506 and/or the coupling plate 3304 to precisely and clearly change image focus, working distance, magnification, etc. To achieve image acquisition flexibility, stereoscopic visualization platform 516 is configured to run one or more calibration, initialization, and/or reset routines. In some embodiments, the stereoscopic visualization camera 300, robotic arm 506, coupling plate 3304, or more generally the stereoscopic visualization platform 516 is calibrated during manufacturing and/or after installation. Calibration of camera 300 with robot arm 506 provides positioning information for camera 300 with respect to robot arm 506 and operator space. In some embodiments, after powering up the stereoscopic visualization platform 516, the camera 300 and/or the robotic arm 506 perform further calibration/initialization to determine the current location and orientation of the camera 300. Configured to verify.

The example processor 4102 is configured to store results from the calibration (eg, calibration data), for example, in memory 1570 and/or memory 4120 of FIG. 41. Calibration results may be stored in calibration registers and/or look-up tables (“LUTs”) within memory 1570 and/or 4120. The stored calibration data provides optical, functional, and/or performance characteristics of the camera 300, robotic arm 506, and/or coupling plate 3304 that are adjustable, measurable, and/or verifiable by an operator or processor 4102. associated with or mapped to the attributes of For example, the working distance actuator motor encoder position of main objective lens assembly 702 (of FIG. 7) is mapped to the working distance in the LUT. In another example, the zoom lens axis position along the linear encoder of zoom lens assembly 716 is mapped to a magnification level in the LUT. For each of these examples, the example processor 4102 is configured to determine the appropriate level of the encoder characteristic, adjust it, and verify that the characteristic provides the specified or desired working distance and/or magnification. . In some embodiments, the LUTs may be multiple, in which multiple performance characteristics are determined by multiple performance characteristics of platform 516 for overall control of all relevant aspects of camera 300, robot arm 506, and/or coupling plate 3304. mapped to an attribute.

The combination of the robotic arm 506 and the example stereoscopic visualization camera 300 provides highly accurate position, orientation, and/or orientation information with respect to the reference frame of the robotic arm 506. The following section describes how stereoscopic visualization camera 300 is calibrated to define visual hints. After the visual cues are determined, the stereoscopic visualization camera 300 is registered to the reference frame of the robot arm 506 and/or the coupling plate 3306. Thus, after calibration and registration, a stereoscopic view of the surgical site is unified using integrated control of the stereoscopic visualization camera 300 in combination with control of the position, orientation, and orientation of the robotic arm 506 and/or the coupling plate 3304. .

In some embodiments, the example processor 4102 of FIG. 41 precisely integrates the registration of the stereoscopic visualization camera 300 with the position, direction, and/or orientation calibration of the robot arm 506, including visual cues. , configured to define a unified position, direction, and/or orientation perception of the acquired stereoscopic images and all points therebetween with respect to a defined coordinate system. The example processor 4102 uses the essential visual imaging data from the stereoscopic visualization camera 300 to adjust or direct the physical positioning and/or orientation of the robotic arm 506 as desired by the operator. Configured to provide visualization. Additionally, such instructions and adjustments provided by processor 4102 are provided to maintain desirable characteristics of the visualization, such as focus, working distance, predefined position, orientation, etc.

In some embodiments, the calibration of stereoscopic visualization camera 300, robotic arm 506, and/or coupling plate 3304 generally addresses (i) inaccuracies in functional parameters of stereoscopic visualization platform 516 that affect stereoscopic images; (ii) calibrating or adjusting the stereoscopic visualization platform 516 to minimize stereoscopic image inaccuracies below a desired level; (iii) stereoscopic images relative to each other or with a calibration template; (iv) using the stereoscopic visualization platform 516 to perform the task, the accuracy of the calibration being verified through simultaneous dual-channel comparison of the images; Levels are detectable and maintained, including using.

In an alternative embodiment, the robot arm 506 is provided with precise calibration of the physical relationships of the joints and links of the robot arm 506 to each other, as well as visual cues of the camera 300 with the robot arm 506 and/or the initial pose configuration. One or more fixed calibration standards are provided that are used to calibrate the relationship. The robotic platform calibration reference can be used to register or integrate the robotic arm 506 with an external environment, such as an operating room, or with a patient or target space within the external environment. The fixed calibration reference may include a dedicated attachment to an external portion of the body of the robot arm 506 or a combination of known external features of the robot arm 506 such as mounting points, joints, corners, etc.

1. Stereo Visualization Camera Calibration Embodiments To match stereo views of a surgical site, the example processor 4102 and/or stereo visualization camera 300 are configured to perform one or more calibration routines. One example routine includes one or more routines stored in memory 1570 that, when executed by processor 4102, cause processor 4102 to determine a lens position corresponding to a particular working distance, magnification (e.g., zoom level), and focus level. It can be specified by the command. The instructions may also cause processor 4102 to operate one or more routines that identify the projection center, stereoscopic optical axis, and/or ZRP of stereoscopic visualization camera 300 at different working distances and/or magnifications. Calibration allows, for example, processor 4102 to remain focused on the target surgical site when magnification and/or working distance changes.

FIG. 42 illustrates an example procedure 4200 or routine for calibrating stereoscopic visualization camera 300, according to example embodiments of the present disclosure. Although procedure 4200 will be described with reference to the flow diagram shown in FIG. 42, it should be understood that many other methods of performing the steps associated with procedure 4200 can be used. For example, the order of many of the blocks may be changed, certain blocks may be combined with other blocks, and many of the blocks described are optional. Additionally, the operations described in step 4200 may be performed on multiple devices, including, for example, optical element 1402, image capture module 1404, motor and lighting module 1406, and/or information processor module 1408 of example stereoscopic visualization camera 300. It can be executed between For example, procedure 4200 may be performed by one of the programs 1560 of information processor module 1408.

The example methodology 4200 begins when the stereoscopic visualization camera 300 is powered on (block 4202) or otherwise initialized. Camera 300 may be mounted on robot arm 506. Alternatively, procedure 4200 can be performed when stereoscopic visualization camera 300 is connected to a fixed mount. The example methodology 4200 then performs ZRP alignment (block 4204), as described above with respect to FIGS. 25 and 26. An example processor 4102 may automatically align the ZRP as described above and/or operate in conjunction with an operator to provide alignment of the left and right optical paths on the image sensors 746, 748. In some examples, the processor 4102 and/or the operator may operate the motor with sufficient precision to make very small adjustments to the tilt of the lens component to move the ZRP into alignment with the pixel grid origin. A small movement or bending of the bend (bend 1300 in FIG. 13) may occur through the bend. During semi-manual registration, processor 4102 may cause left and right images from image sensors 746 and 748 to overlap on display monitor 512. The operator may adjust the image using input device 1410 and move the pixel sets of sensors 746 and 748 accordingly until the ZRP is properly aligned.

During alignment, the ZRP is aligned to the image center and set to avoid pseudo parallax. Alignment to within about 1 pixel on the display is possible. The degree of alignment from the left and right views to the image center is visible in the superimposed images, including during zoom operations. In the 8° FOV example, the use of the 4K image sensors 746, 748 and the corresponding 4K display resolution of the display monitor 512 (which includes approximately 4000 pixels x approximately 2000 lines) results in a system resolution of 8°/4000 pixels = 7 arc seconds. generate. However, ZRP registration can be performed at almost any magnification, where the resolution (same number of pixels (eg, 4000)) is divided by the reduced (or increased) FOV angle. For example, an example embodiment of camera 300 at high magnification produces a FOV angle of approximately 2 degrees. 8K UHD display monitor 512 and sensors 746 and 748 have approximately 8000 pixels in approximately 4000 rows. The resolution of this system is 2°/8000 pixels = 1 arcsec. This is about an order of magnitude better than known systems of the prior art, with accuracy of assembled, individually measured components, with tolerances having resolution measured in minutes. Because image sensors and display monitors generally become denser with smaller pixels in the same physical sensor or display space, the accuracy adjustability of stereoscopic visualization camera 300 scales to smaller pixel sizes. The enhanced precision alignment of the stereoscopic visualization camera 300 provides better and more accurate digital effects.

ZRP registration is complete when the left and right images remain centered within the desired tolerance and the image of the target surgical site is accurate when cycling from low to high magnification. After the ZRPs are aligned throughout the magnification capabilities of stereoscopic visualization camera 300, the pixel set locations and/or lens locations for each magnification level are stored, for example, in LUT 4203 or other data structure. In other examples, processor 4102 writes pixel set locations and/or lens locations for each magnification level to a calibration register.

After the ZRP is aligned, the example processor 4102 is configured to calibrate (block 4206) for working distance and/or magnification (eg, zoom). As discussed above in connection with the working distance and zoom example of FIG. 15, precise knowledge of the working distance is important in camera 300 so that robot arm 506 can precisely position the camera relative to the desired coordinates. be. In some cases, precise fiducials are used in conjunction with the mechanical dimensions of the robot arm 506 to transform the object plane data from the images into a coordinate system indicative of the stereoscopic visualization platform 516, referred to herein as robot space. Ru.

An example calibration procedure 4200 is performed to map the working distance of the optical system of the stereoscopic visualization camera 300, which is a common mode objective ("CMO") lens assembly (e.g., in front of FIGS. 4 and 7). The working distance from the main objective lens 408) to the object plane can be calculated or measured in millimeters. The working distance is mapped to a known measurable parameter, such as a focus motor position measured in counts of a motor shaft encoding device from a known "home" location, such as a physical stop or limit switch trigger position.

Calibration at block 4206 involves processor 4102 sequentially moving the object plane in discrete steps along the optical axis and refocusing the image while recording encoder counts and working distance, as discussed in more detail in conjunction with FIG. It is executed by Processor 4102 externally measures the working distance from the front of the CMO. The mapping of encoder counts and working distances is stored in LUT 4203 or in different LUTs and/or calibration registers. This calibration allows the processor 4102 to output encoder count positions to the motor controller given the desired working distance. The exemplary embodiment of the stereoscopic visualization camera 300 uses a high revolution count shaft encoding device, with working distance resolution on the order of 1 μm for each encoder count. Alternative embodiments may include different encoder resolutions that provide higher or lower working distance resolution as desired.

FIG. 43 illustrates an example stereoscopic visualization camera 300 embodiment that moves an object plane in discrete steps, according to example embodiments of the present disclosure. One example stereoscopic visualization camera 300 includes a main objective lens assembly 702 (eg, single CMO) of FIG. 7, which is configured to provide left and right views of the target surgical site. In the illustrated example, the main objective lens assembly 702 is shown as an achromatic lens refractive assembly having a stationary front working distance lens 408 within a housing 4302 and a movable rear working distance lens 740, with a movable rear working distance lens 740. Lens 740 is movable along the z-axis (or other optical axis). Movement of the rear working distance 704 changes the distance to the front working distance lens 408. The spacing between lenses 408 and 704 determines the overall front focal length 4304 of main objective lens assembly 702, and thus the location of the front focal plane 4306 (or simply "focal plane"). Front focal plane 4306 is positioned at a distance equal to focal length 4304 from main surface 4308 of main objective lens assembly 702 . Measuring the location of major surface 4308 can be difficult, so the distance from the bottom of housing 4302 to the front focal plane is defined as working distance 4310. Thus, working distance 4310 accurately establishes the plane of the target region or scene that is in focus.

Imaging the object at the front focal plane 4306 creates a series of images positioned at infinity behind or behind the main objective lens assembly 702 . Two parallel optical paths, including optics of camera 300 and sensors 714, 716, 718, 744R, and 744L, separated laterally by an interpupillary distance (“IPD”) 4312, provide light from main objective lens assembly 702. Left and right views are generated along left optical axis 4320 and right optical axis 4322 in slightly different directions from axis 4324, respectively. The two optical paths are adjusted such that their respective external left and right axes of convergence are set to coincide with the center of the FOV of image 4330. This point 4330 is referred to herein as the "tip" of the stereoscopic visualization camera 300 at the front focal plane 4306.

Adjusting the position of the rear working distance lens 740 causes the main objective lens assembly 702 front focal length 4304 to change. As shown, the repositioning of the rear working distance lens 740 creates a new working distance 4310' located at the new front focal plane 4306' location. Movement of rear working distance lens 740 also realigns left optical axis 4320' and right optical axis 4322', resulting in repositioning tip 4330' of camera 300. Visualizing an object with camera 300 above or below the focal plane 4306 reduces focus of the object.

Similar to working distance calibration, processor 4102 can construct similar LUTs or additional columns in working distance LUT 4203 by varying the magnification while measuring the image height of an object of known size. . Multipliers can be quantified by determining the counts of a motor shaft encoding device from a known "home" location, such as a physical stop or limit switch trigger position. Processor 4102 can measure image height relative to, for example, the number of sensor pixels at each magnification position. Magnification can be characterized, for example, by dividing the height in pixels by the height in millimeters.

Returning to FIG. 42, after the working distance and magnification of the stereoscopic visualization camera 300 are calibrated, the example processor 4102 is configured to identify a projection center (block 4208). The center of projection (eg, COP) may be identified using one or more routines modeling the stereoscopic visualization camera 300, as described above in connection with FIG. 15. Modeling the physical camera 300 using a mathematical model, often implemented in software, firmware, hardware, and/or GPU code of the processor 4102, to match left and right stereoscopic views of the surgical site. is desirable. Views of 3D computer models, such as MRI images of brain tumors, are often rendered and viewed from user-adjustable directions and distances (eg, as if the images were captured by a synthetic stereoscopic camera). Processor 4102 may use the tunability of the model to match the perspective of the live surgical image; therefore, the perspective of the live surgical image must be known.

The exemplary embodiments of stereoscopic visualization camera 300 and/or processor 4102 are configured to accurately measure and calculate camera model parameters at each value of magnification and working distance. These values are controlled by a separate optical system contained within the stereoscopic visualization camera 300. The dual optics are aligned such that the parallax at the center of the image between the left and right channels/views is approximately zero at the focal plane 4330. Furthermore, the stereoscopic visualization camera 300 is parfocal over a range of magnifications and par central over a range of magnifications and working distances because the ZRP of each left and right channel is located at the center of each pixel grid. (described above in block 4202). In other words, if only the magnification is changed, the image remains in focus in both channels and is trained on the same center point. Similarly, changing only the working distance should result in no vertical disparity in the images if the target and stereoscopic visualization camera 300 remain stationary, and only increase the horizontal disparity between the left and right views.

FIG. 44 depicts a graph 4400 illustrating a routine executable by processor 4102 to identify the COP of stereoscopic visualization camera 300, according to example embodiments of the present disclosure. In the illustrated example, the COP of the pinhole or modeled camera 300 is along the optical axis 4402 at the plane (O) of the pinhole. To identify the COP of the camera model, a virtual pinhole camera model is used, where the processor 4102 is configured to identify the actual focal length 4404 from the COP to the object plane. During the calibration routine, processor 4102 determines, for example, the plane of optical image sensor 744 with an object of height 4408 at three different distances along optical axis 4402, the object plane, a distance “d” less than the object plane distance, and The magnification of the camera 300 is maintained while a measurement of the image height 4406 in pixels at a distance "d" above the object plane distance is recorded. Processor 4102 uses a routine that includes algebra to identify focal length 4404 to COP 4410 based on similar triangles at the two most extreme positions. Processor 4102 may identify the focal length at the alternate magnification based on the ratio of the alternate magnification to the magnification used for calibration.

Returning to FIG. 42, the example processor 4102 is configured to identify COPs at various working distances and magnifications. Processor 4102 associates the lens motor shaft encoder counts in LUT 4203, different LUTs, or one or more calibration registers to COPs at various working distances and magnifications. In some embodiments, processor 4102 may only store COP relationships for one magnification and/or working distance and use one known COP relationship to calculate other magnification and/or working distances. .

After calibrating for the COP, the example processor 4102 is configured to calibrate the stereo left and right optical axes and the interpupillary distance (“IPD”) between the axes of the stereoscopic visualization camera 300 (block 4210). To characterize the optical system of the stereoscopic visualization camera 300, the IPD between the left and right channels/views should be known. In embodiments, the IPD may be designed into a mechanical component that holds the sensor and optics shown in FIG. Therefore, the IPD is set mechanically. However, the actual optical axis may be different from the mechanical axis of the optical element and the optical element's mount. In other embodiments, the IPD can be changed within the stereoscopic visualization camera 300.

In some applications, it is desirable to precisely know the orientation of the stereoscopic optical axis relative to a reference or mechanical axis in the system of stereoscopic visualization camera 300. This allows, for example, processor 4102 to precisely aim stereoscopic visualization camera 300 through mechanical means. Aiming can be characterized by a view vector defined geometrically with respect to the reference frame of the stereoscopic visualization camera 300 and simultaneously focusing on the stereoscopic optical axis. In addition, the clocking of the left and right channels of the optical sensor 744 that makes up the stereoscopic visualization camera 300 is included in the view vector.

FIG. 45 depicts a top view of an optical schematic illustrating how the IPD of stereoscopic visualization camera 300 may be measured and calibrated, according to example embodiments of the present disclosure. In the illustrated example, optical axis 4502 is perfectly aligned with mechanical axis 4504. Right image sensor 746 and left image sensor 748 (approximated by one or more camera models) are separated by IPD 4506. Sensors 746 and 748 are aligned and focused on object 4508 (within the target surgical site). Object 4508 is positioned at a focal distance 4510 from sensors 746 and 748 such that the parallax in the object plane is theoretically zero, as shown in the left or right view representation of the object at focal plane 4512. In this illustrative example, object 4508 is a disc, the front view of which is shown as item 4514, for clarity.

FIG. 45 also shows another example in which object 4508 is displaced a distance "d" along the mechanical axis and is designated as item 4508'. The displacement of the object 4508 creates a parallax that appears as P _L in the left view 4520 display and P _R in the right view 4522 display. In this example, the mechanical and optical axes are coincident and the magnitude of the parallax is equal. Disparity can be measured, for example, by counting the number of pixels of difference between left view 4520 and right view 4522 and multiplying by the magnification factor pixel/mm specified in the COP calibration step. Processor 4102 may calculate the IPD using triangulation. The measurement accuracy of the displacement distance d and parallax of each view contributes to precise knowledge of the IPD of the stereoscopic visualization camera 300.

FIG. 46 shows a top view of an optical schematic illustrating how the optical axis of stereoscopic visualization camera 300 can be measured and calibrated, according to example embodiments of the present disclosure. In this example, optical axis 4502 is offset from mechanical axis 4504 by an angle (α) shown as 4602. The right image sensor 746 and the left image sensor 748 (approximated by one or more camera models) are configured to detect the parallax in the plane of the object 4508 as shown in the representation of the left or right view of the object 4508 at the focal plane 4604. The object 4508 (within the target surgical site) is aligned and focused at a focal length such that 0 is theoretically zero.

FIG. 46 also shows another example in which object 4508 is displaced a distance "d" along the mechanical axis and is designated as object 4508'. The displacement of the object 4508 creates a parallax that appears as P _L in the left view 4610 display and as P _R in the right view 4612 display. In this example, where mechanical axis 4504 and optical axis 4502 do not coincide, the magnitude of the parallax is unequal. The example processor 4102 is configured to calculate the IPD and misalignment angle α (eg, stereo optical axis) via triangulation. The accuracy of measuring the displacement distance d and disparity of each view allows the processor 4102 to accurately determine the IPD and optical axis of the stereoscopic visualization camera 300.

A similar procedure can also be used, for example, to measure the misalignment of mechanical and optical axes in the vertical plane. For example, a combination of misalignments in the horizontal or vertical planes can be combined such that the view vector can be accurately estimated with respect to the mechanical axis. In some embodiments, IPD and optical axis parameters may be measured at various levels of working distance and/or magnification. The relationship between IPD, optical axis, working distance, and/or magnification may be stored by processor 4102 in LUT 4203, another LUT, and/or a calibration register.

Returning to FIG. 42, after the optical axis and/or IPD of the example stereoscopic visualization camera 300 is calibrated, the example processor 4102 completes the calibration process so that the camera 300 can be connected to the robotic arm 506. (block 4212). Procedure 4200 may then end. In some embodiments, at least a portion of the example procedure 4200 is repeated if the camera 300 is reinitialized and/or any calibration cannot be verified or confirmed.

It should be appreciated that in some embodiments, the above steps of procedure 4200 can be performed manually or semi-manually. In other embodiments, the above steps may be performed automatically and sequentially by processor 4200. In some embodiments, measurements can be made through image recognition of a suitable target or any target that has a sufficient number of objects with sufficient contrast to allow discrimination in both left and right views. . In addition, processor 4102 may identify or calculate parallax measurements to evaluate the precise relative positions of optical elements of stereoscopic visualization camera 300. Processor 4102 may perform optical measurements in real time.

In some embodiments, the use of automated iterative techniques to perform these or equivalent methods of calibration and measurement can increase accuracy and reduce the time and/or effort required for calibration and measurements. For example, the working distance (and thus the displacement d) is precisely known by the quantity of encoder counts and LUT 4203, as described above. The magnification, and for example its pixel/mm conversion factor, is also precisely known by the quantity of encoder counts and LUT 4203, as described above. Counting pixels in an image for difference or object size identification can be precisely performed manually or precisely automated, for example as described above using template matching. Measuring and storing these values can be combined to allow the example processor 4102 to accurately estimate stereoscopic camera model parameters and view vectors in near real time.

FIG. 47 shows a diagram of a calibrated stereoscopic visualization camera 300 whose optical parameters are fully characterized. In the illustrated embodiment, left and right optical axes are shown leading to imaginary left and right image sensor positions 4700, as identified via the camera model. FIG. 47 also shows a central stereoscopic optical axis or view vector 4702. The imaginary left and right view components of the camera model are positioned at focal length Z. Additionally, the left and right view components of the camera model are separated by the measured effective IPD. In the illustrated example, objects at the focal plane are viewed from a similar stereo perspective by imaginary left and right view components recorded by image sensors 746 and 748 within stereo visualization camera 300.

2. Calibration of Stereoscopic Visualization Enables Fusion with Additional Images The example processor 4102 not only provides high-resolution clear images, but also integrates live stereoscopic images with one or more images received from an external device 4104. / configured to also use the calibration parameters/information to align with the model. Mapping of calibration data associated with camera model parameters in LUT 4203 and/or calibration registers causes processor 4102 to generate a mathematical model of stereoscopic visualization camera 300 implemented in software, firmware, hardware, and/or computer code. can be created. In one example, processor 4102 is configured to receive, identify, or evaluate camera model parameters using, for example, procedure 4200 discussed in conjunction with FIG. 42. If calibration has already been performed, processor 4102 evaluates camera model parameters from one or more of memories 1570 and/or 4120. The processor 4102 also receives from the device 4104 image data of alternative modalities, such as preoperative images, MRI images, 3D models of the surgical site from MRI or CT data, X-ray images, and/or surgical guides/templates. . Processor 4102 is configured to render a composite stereoscopic image of alternative modality data using the camera model parameters. The example processor 4102 is also configured to provide the composite stereoscopic image for display via the monitor 512. In some examples, the processor 4102 is configured to fuse the composite stereoscopic images with the current stereoscopic visualization to show desired aspects of each modality from the same perspective as if acquired by one visualization device. can be viewed and/or superimposed.

In some embodiments, the parameters shown in FIG. 47 are used by processor 4102 to match synthetic stereoscopic images of alternative modalities, such as MRI image data, with the stereoscopic perspective of stereoscopic visualization camera 300. Accordingly, the example processor 4102 uses stored optical calibration parameters for stereoscopic image synthesis. In one example, processor 4102 uses optical calibration parameters to fuse a live stereoscopic image with a three-dimensional model of a brain tumor imaged preoperatively using an MRI device. The example processor 4102 uses optical calibration parameters to select a corresponding location, size, and/or orientation of a three-dimensional model of a brain tumor that matches the stereo image. In other words, processor 4102 selects a portion of the three-dimensional model that corresponds to the view recorded by stereoscopic visualization camera 300. Processor 4102 may also change which portions of the model are displayed based on detecting how the working distance, magnification, and/or orientation of camera 300 changes.

Processor 4102 may cause the graphical representation of the model to be superimposed on the stereoscopic image and/or the graphical representation of the model may be visually fused and displayed with the stereoscopic image. Image processing performed by processor 4102 may include smoothing the boundaries between the graphical representation of the model and the live stereoscopic view. Image processing may also include increasing the transparency of at least a portion of the graphical representation of the model so that the underlying live stereoscopic view is also visible to the surgeon.

In some examples, processor 4102 is configured to generate and/or render a depth map for every pixel in the stereo image. Processor 4102 may use the calibration parameters to determine tissue depth in the image, for example. Processor 4102 uses depth information for image recognition to note tissue of interest and/or to identify location of equipment to avoid unintentional contact when camera 300 mates with robotic arm 506. It is possible. Depth information is output by processor 4102 to, for example, a robotic suturing device, diagnostic equipment, a procedure monitoring and recording system, etc., for performing a coordinated at least semi-automated surgical procedure.

3. Robot Arm Calibration Embodiments The stereoscopic visualization camera 300 may be connected to the robot arm 506 and/or the coupling plate 3304 after being calibrated as described above. As discussed below, precise knowledge of the working distance relative to the focal length Z provided by the stored calibration parameters may be used by example processor 4102 and/or Used by robot processor 4122. The combination of the stereoscopic visualization camera 300 with the robotic arm is configured to provide seamless transitions between various working distances while maintaining focus or view of the target surgical site.

The robot arm 506 calibration procedure described below may be performed regardless of robot arm type. For example, the calibration procedure can be performed for articulated robotic systems that include mechanical links connected to each other via revolute joints, ranging from just one or two links and joints to joint structures that include six or more joints. It is possible. The calibration procedure can also be performed for a Cartesian robotic system with a gun tray with linear joints that uses a coordinate system with X, Y, and Z directions. The last joint of the Cartesian robot system may include a wrist type swivel joint. The calibration procedure may further be performed for a cylindrical coordinate robot system comprising a rotational joint at the base and one or more additional rotational and/or linear joints to form a cylindrical workspace. Additionally, the calibration procedure may be performed for a polar robot system comprising an arm connected to the base via a joint capable of operating in two or more rotational axes and further comprising one or more linear or wrist joints. . The calibration procedure may further be performed on a Selective Compliance Assembly Robot Arm (“SCARA”) system comprising a selective compliance arm operating in a primarily cylindrical shape that is used for assembly applications.

FIG. 48 illustrates an example procedure 4800 or routine for calibrating a robotic arm 506, according to example embodiments of the present disclosure. Although procedure 4800 will be described with reference to the flow diagram shown in FIG. 48, it should be understood that many other methods of performing the steps associated with procedure 4800 can be used. For example, the order of many of the blocks may be changed, certain blocks may be combined with other blocks, and many of the blocks described are optional. Additionally, the operations described in step 4800 may be performed, for example, on optical element 1402, image capture module 1404, motor and lighting module 1406, information processor module 1408, and/or joints R1-- of example stereoscopic visualization camera 300 of FIG. R9 and among multiple devices, including the robot arm controller 4106 of FIG. For example, procedure 4800 may be performed by a program stored in memory 4120 of robot arm controller 4106.

In some embodiments, coupling plate 3304 is connected to robotic arm 506 (block 4802). If the coupling plate 3304 is not used, the stereoscopic visualization camera 300 is connected directly to the robotic arm 506 connection or coupling interface 3450. If a bonding plate 3304 is used, the stereoscopic visualization camera 300 is connected to the bonding plate (block 4804). As mentioned above, the first end 3702 of the coupling plate 3304 is connected to the robot arm 506 and the second end 3704 of the coupling plate 3304 is connected to the stereoscopic visualization camera 300.

After the example stereoscopic visualization camera 300 is connected to the robot arm 506, the example processor 4102 and/or robot arm controller 4106 calibrates the camera and view vectors to a coordinate system originating around the stationary base 3404 of the robot arm 506. (block 4806). This coordinate system is referred to herein as "robotic space" or "robot space." During this calibration step, the known movements to the robot arm 506 are used by the processor 4102 and/or the robot arm controller 4106 to determine the orientation and location of the view vector and object plane of the camera 300 during visualization of the target surgical site. Identify.

In some embodiments, the mechanical features of camera 300, coupling plate 3304, and robotic arm 506, when mechanically connected together, create a unique relationship between camera 300, coupling plate 3304, and robotic arm 506. Exists as specified and known. In these embodiments, processor 4102 and/or robot arm controller 4106 determine the position, direction, and/or orientation of the view vector from the known mechanical geometry of camera 300, coupling plate 3304, and robot arm 506.

In other embodiments where no mathematical features are present, the example processor 4102 and/or robot arm controller 4106 may be configured to execute a routine that accurately identifies the spatial relationship between the camera 300 and the robot arm 506 in robot space. It is composed of The processor 4102 and/or the robotic arm controller 4106 move the stereoscopic visualization camera 300 to a starting position, which may include a retracted position, a reorientation position, or a surgical position. Stereo visualization camera 300 then moves the camera from the starting position to a position that generally visualizes a calibration target located on stationary base 3404 of robotic arm 506. The calibration target may be placed in a convenient area of the cart 510 at a location within the motion sphere of the robot arm 506, for example. Some examples of calibration targets include, for example, small spheres or other uniquely recognizable objects that can be placed relative to each other (in a two-dimensional or stereoscopic image) in a unique, known orientation. The coordinates of the sphere are fixed and known relative to the cart 510 and stationary base 3404, and therefore known in robot space. Processor 4102 and/or robot arm controller 4106 are configured to store coordinates in memory 4120, for example.

During calibration, processor 4102 and/or robot arm controller 4106 receives view vector data 4807 regarding working distance, magnification, stereo optical axis, and/or IPD. The stereoscopic visualization camera 300 is configured to simultaneously visualize the sphere at the calibration target and locate the sphere through the use of parallax in the stereoscopic image. Processor 4102 and/or robot arm controller 4106 record the position of the sphere in an initial coordinate system, e.g., X, Y, and Z, with respect to a reference at camera 300 (ie, "camera space"). The X, Y, Z location may correspond to an original location and may be defined in a file or LUT as having an origin or other known coordinate values. Processor 4102 and/or robot arm controller 4106 also use output data from joint sensors to identify the positions and orientations of joints and links in robot arm 506. Processor 4102 and/or robot arm controller 4106 also receive position information to determine the position and orientation of coupling device 3304. The position and orientation of robotic arm 506 and coupling device 3304 together enable processor 4102 and/or robotic arm controller 4106 to determine the pose of camera 300. Processor 4102 and/or robot arm controller 4106 performs coordinate transformations between camera space and robot space based on the location of the calibration target sphere recorded by the camera and the location of robot arm 506 and/or coupling plate 3304. configured to run. Processor 4102 and/or robot arm controller 4106 may store coordinate transformations in LUT 4203, a different LUT of robot arm 506, and/or one or more calibration registers.

In some embodiments, camera 300 moves to record images of multiple calibration targets located on cart 510, the ceiling, walls, and/or the surgical area. Each calibration target may have a unique orientation that allows its physical X, Y, Z location to be identified. Processor 4102 and/or robot arm controller 4106 perform additional coordinate transformations for each calibration target and store the transformations in one or more LUTs and/or registers.

In other embodiments, processor 4102 and/or robot arm controller 4106 may use alternative methods to calibrate camera 300 with respect to robot space. In this context, "calibration" is interpreted to mean "registration," where processor 4102 and/or robot arm controller 4106 are configured to calculate registration over a wide space in which registration may vary. It is composed of For example, a system may be used in which a separate stereoscopic camera is used to observe and locate a calibration target on cart 510 and camera 300 and/or a similar calibration target located on the patient or surgical bed. Processor 4102 and/or robotic arm controller 4106 are configured to model and track camera 300, where camera 300 is modeled and tracked as a surgical instrument along with a view vector and working distance. The view vector and working distance define parameters for accurately visualizing the target surgical site. In these other embodiments, other cameras identify and report coordinate system location and orientation information for each such device in a reference frame, such as stereoscopic camera 300. Using linear algebra, the pose and/or location of the instruments relative to each other is then calculated by processor 4102 and/or robot arm controller 4106, thereby calibrating camera 300 with respect to robot space. .

In some embodiments, processor 4102 and/or robot arm controller 4106 are also configured to calibrate with respect to coupling plate 3304. In some cases, coupling plate 3304 includes one or more switches that activate depending on the position of joints R7-R9. The known position of the switch is used by processor 4102 and/or robot arm controller 4106 as part of the coordinate transformation. Additionally or alternatively, coupling plate 3304 is calibrated by moving and orienting robotic arm 506 while images from camera 300 are monitored. In the example where coupling plate 3304 is oriented as shown in FIG. 37, robotic arm 506 is commanded to move in a direction relative to the assumed orientation (e.g., move camera 300 along the z-axis). . If the assumed orientation is that shown in FIG. 37 with camera 300 aimed downward, the object in the image should become larger as camera 300 approaches due to the downward movement of robot arm 506. For example, if the object in the image instead moves sideways or up/down, processor 4102 and/or robot arm controller 4106 may detect the movement and determine that the assumed orientation is incorrect. configured. Processor 4102 and/or robot arm controller 4106 may generate an error, prompt the operator for the correct orientation, and/or determine the correct orientation based on movement detected in the image. As mentioned above, changes in the image from movement of camera 300 are automatically decoded, for example, through the use of image template matching algorithms. In some embodiments, use of a template matching algorithm by processor 4102 and/or robot arm controller 4106 identifies the orientation of the joints in bonding plate 3304, which is stored in a LUT for calibration.

FIG. 49 shows a diagram illustrating how stereoscopic visualization camera 300 and/or robot arm 506 are calibrated to robot space, according to example embodiments of the present disclosure. In the illustrated embodiment, each joint R1-R9 and corresponding link is modeled based on rotational function and/or length. Memory 4120 may store mathematical parameters associated with the model. Additionally, processor 4102 and/or robot arm controller 4106 may use a mathematical model to determine, for example, the current position of robot arm 506 and/or camera 300, which is provided by an operator. It can be used to calculate how the joint should rotate based on the intended movement.

In the illustrated example, joint R1 is provided at coordinate position (0,0,0). The length between joints R1-R9 corresponds to the length of the link. In the illustrated example, stereoscopic visualization camera 300 is modeled as a robotic end effector connected to nine combiners. The three-dimensional space shown in FIG. 49 is modeled using a series of ten homogeneous transformations that may include matrix multiplications. The first six frames or joints R1-R6 represent the forward kinematics of the robot arm 506 and may be calculated using the Denavit-Hartenberg parameters of the robot arm. The next three frames or joints R7-R9 represent the transformation from the tool tip of the robot arm 506 to the tip of the coupling plate 3304. The last frame R10 represents the transformation from the tool tip of the coupling plate 3304 to the control point of the stereoscopic visualization camera 300.

Frame or joint R7 represents the pitch joint of the coupling plate 3304 and can vary from 0° to 90°. Frame or joint R8 represents the yaw joint of coupling plate 3304 and can vary between -90°, 0°, and 90° depending on the yaw configuration. Joints R7-R9 of the coupling plate may include voltage sources and potentiometers. Connector 3450 and/or coupler controller 4130 of robot arm 506 may include an I/O tool tip connector configured to receive a voltage output from a potentiometer. The processor 4102 and/or the robot arm controller 4106 are configured to receive the output voltage and determine the pitch and yaw angles of the coupling plate 3304 in response. Processor 4102 and/or robot arm controller 4106 combine the coupling plate pitch and yaw information with sensor output data from joints R1-R6 of robot arm 506 to calculate the positions of frames R1-R10 and , the coupling plate 3304, and/or the camera 300.

The control point represents the frame 10 at the very end of the kinematic chain and is completely programmable in position based on which features are selected. For example, if the operator selects the assisted drive feature, processor 4102 and/or robot arm controller 4106 may cause control points to represent camera 300 to be within the camera along the axis of rotation of control arm 304. configured. In another example, if the operator selects a target lock feature, processor 4102 and/or robot arm controller 4106 are configured to set the control point of camera 300 to the origin of the optical axis view vector.

Returning to FIG. 48, after calibrating camera 300 with respect to robot space, processor 4102 and/or robot arm controller 4106 are configured to calibrate robot space with respect to patient space (block 4808). Patient space calibration is necessary for the stereoscopic visualization platform 516 to provide accurate visualization of the patient requiring orientation between the robotic system and the patient. In some embodiments, this orientation is fixed. In other embodiments, this orientation is sensed and known if it changes. In some embodiments, the patient is placed in an operating room bed and registered to the bed using one or more fiducials 4809. For example, if a patient is undergoing brain surgery, the patient is secured to a bed and an external frame is secured to the patient's skull. The frame is observable by a stereoscopic visualization camera 300, and two or more non-collinear objects of known locations are visible simultaneously, thereby determining the position and orientation of the frame and thus the patient's skull. The fiducials 4809 may be included in an arrangement such as an arrangement of calibration targets that may be performed. Other embodiments may use fiducials 4809 that are implanted in the patient and visible on MRI or similar images. Such fiducials 4809 can be used to accurately track and register the patient's skull and MRI images to a coordinate system representing patient space. Additionally, other embodiments may use image recognition of the patient's own natural features. For example, facial or similar recognition using biometric data, in-situ x-rays, or similar alternative modality imaging can be used to precisely identify the patient's location and orientation. In another example, a model of the patient's facial surface may be identified using one or more depth map computations and surface matching functions performed by processor 4102 and/or robotic arm controller 4106 as described above. can.

In embodiments, the position and orientation of the operating room bed with respect to the robot space is fixed and specified. Some embodiments include, for example, a rigid frame that mechanically registers the bed with the attachments of the cart 510 in a known position and orientation. Alternatively, the bed can be fixed with respect to the robotic arm 506 and the position and orientation can be determined using fiducials. For example, the robotic cart 510 and bed can be anchored to the floor and remain fixed for the duration of the procedure.

After visualizing patient fiducials 4809 by camera 300, their position and orientation in robot space can be decoded and stored by processor 4102 and/or robot arm controller 4106, where they are transferred from robot space to patient space. Coordinate transformation becomes possible. Note that the coordinate system transformation from one space to another is generally selectable and reversible. For example, it may be more efficient to translate from desired camera motion or pose to robot space to allow processor 4102 and/or robot arm controller 4106 to identify discrete joint motions and orientations. Alternatively, it may be easier and more efficient to present the information to the surgeon on display monitor 512 in the patient space. The patient and vector locations may be determined by processor 4102 and/or robot arm controller 4106 in each of most arbitrary coordinate systems, such as the cart origin, patient reference frame, GPS, and/or any other coordinate system desired. can be converted.

In some embodiments, the processor 4102 and/or the robot arm controller 4106 use automated iterative techniques to perform these or equivalent methods of robot/patient space calibration and measurement to increase accuracy and Configured to reduce calibration time. In the exemplary embodiment, the displacement and orientation of the stereoscopic visualization camera 300 relative to the reference is precisely known by the processor 4102 and/or the robot arm controller 4106. Movements of the robot arm 506 can be performed accurately and subsequent reference images can be accurately analyzed. Visualization and knowledge of calibration parameters can be coupled by processor 4102 and/or robot arm controller 4106 so that measurements, and thus calibration, can be performed automatically and accurately. This is important, for example, to maintain accurate calibration from one surgical procedure and patient to the next.

In some examples, processor 4102 and/or robot arm controller 4106 are configured to determine the boundaries of robot arm 506 and/or camera 300 relative to patient space and/or robot space. Boundaries represent virtual limits implemented in software to prevent robotic arm 506 and/or camera 300 from touching or exiting a defined area or space. In some examples, the limits are defined in one or more LUTs or registers stored in memory 4120 as scaling factors applied to joint movement signals by processor 4102 and/or robot arm controller 4106. The magnitude of the scaling factor decreases to zero as the limit to each individual boundary is approached. For example, the amount and speed of joint rotation may be determined based on operator input. However, processor 4102 and/or robot arm controller 4106 scales the joint rotations with a scaling factor before sending the signals to the appropriate joints. In addition, processor 4102 and/or robot arm controller 4106 may maintain the amount of rotation such that the joint moves the desired amount but decelerates until the joint reaches the boundary. It should be appreciated that a scaling factor may not be applied to a joint in a rotational area to which a scaling factor is applied if the desired movement is away from the boundary. Thus, the processor 4102 and/or the robot arm controller 4106 may apply a scaling factor to a particular joint while applying a scaling factor of '1' to other joints based on the current position and estimated desired movement from the operator. .

The scaling factors are exactly between 0 and 1, allowing them to be chained together, allowing the software to support an infinite number of possible boundaries. The scaling factor may decrease linearly as the boundary is approached, gradually slowing the rotation of joints R1-R9 as the robot arm 506 approaches the boundary. In other examples, the scaling factor may decrease exponentially as the boundary is approached.

Generally, the operator typically focuses on the surgical field or stereo image on display monitor 512. Therefore, the operator is typically unaware of the position of the individual links of the robot arm 506 and/or the coupling plate 3304. Therefore, it is not always intuitive when the robot arm 506 is about to approach a limit or collide with another part of the robot arm 506. Accordingly, the joint limits may be active at all times and may prevent any portion of the robot arm 506 from hitting itself or putting the joint into a singularity configuration such as an elbow lock. The example processor 4102 and/or robot arm controller 4106 are configured to determine the scaling factor based on the current position of the robot arm 506. Processor 4102 and/or robot arm controller 4106 may also take into account the intended movement instructions provided by the operator to determine which scaling factor to apply. Based on the current movement and/or expected movement, processor 4102 and/or robot arm controller 4106 calculates a scaling factor based on distance in joint angular space, e.g., using one or more LUTs. do. The joint angle space may define a particular combination of joint angles that are known to cause the joint lock or robot arm 506 to hit itself. Joint angular space determination is therefore based on identifying and comparing the current (and/or expected) movements of the joints relative to each other.

In addition to the boundaries of the robot arm 506, the memory 4120 can prevent the robot arm 506 from hitting the cart 510, prevent the robot arm from hitting the display monitor 512, and/or prevent the camera 300 from hitting the robot arm 506. can remember the bounds associated with the Cartesian limits. Processor 4102 and/or robot arm controller 4106 may, for example, use the coordinate system discussed in conjunction with FIG. 49 to determine and/or apply Cartesian limits. In some examples, limits may be relative to or anchored to a particular link. Therefore, if the link moves in 3D space, the boundaries around the link will also move accordingly. In other examples, the limit is stationary and fixed to a particular coordinate plane or line within the 3D space shown in FIG. 49. Processor 4102 and/or robot arm controller 4106 may calculate or identify scaling factors in Cartesian space and apply the limits by applying forward kinematics transformations.

The example processor 4102 and/or robotic arm controller 4106 may also determine patient boundaries, which define a virtual configuration that no points of the robotic arm 506 and/or camera 300 can violate. . The patient boundary may be determined by calculating a scaling factor in Cartesian space for the distance of each position joint on the robot arm 506 and/or coupling plate 3304 to the location of the boundary plane. The bounding plane is implemented as an X,Y plane located at the same vertical Z location as the non-pitch configuration, as shown by orientation 5002 in FIG. In a pitch configuration such as a half-seated patient shown in orientation 5004 of FIG. 50, the bounding plane is set as a Y,Z plane positioned at a positive or negative .

The example boundaries described above may be stored in memory 4120 as default boundaries and/or determined by processor 4102 and/or robotic arm controller 4106 prior to the surgical procedure. In some embodiments, particular boundaries may be evaluated or determined based on the input type of surgical procedure being performed. For example, patient boundaries may be determined by camera 300 imaging the patient and using calibration information/parameters to determine patient depth. Processor 4102 and/or robotic arm controller 4106 may then create and apply boundaries to the specified location on or next to the patient. Similar boundaries may be created after detection of monitors, surgical staff, or surgical equipment.

For example, boundaries may be determined around the use of certain surgical instruments, such as instruments of larger size or instruments that pose a particular risk if contacted. An example processor 4102 and/or robotic arm controller 4106 may receive instrument type input and/or detect instruments in stereo images using image analysis. In other examples, processor 4102 and/or robotic arm controller 4106 calculate depth information associated with a surgical instrument to determine its size, orientation, and/or location. The example processor 4102 and/or robotic arm controller 4106 transforms the surgical instrument image into a coordinate system, such as the coordinate system discussed in connection with FIG. 49. Processor 4102 and/or robotic arm controller 4106 also apply a scaling factor having a value less than "1" to an area corresponding to the location of the surgical instrument, thereby causing robotic arm 506 and/or camera 300 to Avoid inadvertent contact. In some cases, processor 4102 and/or robotic arm controller 4106 may track movement of surgical instruments and change boundaries accordingly during the procedure.

FIG. 51 illustrates an example of how rotary joint velocity of the robot arm 506 and/or coupling plate 3304 is scaled based on distance to a boundary, according to example embodiments of the present disclosure. Graph 5102 shows the rotational speed of joint R1, and graph 5104 shows a first zone 5112 corresponding to the area near the boundary where the scaling factor is lowered from the value "1" and at the boundary where the scaling factor is lowered to the value "0". A shoulder angle (eg, rotational position) 5110 is shown associated with a corresponding second zone 5114.

FIG. 51 shows that as the robot arm 506, particularly joint R1, approaches the at least one link and/or stereoscopic visualization camera 300 to the first zone 5112, the rotational speed dynamically scales with respect to the distance to the second zone 5114. Indicates that the Then, when at least one link and/or stereoscopic visualization camera 300 reaches the second zone 5114, the scaling factor is lowered to the value "0" and all revolute joint movement towards the boundary is stopped. In other words, as the robot arm 506 and the stereoscopic visualization camera 300 approach a limit or boundary, the processor 4102 and/or the robot arm controller 4106 reduce the rotational speed of at least some of the joints R1-R9 so that the second zone When 5114 is reached (as shown between 20 seconds and 30 seconds in graphs 5102 and 5104), a speed of "0" degrees/second is finally reached. The graph also shows that if the at least one link and/or stereoscopic visualization camera 300 moves away from the second zone 5114, the processor 4102 and/or the robot arm controller 4106 It is also shown that a scaling factor of "1" is used.

In some embodiments, processor 4102 and/or robotic arm controller 4106 are configured to cause display monitor 512 or other user interface to display one or more graphical icons representing the status of robotic arm 506. For example, if the robotic arm 506 and/or the camera 300 are located in a zone or area where the scaling factor has a value of "1", a green icon may be displayed. Additionally, if the robot arm 506 and/or camera 300 are positioned within the first zone 5112, a yellow icon may be displayed to indicate that the joint rotation speed is reduced. Additionally, if the robot arm 506 reaches the second zone 5114 or boundary/limit, a red icon may be displayed to indicate that no further movement beyond the boundary is possible.

Returning to FIG. 48, after the robot space boundaries are determined, the example processor 4102 and/or robot arm controller 4106 are configured to enable the robot arm 506 to operate with the stereoscopic visualization camera (block 4812). . This may include making the robotic arm 506 and stereoscopic visualization camera 300 available for use during the surgical procedure. It may also include features such as assisted drive and/or target locking. Additionally or alternatively, this may include enabling one or more user controls on one or more of the input devices 1410 of FIG. 41. After the robotic arm 506 is enabled with the stereoscopic visualization camera 300, the example procedure 4800 ends. The example procedure 4800 may be repeated if the stereoscopic visualization platform 516 is reinitialized, experiences a fault detection, or is unable to verify calibration.

D. Embodiments of Stereo Visualization Camera and Robot Arm Operation The example stereo visualization camera 300 is configured to operate in conjunction with the robot arm 506 and/or the coupling plate 3304 to provide enhanced visualization features. . As discussed in more detail below, the enhanced features include focus extension, automating focal tip positioning, providing distance measurements between objects in images, providing robot motion in conjunction with visualization, Includes sag compensation, image fusion, and visualization position/orientation storage. Enhanced visualization features also include assisted drive capabilities of the robot arm 506 and target locking capabilities that allow the camera to be locked to a particular view while the robot arm 506 and/or coupling plate 3304 can be reoriented. can also be included.

1. Focus Extension Embodiments In some embodiments, the robotic arm 506 and/or the coupling plate 3304 may provide focus extension of the camera 300. As described above in connection with FIG. 43, stereoscopic visualization camera 300 includes a main objective lens assembly 702 that changes the working distance. To focus on an object at the surgical site, main objective lens assembly 702 changes focal length from just in front of the object to beyond the object. However, in some cases, the main objective lens assembly 702 reaches the mechanical limits of the front working distance lens 408 before achieving best focus.

The example processor 4102 and/or robot arm controller 4106 detects when a mechanical limit is reached and/or determines that the mechanical limit of the lens 408 is about to be reached and replaces the lens 408 with the robot accordingly. The arm 506 is configured to adjust its position. The processor 4102 and/or the robotic arm controller 4106 are configured to use the robotic arm 506 to calculate a view vector for the camera 300 and extend the focus by actuating the robotic arm 506 along the optical axis. . Processor 4102 and/or robot arm controller 4106 use the above calibration parameters of stereoscopic visualization camera 300 to determine the distance required to achieve focus. For example, as discussed above, the position of the front working distance lens 408 is mapped to the physical working distance of the main objective lens assembly 702 to the target object. This distance provides an estimate of how far the center of camera 300 is from the target object. Additionally, the calibration parameters may include a mapping between the motor or encoder steps of the front working distance lens 408 and the working distance, providing an estimate of the distance required to achieve a particular working distance or focus. Accordingly, processor 4102 and/or robot arm controller 4106 may read the current encoder value of front working distance lens 408 and determine the number of meters representing the vertical distance from camera 300 to the target object. In other words, processor 4102 and/or robot arm controller 4106 convert lens movement (in encoder counts) into physical distance within robot space. Processor 4102 and/or robot arm controller 4106 then determine joint rotation speed, direction, and/or duration (e.g., movement sequence) for moving robot arm 506 the specified distance along the optical axis. . Processor 4102 and/or robot arm controller 4106 then sends one or more signals to the appropriate joints corresponding to the movement sequence to cause robot arm 506 to provide focus expansion. In some cases, processor 4102 and/or robot arm controller 4106 may apply a scaling factor before the signals are sent to joints R1-R9 of robot arm 506 and/or coupling plate 3304.

It should be appreciated that focus expansion causes automatic movement of the robot arm 506. In other words, the robot arm 506 can continue to move the camera 300 through the best focus point. Additionally, movement of the robot arm 506 is performed without input from an operator moving the robot arm, rather from operator images regarding changes in focus. In some cases, processor 4102 and/or robot arm controller 4106 may automatically adjust focus to maintain a clear image.

In some embodiments, the processor 4102 and/or the robotic arm controller 4106 are configured to move the robotic arm 506 along the working distance of the camera in response to a single button press via the input device 1410. . This feature allows the operator to obtain focus by modifying the motor position of the main objective lens assembly 702 and moving the robot arm 506 and/or the coupling plate 3304. This "robot autofocus" feature or procedure, as described above in connection with FIG. This is achieved by Processor 4102 and/or robot arm controller 4106 uses feedback laws to increase the vertical velocity of robot arm 506 (or along the optical axis of camera 300) until the determined distance reaches a value of "0". configured to command speed). Processor 4102 and/or robotic arm controller 4106 may use this autofocus algorithm from time to time during a procedure to focus on a target object. In some embodiments, the processor 4102 and/or robot arm controller 4106 selects the last robot arm 506 and/or coupling plate 3304 from which autofocus was used as a seed or starting point when searching for an autofocus direction. movement can be used to improve the speed and accuracy of focusing on the target object.

An example processor 4102 and/or robot arm controller 4106 may be movable by a front lens set 714, lens barrel, each motor having an encoder count mapped to position, focus, working distance, and/or magnification. It should be appreciated that in addition to or as an alternative to moving set 718 and/or final optical set 742, robotic arm 506 and/or coupling plate 3304 may be configured to move. For example, processor 4102 and/or robot arm controller 4106 may cause robot arm 506 to move toward the optical axis if any of front lens set 714, lens barrel set 718, and/or final optics set 742 are approaching the limits of travel. can be moved along. In some examples, processor 4102 and/or robot arm controller 4106 first move robot arm 506 to a position that is generally at or near focus, and then move front lens set 714, lens barrel set 718, and/or Alternatively, the final optical set 742 may be adjusted to approximately ideally focus the target image.

2. Focus Tip Auto Positioning Embodiments In some embodiments, the robotic arm 506 and/or the coupling plate 3304 may operate in conjunction with the stereoscopic visualization camera 300 to provide focus tip auto positioning. In these embodiments, the processor 4102 and/or the robotic arm controller 4106 are configured to position the camera 300 for visualization of the target surgical site without information or feedback of specific images and their contents. Processor 4102 and/or robot arm controller 4106 may perform open-loop camera positioning using the calibrated camera model parameters described above in connection with FIGS. 42 and 49. Processor 4102 and/or robot arm controller 4106 may position stereoscopic visualization camera 300 on robot arm 506 such that the focal point or focus tip of the camera is within the scene. The stereoscopic visualization camera 300 determines the pointing direction of the camera 300 with respect to the coordinate system based on calibration information regarding the pose of the robot arm 506 and/or the coupling plate and optical calibration parameters of the camera 300. Processor 4102 and/or robot arm controller 4106 may characterize the aim with a geometrically defined view vector that is aligned with the stereoscopic optical axis of camera 300 with respect to the coordinate system of robot arm 506.

In some embodiments, processor 4102 and/or robot arm controller 4106 execute an initialization routine to set calibration parameters and/or other memory data to actual physical references that may be used for tip positioning. configured to align to the position. For example, processor 4102 and/or robot arm controller 4106 may place robot arm 506 and/or coupling plate at “position 0” where all position data fields are set to 0 (or 0,0,0 in three-dimensional space). can be moved to hard spots in the body. Further movements are performed relative to this point and the position data is updated according to encoder counts of the various joint motors of the robot arm 506 and/or the coupling plate 3304, for example.

In other embodiments, processor 4102 and/or robot arm controller 4106 may determine or set the tip position of camera 300 based on one or more visualization parameters. For example, processor 4102 and/or robot arm controller 4106 may use the projection center location as the proximal end of the view vector (eg, the “starting point” for aiming camera 300). In some surgical systems, this point on the surgical instrument is referred to as a "hind" point and may be provided in relation to the tip. The processor 4102 and/or the robot arm controller 4106 calculate view vector directions from the tip and back points to determine the aiming of the camera 300 with respect to the coordinate system of the robot arm 506.

Additionally or alternatively, the processor 4102 and/or the robot arm controller 4106 may determine a focal length that calculates the extent of the focal plane of the stereo image from the projection center. The center of the image in the focal plane is the "tip" point. Processor 4102 and/or robot arm controller 4106 may use the calibrated working distance to determine the actual physical spatial distance from camera 300 to the tip point. Additionally, processor 4102 and/or robot arm controller 4106 may determine the magnification factor as described above with respect to magnification calibration.

3. Distance Measurement Embodiments In some embodiments, the robotic arm 506 and/or coupling plate 3304 operate in conjunction with the stereoscopic visualization camera 300 to provide distance and/or depth measurements between objects in stereoscopic images. It is possible. For example, processor 4102 may positionally determine the focal point or center of the focus tip of camera 300 with respect to the coordinate system of robot arm 506 using optical calibration parameters translated to robot space. As described above in connection with FIGS. 45 and 46, the view vector and left/right disparity information for any point in the image can be used by processor 4102 to perform triangulation regarding the tip or any other point in the image. Its position in three dimensions can be calculated. This triangulation allows processor 4102 to map any point in the image to the robot coordinate system. Accordingly, the processor 4102 can calculate the location and/or depth of multiple objects and/or the location of different parts of the objects with respect to the same coordinate space of the robot arm 506, thereby providing distance measurements and/or Becomes able to measure depth.

Processor 4102 may visually display distance and/or depth measurement information on and/or in conjunction with the stereoscopic images. In some cases, the operator uses input device 1410 to select an object on the screen, or selects two or more objects by pointing directly at the actual object at the patient using a finger or a surgical instrument. obtain. Processor 4102 receives the selection indication and determines the coordinates of the objects and distances between the objects accordingly. Processor 4102 may then display a ruler graphic and/or a distance value (and/or an indication of the selected object) in conjunction with the stereo image.

Furthermore, object tracking allows the location of other objects previously imaged (or previously provided in other images) to be remembered and compared later. For example, camera 300 may move to a location where at least some of the objects are currently outside the FOV. However, an operator can instruct processor 4102 to determine the distance between an object within the FOV and a previously imaged object that is currently outside the FOV.

In some embodiments, the processor 4102 uses the coordinates of the object to fuse with digital images or modes from alternative modality visualizations, such as MRI images, X-ray images, surgical templates or guidelines, preoperative images, etc. obtain. In one example, processor 4102 is configured to use the location of the object in the coordinate plane and depth information to scale, orient, and position the alternate modality visualization accordingly. Processor 4102 may select at least some of the alternative modality visualizations that have the same features (eg, objects) in the displayed stereo image. For example, processor 4102 may use image analysis routines to find blood vessel patterns, scars, deformations, or other visible physical structures or objects in stereoscopic images. Processor 4102 may then find the same feature in the alternate modality visualization. Processor 4102 selects a portion of the alternative modality visualization that includes the same features. Processor 4102 may then use the coordinates, depth, and/or distances between features in the stereo image to scale, rotate, and/or orient selected portions of the alternate modality visualization. The processor may then fuse the adjusted portions of the alternate modality visualization with the stereo image. Processor 4102 may track how the distinguishable objects move relative to each other and/or relative to the FOV and determine how to update the fused image accordingly. For example, movement of camera 300 to another surgical location may cause processor 4102 to select another portion of the pre-operative image for fusion with the stereoscopic image of that other surgical location.

In some cases, processor 4102 and/or robotic arm controller 4106 may move robotic arm 506 to track movement of objects within the FOV. Processor 4102 uses the object's coordinate location to detect movement or obfuscation. In response to detecting movement or darkening, processor 4102 and/or robot arm controller 4106 may track movement of the object or determine how robot arm 506 and/or coupling plate 3304 moves to resolve the darkening. configured to determine whether to do so. For example, processor 4102 and/or robotic arm controller 4106 may move robotic arm 506 in a circular path to visualize points on the patient's retina from multiple directions to avoid reflections or darkening from instruments. .

4. Image Fusion Embodiments As mentioned above, processor 4102 is configured to fuse images from alternative modalities into a live stereoscopic image. For example, if a surgeon is performing surgery on a patient with a deep brain tumor, processor 4102 may display live images from camera 300 to display monitor 512 of the brain tumor at the appropriate location, at the appropriate depth, and in a stereoscopic perspective. can be instructed to visualize an MRI image of. In some embodiments, processor 4102 is configured to use range and/or depth measurement information of one or more objects within the FOV for fusion with alternative modality views. Processor 4102 also uses the stereo optical axis (e.g., view vector), IPD, and/or camera model parameters calculated in the calibration step discussed in connection with FIG. 42 and stored in one or more LUTs. can also provide imaging fusion. The use of optical calibration parameters allows processor 4102 to display alternative modality images as if the images were acquired by stereoscopic visualization camera 300. Processor 4102 uses the camera's optical calibration parameters to adjust the effectiveness of camera 300 such that, given the applied working distance and magnification of camera 300, the alternate modality image is viewed at a distance Z from the focal point at the surgical site. Alternate modality images may be modeled, scaled, or modified based on the IPD.

FIG. 52 depicts a diagram of an example procedure 5200 for fusing images from alternative modality visualizations with stereoscopic images, according to example embodiments of the present disclosure. Although procedure 5200 will be described with reference to the flow diagram shown in FIG. 52, it should be understood that many other methods of performing the steps associated with procedure 5200 can be used. For example, the order of many of the blocks may be changed, certain blocks may be combined with other blocks, and many of the blocks described are optional. Additionally, the operations described in step 5200 may be performed, for example, on optical element 1402, image capture module 1404, motor and lighting module 1406, information processor module 1408, and/or joints R1-- of example stereoscopic visualization camera 300 of FIG. R9 and among multiple devices, including the robot arm controller 4106 of FIG. For example, procedure 5200 may be performed by a program stored in memory 1570 of processor 4102.

Processor 4102 of example procedure 5200 uses optical calibration parameters to render previously generated three-dimensional MRI data of a patient as a stereoscopic image with a suitable perspective as the stereoscopic image recorded by camera 300, for example. It is configured as follows. Processor 4102 may receive alternative modality visualizations, such as MRI data, for example, from device 4104 of FIG. 41 (block 5202). Processor 5202 may also receive input 5203 via input device 1410 indicating that the alternate modality visualization should be fused with the stereoscopic images recorded by stereoscopic visualization camera 300 (block 5204).

During procedure 5200, when the surgeon positions camera 300 in a desired orientation and position for the surgical procedure, pose data 5205 is obtained by processor 4102 (block 5206). Pose data 5201 may include the positions of robot arm 506, coupling plate 3304, and/or stereoscopic visualization camera 300. Processor 4102 also accesses magnification and working distance optical calibration parameters 5207 associated with camera 300 from one or more LUTs, such as LUT 4203 of FIG. 42 (block 5208). Processor 4102 uses pose data 5205 in conjunction with magnification and working distance optical calibration parameters 5207 to determine the stereo axis and IPD of camera 300 (block 5210). Processor 4102 applies pose data, stereo axis data, IPD data, and/or optical calibration parameters to select at least a portion of the MRI data and/or to cause the selected portion to be viewed by stereoscopic visualization camera 300. The selected portions of the MRI data are modified, scaled, oriented, segmented, etc., so as to be provided in perspective of a view of the patient's brain as seen in the image (block 5212). Processor 4102 is configured to apply the stereoscopic optical axis view vector and IPD to render the selected portion of the MRI data into a stereoscopic image corresponding to the current live view of camera 300 (block 5114). Processor 4102 may then fuse the stereoscopic MRI images with live stereoscopic images from camera 300 (block 5216), as discussed herein.

As discussed above, processor 4102 may use objects or features to position or fuse rendered MRI data with stereoscopic images from stereoscopic visualization camera 300. For example, the processor 4102 may use one or more image analysis routines to identify distinct features or objects in stereoscopic images, locate the same distinct features in rendered stereoscopic MRI data, and locate features or objects. The rendered stereoscopic MRI data may be superimposed on the appropriate portion of the camera stereo image so that it is aligned and has the same scale, size, depth, orientation, etc. Processor 4102 may render the rendered stereoscopic MRI data at least partially transparent so that live images can also be viewed. Additionally or alternatively, the processor 4102 may adjust shading at the borders of the rendered stereoscopic MRI data to reduce visual contrast between the rendered stereoscopic MRI data and the camera stereo image. The example procedure 5200 of FIG. 52 may then end.

The example procedure 5200 allows a surgeon to visualize a brain tumor in a precise location relative to the stereo image of the camera 300. Surgeons may use this fused visualization particularly in parts throughout the surgical procedure. For example, a surgeon can see a tumor that has not yet been exposed, best described as "x-ray vision" under the current level of the incision. Control of the transparency of live or rendered stereoscopic MRI images may be adjusted via input device 1410 to optimize clarity of the fused image. One example procedure accordingly allows for safer, more accurate and efficient tumor resection.

In some embodiments, procedure 5200 may be repeated if the FOV, focus, working distance, and/or magnification changes. In these embodiments, the processor 4102 uses the updated pose information and extracts the corresponding stereo axes and IPD from the lookup table to re-render the MRI data into an updated accurate stereo image. configured. Processor 4102 is configured to fuse the newly rendered MRI data with the current stereoscopic image so that the live view and corresponding MRI data are placed in the proper position, depth, and orientation.

In some embodiments, the example processor 4102 is configured to operate with the stereoscopic visualization camera 300, the robotic arm 506, and/or the coupling plate 3304 to generate a live cross-sectional fusion visualization. Cross-sectional visualization of the surgical site provides the surgeon with a greatly improved perspective not otherwise available. FIG. 53 shows a diagram of a patient 5300 with a glioblastoma 5302 showing the interior of the patient's head in phantom. In particular, a glioblastoma 5302 may be within the patient's brain 5304 and is shown as a bright phantom line. The illustration of FIG. 53 is typical of a preoperative diagnostic image from, for example, an MRI device, in which many image slices are stacked to render and visualize a 3D model of the interior of the patient's skull 5306.

In the illustrated example, glioblastoma 5302 is to be removed through brain surgery. FIG. 54 shows a perspective view illustration of a patient 5300 undergoing a craniotomy procedure 5400 to provide access to the cranium 5306. Procedure 5400 also includes brain resection and retraction using surgical instrument 5402. Generally, surgical access site 5404 is created in a deep cone shape to access glioblastoma 5302.

FIG. 55 shows a diagram of a stereoscopic visualization platform 516 that includes a stereoscopic visualization camera 300 and a robotic arm 506 to visualize a craniotomy procedure 5400, according to an example embodiment of the disclosure. As shown, craniotomy procedure 5400 is prepared such that robotic arm 506 is positioned to aim stereoscopic visualization camera 300 along visualization axis 5500 of cone surgical site 5404 through the top of skull 5306. The surgeon's view performing the surgery is generally through the top of the skull 506, as shown in FIG. As can be seen from FIG. 7, the depth of the surgery and, for example, the tip of the surgical instrument 5402 is difficult to see.

The example stereoscopic visualization camera 300 shown in FIG. 55 provides a high precision stereoscopic image looking down the axis 5500 of the conical surgical access site. As mentioned above, the disparity information between the left and right views of camera 300 at all points common to both views at the access site is used by processor 4102 to determine from a known reference depth, e.g., an object plane. Determine the depth of each point in . In the illustrated example, the disparity between the left and right views is equal to the value "0", which allows processor 4102 to identify a depth map for each point in the image. The depth map can be re-rendered by processor 4102 as if the map were viewed from a different angle. Additionally, processor 4102 is configured to make at least a portion of the depth map transparent upon receiving instructions from a surgeon and/or operator. In the illustrated example, a portion of the depth map below section plane AA in FIG. .

FIG. 56 shows an illustration of a local perspective view of a conical surgical access site 5404. The illustrated surgical access site 5404 includes stepped conical segments for clarity in this discussion. In this example, the sweep cone angle of section 5404 is indicated by angle "α."

FIG. 58 shows a view of a conical surgical access site 5404 of a craniotomy procedure 5400. Prior knowledge of the size and shape of surgical instrument 5402, along with image recognition of its position, direction, and/or orientation, enables processor 4102 to generate image data in the cross-sectional view shown in FIG. 58. Recognition of the instrument 5402 in the stereoscopic view depicted in FIG. 57 allows precise placement in the cross-sectional view of FIG. 58 and visualization of the underside of the instrument that is invisible to the surgeon, for example, during surgery on a patient's brain 5304. Become.

In some embodiments, processor 4102 is configured to fuse images of glioblastoma 5302 with quasi-live or live stereoscopic images. As discussed above, the combination of robot arm 506 and camera 300 provides highly accurate position, orientation, and/or orientation information of the view relative to the robot reference frame or robot space. After registering or calibrating the robotic arm 506 and camera 300 to the reference frame of the patient 5300, precise position, orientation, and/or orientation information of the surgical access site 5404 and its respective position relative to the patient is generated by the processor 4102. As shown in FIG. 59, processor 4102 uses image fusion to overlay selected portions of the MRI image of glioblastoma 5302 onto a cross-sectional view. In addition, image fusion allows visualization of other relevant MRI images, including, for example, cerebral vessels or other structures for which orientation to the image is desired. The exemplary surgical procedure proceeds with the surgeon being able to see and understand the depth location of the glioblastoma 5302, as well as the safe spacing and positioning of the instrument 5402.

FIG. 59 shows an illustration of a cross-sectional view of surgical access site 5404. In this example, a portion 5302' of glioblastoma 5302 is visible from stereoscopic visualization camera 300. FIG. 60 shows a cross-sectional view perpendicular to plane AA of FIG. 57. The diagram may show a cross-sectional view generated by processor 4102 based on MRI data fused with a live view of surgical access site 5404. Use of the depth map by processor 4102 allows rendering of surgical access site 5404 in various desired cross-sectional planes and cross-sectional plane combinations, as shown in FIG. 61. Rendering allows the processor 4102 to display the complete glioblastoma 5302, including the visible portion 5402' and the remaining portion from the MRI data. Processor 4102 may display the visualization from the perspective of camera 300 or as a cross-sectional view, as shown in FIG. 61.

5. Embodiments of Robot Motion Coupled with Visualization In some embodiments, the example processor 4102 operates in conjunction with the robot arm controller 4106, the stereoscopic visualization camera 300, the robot arm 506, and/or the coupling plate 3304. to combine visualization with robot movement. In some examples, processor 4012 and/or robot arm controller 4106 operate in a closed loop to provide federated visualization based on robot motion. In these examples, the processor 4012 and/or the robotic arm controller 4106 may direct the robotic arm 506, coupling plate, etc. toward visualization of the surgical site based on the particular image and its content (e.g., objects, identifiable features, etc.). 3304 and/or configured to position the camera 300. As mentioned above, the positions of robot arm 506 and camera 300 are known to processor 4012 and/or robot arm controller 4106. Additionally, the image data recorded by the camera is stereoscopic and provides depth data. As a result, processor 4012 and/or robot arm controller 4106 can determine the location of any visualization point in patient or robot three-dimensional space. Thus, when the robot arm 506 moves the camera 300 in a desired direction from an initial position with an initial image, it is expected that the desired image change will be seen in the second, post-movement image.

Alternatively, the expected post-movement image can be calibrated by processor 4102 and/or robot arm controller 4106 configured to apply an equation representing the desired movement to the initial image data, thereby: A calculated second image is generated. Processor 4102 and/or robot arm controller 4106 compares the actual image after movement to the calculated image using a template matching routine or function, as described above. If an error is detected, processor 4102 and/or robot arm controller 4106 can correct the error by moving robot arm 506 and/or camera 300 accordingly. For example, given an initial image and a desired movement "100 pixels to the right" received from the operator, the image data for the theoretically moved image may be shifted to the right by 100 pixels by processor 4102 and/or robot arm controller 4106. It can be calculated as The physical movement is then performed by executing commands to various adjusted robot joints to reposition the robot arm 506 and/or camera 300 to the theoretical desired position, as disclosed. It will be done. A second image is recorded by camera 300 and compared by processor 4102 and/or robot arm controller 4106 to image data calculated using, for example, a template matching function or its equivalent. If the movement is accurate, the data will be 100% correlated at the tip of camera 300 and both images will be perfectly aligned. However, if the actual image data shows the best correlation at another location, e.g., 101 pixels to the right and 5 pixels above, the movement may be modified by processor 4102 and/or robot arm controller 4106 to The error can be corrected by physically moving the camera 300 one pixel to the left and five pixels down.

6. Sag Compensation Embodiments In some embodiments, at least some of the joints R1-R9 of the robot arm 506 and/or the coupling plate 3304 may experience some sag. Processor 4102 and/or robot arm controller 4106 may be configured to provide compensation for robot arm sag. In some cases, processor 4102 and/or robot arm controller 4106 are configured to perform sag compensation in a series of small movements such that motion accuracy is maintained over the range of motion of robot arm 506. For example, to decharacterize sag, sag compensation is performed in the direction of motion in which a particular robot joint is moved to isolate errors as a function of actual robot joint rotational position. By comparing the error to the torque moment calculated by multiplying the loaded weight of camera 300 by the moment arm (or link) length, the compliance of that joint can be determined. Alternatively, joint compliance may be calculated using analytical techniques, such as finite element analysis ("FEA").

Using and storing the above compliance characterization for all joints at all rotational positions, processor 4102 and/or robot arm controller 4106 may calculate the total droop at a particular camera position. Processor 4102 and/or robot arm controller 4106 may specify a sag correction factor for each camera position into a LUT and/or calibration register. Further, the processor 4102 and/or the robot arm controller 4106 may apply the sag factor to the robot arm movement command or movement sequence (before or after applying the scaling factor) such that the sag correction is incorporated into the movement command/signal. . The correction factor may be calculated during the ongoing motion procedure, thereby allowing accurate tracking and following of the camera 300. This correction factor further eliminates the need for a second camera for calibration/positioning of stereoscopic visualization platform 516, eliminates the need to have a reference target on camera 300, and thus eliminates drape interface problems.

7. Visualization Position/Orientation Saving Embodiments In some embodiments, the example processor 4102 is configured to save visualization parameters to return to a particular orientation and/or position of the stereoscopic visualization camera 300. be done. The visualization parameters may include a view vector, location, magnification, working distance, focus, position, and/or orientation of stereoscopic visualization camera 300, robot arm 506, and/or coupling plate 3304.

In one example, a surgeon may desire high magnification visualization of small structures during anastomosis of a portion of a blood vessel under visible illumination. The surgeon may then want to zoom out to a wider view of the entire vessel under infrared illumination to check patency. The surgeon can then return to magnified visualization to complete the suturing. In this example, processor 4102 is configured to save visualization parameters for each location. Processor 4102 may store locations corresponding to locations viewed consecutively over time periods such as 2 seconds, 5 seconds, 30 seconds, etc. Processor 4102 may also store the position after receiving instructions from the surgeon via input device 1410.

Processor 4102 may display a list of stored locations and/or waypoints. Upon selecting a stored location, processor 4102 and/or robot arm controller 4106 moves robot arm and/or coupling plate 3304 to the previous location, including light illumination and filtering, as previously configured. Adjust optical parameters. Such a configuration allows the surgeon to seamlessly view all stored locations in succession without taking his eyes off the displayed image of the procedure or moving his hands and instruments from the site.

In some embodiments, processor 4102 may be configured to allow an operator to create waypoints or positions/orientations prior to a surgical procedure. Waypoints may be provided in a sequence such that processor 4102 can advance through the designated waypoints after receiving advance input from an operator during a procedure. Processor 4102 may provide a three-dimensional representation of robotic arm 506, coupling plate 3304, and/or camera 300 via touch screen input device 1410a to allow an operator to virtually position stereoscopic visualization platform 516. . This may include providing magnification, working distance, and/or focus relative to the virtualized patient and/or based on alternative modality visualizations of the patient. Processor 4102 is configured to store visualization parameters, for example, in memory 1570 and/or memory 4120 of each waypoint.

In some embodiments, processor 4102 is configured to perform particular visualizations specific to particular procedures. For example, image recognition functionality in processor 4102 may be used to automatically align camera 300 with a target object. The surgical site several images are compared by the processor 4102 with previous images or images of the target object to provide recognition of the desired object and its location and orientation within the stereo image. The processor 4102 and/or the robotic arm controller 4106 may, for example, move the robotic arm 506 toward the object, zoom the camera 300 toward the object, and set desired image view attributes for a particular object and procedure. configured. For example, in ophthalmology, live retinal images can be compared with stored images so that, for example, the optic nerve head of a patient's retina can be found from image recognition. Processor 4102 and/or robot arm controller 4106 then automatically moves robot arm 506 and/or coupling plate 3304 to focus camera 300 such that the tip of camera 300 points to the optic nerve head to be diagnosed; /or change the magnification of the camera 300; Processor 4102 may then set camera 300 and/or monitor 512 to display images without reddening, allowing retinal features to be more easily distinguished from surrounding tissue.

In addition to storing stored visualizations and returning to stored visualizations, motion paths from one view to another may also be stored by an example processor. In the anastomosis example described above, processor 4102 and/or robotic arm controller 4106 may cause robotic arm 506 and/or camera 300 to follow the length of the blood vessel under high magnification to check for an aneurysm or other condition. Processor 4102 may be configured to recognize and follow continuous vessels as desired. Processor 4102 may perform a template matching routine on a limited set of pixels to actively determine the direction of movement of robotic arm 506 and/or camera 300.

The example processor 4102 can also program and store path paths within visualizations of objects made from different viewing angles. For example, a goniometric examination of a patient's eye can be performed by programming processor 4102 and/or robotic arm controller 4106 to pivot around a point within the eye. In this example, robotic arm 506 sweeps camera 300 in a generally conical motion so that the patient's eyes are viewed from many viewing angles. Such movement of surgical site visualization can be used to filter out spurious reflections from illumination or to select the best angle to view around obstacles at alternative viewing angles.

In some embodiments, processor 4102 is configured to reduce occlusion in depth map calculations. Occlusion is inherent in depth map calculations due to the parallax of two views of a stereoscopic image, where the first view sees some different part than the other view. As a result, each view does not see some portion of the other views. By moving the robot arm 506 between various configurations and recalculating the depth map while using knowledge of the three-dimensional location of the image pixels, occlusion is reduced. Depth maps are computed iteratively after known motion steps have been performed, after expected map changes have been computed, after errors have been identified by difference, and after an average map has been constructed. This can make the depth map more accurate.

E. Assisted Drive Embodiments In some embodiments, processor 4102 and/or robot arm controller 4106 execute one or more algorithms, routines, etc. defined by instructions stored in memory 1570 and/or 4120. The robotic arm 506 and/or the coupling plate 3304 are configured to provide motorized joint movement based on the detected force applied by the operator to move the stereoscopic visualization camera 300. In these embodiments, the assisted drive feature allows the robotic arm 506 to operate as an extension of the surgeon by moving the stereoscopic visualization camera 300 to a desired location and/or orientation. As described below, processor 4102 and/or robot arm controller 4106 monitors the forces/torques/travels applied by the operator and the position of the arm joints to infer operator intent and adjust robot arm 506 and/or robot arm controller 4106 accordingly. Or configured to move the coupling plate 3304.

FIG. 62 depicts a diagram illustrating an algorithm, routine, or procedure 6200 for providing assisted driving of stereoscopic visualization camera 300, according to example embodiments of the disclosure. Although procedure 6200 will be described with reference to the flow diagram shown in FIG. 62, it should be understood that many other methods of performing the steps associated with procedure 6200 can be used. For example, the order of many of the blocks may be changed, certain blocks may be combined with other blocks, and many of the blocks described are optional. Additionally, the operations described in step 6200 may be performed between multiple devices, including, for example, information processor module 1408 of example stereoscopic visualization camera 300 of FIG. 14 and/or joints R1-R9 and robot arm controller 4106 of FIG. It can be executed with In some examples, procedure 6200 may be performed by a program stored in memory 4120 of robotic arm controller 4106. The example procedure 6200 may be performed periodically when focus is applied to the camera 300. For example, procedure 6200 may sample force/torque data at every update cycle, which may be 1 ("ms"), 5ms, 8ms, 20ms, 50ms, etc.

In the illustrated embodiment, processor 4102 and/or robot arm controller 4106 receives force/torque output data 6201 from sensor 3306 related to the force applied to camera 300 by an operator. Processor 4102 and/or robot arm controller 4106 are configured to filter received output data 6201 (block 6202). Output data may include force and/or torque vectors. Filtering may include applying a first low pass filter, a second low pass filter, and/or a notch filter targeted to cart vibrations. In other examples, processor 4102 and/or robot arm controller 4106 may use one low pass filter and a notch filter.

The example processor 4102 and/or robot arm controller 4106 also receives joint position data 6203 from one or more joint sensors in the robot arm 506 and/or coupling plate 3304. Processor 4102 and/or robot arm controller 4106 use joint position data 6203 to provide compensation of the filtered force/torque output data (block 6204). The compensation may include gravity compensation and/or point of force compensation. In the case of gravity compensation, the effects of Earth's gravity are filtered out from the data. For force application point compensation, processor 4102 and/or robot arm controller 4106 may generate filtered data (and/or gravity compensated data) based on the point at which a force was applied to camera 300 (e.g., control arm 304). ). As discussed above in connection with FIG. 35, sensor 3306 is positioned an offset distance that is an angle from control arm 304. The offset distance and angle causes the force applied to control arm 304 to be slightly shifted in direction and angle as detected by sensor 3306. Force application compensation adjusts the force value as if the force were applied directly to the sensor 3306 rather than the control arm 304. Force application compensation may be predetermined based on the known angle and/or distance between sensor 3306 and control arm 304. Together, the gravity compensation and force application point compensation modify the filtered force/torque data to produce a force/torque vector that is proportional to the force/torque provided by the operator to the camera's control arm 304.

The example processor 4102 and/or robot arm controller 4106 also uses the joint position data 6203 in conjunction with the compensated and filtered force/torque output data to perform coordinate transformations from the force/torque system to the global system or robot space. (block 6206). The transformation may include one or more predefined equations and relationships based on the known robot space and sensor 3306 orientation. The example processor 4102 and/or robot arm controller 4106 also uses the joint position data 6203 to perform coordinate transformations between the camera system of the stereoscopic visualization camera 300 and the global system or robot space (block 6208). . The coordinate transformation of the camera system may be based on optical calibration parameters mapped to the robot space of the robot arm 506, as described above.

After performing the coordinate transformation, the example processor 4102 and/or robot arm controller 4106 are configured to transform the force/torque vector into one or more translation/rotation vectors using at least one sigmoid function. (block 6210). The creation of translation/rotation vectors generates an estimate of the operator's intended orientation. The translation and rotation information is used to determine how the joints of the robot arm 506 should be rotated to mirror, match, and/or approximate the operator's intended movement.

In some examples, the example processor 4102 and/or robot arm controller 4106 is configured to apply robot velocity scaling to the translation/rotation vector (block 6212). Velocity scaling may be based on the operating conditions of the robot arm 506, for example. For example, velocity scaling may be applied based on, for example, once a surgical procedure has begun, to prevent the arm from unintentionally hitting operating room staff, equipment, and/or the patient at relatively high speeds. If a procedure has not yet begun, the example processor 4102 and/or robotic arm controller 4106 may provide less velocity scaling to calibrate or set up the robotic arm 506 when a patient is not present.

The example processor 4102 and/or robot arm controller 4106 determines potential movement sequences for the joints of the robot arm 506 based on the scaled translation/rotation vectors. While evaluating possible sequences, processor 4102 and/or robot arm controller 4106 identify joint singularities to avoid, thereby excluding corresponding translation motions of robot arm 506 (block 6214). As mentioned above, singularities may include elbow locks or other locations that may be susceptible to hysteresis and backlash. The processor 4102 and/or the robot arm controller 4106 are configured to select a movement sequence (block 6216) after the movement singularity is eliminated using, for example, Jacobian kinematics (e.g., inverse Jacobian matrix). . The Jacobian kinematics equations define how a particular joint of the robot arm 506 and/or coupling plate 506 should be moved based on the scaled translation/rotation vectors. Jacobian kinematics provides velocity control, while inverse kinematics, described below, provides position control. In some embodiments, processor 4102 and/or robot arm controller 4106 may instead use inverse kinematics or other robot arm control routines. Processor 4102 and/or robot arm controller 4106 specify how particular joints of robot arm and/or coupling plate 3304 are to be moved in a coordinated manner, e.g., joint rotation speed, joint rotation direction, and/or Or, determine a movement sequence that specifies the joint rotation duration. The movement sequence can also specify the sequence in which the joints of the robot arm 506 and/or the coupling plate 3304 are to be rotated. Any joints R1-R9 of the robot arm and/or coupling plate 3304 may rotate individually or have overlapping movements, depending on the movement sequence.

After the movement sequence is determined, processor 4102 and/or robot arm controller 4106 are configured to perform collision avoidance using joint velocity scaling and/or clearance. For example, the processor 4102 and/or the robot arm controller 4106 may determine whether the movement sequence bounds one or more joints and/or links of the robot arm 506 and/or the coupling plate 3304 or other areas such as the space around the patient or equipment. The method is configured to determine whether a defined Cartesian limit is approached. As described above in connection with FIG. 49, the processor 4102 and/or the robot arm controller 4106 may estimate the position of the links and/or joints in robot space from the movement sequence within one or more defined boundaries and/or or can be compared to the angular limit. Based on the distance from the boundary, processor 4102 and/or robot arm controller 4106 applies one or more joint velocity limits via scaling values (block 6218). Processor 4102 and/or robot arm controller 4106 may also, for example, prevent links of robot arm 506 from hitting each other and/or prevent robot arm 506, coupling plate 3304, and/or camera 300 from crossing boundaries. One or more joint position limits may also be applied (block 6220). The location immediately before the position limit (e.g., 1 centimeter (“cm”) before, 2 cm before, 10 cm before the position limit, etc.) and/or the location of the position limit is in Cartesian robot space where the value of the scaling factor is “0”. can correspond to a location in .

In some examples, the processor 4102 and/or the robot arm controller 4106 may perform Jacobian kinematics with boundaries provided as input to the equation, with higher cost factors for movement through areas near the boundaries. is provided. Use of boundary cost factors allows processor 4102 and/or robot arm controller 4106 to avoid locations near boundaries, if possible, when determining movement sequences. The cost factor may include an inverse proportionality to the reduction of the scaling factor associated with a particular location in robot space. One scaling factor may be applied to each joint/link, or a separate scaling factor may exist for each joint at the same location in robot space.

After providing collision avoidance, the example processor 4102 and/or robot arm controller 4106 are configured to provide correction for relatively fast reversals of the robot arm 506 (block 6222). Processor 4102 and/or robot arm controller 4106 may implement a zero phase delay algorithm to reject directional shocks that cause one or more joints to quickly change rotational direction. A zero phase delay algorithm may be implemented, for example, with a filter that prevents the robot arm from backtracking or rocking if the operator reverses direction too quickly.

As shown in FIG. 62, the example processor 4102 and/or robot arm controller 4106 is configured to confirm commands for a movement sequence (block 6224). Processor 4102 and/or robot arm controller 4106 may check the command to ensure that the command (or a signal indicative of the command) is within joint motor operating parameters (eg, duration, rotational speed, etc.). Processor 4102 and/or robot arm controller 4106 may also validate commands by comparing them to current thresholds to ensure that robot arm 506 does not consume excessive current during any phase of the movement sequence. .

The example processor 4102 and/or robot arm controller 4106 may also apply one or more anti-noise filters to the movement commands or signals indicative of the movement commands (block 6226). The filter may include a high frequency low pass filter that removes high frequency noise components that may induce transient signals in the joint motor. After any filtering, processor 4102 and/or robot arm controller 4106 transmits one or more commands via one or more signals or messages to the appropriate joints of robot arm and/or coupling plate 3304 according to the movement sequence. to the motor (block 6228). The transmitted commands cause the motors at each joint to move the robot arm 506 and/or the coupling plate 3304, thereby moving the camera 300 as intended by the operator. The example procedure 6200 may be repeated as long as the operator applies force to the camera 300.

FIG. 63 depicts an illustration of an example procedure 6300 for moving an example visualization camera 300 via an input device 1410, according to example embodiments of the present disclosure. The sequence of steps 6300 is substantially the same as step 6200 of FIG. 62, but blocks 6202-6206 related to sensor 3306 are deleted. In the illustrated example, control input 6301 is received from an input device 1410 such as a button on control arm 304, a foot pedal, a joystick, a touch screen interface, etc. Control input 6301 indicates directional movement of the camera within the Cartesian robot space of robot arm 506.

As shown in FIG. 63, control inputs 6301 are combined with joint position data 6203 from one or more joint sensors in robot arm 506 and/or coupling plate 3304 to provide a control input from a camera system to a global system and/or a robot system. A coordinate transformation to space is performed (block 6208). The example procedure 6300 then continues in the same manner as discussed with respect to procedure 6200. Processor 4102 and/or robot arm controller 4106 accordingly move robot arm 506, coupling plate 3304, and/or camera 300 to a desired location and/or orientation based on control input 6301 received from input device 1410.

F. Target Lock Embodiments In some embodiments, processor 4102 and/or robot arm controller 4106 execute one or more algorithms, routines, etc. defined by instructions stored in memory 1570 and/or 4120. , the robotic arm 506 and/or the coupling plate 3304 are configured to enable the target locking feature to be provided. In these embodiments, the target locking feature allows the robotic arm 506 to operate as an extension of the surgeon by allowing the stereoscopic visualization camera 300 to be redirected while locking onto the target surgical site. As described below, the processor 4102 and/or the robot arm controller 4106 monitor the forces/torques/travels applied by the operator and the position of the arm joints to infer the operator's intent and adjust the focus of the camera 300 accordingly. The robot arm 506 and/or the coupling plate 3304 are configured to reorient the robot arm 506 and/or the coupling plate 3304 so that it remains locked or stationary.

The target lock feature allows the camera 300 to be reoriented by constraining all motion to the surface of the virtual sphere. The tip of camera 300 is placed on the outer surface of the virtual sphere (eg, the upper hemisphere of the virtual sphere), and the focal point or target surgical site of camera 300 constitutes the center of the virtual sphere. The example processor 4102 and/or robot arm controller 4106 maintains the camera 300 centered on the sphere so that the operator can maintain the camera 300 while maintaining focus on the target surgical site during movement. 300 can be moved over the outer surface of the virtual sphere. The target locking feature allows the operator to easily and quickly obtain widely different views of the same target area.

FIG. 64 depicts a diagram illustrating an algorithm, routine, or procedure 6400 for providing target locking to stereoscopic visualization camera 300, according to example embodiments of the disclosure. Although procedure 6400 will be described with reference to the flow diagram shown in FIG. 64, it should be understood that many other methods of performing the steps associated with procedure 6400 can be used. For example, the order of many of the blocks may be changed, certain blocks may be combined with other blocks, and many of the blocks described are optional. Additionally, the operations described in step 6400 may be performed between multiple devices, including, for example, information processor module 1408 of example stereoscopic visualization camera 300 of FIG. 14 and/or joints R1-R9 and robot arm controller 4106 of FIG. It can be executed with In some examples, procedure 6400 may be performed by a program stored in memory 4120 of robotic arm controller 4106.

The example procedure 6400 is similar to the assisted drive procedure 6200. However, procedure 6400 provides joint position commands to maintain the focus of camera 300, while example procedure 6200 provides joint velocity calculations. The example procedure 6400 includes identifying a desired force/movement vector input by an operator and moving the camera 300 while one or more joints of the robot arm 506 and/or coupling plate 3304 move to redirect the camera 300. Compute the rotational transformation so that the focus of 300 remains stationary. Redirecting the camera 300 allows the target surgical site to be imaged from different angles. Redirection may be necessary if the first viewing path is occluded, for example by an instrument, and the surgeon wishes to maintain the current focus.

The example procedure 6400 begins when an operator selects a target lock button on the input device 1410, which causes a command message or signal to be sent to the processor 4102 and/or the robot arm controller 4106. After receiving the message, processor 4102 and/or robot arm controller 4106 operate in target lock mode, in which the working distance and/or focus remains stationary while allowing the operator to change the orientation of camera 300. When held, reorientation of camera 300 by an operator causes one or more joints of the robot arm and/or coupling plate 3304 to provide assisted movement. Once the command is received, the example processor 4102 and/or robotic arm controller 4106 may record the current working distance, magnification, force, and/or other optical parameters of the camera 300. Processor 4102 and/or robot arm controller 4106 may also record a current image of the FOV.

After procedure 6400 is initiated, processor 4102 and/or robot arm controller 4106 receives force/torque output data 6201 from sensor 3306 related to forces applied by an operator of camera 300. As discussed in connection with FIG. 62, processor 4102 and/or robot arm controller 4106 filter data 6102 and provide gravity/force application compensation (blocks 6202 and 6204). Also similar to FIG. 62, processor 4102 and/or robot arm controller 4106 may use joint position data 6203 in conjunction with compensated and filtered force/torque output data to determine whether the force/torque system and global or robot space (block 6206). Processor 4102 and/or robot arm controller 4106 also use joint position data 6203 to perform coordinate transformations between the camera system of stereoscopic visualization camera 300 and the global system or robot space (block 6208). The example processor 4102 and/or robot arm controller 4106 also performs a transformation from global or robot space to spherical coordinates corresponding to a virtual sphere (block 6410).

After the coordinate transformation, the example processor 4102 and/or robot arm controller 4106 is configured to scale the trajectory velocity (block 6412) based on, for example, the mode of operation of the camera 300. The scaling may be similar to the scaling performed in block 6212 of FIG. 62. The example procedure 6400 of FIG. 64 continues with the processor 4102 and/or robot arm controller 4106 calculating a sphere end point (block 6414). The calculation of the sphere endpoints provides an estimate of the operator's desired direction of movement and determines how much the camera 300 should be moved across the virtual sphere without rotating the sphere.

FIG. 65 shows a diagram illustrating a virtual sphere 6500 of target locking features, according to an example embodiment of the disclosure. As shown in FIG. 65, stereoscopic visualization camera 300 is virtually placed on sphere 6500 based on the current position specified from joint position data 6203. The view vector of camera 300 points to the tip, shown as an xyz target, located at the center of sphere 6500. Processor 4102 and/or robot arm controller 4106 use the transformed force/torque data to determine how to move camera 300 on the sphere along the surface of sphere 6500 while maintaining the view vector pointing at the xyz target. is configured to determine whether any given point on the sphere is given by an equation that is a function of the rotation sphere angles "v" and "u". If force/torque data is used, the processor 4102 and/or the robot arm controller 4106 may use the "x" and "y" components corresponding to the translational force to determine the virtual sphere to identify the sphere endpoints. 6500 directly determines how the camera 300 should be moved.

Processor 4102 and/or robot arm controller 4106 may identify sphere endpoints differently for different inputs. For example, as shown in FIG. 63, if input is received via input device 1410, processor 4102 and/or robot arm controller 4106 may select "up," "down," "left," and "right." from camera coordinates to robot space coordinates provided as x,y vectors. Similar to force/torque data, the x,y vectors are used by processor 4102 and/or robot arm controller 4106 to directly determine how camera 300 should be moved on virtual sphere 6500 to locate the sphere endpoints. used. It should be appreciated that if the input is received via an input device, the operations discussed in conjunction with blocks 6202-6206, as shown in FIG. 63, may be omitted.

In some examples, processor 4102 and/or robot arm controller 4106 are configured to receive trajectory input data. In these examples, processor 4102 and/or robot arm controller 4106 repeatedly moves virtual sphere 6500 along sphere angle "u" while keeping sphere angle "v" constant. Iterative movement along the sphere angle "u" allows the sphere endpoints at the trajectory input to be identified. Although the input is applied to virtual sphere 6500, it should be understood that in other examples the input may be applied to other shapes. For example, virtual sphere 6500 may alternatively be defined as a virtual cylinder, ellipse, oval, pyramid/frustum, etc.

In other examples, processor 4102 and/or robot arm controller 4106 are configured to receive levelscope input data. In these examples, processor 4102 and/or robot arm controller 4106 repeatedly moves virtual sphere 6500 along sphere angle "v" while keeping sphere angle "u" constant. Repeated movement along sphere angle “v” moves camera 300 to the top of virtual sphere 6500.

Returning to FIG. 64, after the sphere endpoints are identified, the processor 4102 and/or robot arm controller 4106 moves the camera 300 along the virtual sphere to the identified endpoints and then moves the camera 300 along the x, y, z The system is configured to calculate the amount of rotation required to maintain view of the target (block 6416). Processor 4102 and/or robot arm controller 4106 may also provide anti-yaw correction (block 6418) during this calculation. In other words, processor 4102 and/or robot arm controller 4106 determines, given a new position of camera 300 on virtual sphere 6500, that the view vector or tip of camera 300 corresponds to the target surgical site or focal point on the virtual sphere. 6500 is configured to determine how to point the camera 300 so that it is provided with the same x, y, z target pointing to the center of the 6500.

During this step, processor 4102 and/or robot arm controller 4106 determines the joint angles of robot arm 506 and/or coupling plate 3304 required to achieve the desired orientation. After the x, y, z spherical endpoints are calculated at block 6414, processor 4102 and/or robot arm controller 4106 determine roll and pitch amounts for camera 300. In some embodiments, the calculation is a two-step process. First, processor 4102 and/or robot arm controller 4106 calculates an initial 4x4 transformation matrix T that provides movement of camera 300 without rotation, given the x, y, z sphere endpoints. The processor 4102 and/or the robot arm controller 4106 then causes the camera 300 to view the target located in x, y, z (and/or positioned at the x, y, z sphere endpoint) in a subsequent joint rotation cycle. The local roll amount and pitch amount are calculated to maintain the following. The processor 4102 and/or the robot arm controller 4106 may calculate the roll and pitch amounts using equations (4) and (5) below, where T _next corresponds to a 4x4 transformation matrix. . Calculations can be performed on each update cycle (eg 8ms).

In the above equation (4), _{xtaregt_next} , _{ytaregt_next} , and _{ztaregt_next} are constraints on the _Tnext matrix. The above constraints specify that the roll and pitch angles are chosen such that the x, y, z equations above are valid. In other words, the x, y, z location of the target in the next updated cycle of joint rotation needs to be the same location as the x, y, z location of the target in the current cycle. The constraints allow camera 300 to rotate through roll and pitch angles, but remain locked relative to the x, y, z location.

Further, -sin θ in the bottom row of the first matrix of equation (5) corresponds to the pitch angle, while sin θ in the bottom row of the second matrix corresponds to the roll angle. A closed-form representation of pitch may exist given the function cos(roll). The processor 4102 and/or the robot arm controller 4106 use an iterative method to estimate the roll calculated as a function cos(roll) with a pitch equal to fn(cos(roll)) to correct the above equation. roll/pitch solution pairs can be generated.

After the roll and pitch amounts are calculated from the operations described in connection with blocks 6416 and 6418, the example processor 4102 and/or robot arm controller 4106 provides singularity avoidance and calculates inverse kinematics. , are configured to determine joint rotations that achieve a new x, y, z position of camera 300 along virtual sphere 6500, as well as roll and pitch amounts (blocks 6214 and 6420). Inverse kinematics calculations allow processor 4102 and/or robot arm controller 4106 to determine a movement sequence for the joints of robot arm 506 and/or coupling plate 3304.

The example processor 4102 and/or robot arm controller 4106 may apply error corrections to the movement sequence in addition to joint velocity limits and/or position limits (blocks 6418, 6218, 6220). As discussed above in connection with FIG. 62, limits and error corrections ensure that the robot arm 506, camera 300, and/or coupling plate 3304 do not hit themselves, cross one or more boundaries, and/or may not be within acceptable joint positions. Processor 4102 and/or robot arm controller 4106 may also perform commands (or signals indicative of commands) prior to sending commands (or signals indicative of commands) to one or more joints R1-R9 of robot arm 506 and/or coupling plate 3304 based on the movement sequence. , commands to the joints of the movement sequence may also be verified to provide anti-noise filtering (blocks 6224, 6226, 6228). The example procedure 6400 may then terminate if no other movement is detected. Otherwise, procedure 6400 is repeated at regular intervals (eg, 10ms, 20ms, etc.) when operator input is received.

In some embodiments, processor 4102 and/or robotic arm controller 4106 may provide target lock tracking of the equipment. In these examples, the xyz target at the center of virtual sphere 6500 is replaced with a dynamic trajectory corresponding to the moving target. Such features may, for example, enable tracking of spinal instruments. In these embodiments, the device may include one or more fiducials and/or other markers. An example stereoscopic visualization camera 300 records images that include fiducials. Processor 4102 and/or robot arm controller 4106 may perform a coordinate transformation from camera-shaped space to robot space to determine how the instrument is moving along the x, y, and z axes. An example processor 4102 and/or robot arm controller 4106 may track how the fiducial moves in the image and determine a corresponding x, y, z movement vector. In some cases, the x, y, z vectors may be input into the sphere endpoint calculation of block 6414 of FIG. 64 to change the center location of virtual sphere 6500. In response to movement of sphere 6500, processor 4102 and/or robot arm controller 4106 determine how to position robot arm 506 to maintain the same working distance and/or orientation at the new target location. Processor 4102 and/or robot arm controller 4106 may then apply inverse kinematics to determine joint rotation of robot arm 506 and/or coupling plate to track movement of the target. Similar to steps 6200 and 6400, processor 4102 and/or robot arm controller 4106 performs error correction, joint limits, filters, and/or validation before sending commands to joints as specified in the determined movement sequence. can be applied.

Conclusion It will be appreciated that each of the systems, structures, methods, and procedures described herein may be implemented using one or more computer programs or components. These programs and components may be stored in random access memory ("RAM"), read-only memory ("ROM"), flash memory, magnetic or optical disks, optical memory, or other storage media, and combinations and derivatives thereof. may be provided as a sequence of computer instructions on any conventional computer-readable medium, including. The instructions may be configured to be executed by a processor, which, when executing the sequence of computer instructions, performs or facilitates performing all or a portion of the disclosed methods and procedures.

It should be understood that various modifications and variations to the example embodiments described herein will be apparent to those skilled in the art. Such changes and modifications may be made without departing from the spirit and scope of the invention and without diminishing its intended advantages. It is therefore intended that such changes and modifications be covered by the appended claims. Additionally, consistent with current U.S. law, unless "means" or "step" are explicitly recited in a claim, 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. It is to be understood that it is not intended that Article 6, paragraph 6 be invoked. Therefore, the claims are not intended to be limited to the corresponding structures, materials, or acts described herein or their equivalents.

Claims (21)

A robot imaging device,
a base section configured to be connected to a secure structure or cart;
A robot arm,
a first end connected to the base section;
a second end including a coupling interface; and a plurality of joints and links connecting the first end to the second end, each joint configured to rotate the joint about an axis. a robot arm including a plurality of joints and links including a motor configured to and a joint sensor configured to transmit a position of each of the joints;
a stereoscopic camera connected to the robotic arm at the coupling interface and configured to record left and right images of a target surgical site to generate a stereoscopic image stream of the target surgical site; and,
a sensor positioned at the coupling interface and configured to detect translational and rotational forces applied to the stereoscopic camera by an operator and transmit output data indicative of the translational and rotational forces;
one or more instructions and/or data specifying a direction, speed, and duration of rotation of each of the joints of the robot arm based on at least the current position of the robot arm and detected translational and rotational forces; a memory storing at least one algorithm defined by the structure;
a coupling plate having a first end configured to connect to the coupling interface of the robot arm and a second end including a second coupling interface configured to connect to the stereoscopic camera; and,
at least one processor communicatively coupled to the sensor and the robotic arm;
Equipped with
The at least one processor includes:
receiving from the sensor the output data indicative of the translational force and the rotational force;
using the at least one algorithm in the memory to determine a movement sequence for the robotic arm based on the current position of the robotic arm and the output data from the sensor;
rotating the at least one joint of the robotic arm based on the determined movement sequence via one or more motor control signals provided to the at least one joint;
is configured to do
the rotation of the at least one joint provides powered movement of the robotic arm based on the detected translational force and the detected rotational force applied to the stereoscopic camera by the operator;
The coupling plate is selectively movable by the operator and includes one or more joints having a rotatable latching mechanism. 2. The apparatus of claim 1, wherein the at least one processor is configured to determine the current position of the robotic arm based on output data from the joint sensors of the plurality of joints. 3. The apparatus of claim 1 or claim 2, wherein the sensor comprises at least one of a six degrees of freedom haptic device or a torque sensor. the stereoscopic camera includes at least one control arm having a release button to enable the powered movement;
The at least one processor includes:
receiving an input message indicating that the release button has been selected;
determining the movement sequence using the output data from the sensor after receiving the input message associated with the release button;
3. A device according to claim 1 or claim 2, configured to perform. 1 or 2 , wherein the coupling plate includes at least one joint comprising a joint sensor configured to transmit respective positions of the joint and motor controllable by the at least one processor according to the movement sequence. The device according to item 4. 6. The apparatus of claim 5, wherein the sensor is located at the coupling interface or the second coupling interface. The coupling plate includes a second joint that allows the stereoscopic camera to be manually rotated by an operator between horizontal and vertical orientations, the second joint transmitting a position of the second joint. 6. The apparatus of claim 5, including a joint sensor configured to. 5. The apparatus of claim 1 or claim 4, wherein the stereoscopic camera includes a housing including a bottom surface configured to connect to the robotic arm at the coupling interface. A robot imaging device,
A robot arm,
a first end connected to a secure structure;
a second end including a coupling interface; and a plurality of joints and links connecting the first end to the second end, each joint configured to rotate the joint about an axis. a robot arm including a plurality of joints and links including a motor configured to and a joint sensor configured to transmit a position of each of the joints;
an imaging device connected to the robotic arm at the coupling interface, the imaging device configured to record images of a target surgical site;
a sensor positioned at the coupling interface and configured to detect a force and/or torque applied to the imaging device by an operator and transmit force and/or torque output data indicative of the force and/or torque; and,
a coupling plate having a first end configured to connect to the coupling interface of the robot arm and a second end including a second coupling interface configured to connect to a stereoscopic camera; ,
at least one processor communicatively coupled to the sensor and the robotic arm;
Equipped with
The at least one processor includes:
receiving the force and/or torque output data from the sensor;
converting the force and/or torque output data into translational and rotational vectors;
using kinematics to determine a movement sequence of the robot arm based on the current position of the robot arm and the translation vector and the rotation vector, the movement sequence comprising: specifying or determining rotational direction, speed, and duration of at least some of the movements;
rotating the at least one joint of the robotic arm based on the determined movement sequence via one or more motor control signals provided to the at least one joint;
is configured to do
The coupling plate is selectively movable by the operator and includes one or more joints having a rotatable latching mechanism. The processor includes:
determining at least one scaling factor based on at least one of the current position of the robot arm or a characteristic position of the robot arm based on the movement sequence;
applying the scaling factor to at least one joint velocity of the movement sequence;
10. The apparatus of claim 9, configured to perform. the at least one scaling factor is configured based on a distance of the robotic arm or the imaging device from a virtual boundary;
11. The apparatus of claim 10, wherein the at least one scaling factor decreases to a value of "0" as the virtual boundary is approached. 12. The apparatus of claim 11, wherein the virtual boundary corresponds to at least one of a patient, medical equipment, or operating room staff. 12. The apparatus of claim 11, wherein the processor is configured to cause a display device to display an icon indicating that the at least one scaling factor has been applied to the movement sequence. The processor includes:
determining at least one scaling factor based on joint angles between joints or between joint limits of the robot arm;
applying the scaling factor to at least one joint velocity of the movement sequence;
14. A device according to claim 9 or 13, configured to perform. The processor includes:
providing gravity compensation to said force and/or torque output data;
providing force application compensation to the force and/or torque output data to compensate for an offset between a location of the sensor and a location of the imaging device at which the force and/or torque is applied by the operator; ,
14. A device according to claim 9 or 13, configured to perform. The processor includes:
identifying or identifying joint singularities of the plurality of joints of the robot arm to control hysteresis and backlash;
determining the movement sequence based on the kinematics while avoiding robot arm movement through the joint singularity;
16. The apparatus of claim 9, claim 13 or claim 15, configured to perform. the coupling plate includes at least one joint including a joint sensor configured to transmit a position of each of the joints and a motor controllable by the at least one processor according to the movement sequence;
14. Apparatus according to claim 9 or claim 13, wherein the sensor is located at the coupling interface or the second coupling interface. 18. The apparatus of claim 17, wherein the robotic arm includes at least four joints and the coupling plate includes at least two joints. The processor controls the robot arm by sending one or more command signals to the motor of each of the joints indicating the rotational direction, the speed, and the duration of movement specified by the movement sequence. 18. The apparatus of claim 9, claim 13, or claim 17, configured to rotate at least one of the joints of. The processor compares images recorded by the imaging device when the robotic arm is moving during the movement sequence to determine whether the robotic arm is moving as determined during the movement sequence. 18. The apparatus of claim 9, claim 13, or claim 17, configured to confirm that. 18. The apparatus of claim 9, claim 13, or claim 17, wherein the kinematics includes at least one of inverse kinematics or Jacobian kinematics.

Images