JP7138802B2 - Alignment of Preoperative Scan Images to Real-Time Surgical Images for Mediated Reality Views of the Surgical Site - Google Patents

Description

The present invention relates to image alignment, and more particularly to alignment of preoperative scan images to real-time surgical images for a mediated reality view of the surgical site.

When performing surgery, surgeons often rely on preoperative three-dimensional images of the patient's anatomy, such as CT (computed tomography) scan images. However, the usefulness of such preoperative images is limited because the images cannot be easily integrated into the surgical procedure. For example, because the images are captured in a preoperative session, the relative anatomical positions captured in the preoperative images may not match the actual positions during the surgical procedure. Furthermore, utilizing preoperative images during surgery requires the surgeon to divide his attention between the surgical field and the display of the preoperative images. Furthermore, carefully navigating between different layers of preoperative images may require significant attention that takes away from the surgeon's focus in surgery.

1 is an exemplary embodiment of an imaging system; An example of a surgical environment using an imaging system for mediated-reality assisted surgery. FIG. 12A is a top view of an embodiment of a body patch with fiducial markers for aligning real-time surgical images to pre-operative scan images. FIG. 10 is a cross-sectional view of an embodiment of a body patch with fiducial markers for aligning real-time surgical images to pre-operative scan images; FIG. 4 is a flow chart illustrating aspects of a process for generating a mediated-reality view of a surgical site based on pre-operative images; FIG. 4 illustrates aspects of a process for aligning pre-operative images captured before the patient is positioned for surgery with post-positioning images captured after the patient is positioned for surgery;

The drawings and the following description relate, by way of illustration only, to preferred embodiments. From the discussion that follows, it will be readily appreciated that alternative embodiments of the structures and methods disclosed herein will be readily recognized as viable alternatives that may be used without departing from the principles of what is being claimed. It should be noted.

Reference will now be made in detail to certain aspects, examples of which are illustrated in the accompanying drawings. It is noted that, wherever practicable, similar or identical reference markings may be used in the drawings to indicate similar or identical functionality. The drawings depict aspects of the disclosed system (or method) for purposes of illustration only. Those skilled in the art will readily recognize from the ensuing description that alternative embodiments of the structures and methods illustrated herein may be used without departing from the principles set forth herein. .

A mediated-reality system for surgical applications projects pre-operative images and real-time captured images of the surgical site onto a head-mounted display worn by the surgeon during the surgical procedure. Incorporate into the visualization that will be used. The mediated reality system tracks the position of the surgeon's head and generates a real-time image of the surgical site from a virtual camera's perspective corresponding to the surgeon's head position, mimicking a point that would appear natural to the surgeon. Furthermore, the mediated reality system aligns the pre-operative image with a real-time image from the virtual camera's point of view, and a selection of aligned pre-operative three-dimensional images overlaid on the real-time image representing the virtual camera's point of view. Provides mediated-reality visualization of the surgical site with the mediated portion. Thus, the mediated reality system provides the surgeon with an underlying three-dimensional anatomy of the patient prior to making an incision and throughout the procedure, even when anatomical features may be obscured from the surgeon's view in real-time images. make it visible. Furthermore, the technology beneficially provides visualization in a manner that does not distract the surgeon from the surgical site, thus enhancing the surgeon's ability to perform surgery with high efficiency and precision.

In embodiments, patches containing fiducial markers are placed on the patient's body and remain in place during preoperative imaging and after the patient is positioned for surgery. The fiducial markers include patterns that can be recognized by the mediated reality system in both pre-operative images and real-time images captured after the patient is positioned for surgery. The mediated reality system aligns the pre-operative image with the real-time image based on the detected positions of fiducial markers found in both image sets. For example, the mediated reality system may apply one or more transformations to the preoperative image to align fiducial markers detected in the preoperative image with corresponding fiducial markers detected in the real-time image.

In another aspect, a three-dimensional image, such as an ultrasound image or a fluoroscopic image, may be captured after the patient is positioned for surgery, preoperative image scanning and patient positioning for surgery. It can be used to predict changes in the three-dimensional position of anatomical features that occur during The preoperative image then maps the positioning of the anatomical features in the preoperative image scan to the detected locations of the anatomical features seen in the three-dimensional images captured after the patient is positioned for surgery. It can be bent to align to compensate for changes that may occur.

In yet another aspect, a combination of fiducial markers and post-positioning images can be used to align pre-operative images to real-time images in mediated reality visualization. For example, the pre-operative image and the post-positioning image are compared to determine how the positions of the fiducial markers change in the three-dimensional space between the images, and a transform is derived to transform the pre-operative image. to align the fiducial markers with their respective locations in the image after positioning.

In certain exemplary aspects, the method generates a mediated reality view of the surgical site. A preoperative image is received representing a three-dimensional anatomy of the patient at a first position. Based on the preoperative image, coordinates are identified in three-dimensional preoperative image space corresponding to the location of fiducial markers provided on the patch applied to the patient. Real-time images from the camera array are received after the patient is positioned for surgery at the second location. Based on the real-time images, coordinates in the three-dimensional real-time image space are identified and correspond to the locations of fiducial markers provided on the patch applied to the patient. A transform is applied to the preoperative image to substantially align the locations of the fiducial markers in the preoperative image with the locations of the fiducial markers in the real-time image. The transformed pre-operative image is overlaid on the real-time image to generate a mediated reality view, which is provided to the display device for display.

In another particular aspect, the method generates a mediated reality view of the surgical site without necessarily relying on fiducial markers on the patch. In this aspect, a pre-operative image is received representing a three-dimensional anatomy of the patient in a first position. A post-positioning image captured after the patient is surgically placed in the second position is received. Based on the pre-operative and post-positioning images, a set of corresponding features found in the pre-operative and post-positioning images is identified. A first transformation is applied to the preoperative image to substantially align the locations of corresponding features in the preoperative image with their respective locations in the post-positioning image to produce an initial transformed preoperative image. do. A camera array captures real-time images of the patient and overlays the initial transformed preoperative images onto the real-time images to produce an initial mediated reality view provided to the display device for display. Further details regarding the above-described aspects and descriptions of additional aspects are provided below.

FIG. 1 illustrates exemplary aspects of a mediated reality system 100 including an image processing device 110, a camera array 120, a display device 140, and an input controller 150. FIG. In alternate aspects, the mediated reality system 100 may include additional or different components.

Camera array 120 includes multiple cameras 122 (eg, camera 122-1, camera 122-2, . Cameras 122 may be physically arranged in a particular configuration such that their physical locations and orientations are fixed with respect to each other. For example, camera 122 may be structurally secured by a mounting structure that mounts camera 122 in a predefined fixed location and orientation. Cameras 122 of camera array 120 may be positioned such that adjacent cameras may share overlapping views of scene 130 . Furthermore, cameras 122 in camera array 120 may be synchronized to capture images 190 of scene 130 at substantially the same time (eg, within a threshold time error). Still further, camera array 120 may include one or more projectors 124 that project structured light patterns onto scene 130 . In aspects, camera 122 may include a light field camera that captures light field information of scene 130 . Here, the camera 122 captures both the intensity of the light and the direction of the rays representing the scene. Hence, the light field image 190 encodes depth information and enables scene reconstruction as a three-dimensional image.

The image processing device 110 receives the image 190 captured by the camera array 120 and processes the image to synthesize an output image corresponding to the viewpoint of the virtual camera. Here, the output image corresponds to an approximation of the image of the scene 130 that would be captured by a camera placed at an arbitrary position and orientation corresponding to the viewpoint of the virtual camera. Image processing device 110 combines output images from a subset (eg, one or more) of cameras 122 in camera array 120 but does not necessarily utilize images 190 from all cameras 122 . For example, for a given virtual camera viewpoint, image processing device 110 generates a stereoscopic pair of images from two cameras 122 positioned and oriented to most closely match the virtual camera viewpoint. 190 can be selected. Furthermore, the image processing device 110 may detect structured light projected onto the scene 130 by the projector to estimate depth information of the scene. The depth information may be combined with the image 190 from the camera 122 to synthesize the output image as a three-dimensional rendering of the scene 130 as seen from the perspective of the virtual camera. Alternatively, the structured light projector 124 may be omitted, and the image processing device 110 may solely derive the three-dimensional rendering from the images 190 captured by the one or more cameras 122. .

The viewpoint of the virtual camera may be controlled by an input controller 150 that provides control inputs corresponding to the position and orientation of the virtual camera viewpoint. An output image corresponding to the viewpoint of the virtual camera is output to display device 140 and displayed by display device 140 . The output image can be updated at a high frame rate to synthesize a video representing the viewpoint of the virtual camera. Furthermore, the image processing device 110 beneficially processes the input received from the input controller 150 and processes the image 190 captured from the camera array 120 to provide a substantially An output image corresponding to the virtual viewpoint may be generated in real-time (eg, at least as fast as the frame rate of the camera array 120).

Furthermore, the image processing device 110 may receive a pre-operative image 170 representing a three-dimensional volume such as, for example, a CT scan image, an ultrasound image, or a fluoroscopic image. As described in further detail below, image processing device 110 may detect visual features in pre-operative image 170 that correspond to visual features in real-time image 190 captured by camera array 120 . Image processing device 110 then applies one or more transforms to pre-operative image 170 to convert features detected in pre-operative image 170 (or portions thereof) to correspondences detected in real-time image 190 . can be aligned with the features that The image processing device 110 may apply one or more transformations on a frame-by-frame basis such that the pre-operative image 170 is aligned with the real-time image 190 in each frame such that the virtual viewpoint changes. The image processing device 110 overlays the preoperative images 170 with real-time images 190 to provide the surgeon with a mediated reality view that can simultaneously visualize the surgical site and the underlying three-dimensional anatomy of the patient undergoing surgery. give.

In aspects, a scene 130 (eg, a surgical patient's body) may be prepared with patches 160 containing fiducial markers prior to capturing pre-operative images 170 . Image processing device 110 may identify certain features of the fiducial markers that allow identification of correspondence between features in pre-operative image 170 and real-time image 190 . Image processing device 110 transforms preoperative image 170 in such a way that the pattern of fiducial markers in preoperative image 170 will be aligned with the corresponding pattern seen in real-time image 190 from camera array 120 . may apply. For example, pre-operative image 170 may be translated, rotated, and/or bent to align fiducial markers with corresponding fiducial markers in real-time image 190 .

In an aspect, optionally, image processing device 110 also receives one or more captured post-positioning three-dimensional images 180 of scene 130 after the patient has been positioned for surgery. For example, post-positioning images 180 may include ultrasound or fluoroscopy images captured after the patient is positioned for surgery. Image processing device 110 may utilize post-positioning image 180 in determining a transform to apply to pre-operative image 170 to align pre-operative image 170 with real-time image 190 . In aspects, image processing device 110 identifies fiducial markers or anatomical features in post-positioning image 180 and applies one or more transformations to pre-operative image 170 to transform pre-operative image 170 into a post-positioning image. 180 may be aligned. The conversion step just described renders the preoperative image 170 useful for changes in the positioning of anatomical elements that may occur between capturing the preoperative image 170 and positioning the patient for surgery. Corrections may be made.

The image processing device 110 includes a processor and a non-transitory computer-readable medium that stores instructions that, when executed by the processor, perform the functions attributed to the image processing device 110 described herein. There is

For example, display device 140 may include a head-mounted display device or other display device for displaying output images received from image processing device 110 . In aspects, input controller 150 and display device 140 are integrated into a head-mounted display device, and input controller 150 includes motion sensors that detect the position and orientation of the head-mounted display device. A virtual viewpoint can then be derived to correspond to the position and orientation of the head-mounted display device, where the virtual viewpoint would be viewed by a viewer wearing the head-mounted display device. corresponds to Thus, in this aspect, the head-mounted display device is capable of providing real-time rendering of the scene as viewed by an observer without the head-mounted display. Alternatively, the input controller 150 is a user-controlled control device (e.g., mouse, pointing device, handheld controller, gesture recognition controller, etc.) that allows the viewer to manually control the virtual viewpoint displayed by the display device. ).

FIG. 2 illustrates exemplary aspects of the mediated reality system 100 for surgical applications. Here, aspects of the camera array 120 can be placed over the scene 130 (in this case, the surgical site) and placed via a swing arm 202 attached to a workstation 204 . The swing arm 202 may be manually moved or robotically controlled in response to the input controller 150 . Workstation 204 may include a computer that controls various functions of camera array 120 and display device 140, as well as a second computer capable of displaying a user interface for performing various configuration functions. , or may mirror the display in display device 140 . Image processing device 120 and input controller 150 may each be integrated into workstation 204, display device 140, or a combination thereof.

FIG. 3A illustrates exemplary aspects of a patch 160 applied to a patient 300 that may be utilized in the mediated reality system 100 to align preoperative images 170 with images 190 captured in real time. In aspects, the patch 160 comprises a thin, flexible adhesive patch that can be applied to the patient's skin and conformed to the contours of the patient's body. Patch 160 may be placed closest to the anatomy (eg, vertebrate 310) of interest for the relevant surgical procedure. Patches 160 can be uniquely identified by image processing device 110 in both preoperative image 170 and real-time image 190, even when viewed from different perspectives and under different lighting conditions. A pattern of markers 320 may be included. For example, fiducial markers 320 may each include one or more raised surfaces that form a recognizable three-dimensional geometry. In aspects, each marker may be unique within a particular region of patch 160 . In an aspect, the patches 160 each have a unique three-dimensional geometry, are each uniquely recognizable in the pre-operative image 170 and the real-time image 190, and are each located at a particular location on the human body. It contains a grid of fiducial markers (which can be of the same or unequal shape) that can be correlated. The fiducial markers 320 may each comprise different densities of material that can be distinguished by ultrasound or radio opaque images.

In aspects, patch 160 may be divided into pieces separated by perforated boundaries. The perforations allow one or more pieces of patch 160 to be easily removed from the patient without removing the entire patch 160 . For example, in one use case, a surgeon may remove a piece of patch over the location of the desired incision after the patient is positioned for surgery and image processing device 110 performs initial alignment calculations. . Remaining pieces of patch 160 that are not directly over the location of the incision may remain in place. Image processing system 110 may continue to detect fiducial markers 320 in the remainder of patch 160 throughout operation to update the alignment.

FIG. 3B shows an exemplary cross-sectional view of an exemplary fiducial marker 320 that may be integrated with patch 160. FIG. As illustrated, fiducial markers 320 have various three-dimensional structures that allow markers 320 to be distinguished from each other in real-time image 190, pre-operative image 170, and post-positioning image 180. density.

FIG. 4 illustrates exemplary aspects of a process for generating a mediated reality visualization of a surgical site using pre-operative images 170 aligned and overlaid on real-time images 190 . The image processing device 110 receives 402 the pre-operative image 170 . Image processing device 110 identifies 404 respective three-dimensional coordinates corresponding to the locations of fiducial markers in preoperative image 170 . For example, image processing device 110 detects fiducial markers in preoperative image 170 and maps the fiducial markers to a first set of three-dimensional coordinates representing the location of the human body in preoperative image space. For example, image processing device 110 may obtain a predefined mapping between uniquely recognizable structures of fiducial markers and their corresponding locations on patch 160 . The image processing device 110 receives 406 real-time images 190 of the scene captured by the camera array 120 . Image processing device 110 identifies 408 respective three-dimensional coordinates corresponding to the locations of fiducial markers in real-time image 190 . For example, image processing device 110 detects fiducial markers in real-time image 190 and maps each fiducial marker to a second set of three-dimensional coordinates representing a location on the human body in real-time image space. Image processing device 110 then applies 410 one or more transformations to preoperative image 170 to substantially align the pattern of fiducial markers in preoperative image 170 with the pattern in real-time image 190 . For example, in one aspect, the image processing device 110 performs an optimization algorithm to minimize the distance between the transformed coordinates of the pattern in the preoperative image 170 with the corresponding coordinates of the pattern in the real-time image 190. Identify transformations. The image processing device 110 overlays 412 the transformed pre-operative image onto the real-time image to generate a mediated reality view. The mediated reality view is then provided 414 to the display device 140 for display.

FIG. 5 illustrates aspects of a process for aligning a pre-operative image 170 captured before the patient is positioned for surgery with a post-positioning image 180 captured after the patient is positioned for surgery. do. The image processing device 110 receives 502 the pre-operative image 170 . Preoperative images may include, for example, CT scan images. The image processing device 110 receives 504 a post-positioning image 180 captured after the patient has been positioned for surgery. Post-positioning image 180 may include, for example, an ultrasound image or a fluoroscopy image. Image processing device 110 identifies 506 corresponding sets of features (eg, anatomical features or fiducial markers on patches) found in both pre-operative image 170 and post-positioning image 180 . The image processing device 110 then applies 508 the transform to the pre-operative image 170 to substantially align the locations of the corresponding features in the pre-operative image with the locations of the post-positioning image 180 . For example, image processing device 110 may apply translations, rotations, bends, or a combination thereof to determine the distance between the transformed coordinates of features in pre-operative image 170 to that of corresponding features in post-positioning image 180 . May be minimized by coordinates. The transformation beneficially compensates for changes in the location of anatomical features that may occur between capturing the pre-operative image 170 and positioning the patient for surgery. The transformed pre-operative image 170 may then be overlaid 510 on the real-time image 190 captured by the camera array 120 and provided 512 to the display device 140 for display as a mediated reality view of the surgical site.

In yet another aspect, transformations may be applied to pre-operative image 170 based on both post-positioning image 180 and fiducial markers detected in real-time image 190 . For example, in one aspect, a first transform is applied to pre-operative image 170 according to the process of FIG. 5 to obtain a first transform. A second transform is then applied according to the process of FIG. 4 to refine the alignment in each image frame based on the pattern of fiducial markers detected in the pre-operative image 170 and the real-time image 190 .

Upon reading this disclosure, those skilled in the art will appreciate additional structural and functional design alternatives to the disclosed embodiments as disclosed in accordance with the principles herein. Thus, while specific embodiments and applications have been illustrated and described, it is to be understood that the disclosed embodiments are not limited to the precise structures and components disclosed herein. Various modifications, changes, and variations that are apparent to those skilled in the art may be made in the arrangement, operation, and details of the methods and systems disclosed herein without departing from the scope of the described aspects.

Claims (24)

A method for generating a mediated reality view of a surgical site by a processor , comprising:
receiving by the processor a pre-operative image representing a three-dimensional anatomy of a patient in a first position;
Based on the preoperative image, the processor identifies coordinates in a three-dimensional preoperative image space corresponding to the locations of fiducial markers provided on the patch applied to the patient, the patch comprising: positioned on a patient's body, the fiducial marker being positioned over the patch ;
the processor receiving real-time images from a camera array after the patient has been placed for surgery at a second location;
based on the real-time image, the processor identifies coordinates in a three-dimensional real-time image space corresponding to the location of the fiducial marker provided on the patch applied to the patient;
applying a transform to the preoperative image such that the processor aligns the location of the fiducial marker in the preoperative image with the location of the fiducial marker in the real-time image, wherein the transform is applied. to do
performing an optimization algorithm to identify the transformation that minimizes the distance between transformed coordinates of the pattern of fiducial markers in the preoperative image with the coordinates of the pattern in the real-time image. , and
overlaying the transformed pre-operative image onto the real-time image to generate the mediated reality view;
and the processor providing the mediated reality view to a display device for display. 2. The method of claim 1, wherein the patch comprises a compliant adhesive patch placed closest to the anatomy of interest. A method for generating a mediated reality view of a surgical site by a processor , comprising:
receiving by the processor a pre-operative image representing a three-dimensional anatomy of a patient in a first position;
Based on the preoperative image, the processor identifies coordinates in a three-dimensional preoperative image space corresponding to locations of fiducial markers provided on the patient's applied patch, the fiducial markers comprising: a three-dimensional structure comprising one or more bulging surfaces, each of said fiducial markers having a unique geometry with respect to other fiducial markers in said patch, said patch resting on said patient's body; positioned, wherein the fiducial marker is positioned over the patch ;
the processor receiving real-time images from a camera array after the patient has been placed for surgery at a second location;
based on the real-time image, the processor identifies coordinates in a three-dimensional real-time image space corresponding to the location of the fiducial marker provided on the patch applied to the patient;
applying a transform to the pre-operative image to align the location of the fiducial marker in the pre-operative image with the location of the fiducial marker in the real-time image;
overlaying the transformed pre-operative image onto the real-time image to generate the mediated reality view;
and the processor providing the mediated reality view to a display device for display. 2. The method of claim 1, wherein the fiducial markers are arranged in a grid on the patch. A method for generating a mediated reality view of a surgical site by a processor , comprising:
receiving by the processor a pre-operative image representing a three-dimensional anatomy of a patient in a first position;
Based on the preoperative image, the processor identifies coordinates in a three-dimensional preoperative image space corresponding to locations of fiducial markers provided on the patient's applied patch, the fiducial markers comprising: said patch comprising materials of visually distinguishable different densities , said patch being placed on said patient's body, said fiducial marker being placed on said patch ;
the processor receiving real-time images from a camera array after the patient has been placed for surgery at a second location;
based on the real-time image, the processor identifies coordinates in a three-dimensional real-time image space corresponding to the location of the fiducial marker provided on the patch applied to the patient;
applying a transform to the pre-operative image to align the location of the fiducial marker in the pre-operative image with the location of the fiducial marker in the real-time image;
overlaying the transformed pre-operative image onto the real-time image to generate the mediated reality view;
and the processor providing the mediated reality view to a display device for display. 2. The method of claim 1, wherein the patch includes perforated borders for tearing the patch into pieces. A non-transitory computer readable recording medium storing instructions for generating a mediated reality view of a surgical site, the instructions, when executed by a processor, causing the processor to:
receiving a pre-operative image representing a three-dimensional anatomy of the patient in a first position;
identifying, based on the preoperative image, coordinates in three-dimensional preoperative image space corresponding to locations of fiducial markers provided on a patch applied to the patient, the patch being located on the patient's body; and the fiducial marker is positioned over the patch ;
receiving real-time images from a camera array after the patient has been placed for surgery at a second location;
determining, based on the real-time image, coordinates in three-dimensional real-time image space corresponding to the location of the fiducial marker provided on the patch applied to the patient;
applying a transform to the pre-operative image to align the location of the fiducial marker in the pre-operative image with the location of the fiducial marker in the real-time image, wherein applying the transform. teeth,
performing an optimization algorithm to identify the transformation that minimizes the distance between transformed coordinates of the pattern of fiducial markers in the preoperative image with the coordinates of the pattern in the real-time image. , and
overlaying the transformed pre-operative image onto the real-time image to generate the mediated reality view;
providing the Mediated Reality View to a display device for display;
A non-transitory computer-readable recording medium characterized by causing a step comprising : 4. The method of claim 3, wherein the patch comprises a flexible adhesive patch placed closest to the anatomy of interest. 4. The method of claim 3, wherein the fiducial markers are arranged in a grid on the patch. 4. The method of claim 3, wherein the patch includes perforated borders for tearing the patch into pieces. 6. The method of claim 5, wherein the patch comprises a flexible adhesive patch placed closest to the anatomy of interest. 6. The method of claim 5, wherein the fiducial markers are arranged in a grid on the patch. 6. The method of claim 5, wherein the patch includes perforated borders for tearing the patch into pieces. 8. The non-transitory computer readable medium of claim 7, wherein said patch comprises a flexible adhesive patch placed closest to an anatomical structure of interest. 8. The non-transitory computer-readable medium of Claim 7, wherein the fiducial markers are arranged in a grid in the patch. 8. The non-transitory computer-readable medium of claim 7, wherein the patch includes a perforated border for tearing the patch into pieces. A non-transitory computer readable recording medium storing instructions for generating a mediated reality view of a surgical site, the instructions, when executed by a processor, causing the processor to:
receiving a pre-operative image representing a three-dimensional anatomy of the patient in a first position;
identifying, based on the preoperative image, coordinates in a three-dimensional preoperative image space corresponding to locations of fiducial markers provided on patches applied to the patient, wherein the fiducial markers are one or a three-dimensional structure comprising a plurality of bulging surfaces, each of said fiducial markers having a unique geometry with respect to other fiducial markers in said patch, said patch being positioned on said patient's body; wherein the fiducial marker is positioned over the patch ;
receiving real-time images from a camera array after the patient has been placed for surgery at a second location;
determining, based on the real-time image, coordinates in three-dimensional real-time image space corresponding to the location of the fiducial marker provided on the patch applied to the patient;
applying a transform to the pre-operative image to align the location of the fiducial marker in the pre-operative image with the location of the fiducial marker in the real-time image;
overlaying the transformed pre-operative image onto the real-time image to generate the mediated reality view;
providing the Mediated Reality View to a display device for display;
A non-transitory computer-readable recording medium characterized by causing a step comprising : 18. The non-transitory computer readable medium of claim 17, wherein said patch comprises a flexible adhesive patch placed closest to an anatomical structure of interest. 18. The non-transitory computer-readable medium of Claim 17, wherein the fiducial markers are arranged in a grid in the patch. 18. The non-transitory computer-readable medium of claim 17, wherein the patch includes a perforated border for tearing the patch into pieces. A non-transitory computer readable recording medium storing instructions for generating a mediated reality view of a surgical site, the instructions, when executed by a processor, causing the processor to:
receiving a pre-operative image representing a three-dimensional anatomy of the patient in a first position;
identifying, based on the preoperative image, coordinates in a three-dimensional preoperative image space corresponding to the location of fiducial markers provided on the patch applied to the patient, the fiducial markers visually said patch comprising materials of distinguishable different densities , said patch being placed on said patient's body and said fiducial marker being placed on said patch ;
receiving real-time images from a camera array after the patient has been placed for surgery at a second location;
determining, based on the real-time image, coordinates in three-dimensional real-time image space corresponding to the location of the fiducial marker provided on the patch applied to the patient;
applying a transform to the pre-operative image to align the location of the fiducial marker in the pre-operative image with the location of the fiducial marker in the real-time image;
overlaying the transformed pre-operative image onto the real-time image to generate the mediated reality view;
providing the Mediated Reality View to a display device for display;
A non-transitory computer-readable recording medium characterized by causing a step comprising : 22. The non-transitory computer readable medium of claim 21, wherein said patch comprises a flexible adhesive patch placed closest to an anatomical structure of interest. 22. The non-transitory computer-readable medium of Claim 21, wherein the fiducial markers are arranged in a grid in the patch. 22. The non-transitory computer-readable medium of claim 21, wherein the patch includes a perforated border for tearing the patch into pieces.

Images