Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
  • A61B34/10—Computer-aided planning, simulation or modelling of surgical operations
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T19/00—Manipulating three-dimensional [3D] models or images for computer graphics
  • G06T19/20—Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
  • G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
  • A61B34/10—Computer-aided planning, simulation or modelling of surgical operations
  • A61B2034/101—Computer-aided simulation of surgical operations
  • A61B2034/105—Modelling of the patient, e.g. for ligaments or bones
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
  • A61B90/36—Image-producing devices or illumination devices not otherwise provided for
  • A61B2090/364—Correlation of different images or relation of image positions in respect to the body
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/30—Subject of image; Context of image processing
  • G06T2207/30004—Biomedical image processing
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2210/00—Indexing scheme for image generation or computer graphics
  • G06T2210/41—Medical
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2219/00—Indexing scheme for manipulating 3D models or images for computer graphics
  • G06T2219/20—Indexing scheme for editing of 3D models
  • G06T2219/2021—Shape modification

Definitions

• the present invention relates to a method for the pre-operative prediction of a body or a part of a body, e.g. the face, after surgery.
• the invention also relates to a planning system wherein the method can be applied.
• 3D photographic systems can be subdivided into two categories, i.e. those using active methods, which project a specific pattern on the body, and those using passive methods, which acquire a 3D geometric description of the body from one or more images and illumination conditions, with or without the use of a priori geometric knowledge .
• 3D photographic systems deliver the texture of the body, which is used to render the 3D surface.
• Motion simulation can be based on heuristic rules, physics-based knowledge, or it can be image-derived (e.g., building a statistical deformation model based on a set of images from different persons and/or expressions) .
• the result can be natural or artificial.
• the facial motion of one person can be used to drive the facial motion of another person.
• Texture mapping refers to a computer graphics technique wherein a texture image (or texture map) is applied to a polygonal mesh or some other surface representation by coupling the texture image (or texture map) (with associated colour/gray value) to the 3D surface. The result is that (some portion of) the texture image is mapped onto the surface when the surface is rendered.
• Texture is derived from one or more 2D or 3D photographs of the body.
• a texture map is typically delivered simultaneously with the 3D shape description.
• Matching can be done based on a set of corresponding points, or on a metric (e.g., mutual information) that expresses the correspondence between 2D- image-derived features and 3D-shape-based properties.
• the model of body reflectance can be based on skin or skin-like diffuse and specular (mirror-like reflection) properties.
• 2D visualisation has been used to show (a part of) the body under simulated or artificial illumination conditions and for animation by morphing (part of) the body.
• photo-realism is the primary concern.
• the present invention aims to provide a device and method for pre-operatively simulating or predicting an accurate image of the patient's appearance after surgery, in particular maxillofacial or plastic surgery.
• the invention aims to provide a planning system wherein the method can be applied.
• the present invention relates to a method for pre-operatively obtaining a prediction of a post-operative image of at least part of a body, comprising the steps of : determining a 3D pre-operative description of at least part of a body, acquiring a pre-operative 2D photograph of the at least part of the body from any viewing position, - matching the 3D pre-operative description with the preoperative 2D photograph, determining a deformation field for deforming the 3D pre-operative description, typically with a 3D planning system, and - deriving by means of the deformation field and the preoperative 2D photograph a predicted post-operative image of a 3D post-operative description of the at least part of the body.
• the predicted post- operative image is a 2D photograph obtained by deforming the pre-operative 2D photograph using said deformation field.
• the predicted post-operative image is a 3D image.
• a plurality of pre-operative 2D photographs is advantageously acquired and subsequently used in the later method steps.
• the method advantageously further comprises a step of generating from the 3D pre-operative description a 3D pre-operative surface mesh of at least the contours of the at least part of the body.
• the step of deriving the predicted image comprises deriving from the 3D pre- operative surface mesh a prediction of the 3D postoperative surface mesh of at least the contours of the at least part of the body.
• the 3D pre-operative description is obtained using a 3D image acquisition system.
• a 3D image acquisition system can be a Computerised Tomography system, a Magnetic Resonance Imaging system or a 3D photographic system.
• the step of matching is preferably performed by means of a set of corresponding points on said 3D preoperative description and said 2D pre-operative photograph.
• said step of matching is performed by means a metric expressing the correspondence between features derived from the pre-operative 2D photograph and properties based on the 3D pre-operative description.
• the method further comprises the step of taking a picture of a calibration object. Said picture of the calibration object can then be used for calibrating the camera with which the pre-operative 2D photograph are acquired.
• the matching step advantageously a step is performed of creating from the matched pre-operative 2D photographs a texture map for 3D visualisation.
• the 3D pre-operative description comprises typically a soft tissue description of the at least part of the body.
• it also comprises information about the internal structure of the at least part of the body.
• the invention also relates to a surgical planning system for pre-operatively showing a predicted post-operative image of at least part of a body, comprising - means for determining a 3D pre-operative description of at least part of a body, means for matching the 3D pre-operative description with a 2D pre-operative photograph of the at least part of the body, - calculation means for determining a deformation field to deform the 3D pre-operative description and for deriving a predicted post-operative image of a 3D post-operative description of the at least part of the body, display means for showing the predicted post-operative image.
• the predicted postoperative image is a 3D image.
• the predicted post-operative image is a predicted post-operative 2D photograph obtainable by deforming the pre-operative 2D photograph using the deformation field.
• Fig. Ia represents a 3D pre-operative surface mesh, projected onto the 2D pre-operative photographs after registration.
• Fig. Ib and Fig. Ic show 2D pre-operative and post-operative photographs, respectively.
• Fig. Id represents two views of the rendered surface mesh, using a texture map obtained from the set of 2D photographs.
• Fig- 2a represents a 3D pre-operative surface mesh, projected onto the 2D pre-operative photographs after registration.
• Fig.2b and Fig.2c show 2D pre-operative and post-operative photographs, respectively.
• Fig.2d represents two views of the rendered surface mesh, using a texture map obtained from the set of 2D photographs .
• Fig. 3a represents a 3D surface mesh, obtained with a 3D photographic system, projected onto the 2D photographs after registration.
• Fig.3b shows 2D photographs.
• Fig.3c offers six views of the rendered surface mesh using a texture map obtained from the set of 2D photographs .
• Fig. 4 represents a calibration object.
• Fig. 5 represents on the left a set of points specified manually onto the 3D rendered untextured surface, obtained from the 3D surface mesh and on the right a set of (bright) points specified manually onto the 2D photograph, together with the above set of (dark) points obtained by matching the 3D surface with the 2D photograph.
• Fig. 6 represents an iterative improvement of the accuracy by using additional corresponding points on the 2D photograph and the projected surface mesh.
• Fig. 7 gives a schematic representation of the 3D image space I , the 3D camera space C and the 2D photographic image.
• Fig. 8 represents a spherical texture map assembled from the photographs in Fig. 3b.
• Fig. 9 represents from left to right the initial pre-operative skull. The skull is cut into parts that can be repositioned.
• Fig. 10 represents on the top row the preoperative facial skin surface and on the bottom row the predicted post-operative skin surface.
• Fig. 11 illustrates the planning system accuracy. Above: bone displacement field (up to 13.7 mm) .
• the rendered surfaces correspond to positions on the face where the difference between simulated (pre- operative) and real (post-operative) surface are less than
• Fig. 12 represents snapshots of the 3D planning system at work, with facility for soft tissue prediction.
• the face is shown as a 3D rendered texture surface (pre-operative state) . Snapshots of the 3D planning system at work, with facility for soft tissue prediction.
• the face is shown as a 3D rendered textured surface
• Fig. 13 represents a cylindrical texture map assembled from the photographs in Fig.2b.
• Fig. 14 represents the deformation field
• Fig. 15 represents the displacement field (short lines) and boundary of the dilated region (outer contour) . Outside this area, the displacements are zero and the image is not deformed.
• the present invention differs from the prior art in two fundamental aspects. Firstly, in the approach according to the present invention one or more 2D photographs taken from any viewing position can be used. A viewing position is to be considered as a vector having a direction as well as a magnitude, i.e. a viewing position consists of a viewing direction and a camera distance. The number of 2D photographs is thus arbitrary. Secondly, the visualisation is not restricted to 3D visualisation, for which the 2D photographs are used as texture maps, but a single arbitrary pre-operative 2D photograph can be deformed into a simulated post-operative 2D photograph using a physics-based reliable, personalised and accurately predicting 3D deformation field.
• the ability to show the patient's appearance can be integrated into a system for 3D pre-operative planning.
• 'planning system' is meant a software environment that allows a physician to plan or simulate the procedure of an intervention. It can for example be used to predict the outcome of that intervention, to try out different procedures, to optimise the procedure, to prepare it and to improve the communication between the medical staff and with the patient .
• Real time 3D visualisation using texture mapping offers an added value to the surgeon while using the 3D planning system, e.g. when adjusting or repositioning bony structures or an implant . Accuracy and integration in the planning procedure are of primary importance, and photo-realism is of minor importance. Visualisation is possible from any viewing direction.
• a 3D pre-operative image is acquired of (a part of) a patient's body.
• a 3D image acquisition system is preferably used thereto, such as CT (Computerised Tomography) , MRI (Magnetic Resonance Imaging) or any other 3D scanning or photographic system.
• 3D medical imaging modalities such as CT or MRI, offer geometric information of the body contour (further also referred to as the 'soft tissue') and internal structures, such as the bony structures. Based on the volumetric data, the 3D contour of the skin and other tissues, such as bone, are segmented. In the case of skin and bone, segmentation can for example be performed by simple thresholding. Instead of a 3D medical imaging modality, any other 3D scanning device can be used to obtain the outer body contour, such as a 3D photographic system. 3D photographic systems can be subdivided into two categories, i.e. those using active methods, which project a specific pattern on the body, and those using passive methods, which acquire a 3D geometric description of the body from one or more images and illumination conditions, with or without the use of a priori geometric knowledge. In
• a 3D generic face model is interactively fitted to a set of images to acquire the 3D shape.
• a set of (one or more) 2D photographs of (a part of) the body from any viewing direction and camera distance (i.e. any viewing position as previously defined) using any camera is acquired.
• one or more 2D pictures are taken from arbitrarily chosen directions.
• the 3D data are used to generate a 3D surface mesh of the body contour (the ⁇ soft tissue') and, if needed by the planning system, of other tissues such as bone.
• Surface meshes such as the triangular meshes shown in Figs. Ia, 2a and 3a, can for example be created using the marching cubes algorithm.
• a registration method is then applied to match or register the 3D pre-operative surface description with the 2D photographs.
• One way to align or register the 3D surface with a 2D photograph is shown in Fig.5, where a set of corresponding points on the 3D surface and the 2D photograph, respectively, is used.
• the problem then is how to transfer a point from the 3D image space / to the related camera space C and further to the corresponding 2D photographic image. It is assumed that the camera can be modelled as a perspective pinhole camera with its optical centre located at C (see Fig. 7) .
• the geometric relation between / and C can then be expressed by a rotation R and a translation /.
• ky is usually 0. Multiplying the matrices in Eq. (4) and substituting Sx • f, Sy • f and kx ⁇ f by fx, fy and Kx, respectively, yields :
• the transformation matrix in (Eq.5) contains five parameters. When these parameters are known, the camera is said to be calibrated internally. The camera is calibrated externally if the six degrees of freedom of / and R are known. Together, the whole calibration process thus requires eleven parameters to be defined. This can be done by indicating a set of corresponding points on respectively the 3D surface and the 2D photograph (Fig. 5) . Each such point yields two equations. This means that at least six points are needed to calculate the value of the eleven parameters. In practice, more reference points are recommended in order to improve the accuracy of the solution.
• the internal calibration parameters are very sensitive to small errors on the position of the corresponding reference points. As already mentioned, it is therefore recommended to take first a picture of a separate calibration object with accurately known geometry and texture (Fig. 4), to calculate the internal parameters of the camera, and to freeze these settings during the remainder of the photo session.
• the corresponding reference points on the acquired 3D image and 2D photographic image of (part of) the body (Fig. 5) are subsequently used for the external calibration.
• the accuracy of the registration can iteratively be improved by adding corresponding points on the 2D photograph and the projected surface mesh (Fig. 6) .
• registration of the 3D surface with a 2D photograph can also be performed for example based on the optimisation of an objective function that expresses the correspondence between 2D-image-derived features and 3D-shape-based properties (e.g., mutual information) .
• a 2D texture map is created from the registered 2D photographs.
• the surface mesh and corresponding texture are used for 3D visualisation.
• the texture map is then mapped onto the 3D body surface.
• View- dependent using a single texture map for fast displaying, e.g. a virtual sphere enclosing the 3D body contour
• view-independent texture mapping (Fig.8) assume a known relationship between the 3D surface coordinates and the 2D photographic coordinates, as well as a method to calculate the texture values from the colours or gray values in the available photographs.
• Each 3D mesh point corresponds to a point in each 2D photograph and its texture value is nonzero if the 3D mesh point is visible in and front facing at least one of the 2D photographs.
• a corresponding "visible mesh” is generated by removing the vertices that are invisible from the camera position, together with the triangles they belong to.
• the texture value can for example be calculated as a normalised weighted combination of the corresponding colour or gray values in the contributing photographs as proposed in the above-mentioned papers by Pighin or by Iwakiri.
• This weight function should provide a smooth and seamless transition between the photographic patches.
• the weight function ( ⁇ — ⁇ A) 2 has been used, with ⁇ the angle between the surface normal and the line from the surface point to the camera position of the photograph.
• a 3D patient-specific planning system (e.g., for maxillofacial surgery, breast augmentation, nose correction, etc.), including a soft tissue prediction, is used to simulate the post-operative shape.
• the soft tissue prediction is used to deform the pre-operative surface mesh of the soft tissue with associated texture map into a predicted post-operative soft tissue mesh with associated remapped texture.
• the post-operative soft-tissue mesh and corresponding texture map is used for 3D visualisation of the soft tissue.
• Motion simulation can be based on heuristic rules, physics-based knowledge, or it can be image-derived (e.g., building a statistical deformation model based on a set of images from different expressions or a linear combination of a set of textured face meshes each corresponding to a facial expression, such as joy, anger, sadness) .
• the result can be natural or artificial (e.g., the facial motion of one person can be used to drive the facial motion of another person) .
• the invention makes use of a personalised and accurately predicting 3D deformation field for maxillofacial and plastic surgery.
• a 3D planning system for maxillofacial surgery which yields an accurate personalised 3D deformation field of the face.
• Planning a maxillofacial procedure can be subdivided into two separate parts, i.e., the bone-related planning and the soft tissue simulation.
• the bone-related planner allows the surgeon to reshape the skull in a 3D environment. Reshaping the skull implies cutting the skull into different parts and repositioning each of the different parts (Fig. 9) .
• the new facial shape of the patient can be simulated (Figs. 10) .
• a mathematical model is used that is able to accurately simulate the behaviour of the facial tissues.
• Known models are the finite element model (FEM) , the mass-spring model (MSM) and the mass-tensor model (MTM) . Together with one of these models, a set of boundary conditions is used, which are generated from the bone-related planning.
• Fig. 12 shows a few snapshots of the planning system at work on the same patient as used in Figs 9-11.
• the soft tissue with associated texture moves in real time and simultaneously with the bone displacements.
• Fig. 13 shows the associated texture map for this patient.
• the 3D pre-operative and post-operative surface meshes are projected onto the preoperative 2D photographs.
• the vertices of the pre-operative 3D surface meshes that are visible from the camera viewpoint were previously mapped or projected onto the preoperative 2D photographs (Fig. Ia, 2a, 3a) using the registration parameters and matrices previously obtained. For each of these vertices a displacement vector and corresponding vertex in the post-operative 3D surface mesh is known. These corresponding vertices are also projected onto the pre-operative 2D photographs. Since the preoperative 3D surface mesh is deformed into a post-operative mesh, some vertices that were previously visible, may become invisible now. These vertices are also removed as well as their associated vertex in the pre-operative mesh.
• the projected deformation field acquired from the pre-operative and post-operative soft-tissue meshes, is used to geometrically deform the pre-operative
• a patient-specific 3D deformation model is used to deform the 2D photographs geometrically. From the projected pre-operative and postoperative soft-tissue meshes, the 2D displacement of all the projected mesh vertices in the 2D photograph is known. Hence, the 2D geometric deformation vector is known in a discrete number of points in the 2D photograph (Fig. 14) . The displacement in each pixel of the photograph can then be calculated by interpolation between the discrete deformation vectors. Outside the projected mesh, the deformation is in principle zero. However, due to small mismatches between the 2D photograph and the projected preoperative surface mesh, it may be recommended to slightly extrapolate the deformation field outside the mesh.
• Fig. 15 is a typical example.
• the contour of an enlarged region can be used as the zero- deformation borderline.
• interpolation of the discrete deformation field can for example be performed using bicubic spline functions.
• the above method can also be used in practice without the steps of determining a deformation field and using the deformation field to deform the one or more preoperative 2D photographs.
• the latter step results in a predicted post-operative 2D photograph.
• 3D visualisation using texture mapping lacks photo-realism (e.g. unnatural texture blending and hair modelling artefacts (particularly when using medical imaging, such as CT for 3D image acquisition) and the texture map mostly needs retouching.
• the simulated post-operative 2D photograph on the other hand, has intrinsically the same photo-realism as the original pre-operative photograph.

Landscapes

• Engineering & Computer Science (AREA)
• Physics & Mathematics (AREA)
• Health & Medical Sciences (AREA)
• Theoretical Computer Science (AREA)
• General Physics & Mathematics (AREA)
• Life Sciences & Earth Sciences (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Surgery (AREA)
• Molecular Biology (AREA)
• Heart & Thoracic Surgery (AREA)
• Computer Graphics (AREA)
• Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
• Robotics (AREA)
• General Engineering & Computer Science (AREA)
• Architecture (AREA)
• General Health & Medical Sciences (AREA)
• Animal Behavior & Ethology (AREA)
• Medical Informatics (AREA)
• Computer Hardware Design (AREA)
• Software Systems (AREA)
• Biomedical Technology (AREA)
• Public Health (AREA)
• Veterinary Medicine (AREA)
• Magnetic Resonance Imaging Apparatus (AREA)
• Processing Or Creating Images (AREA)
• Image Processing (AREA)
• Measuring And Recording Apparatus For Diagnosis (AREA)
• Apparatus For Radiation Diagnosis (AREA)
• Image Generation (AREA)
• Materials For Medical Uses (AREA)
• Pharmaceuticals Containing Other Organic And Inorganic Compounds (AREA)

Priority Applications (5)

Applications Claiming Priority (2)

Related Child Applications (1)

Publications (1)

Family

ID=34610863

Family Applications (1)

Country Status (7)

Cited By (3)

Families Citing this family (36)

Citations (2)

Family Cites Families (10)

• 2005
  • 2005-04-08 GB GBGB0507204.6A patent/GB0507204D0/en not_active Ceased
• 2006
  • 2006-04-07 AT AT06721556T patent/ATE532155T1/de active
  • 2006-04-07 JP JP2008504587A patent/JP4979682B2/ja not_active Expired - Fee Related
  • 2006-04-07 EP EP06721556A patent/EP1872337B1/en not_active Expired - Lifetime
  • 2006-04-07 WO PCT/BE2006/000035 patent/WO2006105625A1/en not_active Ceased
  • 2006-04-07 ES ES06721556T patent/ES2376905T3/es not_active Expired - Lifetime
• 2007
  • 2007-10-05 US US11/868,313 patent/US8428315B2/en active Active

Patent Citations (2)

Non-Patent Citations (3)

Also Published As

Similar Documents

Legal Events