TW201342305A - Method and arrangement for 3D model morphing - Google Patents

Abstract

A method for morphing a standard 3D model based on 2D image data input, comprises the steps of performing an initial morphing (100) of said standard 3D model using a detection model and a morphing model, thereby obtaining a morphed standard 3D model determining (200) the optical flow between the 2D image data input and the morphed standard 3D model, applying (300) the optical flow (300) to said morphed standard 3D model, thereby providing a fine tuned 3D standard model.

Description

Method and device for deforming three-dimensional model

The present invention relates to a method for three-dimensional model deformation.

It is currently a difficult task to deform a model based on actual dynamic scenes or even images taken with inexpensive cameras. For example, 3D, which will be abbreviated as 3D in the following text, may take a lot of time and effort to create 3D content and 3D animation with high detail and lifelikeness. However, this does not meet the requirements, and in the next generation communication system, for example, it is impossible to visually 3D the images of the people attending the meeting.

Therefore, embodiments of the present invention propose a method and configuration for image model deformation, which can be abbreviated as two-dimensional, and will be abbreviated as 2D, image scene, and even lower-quality actual image capture in the following text. Produces high-quality 3D image models while providing a low-cost, simple and automated solution.

An object of the present invention is achieved by an embodiment of the present invention, wherein a method for deforming a standard 3D model based on 2D image data input is provided, the method comprising the steps of: - utilizing a detection model and a deformation model The standard 3D model performs an initial deformation, thereby obtaining a modified standard 3D model; - determining the optical flow between the 2D image data input and the deformed standard 3D model; - applying the optical flow to the deformed standard 3D model, thereby providing a finely tuned standard 3D model.

In this way, the conventional detection according to the deformation can be strengthened by the deformation of the optical flow, thereby achieving a more realistic model and being completed at the same time.

In one embodiment, the optical flow between the 2D image data input and the deformed standard 3D model is determined based on a standard 3D model of the previous fine-tuned deformation determined by a previous D image frame.

In an embodiment variation, the step of determining the optical flow between the 2D image data input and the deformed standard 3D model comprises: - determining a 2D projection of the standard 3D model between the deformation and the criteria for the prior fine-tuning deformation a first optical flow between the 2D projections of the 3D model; - determining a second optical flow between the actual 2D image frame and the 2D projection of the prior fine-tuned standard 3D model; - combining the first and second Optical flow to obtain a third optical flow between the actual 2D image frame and the 2D projection of the deformed standard 3D model; - adapting the third according to depth information obtained during 2D projection of the deformed standard 3D model Optical flow to obtain the optical flow between the 2D image input and the deformed standard 3D model.

This achieves a high quality and time efficient method.

In another embodiment, the deformation model is between the 2D image assets The optical flow between the material input and the deformed standard 3D model is adapted. This further improves the quality of the resulting model and the association of the model with the input video object.

In another embodiment, the detection model in the initial deformation step is adapted according to the optical flow information determined between the 2D image frame and a previous 2D image frame. In this way, for the input 2D image, the 3D standard model can be shaped and deformed more quickly and more realistically.

In another embodiment variation, the step of applying the optical flow includes an energy minimization procedure.

This further enhances the image quality of the resulting fine-tuned deformation model.

The invention also relates to an embodiment of a device for performing the above method, comprising an image or video processing device of the device, and a computer program product comprising, when implemented on a data processing device, adapted to perform A soft body of the method as described above or in the scope of the patent application.

It should be noted that the term "coupled" as used in the scope of the patent application is not to be construed as limited to direct connection. Therefore, the expression "device A coupled to device B" should not be limited to the input of device A being directly connected to the input of device B. It indicates that there is a path between the output of device A and the input of device B, which may include other devices or means.

It is to be noted that the term "comprising" as used in the scope of the claims is not to be construed as limited to the means or means. Therefore, the expression "a device includes components A and B" should not be limited to the device having only components A and B. It represents that for the present invention, the components associated with the device are A and B.

As described above, in this specification, 2D will be represented by 2D, and 3D will be represented by 3D abbreviation.

Those skilled in the art will appreciate that any block diagrams described herein are representative of exemplary circuit system concepts for implementing the principles of the invention. Similarly, any program represented by any flow chart, flow chart, state diagram, virtual code, and the like can be substantially represented by a computer readable medium and executed by a computer or processor, whether or not there is a special This is the case with a computer or a processor.

Figure 1 shows a high-order diagram of a first embodiment of a device and a corresponding method for generating a high-quality instant 3D model using input 2D video. This embodiment uses an input continuous frame of a video sequence. The steps in Figure 1 are for a particular frame, such as the steps performed at the 2D video frame at time T.

The first mode of operation 100 is a standard 3D model available that is pre-selected or pre-stored in memory for deformation. The module 100 will deform the standard 3D model according to the input 2D video frame of time T. A detailed embodiment of this variant procedure will be described with reference to Figure 2. The output of module 100 is a standard 3D model of deformation at time T.

Parallel to the deformation step 100, the optical flow is determined by a standard 3D model from the 2D video frame at time T to the deformation at time T, this step being performed at module 200, which has time T The 2D video frame is used as input, and the standard 3D model of the deformation provided by the module 100, There is also the output of the device as determined in the previous time step. The previously determined output is a standard 3D model of the fine-tuning deformation determined in the previous time step. In the embodiment shown in Figure 1, the time is T-1 and is provided to the module via the device output with a feedback link. 200. In Figure 1, the feedback loop contains a delay element D that provides the previously determined output. Of course, many other implementations based on simple memory storage can also be employed to reduce the need to use dedicated delay elements. It should also be noted that not only the previous video frame corresponding to time T-1, but also the output determined at other previous times can be used. The delay must be adjusted according to these embodiments.

The embodiment of FIG. 1 further includes another module 300 for applying the optical flow determined by the module 200 to the deformed standard 3D model provided by the module 100. The basic idea of the present invention is to combine the module 100 model-based method, which uses a relatively simple 3D model, with the optical flow of the module 300 as the main deformation detail, and the optical flow itself in the module 200. Internal derivation. In fact, when used for face modeling, the model deformation obtained from the module 100 can usually produce a somewhat artificial face, which is then modified by the optical flow deformation of the module 300, and the optical flow itself It is determined by the module 200.

As mentioned above, the resulting fine-tuned 3D standard model is used to determine the feedback loop of the optical flow.

A more detailed embodiment will be described below by modeling the facial features. Those skilled in the art should be able to use the contents of this manual to modify other deformable objects like animals in video.

Figure 2 shows a more detailed embodiment of the standard 3D deformation block 100 of Figure 1. This module contains an Active Appearance Model (AAM) detection module. However, there are also embodiments in which other detection modules such as Active Shape Model (ASM) detection modules are utilized.

The detection module 110 can detect the facial features of the video frame at time T according to a detection model, such as an AAM detection model. AAM model and AAM detection are known techniques for detecting feature points of non-solid objects in computer vision technology. When inputting 3D video into the system, the AAM deformation can also be extended to 3D positioning, and the AAM detection module can detect feature points on other objects that are not faces. The type of object to be detected can be related to the training phase of the AAM detection module, and the training can be performed offline or in an early training program. In the illustrated embodiment, the AAM detection module 110 is trained to detect feature points on the face of the person in the 2D video frame, such as nose, mouth, eyes, eyelashes, and non-solid objects such as cheeks. The AAM detection model used in the AAM detection module 110 can be selected from a set of models, or can be pre-programmed or offline trained to be universally available to all faces.

When an animal model is deformed like a cat, the training program must be adapted to detect other important feature points for the cat's form/possible expression. Those skilled in the art should be familiar with these techniques.

In the example of face modeling, the AAM detection block 110 typically includes a rough motion to detect a face within the video frame and simultaneously or subsequently detect facial expressions associated with human emotions. The full face in the live video frame The right or absolute position is marked as "location" information in Figure 1. The location information is used to move or rotate the 3D standard model of the face, represented in the module 120 as a standard 3D model. In addition, a limited number of facial expressions can be detected in the module 110 by a rough indication of positions such as nose, eyelashes, and mouth. The output is represented in Figure 1 as a "feature" and will be used in the deformation module 130 to adapt the corresponding facial features of the position adaptation standard model output by the module 120.

The 3D standard model input to module 120 can also typically be found or selected by a standard repository. Such a standard database can include 3D standard models of faces and 3D standard models for animals such as cats and dogs. The standard 3D model is translated, rotated, and/or scaled based on the location information of the module 110.

In the case of face modeling, the position adaptation step causes the 3D standard model to reflect the same pose of the face in the live video input. To further adapt the 3D model to represent the correct facial expression of the 2D frame, the features detected by the module 110 are applied to the partially adjusted 3D standard model at step 130. The deformation module 130 further uses a special adaptation model, which is represented by a "morphing model" in FIG. 2, which may include adapting the information provided by the detection module to adapt the facial features of the standard 3D model. instruction. In the case of using the AAM detection model, the deformation model is usually the AAM deformation model. The same considerations apply to examples of previously used ASM variants.

The result obtained is a standard 3D model of the deformation provided by the module 130.

An exemplary embodiment of the deformation of the model can include repositioning the vertices of the standard 3D model associated with the facial features based on the facial feature detection results of the live video input. Higher-order interpolation or even more complex functions are used when 3D content between facial features can be filled by simple linear interpolation, or when more complex high-order AAM deformation models including facial elasticity are to be included. The information about how the vertices are shifted and how the data between the facial features are filled is included in the deformation model.

It should be noted that no matter how good the image quality of the available detection and deformation model (AAM) is, it is possible to obtain a result that looks artificial, because the commonly used detection model is only used to detect the live video input. The position of the internal facial features is then replaced with the facial features of the 3D position adaptation model based on the position in the live video input. The area between the facial features of the 3D standard model is then interpolated using an AAM deformation model. However, the latter does not know or only know some knowledge that each facial feature affects adjacent facial regions when replaced. General information about facial expressions and their effects on facial areas may be related to elasticity and can be placed in the deformation model, but it will still result in artificially deformed results, simply because everyone is different. Without a very generic model that covers all facial expressions.

Similar considerations are still valid when deforming animals detected in other deformable objects, such as video, according to the 3D standard model.

To further improve the deformed standard 3D model, the artificial appearance deformation model provided by the module 100 can be improved using the optical flow deformation in step 300, which has been previously explained in conjunction with FIG.

Before performing optical flow deformation, the optical flow itself must be determined. The optical flow defined herein is the displacement or pattern of apparent motion of objects, surfaces and edges from a frame to another, or from a frame to a 2D or 3D model. In the embodiments described herein, the method for determining the optical flow focuses on calculating the time between different images, such as T and T-1, at the pixel level, or alternatively, focusing on the calculation. The displacement between the pixels at time T and the corresponding voxels within the 3D model, and vice versa.

Since the light must be applied to the deformed standard 3D model in the module 300 according to the 2D video frame, the optical flow must be calculated from the 2D video frame to the 3D model. In general, optical flow calculations are calculated from one 2D video frame to another 2D video frame, so some extra steps are added to determine the 2D video frame to the 3D deformation model. Additional steps may include using a reference 3D input, ie a previously determined, such as a fine-tuned 3D model determined at time T-1. This information is provided to the module 200 by the output of the device.

Figure 3 shows a detailed embodiment of the implementation module 200. In this embodiment, the first module 250 is adapted to determine a first optical flow between the 2D projection of the deformed standard 3D model and the 2D projection of the previous trimming deformed 3D standard model. The second module 290 is adapted to determine a second optical flow between the actual 2D video frame at time T and the 2D projection of the previous fine-tuned 3D standard model. A combining module 270 calculates a third optical flow using the first and second optical flows. This third optical flow is a 2D projection of the deformed standard 3D model of the actual 2D video frame at time T and time T. Module 280 then further adapts the third optical stream to obtain 2D imagery at time T The desired optical flow between the input and the deformed standard 3D model of time T is input. Further details will be described below.

To determine the first optical flow between the 2D projection of the deformed standard 3D model and the 2D projection of the previous fine-tuned 3D standard model, these 2D projections are processed and provided to the module 200 on individual 3D models. Thus, module 230 is adapted to perform 2D mapping or projection for a deformed standard 3D model provided by module 100, while module 240 is adapted to perform a similar 2D projection for a previous fine-tuned 3D standard model, at 3rd In the embodiment of the figure, it is determined by time T-1. The projection parameters used in these projections are preferably projection parameters corresponding to the video camera that recorded the 2D video frame. It is related to the adjustment parameters of the video camera.

In the embodiment shown in FIG. 3, the module 290 includes three sub-modules. In the module 220 of the module 290, the optical flow between the video frame at the current time T and the previous video frame at time T-1 in this example is determined. The time of the previous 2D video frame is the same as the time of the previously fine-tuned deformed 3D standard model.

Thus, delay element 210 of module 290 introduces the same delay as the feedback loop of the complete device of Figure 1. Of course, other embodiments may provide previous values for 2D video, which may simply be stored in internal memory, eliminating the need for additional delay blocks.

Therefore, the module 220 can determine the optical flow calculated between the successive video frames T and T-1, and then used in the module 260 to determine the 2D projection of the 3D fine-tuning output from time T-1 to the time. The optical flow of the 2D video frame of T. The projection itself will be executed at module 240. Projection parameters are used to map The parameters used by the 2D camera and recorded by the 2D video frame.

In step 260, the process of determining the second optical flow takes into account that the standard model and the live video input sometimes represent different people, which still requires calibration. In some embodiments, the module 260 can include two steps: a first face registration step, wherein the face of the live video input at the previous time T-1 is mapped to the 2D of the previously fine-tuned 3D content of the time T-1. The projected face. This registration step can also use the AAM detector again. Next, the optical flow calculated by the live video input at time T is calibrated by interpolating the face of the 2D projected 3D content at time T-1. These embodiments will be further illustrated in Figures 7 and 8.

The first optical flow between the 2D projection of the deformation standard model of time T determined by module 250 and the previous fine tuning standard model of time T-1 is then combined with the second optical flow determined by module 260. A third optical flow of a 2D projection of a warped standard model that generates a 2D video from time T to a time T. This is the required 2D optical flow information. Since an intermediate common component, i.e., the 2D projection of the previously determined fine tuning model, is removed during this combination, the module 270 is represented by a minus sign "-".

However, since the third optical stream is still the optical stream of the two 2D images, an additional step 280 is required to convert this optical stream from the 2D video frame of time T to the 3D content of the deformed standard 3D model of time T. This may involve the use of a backprojection as a reverse program used in 2D projection, so the same projection parameters are used. To this end, the depth generated by the 2D projection is used to recalculate the 2D transition to the 3D vertices.

It should be noted that instead of using time T and T-1 consecutive video frames and The continuously determined fine-tuned 3D model may have a time difference between the new frame and the previous frame that is greater than the delay of one frame. In this case, a corresponding previously determined output deformation model is used such that the time difference between the actual frame and the previous frame used by the module 200 corresponds to the new output to be determined and the light used to determine the light. The time difference of the previous output of the stream. In an embodiment, the similar delay element D of the feedback loop of FIG. 1 and the module 210 of FIG. 3 can be used.

The module 300 of Figure 1 then applies the calculated optical flow to the deformed standard 3D model, thereby producing a fine-tuned 3D standard model.

As shown in FIG. 4, in the first variant embodiment of the apparatus, there is an additional feedback loop in the output of the module 200 for calculating the deformation of the 2D video and the time T between the time T. The optical flow of the standard 3D model is used in an adjustment module 1000 to perform initial deformation of the standard 3D model. The details of this adjustment module 1000 are shown in Figure 5. Compared with FIG. 2, the module 1000 receives an additional input signal, which is indicated as the "optical flow" provided by the output of the optical flow calculation module 200, and the information is used to adapt the use in the deformation module 130. Deformation model. The additional module 140 within the deformation module 1000 updates the previous version of the deformation model based on the optical flow information. A delay element is also used in the embodiment shown in Fig. 5, but in other embodiments only the manner of storing the previous value may be used.

The usefulness of updating the deformation model using optical flow feedback is that the standard universal deformation model does not know how the displacement of each facial feature affects the adjacent facial area. This is because there is no elasticity in the basic deformation model. Or not enough comments. Provide optical flow information to understand more complex and higher-order deformation models. The idea here is that a perfect deformation model can transform the 3D standard model to make it possible to simulate live video input. In this case, the "optical flow combination" block 270 of the module 200 does not need to be processed. The extra light flow will become redundant.

In another variation of the embodiment shown in FIG. 6, there is another feedback loop for returning an internal signal from the optical flow calculation module 200 to the standard 3D deformation module 100. Figure 7 shows a detailed embodiment: the feedback is actually the optical flow between the 2D video frames of time T and T-1, which is provided to additional AAM or other detection model adaptation modules. Here, it can be assumed that the optical flow between the video frames T-1 and T in the live video input maps the facial features detected by the video frame T-1 to the facial features detected by the video frame T. Since not all facial expressions are found by the detection module, sometimes the facial features in the live video input cannot be successfully detected. This condition can be solved by adapting the detection module to detect facial features, so that it can be incorporated into facial expressions, which can be detected in the future and applied to the 3D standard model.

The embodiment shown in Figure 8 adds all of the feedback loops mentioned so far.

Figure 9 shows another high-order embodiment in which a more biased approach is used to combine the model with optical flow distortion. The model-based module 100 provides precise displacement of the finite dispersion of feature points of the 3D model, while the process-based module provides less accurate 2D displacement estimation, but forms denser points on the model. Combine different classifications with different precision through probability method The accuracy of the observation point, even more accurate results of the 3D standard model for fine-tuning deformation can be obtained. This probability method is implemented by the energy minimization module 400 of FIG.

In the case of face modeling, this probabilistic approach can intuitively fill the basic elastic model of the face in unobserved voids. The face can only be active in some way. These actions are limited. For example, neighbors on the model will behave in a similar manner. Similarly, the point of symmetry on the face is related. It means that if you see the left face laughing, although the right face is not observed, but the right face also has a great chance to laugh.

Mathematically, it can be considered as an energy minimization problem, including two data items and one smoothing term.

E=S+D _FLOW +D _MODEL

D _FLOW is the distance relationship between the proposed candidate for the final fine-tuned 3D model and the optical flow of the 2D input image. It is suggested that the closer the candidate is to the probability distribution, the shorter the distance on the observed dense optical flow diagram. This distance relationship is inversely proportional to the accuracy of the optical flow estimate.

D _MODEL is a similar distance relationship, but is a matching relationship between the candidate solution and the observed AAM-based deformation 3D model. The accuracy of the D _MODEL and AAM algorithms is also inversely proportional.

S is an action that punishes the face that is unlikely to occur. It contains two types of children: absolute and relative punishment. Absolute punishment is directly proportional to the impossibility of moving a little in the direction of the suggestion. The meaning of relative punishment is the same, no It is to see the displacement between adjacent points (or other related points, such as symmetrical points).

The problem of energy minimization can be solved by numerical methods. Examples are gradient ramping, stochastic methods (simulated annealing, genetic algorithms, random walks), as well as graph segmentation, confidence propagation, Kalman filters, and more. The purpose is the same: find the proposed deformed 3D model with the lowest energy in the above equation.

Figure 10 is a more detailed embodiment of Figure 9.

A second probability related embodiment is shown in FIG. In this embodiment, the calibrated optical flow will accumulate over time, combined with the accumulated calibration optical flow and AAM detection/deformation results and allow for simple and realistic 3D library content distortion with energy minimization issues. The offset that can be caused by the accumulated optical flow can be corrected by adding the AAM deformation result. And the deformation result of the artificial appearance can be removed by adding the optical flow deformation result.

It is to be noted that the described embodiment can be used not only for deformation of a human face. With the model-based approach of the present invention, any model of non-solid objects can be built and used for deformation. Moreover, embodiments of the invention are not limited to the use of an AAM model. The initial deformation module 100 can also use other models like the Active Shape Model (ASM).

Although the principles of the present invention have been described with respect to the specific device and device, it is to be noted that the contents of the present specification are only examples, and are not intended to limit the scope of the present invention. Any element of the means for performing a particular function in the scope of the claims should be construed as including any means of performing the function. For example, performed by electrical or mechanical components A combination of functions or any form of software, including, for example, firmware, microcode, or the like, combined with appropriate circuitry to perform the software that performs the function, and any circuitry coupled to the software. The mechanical components of the system. The invention is defined by the scope of the patent application, wherein the functions provided by the various means are combined and combined by the means recited in the scope of the patent application, unless otherwise stated, any physical structure does not have or Only a little importance. The Applicant of the present invention therefore recognizes that all means for providing the above-described functions are considered to be equivalent scope of the present invention.

100‧‧‧First operation module

110‧‧‧Detection module

120‧‧‧ modules

130‧‧‧ deformation module

140‧‧‧Module

200‧‧‧ modules

210‧‧‧ Delay element

220‧‧‧ module

230‧‧‧ modules

240‧‧‧ modules

250‧‧‧ first module

260‧‧‧Module

270‧‧‧ Combined module

280‧‧ steps

290‧‧‧ second module

300‧‧‧ modules

400‧‧‧Energy Minimization Module

1000‧‧‧Adjustment module

2000‧‧‧ module

Those skilled in the art will be able to understand the various advantages of the present invention through the following description and accompanying patent examples, and in conjunction with the accompanying drawings, wherein: FIG. 1 is a high-level embodiment of the method of the present invention; 3 is a more detailed embodiment of some of the modules of the embodiment of FIG. 1; FIG. 4 is a high-level illustration of another embodiment of the method of the present invention; 5th and 6th Further details of some of the modules in the embodiment of FIG. 4 are shown; FIGS. 7 and 8 show two other embodiments; and FIG. 9 shows another high-level implementation of the method of the present invention. Examples; and Figures 10 and 11 show more detailed alternative embodiments.

100‧‧‧First operation module

200‧‧‧ modules

300‧‧‧ modules

Claims (13)

A method for deforming a standard three-dimensional model based on two-dimensional image data input, the method comprising the steps of: - performing an initial deformation (100) on the standard three-dimensional model using a detection model and a deformation model, thereby obtaining a deformation a standard three-dimensional model; - determining (200) the optical flow between the input of the two-dimensional image data and the standard three-dimensional model of the deformation; - applying (300) the optical flow to the standard three-dimensional model of the deformation, thereby providing A standard three-dimensional model of fine-tuning deformation. The method of claim 1, wherein the optical flow between the two-dimensional image data input and the deformed standard three-dimensional model is based on a previous fine-tuned deformation determined by a previous two-dimensional image frame. The standard 3D model is determined. The method of claim 2, wherein the step (200) of determining the optical flow between the input of the two-dimensional image data and the standard three-dimensional model of the deformation comprises: - determining (250) one between a first optical flow between the two-dimensional projection of the deformed standard three-dimensional model and the two-dimensional projection of the standard three-dimensional model of the previous fine-tuned deformation; - determining (290) a criterion between the actual two-dimensional image frame and the previous fine-tuned deformation a second optical flow between the two-dimensional projections of the three-dimensional model; - combining (270) the first and second optical flows to obtain a two-dimensional projection between the actual two-dimensional image frame and the deformed standard three-dimensional model Third optical flow; - adapting (280) the third optical stream according to depth information obtained during the two-dimensional projection of the deformed standard three-dimensional model to obtain the light between the two-dimensional image data input and the deformed standard three-dimensional model flow. The method of any one of claims 1 to 3, further comprising adapting (140) the optical flow between the two-dimensional image data input and the modified standard three-dimensional model at the initial The step of the deformation model used in the deformation step (1000). The method of any one of claims 1 to 4, further comprising adapting the initial deformation according to the optical flow information determined between the two-dimensional image frame and a previous two-dimensional image frame. The steps of the detection model used in the step. The method of any of claims 1 to 3, wherein the step of applying the optical flow comprises an energy minimization procedure (400). A device for deforming a standard three-dimensional model based on two-dimensional image data input, the device being adapted to: - perform an initial deformation (100) on the standard three-dimensional model using a detection model and a deformation model a modified standard three-dimensional model; - determining (200) an optical flow between the input of the two-dimensional image data and the standard three-dimensional model of the deformation; - applying (300) the optical flow to the standard three-dimensional model of the deformation, This provides a standard three-dimensional model of the fine-tuned deformation to one of the outputs of the device. The device as described in claim 7 is more suitable for standard three-dimensional deformation based on previous fine-tuning deformation determined by a previous two-dimensional image frame. The model determines the optical flow between the two-dimensional image data input and the deformed standard three-dimensional model. The device as described in claim 8 is further adapted to determine the optical flow between the input of the two-dimensional image data and the standard three-dimensional model of the deformation by the following steps: - determining (250) one a first optical flow between the two-dimensional projection of the deformed standard three-dimensional model and the two-dimensional projection of the standard three-dimensional model of the previous fine-tuned deformation; - determining (290) a criterion between the actual two-dimensional frame and the previous fine-tuned deformation a second optical flow between the two-dimensional projections of the three-dimensional model; - combining (270) the first and second optical flows to obtain a two-dimensional projection between the actual two-dimensional frame and the deformed standard three-dimensional model a third optical flow; - adapting (280) the third optical flow according to the depth information obtained during the two-dimensional projection of the deformed standard three-dimensional model to obtain the standard three-dimensional input between the two-dimensional image data and the deformation The flow of light between the models. The apparatus of any one of claims 7 to 9 is further enabled to adapt (140) the optical flow between the two-dimensional image data input and the deformed standard three-dimensional model. The deformation model used in the initial deformation step (1000). The device of any one of claims 7 to 10 is further adapted to adjust the optical flow information determined between the two-dimensional image frame and a previous two-dimensional image frame. The detection model used in the initial deformation step. An image processing apparatus comprising the apparatus of any one of claims 7 to 11. A computer program product comprising software for performing the method of any one of claims 1 to 6 when implemented on a data processing device.