TW201320005A - Method and arrangement for 3-dimensional image model adaptation - Google Patents

Abstract

1. Method for adapting a 3D model (m) of an object, said method comprising the steps of performing at least one projection of said 3D model to obtain at least one 2D image model projection (p1) with associated depth information (d1), performing at least one state extraction operation on said at least one 2D image model projection (p1), thereby obtaining at least one state (s1) adapting said at least one 2D image model projection (p1) and said associated depth information (d1) in accordance with said at least one state (s1) and with a target state (s), thereby obtaining at least one adapted 2D image model (p1') and an associated adapted depth (d1') back-projecting said at least one adapted 2D image model (p1') to 3D, based on said associated adapted depth (d1') for thereby obtaining an adapted 3D model (m').

Description

Method and configuration for 3D image model adjustment

The present invention relates to a method of three-dimensional (Three Dimensional; hereinafter referred to as 3D) image model adjustment.

3D model adjustments are usually performed manually, which is often undesirable. Another way to adjust the 3D model is to use state adjustments, which are related to adjusting the 3D model to conform to a certain state. These states affect the 3D position of the shape and/or the appearance of textures such as certain portions or features of the model. One of the main problems with the prior art of 3D model state adjustment is that the number of features to be adjusted in 3D is typically quite large, and thus manual intervention is still typically required due to insufficient computing resources. In addition, the best technology at present is limited to the use of a rigged model, which is severely limited when used in a dynamic system. In a dynamic system, the model can be learned so that its shape can also be changed during the learning process.

Accordingly, it is an object of an embodiment of the present invention to provide a method and arrangement that can be fully automated and that can be adjusted using a 3D image model that dynamically adjusts the model.

According to an embodiment of the invention, the object is achieved in a method of adjusting a 3D model of an object, the method comprising the steps of: Performing at least one projection of the 3D model to obtain at least one 2D image model projection (p1) having associated depth information (d1); performing at least one state extraction operation on the at least one 2D image model projection (p1), thus Obtaining at least one state (s1); adjusting the at least one 2D image model projection (p1) and the associated depth information according to the at least one state (s1) and a target state (s), thereby obtaining at least one adjusted a 2D image model (p1') and an associated adjusted depth (d1'); the at least one adjusted 2D image model is backprojected to 3D based on the associated adjusted depth (d1') Get an adjusted 3D model (m').

By adjusting the state of at least one 2D projection and the associated depth information of the 3D image model, less computing resources are used, thereby eliminating the need for manual intervention in the process. Backprojection to 3D ensures that the 3D model itself is adjusted as realistically as possible.

In an embodiment, the adjusted 3D model (m') is further determined based on the initial 3D model (m) information.

This way a smooth morphing of the adjusted model can be obtained.

In another embodiment, the limit imposed from the outside determines the target state (s).

It may be such as to contain high-level information about features such as the shape of the nose or the color of the eye.

In another embodiment, the state of input (IV) from an external image (se) Get the target state (s).

When the state (se) of the external image input (IV) is combined with the at least one state (s1) to obtain the target state, the manner may allow a 3D model to be smoothly adjusted to, for example, a scene on a live video. The changing characteristics of the object, or adjusted to resemble this object appearing on the still image.

In a preferred variant, the external image input (IV) comprises a 2D image input, and the at least one 2D projection of the 3D model is performed according to a virtual camera from the external image input (IV) interpretation One of the 2D projections.

This method is suitable for obtaining an optimal relationship between the external image input and the 3D model.

In yet another variation, the external image input can include a 2D+disparity input, which means providing 2D and disparity information from the outside, such as a stereo camera. The depth information can then be derived directly from the disparity information using the formula of depth x disparity = constant.

This way the depth data from this input can be used to update the associated depth.

The present invention is also directed to an embodiment of a configuration embodied in an image or video processing device for performing the method, and relating to an embodiment comprising, when executed by a data processing device, adjusted to perform the foregoing or the scope of the patent application Method of software software for computer programs.

Please note that the term "coupled" as used in the scope of the patent application should not be construed as limited to direct connection. Therefore, the scope of the phrase "a device A is coupled to a device B" should not be limited to an output of device A being directly connected to An input device or system of device B. This means that a path between an output of device A and an input of device B may be a path including other devices or mechanisms.

Please note that the term "comprising" as used in the scope of the patent application should not be construed as being limited to the Therefore, the scope of the phrase "a device includes mechanisms A and B" should not be limited to that the device only includes components A and B. This means that with regard to the invention, only the relevant components of the device are A and B.

In all of the texts, Two-Dimensional will be referred to simply as 2D, and as described above, three-dimensional will be referred to simply as 3D.

It will be understood by those skilled in the art that any <RTIgt; </ RTI> <RTIgt; </ RTI> <RTIgt; Similarly, we should be able to understand that any schematic representation of flowcharts, flow diagrams, state transition diagrams, and virtual code, etc., can be rendered substantially in computer readable media and thus can be used by a computer or processor (regardless of Whether the computer or processor is explicitly shown) various programs that are executed.

Figure 1a shows the steps of the first variant of the method for adjusting a 3D model m.

In a first step, the 3D model is projected to 2D. The parameters used for this projection are parameters used in accordance with the conventional pinhole camera model, which is described in the teaching manual "Multiple View Geometry in Computer Vision" by Richard Hartley and Andrew Zisserman (Cambridge University). Press, second Edition 2003, ISBN 621 54051 8) Chapter 6.

It is thus related to projecting points in a 3D space onto a plane via a central "pinhole". In this model, the plane corresponds to the projection plane of the camera, and the pinhole corresponds to the diabragma opening of the camera, also commonly referred to as the camera center. The result of this projection step is denoted by p1, d1, where p1 indicates the 2D projection itself, which may be represented by a 2D matrix of pixel values containing color information, and wherein d1 indicates a projection depth map, which may also be related The 2D matrix of the depth values of the joints is represented. These associated depth values are calculated from the original depth values and camera positions in accordance with some conventional equations that will also be provided hereinafter.

In an alternative embodiment, the projection and the depth map may be represented within a large 2D matrix, wherein for each projected pixel, color information and associated depth information are presented in corresponding matrix columns and rows. .

The projection itself is shown schematically in Figure 2a, which shows a point A with three spatial coordinates x _A , y _A , and z _A relative to the origin O, where the reference is used to define a reference The three axes x, y, z of the coordinate system define these coordinates. The pinhole camera is indicated by a camera center position C of a pinhole camera having coordinates x _C , y _C , and z _C relative to the same reference origin and reference coordinate system. Projection of point A is performed on a projection screen associated with the camera, designated S. The projection of point A via the pinhole C to the screen is labeled p(A) with associated coordinates (x _PA , y _PA ). However, these coordinates are defined in a manner related to the two-dimensional axes x _P and y _P defined within the projection plane S.

In order not to overload the 2a map, it is assumed here that the camera does not rotate relative to the three reference axes x, y, z. However, conventional formulas are also applicable to such more general cases, and these formulas are used in accordance with embodiments of the present invention to calculate projections and associated depth maps. As shown schematically in Figure 2b, the rotation of the camera is labeled θ _x , θ _y , θ _z to show the rotation of the camera center about the x, y, and z axes, respectively. These rotations are shown in the case where the origin O coincides with the center C of the camera.

In the most general case, C may translate and rotate relative to the reference origin O and the reference axes x, y, z.

In an embodiment in accordance with the invention, the projection of a 3D model will thus contain the color or texture information of the projected 3D point of the model as long as the projected 3D points fall within the contour of the screen area S and are not Another projection of another 3D point of the model is blocked. The 2D projection of the 3D object does almost inherently block, and more than one 3D point that blocks the model will be projected onto the same 2D point on the projection screen.

The depth map associated with the projection will thus include the respective relative depth values associated with the position of the camera for each of the projected pixels of the projected pixels p(A). It is marked as

Where θ _x , θ _y , θ _z indicate the respective rotation of the camera about the reference axes shown in Figure 2b,

Where a _x , a _y , a _z represent the coordinates of a point a in the reference coordinate system, where c _x , c _y , c _z represent the coordinates of the camera center c in the reference coordinate system, and wherein d _z represents point a relative to The depth associated with the center of the camera c.

If the camera does not rotate relative to the reference coordinate system x, y, z in the reference origin O, then these angles of rotation are all zero, so the equation (1) will be reduced to: d _z = a _{z -} c _z ( 2)

When the equation uses the notation in Figure 2a, it will correspond to: d(A)=z _A -z _c (3)

It is also shown in Figure 2a.

In general, the projections will be selected such that the features in the 3D model to be 3D adjusted will be part of the projection at a sufficiently high resolution, or such features will best fill the projected image. . This can be done tentatively by attempting a set of pre-determined projection positions and then selecting a projection position that provides the best results.

In another embodiment, it can be further determined via an intermediate step in which an approximation of the 3D surface of the model will be calculated using the 3D triangle. In general, the model will only be calculated in this model with such 3D triangles. An approximation of the part of the feature to be adjusted. Determines the normal to the vertical direction of each of these triangles. In an ideal projection, the normal should be oriented 180 degrees from the camera to the direction of the triangle. For each camera position, the sum of the cosines of the normals on the respective triangles of all triangles and the direction of the camera to the center of the triangle should be minimal. An optimum direction can be calculated by calculating the sum of some possible camera positions and selecting the position at which the minimum of the sum can be obtained. In an alternate embodiment, the minimum problem can be solved to determine, for example, the optimal camera orientation.

Of course, many other techniques can be used as is known to those skilled in the art.

In one step at a time, the state is extracted from the projection. State means the configuration of one of the object features, and a set of values represents the features themselves. These values may thus describe the potentially variable characteristics or characteristics of the object. The set of values can be arranged as a vector, but other representations for such a state are of course also possible. State extraction thus means determining some of the state parameters of the state of an object used to represent an image (in this example, a projection of a 3D model). As shown in the examples that will be described below, the above steps may be performed via some calculations based on 3D model information, or using some more general methods, for example, first including identifying/detecting one of the objects considered A step, which may (but not necessarily) perform some segmentation operations, and then perform a depth analysis on the object thus identified/detected.

However, in most embodiments according to the present invention, the 3D model itself is known, so that the state can be greatly reduced according to the state of the 3D model. The calculation of the extraction. If the 3D state is a coordinate with respect to certain features (which may be facial features in the case of a 3D model of the human head), the 2D projection of these 3D points may immediately result in a state parameter of the 2D image.

If the state of the 3D model is not yet known, then the identification step described above may be followed by further analysis, such as involving the use of an Active Appearance Model (AAM). The AAM can determine the shape and appearance of the facial features of the 2D projected image by fitting a 2D AAM internal shaping model in a situation such as being updated as an object model. It can initially compare the 2D projection to the initial value of a 2D AAM model and then gradually change the AAM model itself to find the best fit. Once a good match is found, the parameters determined according to the AAM adjusted model, such as face_expression_1_x and face_expression_1_y, are output.

In Figure 1a, the state of the projected image is labeled s1 and is used in a target state synthesis step. This state s is obtained from the state s1 of the 2D projection and from the external state information. The external status information indicated as se may be predetermined in an off-line manner from other description information such as a still image input or high-order semantic information related to features such as nose shape, eye color, and facial expression. In this case, the external status information can be pre-stored in a memory.

In an alternate embodiment, the "on-the-fly" decision may be made, such as based on external video image input data that may change rapidly as time elapses. The external status information se. In this case, the subsequent frame, usually a video sequence, determines the external state se.

The external state information and the state s1 of the 2D projection are used to obtain the target state.

The method of determining the target state indicated as s in Figure 1a from the input states s1 and se may include the step of performing a weighted combination of the weights of the trust levels used to reflect the states of the states, for the s1 and se values, Among them, the level of trust is determined during the state extraction. In the above example of the AAM method of determining the s1 parameters, each of the parameters used to identify the result of the match may then be selected as the level of trust level.

Another method of determining the target state may include only selecting a result such as se, and the result of the interpolation or weighted combination as described in the previous example in the different states indicates that the result of the interpolation is outside the predetermined limit, The choice is preferred.

In describing the embodiment shown in Figures 4a-b, a particular embodiment of the determination of the state and target state will be further described.

After determining the target state labeled s in Figure 1a, the 2D projection p1 and the associated depth map d1 will be converted according to the target state s. In one example, a method of using a triangle to represent features such as facial features can be used. An image conversion can be produced by interpolating the distances defined by the triangles and causing some of the features previously attributed to the pixels at the previous location to now belong to the pixels at these new locations. This method is extremely suitable for the case where many such triangles are used.

In a similar method, the projected image will be calculated according to the new state. The updated 2D coordinates of the pixels associated with the features. The color and texture information of the pixels between the triangles defined on the original 2D projection will be attributed to the pixels between the triangles located at these new locations in the updated image. If the two points on the 2D projection have internal coordinates (100, 100) and (200, 200) and the two points will be converted to coordinates (50, 50) and (100, 100) on the converted projection, then on the coordinates (150, 150) The color of the original pixel will be attributed to the pixel on the coordinates (75, 75) in the converted image.

In the description of Figures 4a-b, a more detailed embodiment will be further described.

The adjusted 2D projection is labeled p1'.

The associated depth values of the associated depth map are also adjusted in parallel based on the target state. In some embodiments, the determination of the target state is directly related to the calculation of the adjusted depth values for certain pixels of the projection. Adjustments to other depth values according to the target state may also be performed via interpolation between the calculated adjusted depths, as described above, with respect to the adjustment of the adjusted color values of the projected pixels.

The adjusted depth map is labeled d1'.

Re-projection or back-projection to 3D can be performed based on the converted depth map and the converted 2D projection, which typically includes the adjusted 2D image model, by using the conversions that are the opposite of those used in 3D to 2D projection. , but now the adjusted associated depth values are used for each 2D pixel of the adjusted projected image.

The result of this back projection is indicated as p3d_1.

In some cases, the 3D back projection point is sufficient to form an updated 3D model.

In other embodiments, the back projection to 3D is merged with the original 3D model m to obtain an updated or adjusted 3D model m'.

Figure 1b shows a configuration A for performing one of the embodiments of the method.

Figure 3a shows a variant embodiment in which more than one projection is performed on the initial 3D model m. The projections themselves may be selected depending on the form and shape of the model and the amount of blocking that occurs when the first projection is selected, or one of the methods described above for determining the projection parameters themselves. A possible implementation may thus be based on an approximation of a 3D surface that will be modeled using a set of triangles in 3D. Calculate the vertical direction of each of these triangles. A 3D "normal" vector that can point outside the body of the 3D model represents the vertical direction. By calculating the difference between the 3D vector and the camera's projection direction, a simple way to determine the blocking is obtained. As for the unobstructed surface, the projection direction should be opposite to the normal vector. Thus, the camera projection can be adjusted and may thus become: several projections may be required in order to obtain a sufficiently good projection of sufficient resolution of all features to be modeled. In an alternate embodiment, a predetermined projection of 3 times may be used to mitigate the trial error calculation for the optimal camera position.

These different projections are labeled p1, p2, to pn, and the associated depth maps are labeled d1, d2, to dn. Each of these projections thus has a position, rotation, and phase as shown in Figures 2a-b. Associated with the width and length of a virtual camera associated with the camera.

Each of these different projections p1 to pn will also undergo a state extraction operation, thus resulting in respective determined states s1, s2 to sn. In some embodiments, particularly in the case where the features to be adjusted are directly related to the coordinates or pixel locations of the features in question, the state of these respective projections can be calculated in the manner previously described.

These respective determined states s1, s2 to sn may (but are not necessarily) used in conjunction with the external state input se as separate inputs for determining a target state s. The decision of the target state can include performing a weighted combination of the input states with a level of trust that is used to reflect various input states, wherein the trust levels are determined during state extraction. In the above example of the AAM method of determining the s1 parameters, each of the parameters used to identify the result of the match may then be selected as the level of trust level.

Another method of determining the target state may include selecting only one of the input states or selecting the external state, and the result check of the interpolated or weighted combination as described in the previous example of the different states indicates the interpolation. The result is that in the case of outside the predetermined limits, the choice is preferred.

The target state s forms the basis for the n respective individual projections and their respective associated depth maps to be updated. The updated projections are labeled p1', p2' to pn', and the updated depth maps are labeled d1', d2' to dn'.

These updated projections p1', p2' to pn' are then backprojected to 3D based on the updated depth map values associated with each 2D pixel in the projections. Combine these back projections with the original model An updated or adjusted model.

Figure 3b shows an embodiment of a configuration for performing the method of deformation.

Figure 4a shows an embodiment of a 3D model adjustment for adjusting a human head. In this embodiment, the state of the model is related to facial expressions, but in other embodiments, the state may also be related to the color of the hair, eyes, skin, and the like. The goal of this particular embodiment is to depict the 3D model using facial features provided by an input 2D video.

The input video is labeled IV in Figure 3a. For each frame of the video, the zoom and orientation of the object is estimated in a manner related to the scaling and orientation of the object of the 3D model. This approach is preferred for determining the first projection associated with the virtual camera viewpoint of the 3D model to a 2D plane, wherein the projection should be as similar as possible to the 2D projection used in the camera used to capture 2D video. This need not be the case for a particular choice of the first projection, but may be advantageous for a simple update. For a particular projection, the projection of the 3D model to a 2D plane should therefore use a virtual camera with some associated projection parameters, and these projection parameters are as similar as possible to those projections of the camera used to capture the 2D image of the input video. parameter.

The calculation of these projection parameters is performed in accordance with some conventional techniques, such as will be described hereinafter: a 3D library model of the input system face that determines the parameters for the parameters of the virtual camera and a live 2D video feed. Since the 3D position of the facial features of the 3D database model is known, in the live video feed The 2D position of the facial features, and the projection matrix of the webcam and the virtual camera, so the data should be sufficient to calculate the 3D position of the facial features of the face in the live video feed. If the 3D positions of the facial features in the live video feed and the 3D position of the corresponding facial features of the library model are thus known, then a 3D transition between the corresponding 3D positions can be calculated (translation and Turn). In an alternative embodiment, it is thus also possible to calculate a 3D transition (translation and rotation) required by a virtual camera to capture the same 2D field of view as seen in the live video feed of the 3D library model. The minimum number of feature points required for the calculation of the transition to be applied to the virtual camera is three. Because the face is not a rigid object due to changing and different emotions, the problem of minimization will need to be solved when obtaining facial features. Therefore, three stable points such as the left side of the left eye, the right side of the right eye, and the top of the mouth are used. The 3D position of the three facial features in the database model, and the 2D position of the live video feed and corresponding facial features in the webcam projection matrix are thus input to the conventional Grunert algorithm. The algorithm will provide the calculated 3D position of these corresponding 3 facial features. It can then be used to move the virtual camera around the 3D library model to retrieve the same library model 2D picture as the 2D picture provided by the face in the live video feed.

In some embodiments as shown in Figure 4a, it is preferred to use another projection of the 3D model. This is the case where the first projection using the camera parameters results in an optimal projection of one of the images similar to the video feed but still does not produce sufficient pixel data (eg, on the projected image) When one part of the department is blocked by the nose, it may be desirable.

Figure 5a shows this situation, in which the "real" person's video captured by the real "camera" is shown in the left rectangle, and the left part of the right rectangle is shown as the first of the virtual camera 1. Projection of a 3D model performed by a virtual camera. As shown, the virtual camera's projection of the 3D model matches the projection conditions used by the "live" 2D camera. However, some pixels of the left part of the face are still blocked by the nose. Therefore, another projection is performed by another virtual camera, which is labeled "Virtual Camera 2". The parameters of the virtual camera 2 are determined based on the blocked pixels of the previous camera position. The parameters of the virtual camera 2 can be determined based on the intrinsic parameters such as facial points and the extrinsic parameters of the virtual cameras, and based on the knowledge of the 3D model. This information will be able to decide whether to project two voxels or 3D points of the features of the 3D model to be modeled into the same pixels in a 2D projection. If this is the case, it is clear that blocking will occur. Based on this information, another virtual camera position can then be calculated, and at least a different projection of the voxel can be present. By performing such an inspection of all of the projected pixels, the presence of the barrier can be determined and the virtual camera position and rotation can be determined accordingly.

In another embodiment, some predetermined virtual cameras or virtual cameras selected from the virtual cameras may be used to obtain a projection of the feature of interest. Moreover, in an alternative embodiment, a standard configuration of a virtual camera that provides a front view and two 90 degree side views, respectively, may be used, and all projections may be used depending on the features to be modeled. A subset of the projection.

If only two projections are used, the right portion of the right rectangle of Figure 5a shows the result of the second projection. The associated depth maps labeled d1 and d2 are also generated along with the projections p1 and p2. These depth maps indicate the relative depth of each 2D projected pixel, including the rotational information observed from the viewpoint of the respective virtual camera 1 or 2 associated with the respective camera position obtained using equation (1). A depth map of each of the two projections is indicated in the lower graph of the right rectangle.

In the next step, the states of the two projections p1 and p2 and the subsequent frames of the input video are extracted. Since the state is related to facial expressions in this embodiment, these facial expressions are characterized. The current best techniques described above, such as AAM techniques, are used to extract features associated with these facial expressions from subsequent frames of the input video and the 2D projections. The state of the projection can also be calculated from the 3D state of the model and the corresponding voxel projections in the manner previously described. This situation is illustrated in Figure 5b, while in the left rectangle, the position of the different pixels of the mouth and the edge of the eye on the live 2D frame is shown. These positions of these same features on the projections are thus also determined. In the right part of Figure 5b, which is shown only for projection p1, it also occurs significantly at projection p2, but is not shown in this figure so as not to overload the pattern. In this particular embodiment, the respective states correspond to the locations of pixels associated with p1, p2, and those features appearing on an input box. These states are labeled s1, s2, and se, respectively. Since only p1 is shown in Fig. 5b, only s1 is also shown. These three states are used to determine the target state, which in this embodiment corresponds to state se. Although in this embodiment, the respective states s1 And s2 are thus not used to determine the target state, but during transitioning the projections according to the target state, the respective states s1 and s2 are still used. This target state is thus also used to adjust the 2D projections p1 and p2. For virtual cameras corresponding to "real" video cameras, this adjustment can be easily performed by replacing the pixel locations of the selected features with corresponding pixel locations of the features appearing in the video frame. By selecting the virtual camera 1 to map to the real camera, it can be executed extremely simply. In order to adjust the 2D projection p2 obtained by the other virtual camera 2, a possible method comprises the steps of calculating the position of the adjusted features of p2 first determined in 3D. The above steps can be performed according to the adjusted projection p1' and the adjusted depth map d1'. This approach determines the calculation of the 3D position of these features visible on p1'. By using the projection parameters for the second projection, the corresponding position on p2' can be identified. For blocked features from p1 and p1', interpolation techniques can be used to calculate the adjusted projection and the adjusted depth map.

Once the new locations of the key features of p1 and p2 are known, gradient techniques such as weighted interpolation can be used to determine the color and depth of pixels that are not critical features.

The adjustment of the projection p1 is shown in the bottom view of the right rectangle of Fig. 5b. The projection is shown in the "laughing" face expression that appears to be adjusted to the left rectangle of the input video frame. This also occurs in projection p2 (not shown in Figure 5b).

The adjusted depth map is then used to re-project the adjusted projections p1' and p2' to 3D and merge to replace or update the old data. The data of d1' can be calculated from the following approximation: the adjusted depth is equal to the initial depth, and thus the initial depth d(A) of pixel A associated with the feature under consideration and having projection coordinates x _PA , y _PA will now be assigned The coordinates of the coordinates x _PA ', y _PA ', because x _PA 'and y _PA ' are the adjusted coordinates of the feature under consideration.

In this regard, it should be mentioned that all backprojections of the adjusted 2D images should be consistent in the 3D domain. It basically means that when the backprojection is a transformed feature visible in more than one 2D projected image, this feature should be backprojected from all projections to the same 3D position. Therefore, if the corner of the mouth is converted and the corner of the mouth appears in several of these projections, then all of the backprojected coordinates should be the same.

Suppose x_3d is a feature considered in a 3D object (for example, the tip of the nose), x_3d is a vector with information (x, y, z, color), and x_2dz is a feature in the 2D+Z domain, then it is A vector containing information (x-2d, y_2d, depth, color).

The model of 3D to 2D+Z projection is established by function p according to a virtual camera c1: p(c1, x_3d)=x_2dz_c1

Now consider the 3D model with the state adjusted. The expected 3D feature after the state adjustment is referred to as x'_3d. The 3D state transfer function is m_3d: x'_3d=m_3d(x_3d)

This means: x'_2dz_c1=p(c1,x'_3d)=p(c1,m_3d(x_3d))

Since the state-related adjustments are performed on the projections, the m_3d function cannot be used in the 2D+Z domain. This can be approximated by using a m_2dz function: x"_2dz_c1=m_2dz(c1,x_2dz_c1)

It can be consistent with the 3D state only under the following conditions: x'_2dz_c1=x"_2dz_c1

This means that the function p(c1, m_3d) and the function m_2dz(c1) are actually identical in the domain under consideration.

If this is the case, there is no problem and the method described above can be used without any problems; if not, an additional step must be performed.

In order to take this into account, careful selection of projection parameters will solve the problem from the beginning.

However, if it is not processed, an inconsistent condition may occur. One of the problems is that the backprojection of these sources must be "consistent" when using multiple 2D+Z sources to recreate 3D models. "." When the functions are consistent in the 3D state, there is no problem (this is because all 2dz functions actually implement a specific 2dz version of the 3d state transition function). When the functions are inconsistent in the 3d state It is necessary to enforce consistency via a "correct" 3d state transfer function or an approximation of the function. It is possible to, for example, select a reference 2DZ state transfer function and project all other state transfer functions to the reference, and perform the above steps :x'_2dz_c1ref=m_2dz(c1ref,x_2dz_c1ref)

Now consider using m_2dz(c1ref) as the reference 2dz state transfer function. Other functions can be established by moving through the 3D domain: x'_3d=p_inv(c1ref,x'_2dz_c1ref)=p_inv(c1ref,m_2qz(c1ref,x_2dz_c1ref) m_2dz(c2,x'_2dz_c2)=p(c2, X'_3d)=p(c2,p_inv(c1ref,m_2dz(c1ref,x_2dz_c1ref))) Note that not all features from 3D objects will have valid values after moving through p(c, x_3d). For example, not Points within the virtual camera's field of view or points blocked by other features in the object have no valid values. In order for these points to have consistent transformed features, additional reference cameras will be required.

The second embodiment is a variant of the first embodiment and also relates to the state adjustment of the 3D model of the face, but differs from the previous embodiment in that the second embodiment does not use a 2D camera but uses a 2D+Z camera. , For example, a stereo camera, or a time-of-flight camera such as Microsoft Kinect or the like is used. In this case, a facial feature point of the 3D coordinate instead of the 2D coordinate can be used. The 2D+Z projections of the many offline models required are again taken to cover all the points modified by the field data and to infer the state to these projections. The gradation technique of the previous embodiment can be used, for example, for the "offline" 2D+Z data, but the data is merged, but the modified Z data is now also used for the feature points.

In these embodiments, the problem of 3D state adjustment can be reduced. When the transition from one or more 2D images to a full 3D model begins, the state transitions from 2D to 2D+Z are now reduced, making these operations easy to apply to real-time applications.

While the invention has been described above in terms of the specific embodiments of the invention, it is to be understood that the description of the present invention is provided by way of example only and not limitation of the scope of the invention defined in the scope of the appended claims. In the scope of the present invention, any element that is stated as a means for performing a specified function will include any means for performing the function. It may include, for example, a combination of electrical or mechanical components used to perform the function, or any form of software including firmware or microcode therein, and suitable circuitry for performing the function in conjunction therewith, And mechanical components that are coupled to the software controlled by the software (if such a circuit is present). The invention as defined by the scope of the claims is based on the fact that the functions provided by the various devices are combined and combined in the manner required by the scope of the claims, and unless otherwise explicitly defined, The novelty of the invention is only very Less importance or no importance. Applicants of the present invention thus consider any device that provides those functions to be equivalent to those devices shown in the present invention.

The foregoing and other objects and features of the present invention will become more fully understood from In the drawings: 1a-b shows a first variant of the method and apparatus of the present invention; and 2a-b shows a geometrical model according to an embodiment of the invention; 3a-b The figure shows a second variant of the method of the invention; the 4a-b diagram shows the third and fourth embodiments of the method of the invention; and the 5a-c diagram shows the embodiment shown in Fig. 3a Different steps performed in the case of an additional 2D video input.

Claims (10)

A method of adjusting a 3D model (m) of an object, the method comprising the steps of: performing at least one projection of the 3D model to obtain at least one 2D image model projection (p1) having associated depth information (d1); The at least one 2D image model projection (p1) performs at least one state extraction operation, thereby obtaining at least one state (s1); adjusting the at least one 2D image model projection according to the at least one state (s1) and a target state (s) (p1) and the associated depth information (d1), thereby obtaining at least one adjusted 2D image model (p1') and an associated adjusted depth (d1'); and adjusted according to the association The depth (d1') backprojects the at least one adjusted 2D image model (p1') to 3D, thus obtaining an adjusted 3D model (m'). The method of claim 1, wherein the adjusted 3D model (m') is further determined based on the initial 3D model (m) information. The method of claim 1 or 2, wherein the target state (s) is derived from externally applied semantic information. The method of claim 1 or 2, wherein the target state (s) is derived from a state (PS) of an external image input (IV). The method of claim 4, wherein the target state is obtained by combining the state (PS) of the external image input (IV) with the at least one state (s1). The method of claim 4, wherein the 2D projection of the at least one 2D projection of the 3D model is performed according to a virtual camera derived from the external image input (IV). The method of any one of claims 4 to 6, wherein the converting of the live video extracted from the outside and the key features of the projected 2D image is performed, and wherein the keyness of the projections is The new location of the feature is determined based on the location of the critical features of the live video. A configuration (A1) that is adjusted to perform the method of any one of claims 1 to 7. An image processing apparatus comprising the configuration of claim 8 of the patent application. A computer program product comprising software, which is adapted to perform the method steps of any one of claims 1 to 6 when executed on a data processing device.