SG192751A1

SG192751A1 - A method for reconstructing a 3d surface from a 2d sketch

Info

Publication number: SG192751A1
Application number: SG2013061429A
Authority: SG
Inventors: Hock Soon Seah; Xue-Cheng Tai; Jie Qiu; Jooyoung Hahn; Eiji Sugisaki; Lei Jia
Original assignee: Univ Nanyang Tech
Priority date: 2011-03-03
Filing date: 2011-03-03
Publication date: 2013-09-30
Also published as: WO2012118439A1; CN103460252B; CN103460252A

Abstract

The present invention relates to a method for reconstructing a three-dimensional surface in a domain wherein the three-dimensional surface is reconstructed from a two- dimensional sketch comprising at least one stroke with a vector field assigned to the at least one stroke. The method comprises the steps of: interpolating the vector field to form a dense vector field; and reconstructing the surface using the dense vector field; wherein interpolating the vector field comprises minimizing a non-linear functional of the vector field with a curl-free constraint on the dense vector field, the non-linear functional of the vector field involving terms defined over the domain and a term defined over the at least one stroke, and wherein the non-linear functional of the vector field is minimized using an augmented Lagrangian method.

Description

Field of the invention : | oo

The present invention relates to a method for reconstructing a three-dimensional (3D) surface from a two-dimensional (2D) sketch. This method may be used in a sketch- based surface modeling system for various applications. :

Background of the Invention a ; : Although geometric models may be constructed and/or manipulated using currently

Co available modeling systems such as Maya [1], AutoCAD [2], SolidWorks [3] and CATIA 3 n [4], these currently available modeling systems often employ the WIMP (Window, Icon,

Co Menu, Pointer) interface paradigm whereby to form and/or manipulate a 3D model, users are required to select operations from menus and floating palettes, enter parameters into dialog boxes, and move contro! points [5]. Such a modeling process is not intuitive and hence, requires a lot of time and effort, especially if the desired 3D - ~~ model is complex. On the other hand, sketching is a natural, easy and fast way to communicate ideas. By sketching, complex shapes can be formed in a few pencil strokes. Thus, in computer modeling, sketching is often used to form a design in the ; early prototyping stages and the depicted design is then converted into a 3D model. : Motivated by the ease of sketching, a recent research direction known as sketch-based interfaces for modeling (SBIM) aims to automate and assist a sketch-to-3D translation process. SBIM systems have been explored by researchers because of their perceived efficiency and intuitiveness. These systems [5] range from early systems such as

SKETCH [12] and Teddy [13] to more recent systems such as SmoothSketch [14] and

Bh FiberMesh [15]. Without lighting or shading cues (such as those in photometric stereo

[18], [19], [20] or shape-from-shading [21], [22]), the task of 3D model reconstruction 30° from 2D line drawings is more challenging than image-based 3D reconstruction.

EE Although attempts based on contours [13], hidden contours [14], and structured annotations [23] have been made to deal with this challenging task, existing SBIM - systems can still only model a limited range of objects with low complexity. These objects can take only a few shapes and can contain only either rough, smooth, or

Bh 2 oo rounded structures. Moreover, in existing SBIM systems, little consideration is given to | oo oo surface details such as crease structures. Even though efforts have been made in oo - - existing SBIM systems to include surface details, these efforts are mostly unsuccessful.

Currently, if surface details are desired, they have to be added onto the modeled object oo 5 by for example, surficial augmentation techniques [24]. | - . Many surface reconstruction algorithms [25], [26], [27], [28] enforce the integrability - » condition for producing correct surface heights from a gradient field. For single-view

Co Co modeling, the method in reference [9] uses a constrained optimization with many types of geometrical constraints whereas the single-view modeling system [10] uses a close 5: - N form method to reconstruct curved 3D surfaces based on apparent contour, inflation

B oo constraints, and normal specification in the parameter space. However, the method in i - 2) requires -a large amount of user interaction to provide sufficient constraints for - modeling a desired 3D surface (as shown by the authors of reference [7]) and the “methods in [71, [9] and [10] encounter problems in handling sparse gradients, preserving sharp features, and preventing surface distortion [16]. .. ~~ Based on the observation that humans are good at assigning local surface normals for specifying local shapes [29], stroke-based surface reconstruction is implemented in [6],

[7] in two steps. In the first step, dense surface normals are obtained via linear vector interpolation methods from sparse user-input vectors along given strokes. In the second step, the dense surface normals are integrated for reconstructing a height map. - In LUMO [6], the vectors are interpolated based on a Telegrapher's equation (the damped wave equation) whereas in ShapePalettes [7], an energy functional minimization is used and the main mechanism of interpolation is based on a parabolic oo type partial differential equation (PDE) with a fourth order term related to the surface curvature. in both the LUMO and ShapePalettes methods, the governing equations are ~~ linear PDEs so numerical solvers such as the muitigrid method or the Fast Fourier

Transform (FFT) may be applied for the vector interpolation. However, both the LUMO ~ and ShapePalettes methods are based on the Laplace operator which enforces 3 “smoothness in the interpolated vector field. This causes the assigned vectors to be diffused into the whole domain and then eventually averaged out in the final steady state, yielding only smooth reconstructed surfaces without surface details.

Summary of the invention : ~~ This present invention aims to provide a new and useful method for reconstructing a - 3D surface from a 2D sketch. = oo Co so oo | -

In general terms, the present invention proposes a non-linear vector interpolation for reconstructing the 3D surface and further proposes numerical solvers for this non-linear © vector interpolation. : 10 A first “aspect of the present invention is a method for reconstructing a three- oo = dimensional surface in a domain wherein the three-dimensional’ surface is ) ol “reconstructed from a two-dimensional sketch comprising at least one stroke with a g vector field assigned to the at least one stroke and the method comprises the steps of: interpolating the vector field to form a dense vector field; and reconstructing the surface using the dense vector field; wherein interpolating the vector field comprises minimizing a non-linear functional of the vector field with a curl-free constraint on the dense vector field, the non-linear functional of the vector field involving terms defined over the domain and a term defined over the at least one stroke, and wherein the non-linear functional of the vector field is minimized using an augmented Lagrangian method. ) The first aspect of the present invention is advantageous as it is a fast and efficient way of reconstructing a three-dimensional surface. Furthermore, it achieves a reconstructed surface which closely reflects the assigned vector field. It also helps to preserve surface details without surface distortion in the reconstructed surface. Hence, more desirable structures may be achieved. Furthermore, the non-linear functional is solved - using an augmented Lagrangian method which is capable of solving the functional - more quickly. 3 ~The invention may alternatively be expressed as a computer system (for example, a sketch-based surface modeling system) for performing such a method. This computer - | system may be integrated with a device for recording sketches whereby the device “may further comprise a user interface for generating or extracting sketches. The - invention may also be expressed as a computer program product, such as one

~ WO 2012/118439 PCT/SG2011/000082 . recorded on a tangible computer medium, containing program instructions operable by - a computer system to perform the steps of the method. oo -

Brief Description of the Figures : oo os -

An embodiment of the invention will now be illustrated for the sake of example only with _ reference to the following drawings, in which:

CT Fig. 1 illustrates a flow diagram of a method for reconstructing a 3D surface : 10 from a 2D sketch according to an embodiment of the present invention;

Ca Lo Fig. 2(a) illustrates a user's sketch and Figs. 2(b) - (d) illustrate the results of - oo applying steps in the method of Fig. 1 on the user's sketch in Fig. 2(a); - | Fig. 3(a) illustrates a user's sketch and Figs. 3(b) - (d) illustrate the results of applying steps in the method of Fig. 1 on the user's sketch in Fig. 3(a);

Figs. 4(a) — (c) illustrate examples of different stroke types used in the method of Fig. 1; - Fig. 5(a) illustrates a HSV color map and Figs. 5(b) - (d) illustrate vectors oo assigned along example strokes in the method of Fig. 1;

Cl Figs. 6(a) — (c) illustrate vectors assigned to a user's sketch and Figs. 6(d) — (f) respectively illustrate surfaces reconstructed with the method of Fig. 1 using the : assigned vectors in Figs. 6(a) — (c);

Figs. 7(a) — (b) respectively illustrate conceptual drawings of a ridge stroke and a valley stroke; - Figs. 8(a) — (c) illustrate base strokes, strokes of interest and a reconstructed surface for a jump stroke, a bump stroke and a dip stroke respectively; oo Figs. 9(a) — (d) illustrate an example of how a weighting function used in the method of Fig. 1 affects the reconstructed surface;

Co Figs. 10(a) — (f) illustrate another example of how the weighting function used in the method of Fig. 1 affects the reconstructed surface; 30° Figs. 11(a) — (f) illustrate an example of how imposing a curl-free condition on the interpolated vectors in the method of Fig. 1 affects the reconstructed surface;

Co Co Figs. 12(a) — (c) respectively illustrate relative residues, relative errors of - Lagrange multipliers and numerical energy values plotted against the number of iterations in an iterative method used in the method of Fig. 1:

a | 3 | | -

Co Fig. 13(a) illustrates a rule for indexing variablés in an augmented Lagrangian ~~. functional used in the method of Fig. 1 and Fig. 13(b) illustrates an example of - oo implementing the rule in Fig. 13(a) in a discrete computational domain; : g Co “Figs. 14(a) and 14(b) respectively illustrate a user's sketch and a surface : reconstructed from the user's sketch with the method of Fig. 1, and Figs. 14(c) and : - 14(d) respectively show the user's sketch with added strokes and the reconstructed : . surface with modifications corresponding to the added strokes; : Figs. 15(a) and (e) respectively illustrate a user's sketch and initial vectors - ~~ assigned to the user's sketch, and Figs. 15(b) and (f), (c) and (g), and (d) and (h) 10° ‘respectively illustrate a dense vector field and a surface reconstructed from the user's oo sketch Using the LUMO method, the ShapePalettes method and the method of Fig. 1; - ~~ Fig. 16(a) illustrates a user's sketch and Figs. 16(b) - (d) respectively illustrate a

SE surface reconstructed from the user's sketch using the LUMO method, the ‘ShapePalettes method and the method of Fig. 1; ; Fig. 17(a) — (c) respectively illustrate a user's sketch, vectors assigned to the user's sketch and a surface reconstructed from the user's sketch using the method of

Fig. 1; - | Figs. 18(a) — (d) illustrate user's sketches and the respective surfaces reconstructed from the user's sketches using the method of Fig. 1; :

Figs. 19(a) — (b) illustrate another example of how the weighting function used in the method of Fig. 1 affects the reconstructed surface and Figs. 19(c) - (d) illustrate an example of how different stroke types affect the reconstructed surface in the method of Fig. 1;

Figs. 20(a) — (d) illustrate frames of an example animation sequence created using the method of Fig. 1.

Detailed Description of the Embodiments - Referring to Fig. 1, the steps are illustrated of a method 100 which is an embodiment of “the present invention and which reconstruct a 3D surface from a 2D sketch comprising at least one stroke. : ~ In step 102, stroke(s) are drawn to form a user's sketch and these stroke(s) are then - classified based on their stroke types. In step 104, vectors are assigned along the stroke(s) in the user’s sketch. Next, in step 106, the assigned vectors are interpolated

Co 6 ) to form a dense vector field and in step 108, a 3D surface is reconstructed using the - dense vector field. - The user's sketch may comprise basic 2D shapes formed from strokes sketched in step 102. In one example, simple line drawings which reflect the characteristics of the “desired 3D surface are used to form the sketch. in another example, any shading, . highlighting, hatching strokes and stippling in the user's sketch are ignored. The

Lo strokes of the user's sketch may be recorded as input into method 100 using a vector g format. This recording may be performed using the drawing module of a vector-based : animation system CACANi. After each stroke is recorded, the user may draw further - : strokes, trace the recorded stroke or adjust control points on the recorded. stroke to modify his or her sketch.

An example user's sketch in the form of a line drawing of an ancient Chinese phoenix is shown in Fig. 2(a) whereby the strokes are drawn in the style of silhouette drawings and are in the form of cleaned-up vectorized strokes. Figs. 2(b) = 2(d) respectively show the results after applying steps 104, 106 and 108 on the user's sketch in Fig. 2(a). In particular, Fig. 2(b) illustrates a needle map of a portion of the sketch in Fig. 2(a) whereby the needle map shows initial vectors automatically assigned to the strokes in the portion of the sketch, Fig. 2(c) illustrates a dense vector field obtained ~ after interpolating the assigned vectors in Fig. 2(b) and Fig. 2(d) illustrates a surface reconstructed using the dense vector field in Fig. 2(c).

Another example user's sketch is shown in Fig. 3(a). Similarly, Figs. 3(b) — 3(d) respectively show the results after applying steps 104, 106 and 108 on the user's sketch in Fig. 3(a) whereby Fig. 3(b) shows vectors assigned to the strokes in the sketch, Fig. 3(c) shows a dense vector field obtained from the assigned vectors in Fig. 5 3(b) and Fig. 3(d) shows a surface reconstructed using the dense vector field in Fig. 3c).

Steps 102 — 108 wil now be described in more detail.

oo oo oo ; n ~~ Step 102: Draw stroke(s) to form a user’s sketch and classify these strokes :

In step 102, strokes are drawn to form a user's sketch. Different types of strokes may be provided to the user for forming the user's sketch. oo + The types of strokes drawn by the user in step 102 determine the surface geometry (or ‘geometrical structures) to be recreated from the strokes. In step 102, the strokes in the ~ - user's sketch are then classified according to their stroke types: In one example, the = users are able to intuitively imagine from the stroke types provided to them the surface ‘geometries that wil be recreated from the stroke types. A recent study in which lines . . are drawn by artists to convey 3D shapes [33] shows that humans can easily detect . B jumps, ridges, valleys, bumps and dips on 3D surfaces. Hence, in one example, the stroke types provided to the users comprise jump strokes, ridge strokes, valley strokes, bump strokes and dip strokes for reconstructing these distinctive local structures. The stroke types may further comprise regular strokes which serve to determine an overall shape of the reconstructed surface. In other words, each stroke in the user's sketch : may be classified as one of the following: a regular stroke, a jump stroke, a ridge : stroke, a valley stroke, a bump stroke or a dip stroke. In one example, the strokes are -

Co drawn in the style of silhouette drawings (for example, as shown in Fig. 2(a)) and are set as regular strokes by default. In this example, the user may edit the sketch by selecting one or more of the regular strokes and changing the stroke types of these selected strokes. This helps to form and edit the sketch in a fast and straightforward manner. + Figs. 4(a) — (c) illustrates examples of different stroke types (regular strokes 402 (black solid lines), ridge strokes 404 (gray dash-dot lines), valley strokes 406 (black dash-dot : lines), jump strokes 408 (black dotted lines), bump strokes 410 (gray dashed lines), dip ) : “strokes 412 (black dashed lines)) provided in step 102 for forming the user's sketch - and the respective surface geometries reconstructed from these stroke types. As shown in Figs. 4(a) — (c), the surface geometries formed from bump and dip strokes comprise discontinuities in the surface normals whereas the regular strokes translate to smooth changes in the height map. Note that although regular strokes appear to result in different structures in Figs. 4(a) — (c), this is due to the ambiguity in the view angles and the surface geometries recreated by regular strokes are in fact the same.

Step 104: Assign vectors along stroke(s) in the user's sketch CG ~ In step 104, a plurality of vectors is assigned along each stroke in the user's sketch.

The assigned vectors (except those along jump strokes) represent the projection of 3D : 5 "surface normals onto the 2D plane on which the sketch lies. : : - - In one example, a plurality of initial vectors is first assigned automatically along each 3 : : stroke in the user's sketch. The term “automatic” is used in this document to mean that,

Co although human interaction may initiate the process, human interaction is not required while. the process is being carried out. This assignment of initial vectors may be - performed using a predefined scheme. In this example, the initial ‘vectors are - orthogonal to the tangents of the stroke (i.e. stroke tangents) and may have equal or varying magnitudes (for example, in the case of a T stroke as will be elaborated with - reference to Fig. 5(d)). The initial vectors may then be modified (by modifying their magnitudes and/or directions) to more accurately indicate the desired surface geometries. The modification may be performed manually by a user using for example, a keyboard or a mouse based on the user’s imagination of the desired surface. This is

Co more intuitive than inputting numerical values indicating the desired surface heights. : oo Furthermore, users may reconstruct a 3D surface from the initial vectors before deciding how the vectors should be modified. Alternatively, the modification may be : performed automatically based on preloaded user settings or predefined schemes. ‘Note that the modification of the initial vectors is optional. ~ Fig. 5(a) shows a black-and-white print out of a HSV (Hue-Saturation-Vaiue) color map with V = 1. This map may be used to illustrate vectors assigned along strokes in a user's sketch as shown in Figs. 5(b) — (d). Fig. 5(b) shows the initial assigned vectors

Co | perpendicular to the stroke tangents and having equal magnitudes whereas Fig. 5(c)

B shows ‘the initial vectors with their magnitudes and directions adjusted based on a + particular user-defined setting. Fig. 5(d) shows how strokes joined at a T-junction may be handied. For a T-junction, the initial vectors may be assigned such that the - magnitudes for the vectors near the T-junction along the visible part of the occluded stroke 502 are reduced to avoid affecting the vectors near the T-junction along the other stroke 504. Such a T junction may be automatically detected for the assignment of these initial vectors with varying magnitudes. :

The vectors assigned to each stroke type in step 102 are geometrically meaningful. For - example, a jump stroke indicates that a jump in the form of a step-like structure (i.e. a ~ discontinuity or a sudden decrease in surface height) is to be formed in the : reconstructed surface, and the directions and magnitudes of the vectors assigned along the jump stroke respectively represent the direction (i.e. on which side of the

Lo ~ stroke the surface height should decrease) and magnitude of the jump to be formed. - - Furthermore, strokes belonging to the remaining stroke types indicate that a slope (i.e. ‘ gradual ‘decrease in surface height) is to be formed in the reconstructed surface, and - ~ “the directions and magnitudes of the vectors assigned along each of these strokes | respectively represent the direction (i.e. on which side of the stroke the surface height ~ should decrease) and gradient of the slope to be formed. Larger magnitudes indicate . : that a steeper slope, in other words, a higher rate of decrease in the surface height, is . desired. - :

Regular strokes are strokes determining the overall shape of the reconstructed surface.

In one example, initial vectors are assigned to regular strokes based on a default oo height map. Fig. 6(a) shows a sketch with initial vectors assigned along its regular : strokes whereas Fig. 6(d) shows the 3D surface reconstructed using the vectors in Fig. -6(a). Fig. 6(b) shows only the vectors assigned along the strokes representing the nose 602 in the sketch. As compared to the corresponding vectors in Fig. 6(a), the vectors in

Fig. 6(b) have increased magnitudes. Fig. 6(e) shows the 3D surface reconstructed using the vectors in Fig. 6(b) whereby it can be seen that the increased magnitudes of the vectors have resulted in a higher nose. Fig. 6(c) shows only the vectors assigned to strokes representing the left ear 604. These vectors are rotated 15 degrees anti- clockwise as compared to the corresponding vectors in Fig. 6(a). Fig. 6(f) shows the 3D “surface reconstructed using the vectors in Fig. 6(c) whereby it can be seen that the - + rotation in the vectors have resulted in a twisted shape of the surface, leading to a ~ twisted face.

A ridge (or valley) stroke indicates that a sharp ridge (or valley) on the reconstructed surface is desired. in one example, a ridge (or valley) stroke is in the form of two : adjacent parallel regular strokes as shown in Figs. 7(a) — (b). Note that Figs. 7(a) — (b) are conceptual drawings and the space between the two parallel regular strokes is usually not visible. In fact, the ridge (or valley) stroke is usually not in this form in the

- | 10 | oo user's sketch. The stroke to be drawn for forming a ridge (or valley) stroke in the user's sketch is preferably in the form of a single line as this is more user-friendly (for ~~ example, as shown in Fig. 4, the ridge (or valley) stroke is in the form of a gray dash- - dot line (or a black dash-dot line). As shown in Figs. 7(a) — (b), the initial vectors assigned to the two parallel regular strokes of the ridge (or valley) stroke are orthogonal to the stroke tangents and are set either in repulsive directions to form a 3 ridge or in attractive directions to form a valley. In other words, the direction of the initial . vectors assigned along a ridge stroke differs from the direction of the initial vectors - ay assigned along a valley stroke by 180 degrees. In one example, the user can change "only the magnitudes of the initial vectors assigned to the ridge (or valley) strokes and : ~ not the directions of these vectors. If the magnitudes of the vectors are increased by > the user, steeper ridges (or deeper valleys) would be created in the reconstructed . surface. .

Jump, bump and dip strokes are useful in generating exquisite structures in the reconstructed surface. The structures may be formed on top of a given surface as shown in Figs. 8(a) — 8(c). This given surface may be an initial surface reconstructed ~~ from regular strokes, ridge strokes and/or valley strokes in the user's sketch.

Figs. 8(a) - (c) illustrate a base stroke, Strokes Of Interest(s) (SOls) and a : reconstructed surface for a jump stroke, a bump stroke and a dip stroke respectively.

Each of a jump stroke, a bump stroke and a dip stroke comprises a base stroke, and either one adjacent SOI (in the case of a bump stroke or a dip stroke) or two adjacent

SOls (in the case of a jump stroke). The first, second, and third rows in Figs. 8(a) ~ (c) are conceptual drawings. In particular, the first row shows the given surfaces without

Co jump, bump, and dip structures, the second row shows the changes in surface normals 2 “along the base strokes to form the respective structures and the third row shows the - | surfaces with the jump, bump and dip structures. The fourth row of Figs. 8(a) — (c) a | shows the reconstructed 3D surfaces with jump, bump and dip structures. The dashed, solid and dotted arrows in Figs. 8(a) — 8(c) respectively indicate the normal directions - for the regular stroke in the user's sketch, the given surface reconstructed from the ~ regular stroke and the base strokes of the jump, bump and dip strokes.

oo | no - As shown in Fig. 8(a), a jump stroke comprises two SOls which are parallel and adjacent to its base stroke. To form a desired jump, the surface height at the start of . the vector assigned to the base stroke is decreased to form a right angled slope. Note . that in this case, the magnitude of the vector assigned to the base stroke does not indicate the gradient of a slope. Rather, it indicates the size of the desired jump (i.e. the - : amount of decrease in surface height) in the reconstructed surface. If there is a given oo : surface on which the jump is to be built on, vectors identical to the surface normals of

N the given surface are assigned to the SOls, indicating the geometry of the given : - | surface. If such a given surface is not present, the vectors assigned along the SOls are ‘initialized to zero. . As shown in Figs. 8(b) and 8(c), a bump (or dip) stroke comprises one SOI which is parallel and adjacent to its base stroke. The bump (or dip) stroke indicates that a slope . is to be formed in the reconstructed surface and the magnitude of the vector assigned to the base stroke indicates the desired gradient of the slope. Similarly, if there is a given surface on which the bump (or dip) is to be built on, vectors identical to the : surface normals of the given surface are assigned to the SOls. If no such given surface oo is present, the vectors along the SOls are initialized to zero.

An example procedure of using different strokes for reconstructing a 3D surface is as 3 follows. Firstly, the user draws a rough shape of the desired surface using regular strokes, ridge strokes and/or valley strokes. Initial vectors are then assigned to these - strokes and a surface is reconstructed from these assigned vectors. Based on this reconstructed surface, the user then modifies the magnitudes and/or directions of the initial vectors to obtain a more desired reconstructed surface. Jump, bump and dip strokes are then drawn to form detailed geometrical structures on the reconstructed

Co surface. Vectors are then assigned to the base strokes and the SOls of these jump, . bump and dip strokes as shown in Figs. 8(a) — 8(c) whereby the vectors assigned to the SOis represent the projected surface normals from the initial reconstructed surface (i.e. the given surface). in one example, only the initial vectors assigned along regular strokes can be rotated by the user. This is because even though it is possible to adjust the geometry of the : reconstructed surface by rotating the initial vectors assigned along other stroke types, it

~ WO 2012/118439 PCT/SG2011/000082 12 ) | - is not intuitive to know the outcome of such rotations especially for users who do not understand the mathematical meaning of the vectors. Thus, to make method 100 more intuitive and user-friendly, it is preferable to disallow rotations of the initial vectors - | assigned along the other stroke types. oo 5 . +. Step 106: Interpolate the assigned vectors to form a dense vector field

Co “In step 1 06, the vectors assigned in step 104 are interpolated to. obtain a dense vector field. - - oo = The sparse strokes in the users sketch may be expressed as I —Q where

Qc Ris the computational domain in which the 3D surface is to be reconstructed.

The vectors assigned along the strokes I" may be expressed in the form of a vector © fieldn® = (n;,n:)". In step 106, this vector field Nn" is interpolated to obtain a dense - 15 vector field n=(n,n) on the computational domain Q. The 3D surface to be reconstructed is associated with a height map z = I(x, x,) whereby z is the desired

Co height at the point (x,,x,) in the 2D plane on which the sketch lies. The assigned vectors Nn” along the strokes (except jump strokes) may be regarded as the projection of surface normal vectors of the 3D surface (with the desired height map z) onto the 2D plane. Since the direction of the surface normal vectors of the 3D surface can be - expressed as (—0,1,—0,1,1), the assigned vectors mn” along the strokes (except © jump strokes) can be expressed as — VI . :

In vector calculus, if a computational domain QQ < R%is simply connected, then

Equation (1) holds if and only if 3/:Q — R such that VI =n . Equation (1) represents the curl-free condition typically known as the integrability condition in visible ol surface reconstruction.

Vxn=0,n —-0,n,=0 (1)

~ WO 2012/118439 PCT/SG2011/000082 . oo 13

Since a surface associated with the height map z =1 (x,,X,) whose gradient fits the ~ interpolated vectors n is to be reconstructed, the curl-free condition on the interpolated : - (or dense) vector field nis preferably satisfied. j Step 106 is performed using a ‘weighted combination of a Total Variation (TV)

ES | regularization method for preserving sharpness of the reconstructed 3D surface and a

SENS 5 H' regularization:method for preserving smoothness of the reconstructed 3D surface. In particular, this may be performed using an energy minimization functional with a curl- : free constraint as shown in Equation (2). Equation (2) shows minimizing a non-linear | functional of the vector field n" involving terms defined over the domain QQand a term i defined over the strokes I. In other words, Equation (2) implements non-linear vector interpolation. Note that Equation (2) is to be solved with a curl-free constraint on the dense vector field nn. : mine, =[ (1-2) Vn + g|Vaf +nfin-n | with Vxn=0 2 ~~ The energy minimization functional in Equation (2) is a weighted combination of the TV regularization method and the H' regularization method. In Equation (2), a dense curl- : free vector field I is obtained from the vector field n” assigned along the strokes I’ by minimizing an energy function & (m). |.| is the Euclidean norm in R” and |.}, is the Frobenius norm as shown in Equation (3) below. Note that the integration domain ~. Tincludes both closed and open strokes. In other words, the integration domain - T includes all the SOs and base strokes in jump, bump and dip strokes. oo Vn, ’ 22 | Vn |2= =>>(0@n) 3)

Vn, j=1 i=l . . - F .

In Equation (2), the first term [| (1-g)|Vn|, represents the TV regularization method whereas the second term |, g|Vn|. represents the H' regularization

; | EE VE ~~ method. The third term n| | n—n"| (where 7 is a positive constant) is a penalization - parameter to preserve n° on Tin the interpolated vector field n. . E Generally, the H' regularization method reconstructs a smoother surface while the TV

B 5 oo regularization method reconstructs a sharper surface. This is because discontinuities in § - the vector field assigned along the strokes are usually well interpolated into the domain . io 0 under the TV regularization method whereas they are smeared into the domain Q . | under the H' regularization method. Thus, the first term representing the TV - regularization method in Equation (2) serves to generate nonlinear diffusion and preserve discontinuities for example, discontinuities in creases or jumps in the interpolated vector field whereas the second term representing the H' regularization method in Equation (2) serves to generate linear diffusion and smear out discontinuities ~ inthe interpolated vector field.

Using the TV regularization method to preserve discontinuities is advantageous. This is because while other algorithms (for example, the algorithm in [20]) require information

Co indicating the locations of the discontinuities in the domain Q, the TV regularization oo method does not. Since it is usually very difficult to find the locations of the discontinuities from just the vectors assigned along the strokes, using the TV regularization method in Equation (2) allows an easier implementation of the method 100. - in Equation (2), the TV regularization method and the H' regularization method are “weighted using a weighting function (or indicator function) g which may take on any value between 0 and 1. The value ofg (i.e. the weight) determines how smooth or . “sharp the reconstructed 3D surface would be i.e. it determines the texture of the reconstructed 3D surface. Thus, by combining the TV regularization method and the H' regularization method as shown in Equation (2) with different values of g, various surface geometries can be generated in the reconstructed surface. Note that when g=1, only the H' regularization method is implemented Whereas when g = 0, only the TV regularization method is implemented. In one example, the value of g for the sketch is set based on a desired texture of the surface to be reconstructed and the

- | 15 - assigned vectors are interpolated by weighting the combination of the TV and H' regularization methods using this value of g. " ~The value of g for the sketch (including each stroke and the spaces between the strokes in the sketch) may be set by the user using for example, a painting tool. In one 5 ) example, the painting tool is used to paint the sketch in a color associated with a i LC variable alpha value and the value of g for the sketch may be set by varying the alpha oo - value. To perform this, the value of g ([0, 1]) may be mapped with the alpha value (either [0, 1] to 0,255] or [0, 1] to [255,0)). : | | SE

In one example, after setting the value of g for the sketch, the value of g is modified for each stroke in the sketch based -on the classification of the stroke. Considering the . differences between the TV regularization method and the H' regularization method as

Co mentioned above, g may be modified to 0 for jump, ridge, valley, bump, and dip strokes (to preserve the discontinuities in the resulting structures) and may be modified to 1 for regular strokes (to generate a smooth surface). - Alternatively, the user's sketch may be divided into a plurality of portions and a different : weight may be set for each portion based on a desired texture of the surface to be reconstructed for the portion. In one example, g is set to 0 in a first portion fo preserve discontinuities of surface gradients in the first portion and is set to 1 in a second portion to generate a smooth surface in the second portion. Each portion may be painted with "a color associated with an alpha value which may be varied to set the weight for the portion. Similarly, the value of g may be mapped with the alpha value (either [0, 1] to 0,255] or [0, 1] to [255,0]) and the value of g for each stroke in each portion may be :

Co modified according to the classification of the stroke. :

Figs. 9(a) — 9(d) and 10(a) — 10(f) show how the value of the weighting function g affects the reconstructed 3D surface.

Figs. 9(a) and 9(b) respectively show the reconstructed surface from a user's sketch when g =1 and when g = 0. Fig. 9(c) shows the division ‘of the user’s sketch into two oo | 16 portions: a left half and a right shaded half. Setting g=1 on the left half of the sketch and g =0 on the right shaded half of the sketch results in the reconstructed surface in

E Fig. 9(d). As shown in Fig. 9(d), the reconstructed surface is a natural combination of 3 - the respective portions of the reconstructed surfaces in Figs. 9(a) and 9(b). so EE : } : Figs. 10(a) and 10(b) respectively show a user's sketch comprising regular strokes and 3 initial vectors assigned to the strokes. Note that the vectors assigned to the cross in the rectangle are parallel to the stroke tangents in the cross. In the initial vector field, four

N discontinuities are present at the four corners of the outer rectangle and three 10 . discontinuities are present on the cross. These seven discontinuous points are indicated by the arrows in Fig. 10(b). Fig. 10(d) shows the reconstructed surface by oo ‘setting gto O (i.e. by using the TV regularization method alone) for the entire sketch. a As shown in Fig. 10(d), the discontinuities in the initial vector field are well interpolated and are in the form of sharp ridges. Fig. 10(e) shows the reconstructed surface by setting gto 1 (i.e. by using the H' regularization method alone) for the entire sketch.

As shown in Fig. 10(e), smooth surfaces are obtained in the regions corresponding to

Co “the interpolated vectors. Fig. 10(c) shows the division of the user's sketch into two oo portions: a textured portion and a non-textured portion. Setting g = 0 for the textured portion and g =1 for the non-textured portion results in the reconstructed surface of

Fig. 10(f). As shown in Fig. 10(f), the reconstructed surface is a natural combination of the respective portions of the reconstructed surfaces in Figs. 10(d) and 10(e).

The model in Equation (2) is implemented with a curl-free constraint. imposed on the ~~ dense vector field n. This helps to reconstruct the height map / with VI =n. It also ‘25 helps to interpolate sparse vectors over a large domain. Figs. 11(a) — 11(f) illustrate the advantages of having the curl-free constraint imposed on the interpolated vectors. Fig. 11(a) illustrates a user's sketch comprising regular strokes whereas Fig. 11(b) illustrates the initial vectors assigned along strokes of the user's sketch. Figs. 11(c) and

Fig. 11(d) illustrate reconstructed surfaces obtained without the curl-free constraint (with Fig. 11(c) illustrating a reconstructed surface obtained using the H' regularization method and Fig. 11(d) illustrating a reconstructed surface obtained using the TV regularization method). Figs. 11(e) and Fig. 11(f) illustrate reconstructed surfaces obtained with the curl-free constraint (with Fig. 11(e) illustrating a reconstructed surface oo 17 obtained using the H' regularization method and Fig. 11(f) illustrating a reconstructed ~ surface obtained using the TV regularization method). Comparing Figs. 11(c) and 11(d) against Figs. 11(e) and 11(f) respectively, it can be seen that with the curl-free

So o constraint, the weighting function has a greater effect on the reconstructed surface. 50 Furthermore, with the curl-free constraint, the reconstructed surfaces reflect the initial : g vector settings more accurately and sharper surface details can be obtained. - However, it is difficult to apply the curl-free constraint such that the constraint is - satisfied within the entire domain {2 because this constraint is usually violated at discontinuities in the interpolated vector field. The following describes an example method to overcome this difficulty. | oo © Augmented Lagrangian method to minimize the energy function in Equation (2) “In one example, an augmented Lagrangian method is used to minimize the energy functional (i.e. non-linear functional of the vector field) in Equation (2). This overcomes the difficulty in applying the curl-free constraint and allows the curl-free constraint to be

SE satisfied almost everywhere in the domain (2. This method is elaborated below.

Using an operator splitting method with two variables as shown in Equation (4), ; Equation (2) is transformed into a constraint minimization problem as shown in

Equation (5). . . ’ 5 Ss

DE p=)" Pans s= | @ co . Pp, Pa Pa 3, : } , , min | (1-2) | P|. +] g| Pf; +n] |s-n"

SE with P=Vn,s=n,and Vxn=0 (5)

To solve the constraint minimization in Equation (5), an augmented Lagrangian : functional L,(s,n,P;% , A, A, )as shown in Equation (6) is formed where cc, and c, are positive penalty parameters, and 2 _, A. and A are Lagrangian multipliers. If oo | 18 LL oo the weighting function gin Equation (2) is setto 1 on Q, it is not necessary to use c_, ~ A,and P.

Co LO=[U-9IP +g |P[ +n) is-n’] el ” oc i : = | a +A, -(P=Vn)+=[[P-Vn], : | » Cy 2 5. +h s—m+—-[ls-n oo + [0.2 (Vxm) +2 [ (Vn) | oo © - | o, in Equation (6) is a weighting function. @, is adaptively chosen depending on the . - geometrical structure to be reconstructed which in turn depends on the stroke types. In one example, w, is set to 0 for ridge, valley, bump, dip and jump strokes and is set to 1 otherwise because the interpolated vector field n violates the curl-free constraint on discontinuities in surface height and gradient. In one example, the weighting function @, is determined automatically from prior knowledge of the stroke types on the user's sketch. Note that although crease structures which violate the curl-free constraint may arise when @,=1, surface distortion is reduced because of the presence of the TV . regularization method in Equation (2) (see Fig. 9(d)). In one example, since @. is a smooth function, a, is set as the convolution of a 2D Gaussian kernel with a smooth factor 2. in other words, Gaussian linear filtering with a scale parameter 2 is performed to obtain @.. The same kernel may also be used as the weighting function gin 20. Equation (2). Having @_ in Equation (6) is useful as without this variable, it may be . necessary to choose a substantially small penalty parameter c, for strokes which violate the curl-free constraint. A small ¢_ leads to a slow convergence for the iterative procedure described later on and thus, is not practical.

The following is an example algorithm incorporating an iterative procedure to find the saddle point of the augmented Lagrangian functional in Equation (6) via maximizing the ~ Lagrangian multipliers and minimizing variables §, nandP. ~ Algorithm 1: Augmented Lagrangian method for Equation (6) 3 Step 1: Initialize s',n’, P00, A and Ato 0. A variable k (representing the iteration the algorithm is at) is also initialized to 0.

Step 2: Increase the value of k by 1 and if k 2 1, compute an approximate minimizer - (s",n",P*) of the augmented Lagrangian functional LATA A using an alternating minimization method. This is performed according to Equation (7) whereby s,m, Pare the variables to be minimized: : 3 (s*,n",P*) ~argmin L (s,n,P; A SATAY (7)

Step 3: Update Lagrange multipliers using the following equations (8) — (10) k k-1 k k - A, =L] +c, (s"—n") (8) an k 1, CE. k r=A +—=(Vxn") (9) oo Af =A" +c (PF —Vn) | (10) © Notethatif @,= 0 at a point, then A = A at the point

Step 4: Calculate relative residuals R' using Equation (11). If the relative residues R! are larger than an error bound &, repeat steps 2 — 4. Note that the relative residues do ‘not depend on the size of the domain or the penalty parameters and are hence suitable : to be used for determining if the iteration should stop. -

oo 20 . } 1 ~ : ’ R, = . I R, I, Vi € {1,2,3} ’ (11) oo n | | Q2] 3 where |} - [is the L norm on Q, | Q}is the area of the domain and oo - oe | (R!, RY, RL) =(s* —n*,V xn*,P* —Vn*) (12) "© In one example, the relative errors of the Lagrange multipliers ‘are also calculated in - step 4 of Algorithm 1 to determine if the iteration should stop (i.e. if steps 2 — 4 should ‘be repeated). These relative errors are calculated as shown in Equation (13). In a

N further example, the numerical energy is also calculated to determine if the iteration

B ~should stop. The numerical energy is calculated as shown in Equation (14). Similarly, : “the relative "errors and/or numerical energy values are compared against respective i. error bounds to determine if the iteration should stop. Co 3

Co k k-1 k k-1 k k-1 ’ oo IF — | 2, - A I, LF — I A, = A, I, J 5 — I A] = A’ I, (13)

Co we, IA . : : f r < ba r Lo - ; : kp . : el =[ 0-9) P| +g | PF nfls-n’| (19 ~ Fig. 12(a) shows pilots of the relative residues (calculated using Equation (11)) against the number of iterations (the plots in Fig. 12(a) show a total of 10,000 iterations (in the log scale, this is log10000 = 4), Fig. 12(b) shows plots of the relative errors of the ~~. Lagrange multipliers (calculated using Equation (13)) against the number of iterations "and Fig. 12(c) shows plots of the numerical energy values (calculated using Equation :. (14)) against the number of iterations for the examples in Figs. 2(d) (top) and 18(b) (bottom). The axes of Figs. 12(a) and (b) are in the log scale whereas in Fig. 12(c), the y-axis is in the log scale and the x-axis is in the decimal scale. In Fig. 12(a), R1, R2 and R3 respectively correspond to R',R* R" in Equation (12) whereas in Fig. 12(b),

L;, Lc, L; respectively correspond to LL, L, in Equation (13).

Co : | : 21 BE 2 The plots in Figs. 12(a) — (c) numerically show that the above numerical algorithms are convergent. The relative residues and the relative errors of Lagrange multipliers have - the same convergence order. Furthermore, the numerical energy reaches a steady i | : state. Note that the results shown in the figures in this application are obtained in much less than 1000 iterations; see Table 1. In example numerical experiments, it was found - Co ‘that hardly any visual difference in the reconstructed surface can be detected after the oo condition for the relative residues to be less than an error bound of 0.001 is satisfied.

As shown in Figs. 12(a) — 12(c), the relative residuals and the relative errors of the

Lagrange multipliers have the same convergence order whereas the plot for the numerical energy values reaches a steady state with less iteration. Note that the speed

IN at which the plots converge is affected by the values of the penalty parameters. Larger : .* penalty parameter values will result in the numerical energy values decreasing more ~~. slowly across the iterations (i.e. the numerical energy plot takes a longer time to . converge) whereas smaller penalty parameter values will result in the relative errors in

Lagrange multipliers and the relative residues decreasing more slowly across the iterations (i.e. the relative errors plot and the relative residues plot take a longer time to converge). lt is preferable to tune the penalty parameters such that the relative errors plot and the relative residues plot converge faster. - . The following is an example algorithm for an alternating minimization method to solve

Equation (7) in Algorithm 1 as described above. - Algorithm 2: Alternating minimization method to solve Equation (7) - | Step 1: Initialize the variables: §° =s"" A’ = n'’ and P° = P*

Step 2: For /=1,---,L and fixed Lagrange multipliers 2, =272, = 2" and

A, =A!" solve the following minimization problems alternatively: s' =argmin L(s.5™, P70, ,2,A,) (15) . n’ =argmin L(3.nP™h 2, A) (16)

P' =argmin L(s',a' Ph A A) (17)

; | | 0 3 The minimizers §*, 1’ and P’ above are found by minimizing the energy functionals (18), (19)and (20). | .

Ce ee g

SE &®=]~ |s—n" |" +h, s+ nll s-n | (18) 0 c, ~. c, , : oo m=] IP —Vn[; -A,-Vn+=(Vxn) +a, (V xn) y CLC _np +—=—|8'-n| -A,-n ! : (19) c ~ e(®)=[ (1-2) |P| +g|PL + | P-VE I: +A, -P | (20) n More détails on how the minimization of the functionals (18) — (20) may be performed cl are described below. co : : Co

Step 3 After L iterations, (s*,n",P*) is updated by setting ~ (s*,n*,P")= (55,85, PY) . In one example, only one iteration (i.e. L = 1) is performed and this is sufficient to achieve desirable results.

Minimization of functionals (18) — (20) “15 3 The minimization of functionals (18) and (20) (c.f. Equations (15) and (16)) may be : solved by a soft thresholding method [25] involving simple arithmetic computation oo whereas the minimization of functional (19) (c.f. Equation (17)) is terminated by solving oo ~ a system of linear equations over the whole domain £2. Since the grid for this system of linear equations is uniform, Fast Fourier Transform (FFT) can be used to solve these equations at a low computational cost.

The variables s,n, P, ho, As An the augmented Lagrangian functional in Equation (6) may be defined on a discrete computational domain C2 =[1, NV, ]x[1,N,]. Figs. =~ ~ 13(a) — 13(b) show a staggered grid system. Physically different locations in the staggered grid system are used to evaluate the values of the variables. - -

~ 23 : “Fig. 13(a) shows the rule of indexing variables s,n, P, hoy As Ain the augmented - Lagrangian functional in one example. In this ‘example, the first and second : i components of m, sand A are defined at squares [I and hollow dots o respectively - “whereas p,,, p,, A, and A, are defined at solid dots ®. The other variables are : 5 ‘defined at triangles A whereby Ai, J) is at the triangle .1302, p,, (i,j) and ~~ A,,(i, j)are at the triangle 1304, and p,,(i, j) and A, (i, j) are at the triangle 1 306. These rules of indexing may be used in the discretization of the Euler-Lagrange equations for (19). Fig. 13 (b) shows an example of implementing the rule in Fig. 13(a) in a discrete computational domain of size 5 x 4 (i.e. [1, 5] x [1, 4]). :

To Co a Details of examples for minimizing the functionals (18) — (20) in step 2 of Algorithm 2 + areas follows, - : oo

Minimization of &,(s)in Equation (18) . Denoting n = in’, the functional in & (8) can be represented in two parts as shown in

Equations (21) — (23). oo £,(8) = &qr (8) + &1(8) (21) where :

Eq (8) = [. A, -s+ ls - n| : | (22) : . * | Cc, * 2 . er) = [nfs nm +d, s+ isn] (23)

The minimizer in the first energy functional £,.(S) is easily obtained since the integrant is a quadratic polynomial in terms of §. To obtain the minimizer in the second “energy functional &.(s), Equation (23) is.reformulated as Equation (24). in Equation (24), the value of Cis independent of the minimization. : : -

SE : 24

C2 oo JC I

REE g.(s)=[ns-n'|+-Lls—n' +L +C (24)

For each coordinate (i, j) eT, a soft thresholding method [32] is used and the : Co minimizer $s’ in Equation (15) is obtained by the following Equations (25) — (26) where n Lo A, a=max|0]l-———|and Xx, =n ——.

CU Cc, | X, | ¢, cee tis Coa Lo a

Co @,/)el'=s%s (7) =nG, j) = —4,(, J) (25) Co

CG HeT=80GH=0G) al Hx 6GH 0 28)

Minimization of &,in Equation (19) : | : oo For a fixed P = p+ and s =7%', the Euler-Lagrange equation of (19) yields a system of linear partial differential equations (PDEs) as shown in Equation (27) where . n=(n,n,), V'=(-0,,0)". V-P =(V-p,,Vp,). lis a 2x2 identity matrix,

D =(0*+&) and D, =V*V*' x —(¢.D, +c.D,—c,Im=V-A ~V(@2)+h, -V-P+cs (27)

Since D, is a Laplacian operator and operates on each component of n, its presence in Equation (27) makes Equation (27) a coupled equation and results in the diffusion of n’ on the strokes. However, the differential operator Din Equation (27) (present due to the enforcement of the curl-free constraint on the dense vector field) generates extra force to interpolate n” onto the whole domain. Co

So | 25

Using the standard finite difference scheme with the rule of indexing variables as no shown in Fig. 13(a), nin Equation (27) is discretized at a square [J whereas hn, in . "Equation (27) is discretized at a hollow dot ©. Applying the periodic boundary condition oo to all variables, the discrete Fourier Transform F may be used to achieve the: * discretization of Equation (27). The shifting operators comprising the finite differences in Equation (27) essentially represent a discrete convolution whose discrete Fourier - Transform is a component-wise multiplication in the frequency domain.

For discrete frequencies u,u,,a system of linear equations as shown in (28) may be © 10 © formed. :

RE (a, a, Fnu,u,) filu,u,)) Co

Co = (28) ay ap NF uu) Susu) | oo

Denoting

Loa VY. . :

S, =2sin” = +~-lsmy, (29) ’ | . . 2 V, . . : Co ¢, =2sin” —++-=1sinv (30) . oo J 2 J C 2 27 : where v. =—u, (u.=1,---,N)and v, =—u, (u =1,---,N,), the coefficients . ! N ! I J N, J J 2 1 . in the system of linear equations in (28) are ~~

L,Y Lo, v a, =4c, sin’ —+4(c, +c )sin’ = +c, 2 2 oo a, = ACP d, =¢, (5s; . 5 Vv, . Vv, . . : a,, =4(c, te sim + 4c, sin > te (31)

BEN SE | 26 : Furthermore, fi and f,on the right hand side of Equation (28) can be expressed as: . 3 follows. - : | 2 | : - = ~~ f=gFa,+iFa,-Fa Ee f,=¢ Fa, +g Fa; - Fa, | (33) : = where

Ca =A een) HED RA ey co - ‘ } - - a’ = A + C,S, } oo (36) Se - There are N,N, 2x2 systems and the determinant of the coefficients matrix in the - above equations for all discrete frequencies is given by Equation (37) where oo 2 Y 2 VY : | : : - 15 B; = Sin osm oe This determinant is not zero since the penalty parameters oo ¢,, C,, and c are positive. After the system of linear equations is solved for each . a frequency, the discrete inverse Fourier transform is used to obtain Ti”.

Minimization of &,(P) in Equation (20)

Denoting n = 1’, the functional in Equation (20) can be reformulated as follows. In ~~ Equation (38), the value of C is independent of the minimization. : eP)=[ (1-2) |P| +——"=P-———(c,Vn-A) +C (38)

Co “ r 2 c +2g .

g The following approach [8] may be applied to find the closed form of the minimizer at each point Gj) eQ:

BG.) = mas 0 lan we 9)

Ce IWEDL TT 5 : ~~ where

SE W=—"(cVn-1) (40)

EE c, t2g . Step 108: Reconstruct a 3D surface from the dense vector field N 0 SE

SO in step 108, a 3D surface associated with a height map z= I(x, ,X, ) is reconstructed

Co on the computational domain £2 from the dense vector field n obtained in step 106.

This is performed by locating a height map / whose gradient field fits the dense vector © field n. - in one example, the 3D surface is reconstructed via a weighted combination of two :] methods, a TV regularization method for preserving the sharpness of the reconstructed surface and a H' regularization method for preserving the smoothness of the - reconstructed surface. This may be based on the minimization of energy functional &,(/) as shown in Equation (41) which implements a weighted combination of the TV regularization method and the H' regularization method. In Equation (41),

I, represents desired height values on <Q and kis a weighting function (or indicator function) which may take on any vaiue between 0 and 1. Similar to Equation 3 (2), Equation (41) shows minimizing a non-linear functional of a height map, the non- linear functional involving terms defined over the domain Q (the domain in which the oo surface is to be reconstructed) and a term defined over the domain X(the domain in ~ which prior information regarding heights is present). The prior information regarding oo heights, in other words, the initial height information, may be specified by the user, : making the modeling process similar to a morphing process from one height map to another. Furthermore, certain formats of the initial height information may comprise

SEE | | 28 : vectors on a part of the domain. In one example, if no prior information regarding "heights is available, &is set to 0. Furthermore, the convolution of a 2D Gaussian kernel © with a smooth factor 2 may be used for the weighting function 7 - Co BE : i in (= [A= hVI-nl+ [AVI <£[1-1)} @) min ¥,( )=|,A=m\VI-n|+ [ WVI-n] +&[|I-1] (41)

In Equation (41), the term VI —n| represents the TV type norm (i.e. the TV

EE regularization method) whereas the term \Z —n| represents the H' type norm (i.e.

CT the H- regularization method). The TV type norm preserves sharpness of the = reconstructed surface and reconstructs discontinuities (for example, jump" . . discontinuities) without surface distortion whereas the H' type norm enforces oo - smoothness of the reconstructed surface by enforcing the C° continuity of the height map (since [. hVI —n| yields the Laplacian operator in the Euler-Lagrange equation). : ~ It is preferable not to apply the H' type norm to the entire domain C2 (i.e. not to set ~ h=1 for the entire domain {2) as problems of overshooting or undershooting may be

SA “encountered with the jumps in the reconstructed surface. On the other hand, the use of the TV type norm when / #1 helps to prevent these overshooting or undershooting problems. In one example, considering the different effects from the TV and H' type - | norms, the value of the weighting function A (ie. weight) is set individually for each stroke based on the classification of the stroke. For example, it may be set as 0 for jump strokes and 1 for other strokes. In one example, the location of the jump strokes are known beforehand and % is set automatically based on this prior knowledge. | oo ~ Minimization in Equation (41)

In one example, the minimization problem of Equation (41) is solved by an augmented - Lagrangian method as follows. Note that there are other methods for minimizing the energy function in Equation (41) as discussed in references [39] — [42]. The same applies for minimizing the energy function in Equation (2).

CL . 29

The minimization problem of Equation (41) is first transformed into a constraint oo : minimization problem as shown in Equation (42) whereby q=Vland J =1.

SE . ) : ~~ min d-Mig-n{+] hlg-n| +&[1J = 1, | (42)

The minimization problem is then cast into an unconstrained problem as shown in

Equation (43) and a saddle point of the augmented Lagrangian functional in Equation (43) is found. In Equation (43), d, and d ,are positive penalty parameters and pt, and , are Lagrange multipliers. ~~. oo wo | Co oo oo © L(=[,0-hlq-n|+hlq-n[ oo - +n (@-VD +] q-vrp . Q r 2 Q .

J—-1 4, J=D+&|| 1-1 oo +o I =D =D E11, | (43)

The optimality condition for Equation (43) may be found using an iterative algorithm

C similar to Algorithm 1 and Algorithm 2 as described above. More details on this iterative, algorithm are described in reference [11], the contents of which are incorporated by reference herein. This iterative algorithm is typically used when & #0 or when - h#lonQ). oo .

If no prior information about the desired heights is available and no jump strokes are present, £ =0 and h is set as 1 =1, and Equation (41) simply comprises the term : representing the H' type norm. In this case, the Euler-Lagrange equation for [ is oo simply a Poisson equation as shown in Equation (44). Thus, Equation (41) need notbe . ‘25 solved by an iterative algorithm. Instead, the minimization of the energy functional in

Equation (41) can be simply achieved by solving the Poisson equation in Equation (44). in other words, a solution of the Poisson equation in Equation (44) is a reconstructed height map whose gradient fits the vector field n (this is due to the curl-free constraint imposed on Equation (2)).

Co CL 30 oo V-(V)=V-n | (44) 2 ‘Surface distortion is eliminated in the solution of Equation (44) since the interpolated '5 vector field n satisfies the integrability condition. Hence, a surface associated with a height map whose gradient fits the vector field n may be reconstructed without surface . distortion. - ~~ Step 110: Perform local editing on the reconstructed 3D surface oo oo oo oo i. a Step 110 is an optional step for modifying surface details on the reconstructed surface. - A real time computation may be performed in step 110 since the computational cost for step 110s not high.

The entire domain of the reconstructed surface need not be interpolated again to oo modify surface details in the reconstructed surface. Instead, in one example, the user's sketch is first modified by adding at least one further stroke to the sketch and the ~~ computational domains are re-defined based on where the modifications in the user's : sketch are. Equations (2) and (41) are then implemented using the re-defined computational domains.

Figs. 14(a) and 14(b) respectively show a user's sketch and a reconstructed surface from the user's sketch whereas Figs. 14(c) and 14(d) respectively show the user's sketch with added strokes (i.e. with local editing) and the reconstructed surface with additional surface details corresponding to the added strokes. . As shown in Fig. 14(c), several strokes are added to the sketch within a local domain 1402 and a bounding rectangle 1404 around the strokes in the local domain is located (either manually or automatically). A portion of the reconstructed surface corresponding . . .30 to the bounding rectangle 1404 in the sketch is then modified as follows. :

To incorporate the additional surface details into the reconstructed surface based on the additional strokes in Fig. 14(c), the computational domains are re-defined. in one example, {) is re-defined as the domain enclosed by the bounding rectangle 1404

EE 3 3 | whereas [is re-defined as the strokes in the bounding rectangle 1404 (comprising the . newly added strokes, previously present strokes, and the region 1406). The region : 1406 comprises the region within the bounding rectangle 1404 minus the region within - the boundary 1402. Vectors are then assigned along the strokes I and the assigned vectors are then interpolated using Equation (2) to form a dense vector field. The portion of the reconstructed surface corresponding to the bounding rectangle in the sketch is then modified by reconstructing this portion using Equation (41). Equations (2) oo and (41) are computed using the re-defined domain €) and strokes I. Furthermore, in . Equation (41), Z can be set as the region 1406 and the height information on can be used as I,. As shown in Fig. 14(d), the additional surface details are incorporated : ‘without changing the geometry of the previously reconstructed surface. In the example shown in Figs. 14(a) — 14(d), the domain size is 64x64 pixels and the local editing step . uses less than 1 second. Note that although a single local domain 1402 is shown in Fig. 14, two or more separate local domains may be defined in a single sketch. | : : Comparison of method 100 with existing algorithms - “As compared to existing WIMP-based systems, using method 100 to reconstruct 3D surfaces is more intuitive, less time-consuming and less labor-intensive. Furthermore, the 3D surfaces reconstructed by existing SBIM systems such as SKETCH [12], Teddy

[13], SmoothSketch [14] and FiberMesh [15] are usually either entirely rough or entirely smooth and rounded. in other words, the 3D surfaces reconstructed by these existing ~ SBIM systems usually do not comprise surface details, for example the shapes of creases. Method 100 does not have these limitations. : Method 100 is also advantageous over the methods in LUMO [6] and ShapePalettes

[7]. The vector interpolation methods in LUMO and ShapePalettes are linear and the ~~. reconstructed surfaces do not faithfully reflect the assigned sparse vectors. On the other hand, method 100 can faithfully reflect the vector settings in the reconstructed surface due to the presence of the curl-free condition. Although this requires nonlinear vector interpolation, a fast solver based on augmented Lagrangian method [8] is used in method 100 to allow the nonlinear vector interpolation to be performed at a fast speed.

oo WO 2012/118439 PCT/SG2011/000082

Clr 32 2 ) In one example, the drawing as shown in Fig. 15(a) [34] is traced by an-artist in the form of a sketch and Fig. 15(e) shows the assigned vectors along strokes in this - sketch. The dense vector fields and reconstructed surfaces obtained for this sketch . using the LUMO method, ShapePalettes method and method 100 are shown . respectively in Figs. 15(b) and (f), (c) and (g), and (d) and (h). As shown in Figs 15(f) - and (g), the ripples on the clothes are “smoothed out” in the reconstructed surfaces ‘because of the linearity and the lack of appropriate enforcement of the curl-free constraint in the LUMO and ShapePalettes methods. In the ShapePalettes method, the on curl-free constraint leads to surface distortion in the reconstructed surface as the constraint is not satisfied at jump discontinuities and discontinuities in the surface oo = normals. If prior information regarding the locations of such discontinuities is available, - - it may be possible to prevent the surface distortion from being formed with the use of a - ) proper weight function. However, such prior information is seldom available. On the ~ other hand, the surface reconstructed using method 100 clearly comprises the ripples | on the clothes due to the non-linear vector interpolation with the enforcement of the

Co curl-free constraint, the use of the TV regularization method and the use of the augmented Lagrangian method.

In another example, a surface is reconstructed from a sketch of a cat as shown in Fig. 20. 16(a) using the prior art methods and method 100. The reconstructed surfaces using - the LUMO method, ShapePalettes method and method 100 are respectively shown in ; Figs. 16(b) — (d). As shown in these figures, the LUMO and ShapePalettes methods can only reconstruct 3D surfaces with limited shapes and are unable to preserve surface details. On the other hand, due to the curl-free condition and the non-linear interpolation used in method 100, the surface details in the reconstructed surface in

Fig. 16(d) are sharper i.e. better preserved. Note that although the LUMO and

ShapePalettes methods are also capable of generating a reconstructed surface similar to that in Fig. 16(d), a large amount of additional effort (for example, drawing extra : strokes and implementing additional vector settings) is required. These additional steps ~~ may not be intuitive. | EE oo Furthermore, compared with most single-view modeling methods [9,10] that require a : large amount of user interaction (hence, time) for constraint specification, the vector assignment approach in method 100 is intuitive and user-friendly. Also, the algorithms

SE oo 33 : in [71, [9], [10] have difficulties ‘handling sparse gradients, preserving sharp features,

B and preventing surface distortion in surface-from-gradients construction as pointed out - by the authors of reference [16] (which uses a Gaussian kernel approach without

EE discrete integrability enforcement to overcome the difficulties). On the other hand, method 100 is not faced with the same difficulties due to the use of the TV : regularization method and the imposition of the curl-free condition with the nonlinear vector interpolation. On top of this, method 100 provides a very fast and efficient 5 ‘method to solve the energy functionals in the nonlinear vector interpolation via an cE augmented Lagrangian method. . - . Google SketchUp [17] is a sketch-based 3D modeling software and enables intuitive ] oo and fast modeling of 3D surfaces. However, to use the program in SketchUp, the : height of the 3D surface needs to be manually specified by users. This is performed by - using a mouse dragging operation or a keyboard input. On the other hand, the height of | the 3D surface reconstructed in method 100 may be automatically estimated. oo Furthermore, the system in SketchUp is not a single-view modeling system whereas such a system may be implemented using method 100. Furthermore, the process of ~~ using method 100 is also more aligned with traditional design stages, as designers usually sketch in 2D space during the early design stage before modeling and manipulating a model of the sketch in 3D space. In addition, method 100 is also - capable of modeling complex surfaces as illustrated in Figs. 17(a) — (c) using simple . strokes and a fast vector assignment. On the other hand, it may be difficult to create - such freeform surfaces using SketchUp.

Advantages of the embodiments of the present invention ~The key advantages of method 100 are as follows. = Combination of TV and H' requlaizalion methods in method 100 130

Previously, surface reconstruction from sparse information based on an energy minimization approach has been performed using visible-surface representations [30] whereby the weighted H' norm of data, together with its gradient with discrete fidelities, controls the local smoothness of the reconstructed surface. However, it is difficult to

IEE 3 approximate a proper weight function for this. Another previous method to control local : oo smoothness is the TV norm of data [31]. As jump discontinuities under the TV norm of . oo data are easily preserved without weighting functions, discontinuities of surface oo . ~~ gradients.can be recovered by the TV norm of data gradient. . ) In method 100, the TV regularization and H' regularization methods are combined in a . weighted manner to implement stroke-based surface reconstruction. The texture of the 3 - reconstructed surface can be easily controlled by varying the weight used for the g combination. This not only provides artists more freedom to create various types of . 3 ‘surface. geometries, it also helps in the reconstruction of geometrically “crucial ~ 1. structures (see Figs. 10 and 1 9) and the preservation of surface details (so that the » reconstructed surface comprises sharper surface features) without causing surface distortion. A few of these advantages arise due to the use of the TV regularization “method. The TV regularization method not only helps to preserve discontinuities in the vector field, but also helps to preserve discontinuities in the gradient of the ~ | reconstructed surface without causing surface distortion. Thus, jump discontinuities can - - be preserved on the reconstructed surface without overshooting or undershooting. oC © Use of the nonlinear vector interpolation with the curl-free condition in method 100 : in method 100, a nonlinear vector interpolation is performed to obtain a dense vector 3 field from a sparse vector field assigned along strokes in the user's sketch. This interpolation is performed with a curl-free condition imposed on the dense vector field.

The nonlinear vector interpolation in method 100 achieves a faithful refiection of the vectors’ settings and the resulting interpolated vector field in method 100 satisfies the - - integrability (curl-free) condition almost everywhere in the domain. This preserves oo surface details better without causing surface distortion in the reconstructed surface, hence achieving more desirable structures corresponding to the given strokes and vector settings (see Fig. 11).

IE - 35 oo ~ Method 100 is intuitive and user-friendly )

Method 100 is capable of creating a reconstructed surface as shown in Fig. 2(d) by using only regular strokes and default initial vector settings. in other words, method 100 can reconstruct a surface using strokes in accordance with the typical drawing style of - artists. It is common for artists to draw strokes in their own styles, instead of spending - - time to set stroke types or modify vectors to create more desirable surfaces. Thus, oo | ‘method 100 is intuitive and user-friendly. | | LL oo

Initial vectors may be automatically assigned based on for example, default initial ~ vector settings, to strokes in the user's sketch. In one example, the assigned vectors - are orthogonal to the stroke tangents and are of the same magnitude. Automatically assigning vectors to strokes in the user's sketch is advantageous over manually : | assigning the vectors (for example, by assigning them based on a 3D reference model © as performed in [7]) since it reduces the amount of work the user has to perform, hence allowing method 100 to be more user-friendly. - | Co However, the automatically assigned vectors can still be adjusted by the user to form a more detailed surface shape. in one example, the user can modify the magnitudes and directions of the assigned vectors using a keyboard or a mouse based on his or her } imagination of the desired surface. This is more intuitive than inputting numerical values for the surface heights.

Ability to construct different types of surface geometries using different stroke types and/or the weighting function g

To reconstruct more complicated surfaces, users have the option of using various stroke types to generate different geometrically crucial structures such as ridge, valley, jump, bump and dip structures. In method 100, the strokes are classified according to the desired surface geometry to be formed. The different types of strokes help users create desired structures on the reconstructed surface. These structures can be intuitively imagined from the corresponding 2D strokes (see Figs. 18 and 19) and thus, allows artists to create surfaces which they can intuitively perceive from the 2D strokes. : Geometrically meaningful vectors are assigned to each type of stroke based on the desired surface geometry. These assigned vectors may be a projection of surface oo | | 36 oo normal vectors onto the 2D plane on which the user's sketch lie. Method 100 can - successfully transform these sparse assigned vectors into desired structures in the reconstructed surface.

More complicated surfaces may also be formed by adjusting the weighting function } “gin Equation (2). For example, to reconstruct a surface comprising discontinuity in its - - gradient, a sketch may be divided into different textured regions and a user simply has Co on to decide which of the textured regions should contain the discontinuities. This is less =~ - . tedious than existing methods (for example, the algorithm in reference [16]) whereby it is necessary to indicate where exactly gradient discontinuities are located in the ~~ domain. ~ | oo - : ~ Fig. 19(a) illustrates a user's sketch of a monkey. in Fig. 19(a), the user's sketch is “divided. into two portions: a textured portion on the right and a non-textured portion on the left. Setting g=0 in the textured portion on the right and g=1 in the non-textured portion on the left results in the reconstructed surface of Fig. 19(b). As shown in Fig. - 19(b), this reconstructed surface comprises different geometrical structures which - appear sharp on the right and smooth on the left.

Figs. 19(c) — 19(d) illustrates the use of various stroke types to reconstruct a surface with gradient or jump discontinuities. Fig. 19(c) illustrates a user's sketch of a monkey with different strokes whereas Fig. 19(d) illustrates the reconstructed surface from the user's sketch in Fig. 19(c). As shown in Fig. 19(d), the reconstructed surface preserves discontinuities without any distortion and clearly comprises the intended features according to the stroke types in the user's sketch. :

Speed and low computational cost of method 100° BE - : 3 Fewer strokes are required by method 100 to generate reconstructed surfaces of a "30 particular quality as compared to for example the algorithm in reference [16].

Furthermore, local editing may be performed in method 100 whereby to add a few surface details to a previously reconstructed surface, it is not necessary to interpolate the entire domain. Rather it is simply necessary to re-define the domains based on where the modifications in the user's sketch are.

37 | oo

Furthermore, using the augmented Lagrangian method in method 100 helps to overcome the difficulties in solving the constraint minimization problem. Even though a nonlinear process is implemented for both the vector interpolation and the height map reconstruction in method 100, results can still be achieved quickly using very fast and efficient numerical solvers based on the augmented Lagrangian method [32]. This oo - reduces the required computational time and increases the speed of the- modeling = . - process fo reconstruct the 3D surface. The non-linear vector interpolation in method

Co 100 can be implemented at a low computational cost and at a high computational speed using the augmented Lagrangian method and a CUDA-based GPU computation.

In one example, using a nVidia Quadro FX 1800, a computational time of less than 15 : and 5 seconds respectively for a domain size of 512 x 512 pixels and 256 x 256 pixels . is required. ~ A lower computational time may be achieved depending on the user's sketch. = : : : oo

Co Table 1 shows the number of iterations performed and the computational time (Intel(R) - Xeon(R) CPU E5520 @ 2.27GHz, NVIDIA Quadro FX 1800) used for creating the : ‘reconstructed surfaces shown in some of the above figures. © To reconstruct the surfaces in the figures shown in Table 1, if the sketch comprises jump, bump or dip strokes, two interpolations are performed. The first interpolation is performed for reconstructing a 3D surface from regular, ridge and valley strokes and the second interpolation is performed for creating jump, bump and dip structures on top of the reconstructed surface from the first interpolation. This preserves the geometry surrounding the jump, bump or dip structures as shown in Figs. 2(d), 10, 18 and 19, in

Table 1, “1” indicates that the results in that figure are obtained after the first . . interpolation whereas “-2” indicates that the results in that figure are obtained after the =~

LL second interpolation. :

If &=0and h=1on Q (i.e. the user's sketch does not comprise jump strokes), an iterative algorithm is not required to solve Equation (41) since the height map is simply obtained using Equation (44). In Table 1, a dash *“* in the “Number of iterations” column for step 108 indicates that no iteration is required to solve Equation (41) to obtain the results in the figure. A dash “-* in the “Time” column for step 108 indicates

Lo | 38 g ‘that step 108 is not performed. To reconstruct the surfaces in the figures shown in

Table 1, if jump strokes are not present in the user's sketch, the following parameters n=100,c, =10,c, =1and ¢, =1are used in Equation (6). If jump strokes are present in the user's sketch, the following parameters &=0and pu, =0.01are used in the reconstruction step i.e. in Equation (43). ~ . Furthermore, for computing the statistics in Table 1, the error bound ¢ in Algorithm 1is’ = E set as 0.001 in step 106 and 0.05 in step 108 (if the iterative algorithm is necessary) for g all the figures shown in Table 1. This maintains the coherence of the computational : ~. cost statistics for-comparison using Table 1. However, note that bigger error bounds may be selected to reconstruct some of the surfaces to reduce the computation time

RL ‘required. : }

As shown'in Table 1, little iteration is required to reconstruct the surfaces in the above figures. Furthermore, the reconstructed surfaces are of reasonably high quality and can be obtained in a short time. Thus, it can be concluded that method 100 can be implemented quickly and at a low computational cost.

Figure Size of domain | Number of Time Number of Time :

Feme |we | we | ew || ome

Rese | me | w | ow ome [Eww wees |e [mew | w |am [mer me ww [Emer ww | ® | wm | oem - Table 1 oo 39

Multiple applications of method 100

Method 100 is not only capable of reconstructing a static 3D surface. It is also capable - of creating an animation sequence made up of a series of static 3D surfaces by gradually changing the vectors assigned to a user's sketch. For example, a plurality of . frames for an animation sequence may be created by iteratively reconstructing a 3D

Co surface from a user's sketch whereby the assigned vectors along the strokes in the =

So ‘user's sketch for each iteration are different. Thus, method 100 can be used in an - animation tool for creating short animation clips. Figs. 20(a) — (d) show various frames “selected from an example animation sequence formed using method 100. This

Co animation sequence is generated by changing the magnitudes of the assigned vectors - -along the strokes forming the fist and reconstructing the 3D surface each time the

Co magnitudes -of the vectors are changed. Each reconstructed 3D surface is a frame of . the animation sequence. A more complicated animation sequence may be formed by | changing not only the magnitudes of the assigned vectors but also their directions.

E “Furthermore, method 100 may be used in other applications, for example, 2D cartoon - shading [35], [36], [37] or geometric error fixing [38], etc. A sketch-based surface modeling software may also be developed using method 100 and this software can be used in various design applications for reconstructing complex 3D surfaces based on user's sketches easily, quickly and accurately.

oo 40 Co

REFERENCES

[1] AutoDesk Maya. ~- http//usa. autodesk.com/adsk/serviet/pc/index?id=13577897&sitelD=123112.

[2] AutoCAD. . hito//usa.autodesk.com/adsk/serviet/oc/index?id=13779270&sitelD=123112. 5 13] SolidWorks. http://www. solidworks.comy/. oo Co Co : : - RZ CATIA. http://www. 3ds.com/products/catia. - SE - [5] L. Olsen, F. F. Samavati, M. C. Sousa, and J. A Jorge. Sketch-based modeling: A survey. Computers & Graphics, 33(1):88—103, Feb 2009. oo [6] S. F. Johnston. 2002. Lumo: illumination for cel animation. In’ NPAR 02:

Proceedings of the 2nd international symposium on Non-photorealistic animation and oo rendering, ACM, New York, : © NY, USA, pp. 45-52. SE - [7] T.-P. Wu, C.-K. Tang, M. Brown, and H.-Y Shum. 2007. Shapepalettes: interactive - | normal transfer visa sketching. In ACM Transactions on Graphics volume 26 Issue 3 07,

Article No.44. oo [8] X.-C. Tai and C. Wu, Augmented Lagrangian method, dual methods and split

Bregman iteration for ROF model, in SSVM '09: Proceedings of the Second

International Conference on Scale Space and Variational Methods in Computer Vision,

Berlin, Heidelberg, 2009, Springer-Verlag, pp. 502-513.

[9] Zhang L., Dugas-Phocion G., and Samson J.-S. 2002. Single-view modeling of free- form scenes. The Journal of Visualization and Computer Animation 13, 225-235.

[10] M. Prasad, and A. Fitzgibbon. 2006. Single view reconstruction of curved surfaces. “In Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision "and Pattern Recognition - Volume 2, 1345 — 1354. -

[11] J. Hahn, C. Wu, and X.-C. Tai. 2010. Augmented Lagrangian method for generalized TV-Stokes models. In UCLA CAM Report 10-30, 2010.

[12] R. C. Zeleznik, K. P. Herndon, and J. F. Hughes. Sketch: an interface for sketching oo 3d scenes. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, pages 163-170, August 1996. oo [13] T. Igarashi, S. Matsuoka, and H. Tanaka. Teddy: a sketching interface for 3d + freeform design. In Proceedings of the 26th annual conference on Computer graphics ha and interactive techniques (Siggraph 1999), pages 409-416, 1999.

oo | 41 oo [14] O. A. Karpenko and J. F. Hughes. SmoothSketch: 3D free-form shapes from - complex sketches. In Proceedings of the 33th annual conference on Computer graphics and interactive techniques (Siggraph 2006), pages 589-598, 2006.

Lo [15] A. Nealen, T. Igarashi, O. Sorkine, and M. Alexa. FiberMesh: designing freeform surfaces with 3D curves. ACM Transactions on Graphics (Siggraph 2007), 26(3):Article

No. 41, July 2007. SE oo ~~ [16] H.-S. Ng, T.-P. Wu, and C.-K. Tang. 2009. Surface-From-Gradients Without oo Discrete. Integrability Enforcement: A Gaussian Kernel Approach. IEEE Transactions on Pattern Analysis and Machine Intelligence. | Co

[17] Google SketchUp. http./sketchup.google.comy/.

[18] R. J. Woodham, “Photometric method for determining surface orientation from

Lo ‘multiple images,” Shape from shading, vol. 27, pp. 513-531, 1989.

[19] A. Herzmann and S. M. Seitz, “Example-based photometric stereo: Shape - ‘reconstruction with general, varying brdfs,” IEEE Transactions on Pattern Analysis and

Machine Intelligence, vol. 27, pp. 1254 — 1264, 2005. : 120] T-P. Wu and C.-K. Tang, “Dense photometric stereo by expectation “maximization,” Computer Vision - ECCV 2006, vol. 3954/2006, pp. 159-172, 2006.

[21] R. Zhang, P.-S. Tsai, J. E. Cryer, and M. Shah, “Shape-from shading:a survey,” - "IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 21, no. 8, pp. 690-706, Aug 1999. : [22] E. Parados and O. Faugeras, Shape from shading. Springer, 2006.

[23] Y. Gingold, T. Igarashi, and D. Zorin, “Structured annotations for 2d-to-3d modeling,” ACM Transactions on Graphics (Siggraph Asia 2009), vol. 28, no. 5, p.

Article No. 148, Dec 2009.

[24] L. Olsen, F. F. Samavati, M. C. Sousa, and J. A. Jorge, “Sketchbased mesh oo augmentation,” in Proceedings of the 2nd eurographics workshop on sketch-based interfaces and modeling (SBIM), 2005.

[25] A. Agrawal, R. Raskar, and R. Chellappa, “What is the Range of Surface

Reconstructions from a Gradient Field?” Computer Vision C ECCV 2006, pp. 578-591, 2006. . ir [26] R. T. Frankot, R. Chellappa, and S. Member, “A Method for enforcing integrability in shape from shading algorithms,” IEEE Transactions on Pattern Analysis and

Machine Intelligence, pp. 118-128, 1987.

Co | 9

[27] N. Petrovic, I. Cohen, B. J. Frey, R. Koetter, and T. S. Huang, “Enforcing . integrability for surface reconstruction algorithms using belief propagation in graphical models,” Computer Vision and Pattern Recognition, IEEE Computer Society - Conference on, vol. 1, p.743, 2001.

[28] T. Simchony, R. Chellappa, and M. Shao, “Direct analytical methods for solving poisson . equations in computer vision problems,” IEEE Transactions on Pattern - 5 Analysis and Machine Intelligence, vol. 12, pp. 435-446, 1990.

[29] J. J. Koenderink, “Pictorial relief,” Philosophical Transactions: Mathematical,

Physical and Engineering Sciences, vol. 256, no. 1740, pp. 1071-1086, May 1998. :

[30] D. Terzopoulos, “The computation of visible-surface representations,” IEEE Trans. : Pattern Anal. Mach. Intell, vol. 10,-no. 4, pp. 417-438, 1988. ~~. [31] L.'l. Rudin, S. Osher, and ‘E. Fatemi, “Nonlinear total variation based noise 3 removal algorithms,” Physica D, vol. 60, pp. 259-268, 1992. oo 132] C. Wu, J. Zhang, and X.-C. Tai, “Augmented lagrangian method for total variation : ‘restoration with non-quadratic fidelity,” UCLA CAM Report 09-82, Tech. Rep., 2009.

[33] F. Cole, A. Golovinskiy, A. Limpaecher, H. S. Barros, A. Finkelstein, T. Funkhouser, and S. Rusinkiewicz, “Where do people draw lines?” in Proceedings of the 35th annual + conference on Computer graphics and interactive techniques (Siggraph 2008), 2008.

[34] H. Winnemoller, A. Orzan, L. Boissieux, and J. Thollot, “Texture design and draping in 2d images,” no. 4, pp. 1091-1099, 2009. : [35] K. ichi Anjyo and K. Hiramitsu, “Stylized highlights for cartoon rendering and animation,” IEEE Computer Graphics and Applications, vol. 23, no. 4, pp. 54-61, 2003.

[36] K. ichi Anjyo, S. Wemler, and W. Baxter, “Tweakable light and shade for cartoon oo animation,” in Proceedings of NPAR 2006, 2006, pp. 133-139.

[37] H. Todo, K. ichi Anjyo, W. Baxter, and T. Igarashi, “Locally controllable stylized . shading,” ACM Transactions on Graphics (Siggraph 2007), vol. 26, no. 3, p. Article No. 17, August 2007.

[38] J. Tao, “Fixing geometric errors on polygonal models: A survey,” Journal of

Computer Science and Technology, vol. 24, pp. 19-29, 2009.

[39] A. Chambolle, “An algorithm for total variational minimization and applications,” J.

Math. Imaging Vis., vol. 20, pp. 89-97, 2004.

[40] T. Chan, G. Golub, and | P. Mulet, “A nonlinear primal-dual method for total " variation-based image restoration,” SIAM J. Sci. Comput., vol. 20, pp. 1964-1977,

“WO 2012/118439 PCT/SG2011/000082 - : 43

[41] T. Goldstein and S. Osher, “The split Bregman method for L1-regularized problems,” SIAM J. img. Sci., vol. 2, pp. 323-343, 2009. . [42] G. Steidl and T. Teuber, “Removing multiplicative noise by douglas-rachford splitting methods,” J. Math. Imaging Vision (accepted). oo Lo

Claims

1. A method for reconstructing a three-dimensional surface in a domain wherein the - three-dimensional surface is reconstructed from a two-dimensional sketch comprising ) at least one stroke with a vector field assigned to the at least one stroke and the - - method comprises the steps of . | | - ~ interpolating the vector field to form a dense vector field; and . ‘reconstructing the surface using the dense vector field; ~ : wherein interpolating the: ‘vector field comprises minimizing a non-linear functional of the vector field with a curl-free constraint on the dense vector field, the ‘non-linear functional of the vector field involving terms defined over the domain and a : term defined over the at least one stroke, and wherein the non-linear functional of the : E ~ vector field is minimized using an augmented Lagrangian method.

2. A method according to claim 1, wherein interpolating the vector field further comprises using a weighted combination of a TV regularization method for preserving a sharpness of the reconstructed surface and a H' regularization method for preserving oo smoothness of the reconstructed surface.

3. A method according to claim 2, wherein interpolating the vector field further : comprises: setting a weight for the sketch based on a desired texture of the surface to be reconstructed; and interpolating the vector field by weighting the combination of the TV regularization method and the H' regularization method using the weight.

4. A method according to claim 3 wherein the step of setting the weight for the sketch : further comprises the sub-steps of: . painting the sketch in a color associated with a variable alpha value; and varying the alpha value to set the weight for the sketch.

5. A method according to claim 2, wherein the method further comprises the step of - dividing the sketch into a plurality of portions and interpolating the vector field further comprises the following sub-steps for each of the plurality of portions:

setting a weight for the portion based on a desired texture of the surface to be reconstructed for the portion; and Co _ interpolating the vector field in the portion by weighting the combination of the - ™v regularization method and H' regularization method using the weight set for the = S50 portion. oo A Co

-

6. A method according to any of the preceding claims, wherein reconstructing. the - 3 surface comprises using a weighted combination of a TV regularization method for : preserving sharpness of the reconstructed surface and a H' regularization method for preserving smoothness of the reconstructed surface.

B

7. A method according to any of the preceding claims, wherein reconstructing the = surface" comprises minimizing a non-linear functional of a height map using an SE “augmented Lagrangian method, the non-linear functional of the height map involving 1 5 terms defined over the domain and a term defined over a further domain in which prior information regarding heights is present. Co

8. A method for reconstructing a three-dimensional surface in a domain wherein the three-dimensional surface is reconstructed from a two-dimensional sketch comprising atleast one stroke and the method comprises the steps of: assigning a plurality of vectors along the at least one stroke to form the vector - field; and oo reconstructing the three-dimensional surface in the domain by a method according to any of the preceding claims. : oo | IE

-

9. A method according to claim 8, wherein prior to assigning the plurality of vectors - along the at least one stroke, the method further comprises the step of classifying the : at least one stroke according to a surface geometry to be reconstructed from the at : least one stroke. h ~

10. A method according to claim 9, wherein the step of classifying the at least one - stroke further comprises classifying the at least one stroke as one of a regular stroke, a 3 ridge stroke, a valley stroke, a jump stroke, a dip stroke and a bump stroke.

11. A method according to any of claims 8 — 10, wherein assigning the plurality of vectors along the at least one stroke further comprises the sub-steps of: assigning a plurality of initial vectors along the at least one stroke wherein the - - plurality of initial vectors are orthogonal to the tangents of the at least one stroke; and : - 5 - ; modifying the plurality of initial vectors to form the plurality of vectors assigned .. oo “along the at least one stroke. CL :

12. A method according to claim 11, wherein the sub-step of modifying the plurality of =~.

i. initial vectors further comprises modifying one or both of the magnitudes and directions ~ of the plurality of initial vectors. >

13. A method according to any of claims 8 — 12, wherein the at least one stroke : indicates that a jump is to be formed in the reconstructed surface, and the directions and magnitudes of the plurality of vectors assigned along the at least one stroke. | respectively represent a direction and magnitude of the jump to be formed. -

14. A method according to any of claims 8 — 12, wherein the at least one stroke ~~ indicates that a slope is to be formed in the reconstructed surface, and the directions and magnitudes of the plurality of vectors assigned along the at least one stroke respectively represent a direction and gradient of the siope to be formed.

15. A method according to claim 9 when dependent on claim 3, wherein the step of setting the weight for the sketch comprises setting a weight for the at least one stroke and the method further comprises the following step prior to the step of interpolating

25. the vector field: : oo | . modifying the weight for the at least one stroke based on the classification of - the at least one stroke. -

16. A method according to any of the preceding claims, further comprising the steps of: modifying the sketch by adding at least one further stroke to the sketch: locating a bounding rectangle around the at least one further stroke in the sketch: and ‘modifying a portion of the reconstructed surface corresponding to the bounding rectangle in the sketch.

17. A method according to claim 16, wherein the at least one stroke further comprises a stroke lying within the bounding rectangle and modifying the portion of the B - reconstructed surface corresponding to the bounding rectangle in the sketch further Lo : comprises the sub-steps of: oo Co : : 5 oo “assigning a further plurality of vectors along the at least one further stroke and the stroke lying within the bounding rectangle; oo interpolating the further plurality of vectors to form a further dense vector field; . and = 3 | ‘modifying the: portion of the reconstructed surface by reconstructing it-from the further dense vector field. 3 C

18. A method for generating an animation sequence from a two-dimensional sketch 3 ‘comprising at least one stroke, the method comprising the step of: - | - creating a plurality of frames for the animation sequence by iteratively reconstructing a three-dimensional surface from the two-dimensional sketch by a ‘method according to any of claims 8 — 17 wherein the assigned plurality of vectors along the at least one stroke are different for each iteration. oo

19. A computer system having a processor arranged to perform a method according to any of claims 1 - 18.

20. A computer program product such as a tangible data storage device, readable by a computer and containing instructions operable by a processor of a computer system to "cause the processor to perform a method according to any of claims 1 — 18.