Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T11/00—2D [Two Dimensional] image generation

Definitions

• the present invention relates to a mesh coding technique.
• Bezier function spline function, or the like. This method achieves high-speed processing and is popular, but is not good at a complicated gradation representation.
• Rendering a resolution-free object containing, a complicated gradation generally adopts a gradient mesh tool available from Adobe Illustrator®.
• a gradient mesh can render a complex object by giving a color and gradient to a mesh to generate a cubic function (patent references 1 and 2) .
• Patent Reference 1 Japanese Patent Laid-Open No. 11-345347
• Patent Reference 2 Japanese Patent Laid-Open No. 10-320585
• the method of subdividing Bezier patches to reduce an approximation error uses a high-order mesh and can faithfully approximate an object boundary.
• the subdivision of a mesh to reduce an approximation error increases the number of meshes and the data amount at a portion where the object color changes complicatedly .
• the method of solving an optimization problem and creating a gradient mesh can provide a resolution-free representation with a smaller number of meshes even at an object boundary and a portion where the object color changes complicatedly.
• the user needs to decide representative points in mesh generation and the number of meshes in mesh control in a processing sequence. Hence, this method cannot be employed readily. If representative points and the number of meshes are not set appropriately, a mesh satisfactory for the image quality, data amount, and calculation time cannot be generated.
• mesh control the number of meshes selected in accordance with the feature of an object influences the image quality, data amount, and calculation time, which is the second problem.
• a larger number of meshes generally improve the image quality, but leads to a larger data amount and longer calculation time.
• Efficient mesh control can be done as follows: a large number of meshes are arranged at a characteristic portion of an object to improve the image quality; and a small number of meshes are arranged at a less characteristic portion to reduce the data amount and calculation time while suppressing degradation of the image quality.
• the present invention has been made to solve the conventional drawbacks, and provides a technique for efficiently, optimally deciding the shape and number of meshes when performing mesh coding for an object having a gradation.
• the present invention provides a technique for setting representative points to minimize the mesh distortion.
• an image processing apparatus comprising: a unit which extracts an area of an object, as an object area, from an image; a division unit which divides the object area into a plurality of meshes and calculates, for each of the respective meshes, position information, gradient information, and color information for each vertex of the mesh; a decision unit which decides unnecessary vertices based on a difference between color information of the respective meshes and color information in areas, in the object area, corresponding to the respective meshes; and a calculation unit which sets, as necessary vertices, vertices other than the unnecessary vertices, and decides position information, gradient information, and color information of the necessary vertices by performing iterative calculation to change the position information, gradient information, and color information of the necessary vertices so as to minimize a difference between color information of meshes defined by the unnecessary vertices and color information of areas, in the object area, corresponding to the meshes defined by the unnecessary
• an image processing method to be performed by an image processing apparatus comprising: a step of extracting an area of an object, as an object area, from an image; a division step of dividing the object area into a plurality of meshes and calculating, for each of the respective meshes, position information, gradient information, and color information for each vertex of the mesh; a decision step of deciding unnecessary vertices based on a difference between color information of of the respective meshes and color information in areas, in the object area, corresponding to the respective meshes; and a calculation step of setting, as necessary vertices, vertices other than the unnecessary vertices, and deciding position information, gradient information, and color information of the necessary vertices by performing iterative calculation to change the position information, gradient information, and color information of the necessary vertices so as to minimize a difference between color information of meshes defined by the unnecessary vertices and color information of areas, in the object area, corresponding
• FIG. 1 is a block diagram exemplifying the functional arrangement of an image processing apparatus according to the first embodiment of the present invention
• FIG. 2 is a flowchart of processing executed by a mesh generation unit 102 using object area data
• Fig. 3 is a view exemplifying an extracted object area
• Fig. 4 is a view for explaining processes in steps S201 and S202;
• Fig. 5 is a view for explaining processes in steps S203 and S204;
• Fig. 6 is a view for explaining processes in steps S205 and S206;
• Fig. 7 is a view for explaining a process in step S207;
• Fig. 8 is a flowchart of processing executed by a mesh control unit 103;
• Fig. 9 is a view exemplifying a parametric surface obtained by executing the processing according to the flowchart shown in Fig. 8 for a parametric surface shown in Fig. 7 in both the u and v directions;
• Fig. 10 is a flowchart of processing executed by an optimization unit 104;
• Fig. 11 is a view showing an energy function
• Fig. 12 is a view exemplifying a parametric surface obtained by performing the processing according to the flowchart shown in Fig. 10 for the parametric surface shown in Fig. 9;
• Fig. 13 is a block diagram exemplifying the hardware configuration of a computer applicable as the image processing apparatus according to the first embodiment of the present invention
• Fig. 14 is a flowchart of processing executed by the mesh generation unit 102;
• Fig. 15 is a view for explaining processes in steps S1401 and S1402;
• Fig. 16 is a view for explaining processes in steps S1403 and S1404.
• Fig. 1 is a block diagram exemplifying the functional arrangement of an image processing apparatus according to the first embodiment.
• the image processing apparatus according to the first embodiment includes an object input unit 101, mesh generation unit 102, mesh control unit 103, optimization unit 104, and coding unit 105.
• the object input unit 101 receives an input image containing at least one object.
• the "object” means a text, object image, and the like.
• the embodiment assumes that the "object” has a color gradation. That is, a purpose of the embodiment is to perform mesh coding for an object having a color gradation.
• the object input unit 101 extracts the area of an object as an object area from the input image.
• Fig. 3 is a view exemplifying an extracted object area.
• Data of the extracted object area contains the coordinate positions, on an input image, of pixels which form a boundary 303 of an object area 301, and pixel value data of pixels which form an inside 302 of the object area 301.
• the pixel value is given by 8 bits for each of R, G, and B color components.
• the types of color components and the number of bits which form one color component are not limited to them.
• the object area may have another data structure as long as the coordinate positions of pixels which form the boundary of an object area and the pixel values of pixels which form the inside of the object area can be derived.
• the object input unit 101 sends data of the extracted object area to the subsequent mesh generation unit 102.
• Fig. 2 is a flowchart of processing executed by the mesh generation unit 102 using object area data.
• step S201 the mesh generation unit 102 calculates a reference position in the object area using the coordinate positions of pixels which form the boundary of the object area.
• the barycentric position (center position) of the object area is attained as the reference position.
• the reference position in the object area is not limited to the center position, and the processing to calculate the center position is not limited to this.
• step S202 the mesh generation unit 102 sets, in the object area, two axes (x- and y-axes) which are perpendicular to each other at the center position calculated in step S201. Then, the mesh generation unit 102 divides the object area into four by the two axes, that is, sets four divided regions.
• Fig. 4 is a view for explaining the processes in steps S201 and S202. In Fig. 4, a center position 401 is one calculated in step S201. Two axes 402 and 403 are perpendicular to each other at the center position 401 and are along the x- and y-axes of an input image, respectively. By setting the two axes 402 and 403, the object area 301 is divided into divided regions 404, 405, 406, and 407.
• Fig. 5 is a view for explaining the processes in steps S203 and S204.
• reference numeral 501 denotes a boundary in the divided region 404; 502, that in the divided region 405; 503, that in the divided region 406; and 504, that in the divided region 407.
• the mesh generation unit 102 sets a representative point on the boundary 501 in the divided region 404 and a representative point on the boundary 503 in the divided region 406 opposite to the divided region 404. At this time, the mesh generation unit 102 decides the positions of the respective representative points to maximize the distance between the representative points. As a result, a representative point 507 is set as the representative point on the boundary 501, and a representative point 509 is set as one on the boundary 503, as shown in Fig. 5. [0047] Similarly, the mesh generation unit 102 sets a representative point on the boundary 502 in the divided region 405 and a representative point on the boundary 504 in the divided region 407 opposite to the divided region 405.
• the mesh generation unit 102 decides the positions of the respective representative points to maximize the distance between the representative points. Consequently, a representative point 508 is set as the representative point on the boundary 502, and a representative point 510 is set as one on the boundary 504, as shown in Fig. 5.
• step S205 the mesh generation unit 102 calculates a cubic Bezier spline curve (parametric curve) connecting adjacent representative points.
• the mesh generation unit 102 performs this processing between the respective representative points, attaining four cubic Bezier spline curves.
• the shape of the boundary of the object area can be approximated using the cubic Bezier spline curves .
• step S206 the mesh generation unit 102 generates Coons patches from the four cubic Bezier spline curves attained in step S205, calculating one parametric surface from the four cubic Bezier spline curves .
• Fig. 6 is a view for explaining the processes in steps S205 and S206.
• a cubic Bezier spline curve 605 connects the representative points 507 and 508.
• a cubic Bezier spline curve 606 connects the representative points 508 and 509.
• a cubic Bezier spline curve 607 connects the representative points 509 and 510.
• a cubic Bezier spline curve 608 connects the representative points 510 and 507.
• the mesh generation unit 102 After calculating the four cubic Bezier spline curves in step S205, the mesh generation unit 102 represents a region surrounded by the four cubic Bezier spline curves as a parametric surface using Coons patches, thereby calculating a parametric surface 609. This processing is a well-known technique, and a description thereof will be omitted.
• step S207 the mesh generation unit 102 divides the parametric surface calculated in step S206 equally in the u and v directions, generating an initial mesh. More specifically, points on the parametric surface are sampled two-dimensionally in the u and v directions, and the parametric surface (object area) is divided into meshes each defined by four sampled points. For each vertex which constructs a mesh, position information representing the position of the vertex, gradient information at the position, and color information at the position are attained upon the division processing.
• Fig. 7 is a view for explaining the process in step S207.
• cubic Bezier spline curves 701 and 703 run in the u direction (i.e., direction defined by the parametric surface)
• cubic Bezier spline curves 702 and 704 run in the v direction (i.e., direction defined by the parametric surface) .
• points on the parametric surface are sampled two- dimensionally in the u and v directions.
• the parametric curve is divided into meshes each defined by four sampled points, generating an initial mesh 705. That is, the parametric surface is divided into a plurality of meshes.
• the parametric surface is expressed by a parameter which takes a value of 0 to 1 in each of the u and v directions. For example, when dividing the parametric surface into 10, "divide equally" means sampling points at respective positions set by changing the parameter in steps of 0.1.
• the division number may be set arbitrarily. In Fig. 7, the division number is set to 30 in both the u and v directions.
• the mesh control unit 103 starts processing according to a flowchart shown in Fig. 8.
• Fig. 8 is a flowchart of processing executed by the mesh control unit 103.
• the processing based on the flowchart shown in Fig. 8 is done in each of the u and v directions. Processes for the u and v directions are the same, and that for the u direction will be exemplified. The following description also applies to processing in the v direction.
• step S801 the mesh control unit 103 refers to vertices calculated in step S207 for each line in the u direction.
• the mesh control unit 103 specifies meshes (meshes aligned in the u direction) defined by vertices forming the referred line (reference line) and vertices forming a line (adjacent line) adjacent to the reference line. For each specified mesh, the mesh control unit 103 calculates the difference (error) between color information of the mesh (color information calculated by interpolation from color information of vertices forming the mesh) and color information in the object area that corresponds to the mesh. The mesh control unit 103 calculates the average value (average error) of errors calculated for the respective meshes, attaining the average error of each line in the u direction.
• step S803 the mesh control unit 103 determines whether a maximum value (maximum average value) among average errors calculated in step S801 for the respective lines in the u direction is larger than a target error (specified value) . If the maximum average value is equal to or smaller than the target error (specified value) , the process advances to step S802.
• step S802 the mesh control unit 103 decides that vertices belonging to a line having a minimum average error (minimum average value) are unnecessary vertices.
• the mesh control unit 103 deletes position information, gradient information, and color information of the unnecessary vertices.
• the process then returns to step S801 to execute the following process for the remaining vertices.
• the processing according to the flowchart shown in Fig. 8 is also done in the v direction, in addition to the u direction. As a result, unnecessary vertices can be decided in both the u and v directions.
• the mesh control unit 103 reconstructs a parametric surface using vertices (vertices (necessary vertices) whose position information, gradient information, and color information are not deleted) , other than unnecessary vertices .
• Fig. 9 is a view exemplifying a parametric surface obtained by executing the processing according to the flowchart shown in Fig. 8 for the parametric surface shown in Fig. 7 in both the u and v directions.
• an adaptive initial mesh 905 is generated by deleting unnecessary vertices from the initial mesh 705.
• the optimization unit 104 starts the following processing. More specifically, the optimization unit 104 optimizes the following pieces of information to minimize an error between a color in a mesh that is calculated from a plurality of meshes on the parametric surface reconstructed in the above-described way, and a color on the input image that corresponds to the position of the mesh:
• Fig. 10 is a flowchart of processing executed by the optimization unit 104.
• step SlOOl the optimization unit 104 subdivides each mesh into triangular patches by dividing it equally in the u and v directions.
• the division number can be set arbitrarily, and in this example, is set to 5 in both the u and v directions.
• step S1002 the optimization unit 104 sets the triangular patch as the evaluation unit of an energy function. The optimization precision is improved by calculating an optimization error not for each pixel but for each triangular patch.
• step S1003 the optimization unit 104 sets a variable i representing the iteration count.
• step S1004 the optimization unit 104 derives a matrix from the energy function.
• Each element (scalar value) of the matrix is a numerical value calculated using all pieces of mesh information.
• Optimum mesh information is unknown, and mesh information having a minimum scalar value needs to be calculated by changing the mesh information to various values. More specifically, a matrix with a size corresponding to unknowns (number of mesh vertices x number of parameters to be changed) needs to be derived and calculated.
• Several methods are proposed to derive the matrix. In this example, the matrix is derived using a known method called a Levenberg- Marquardt method.
• Fig. 11 shows the equation of the energy function, where M is a gradient mesh formed from P object areas, u and v are parametric coordinates, p is an object area, I p is the color of an original image in the area p, m(u,v) is a coordinate point on the gradient mesh, f p (u,v) is the color of the gradient mesh in the area p, s and t be parametric coordinates, and ⁇ is a balance with the first term.
• the matrix is derived by the Levenberg-Marquardt method.
• the unknowns are position information (two- dimensional coordinates) of a mesh (each vertex) , gradient information of the mesh (each vertex) , and color gradient information.
• step S1005 the optimization unit 104 calculates a solution ⁇ u of simultaneous linear equations .
• step S1006 the optimization unit 104 increments by one the variable i representing the iteration count.
• step S1007 If the sum (mesh update amount) of the absolute values of ⁇ u becomes equal to or smaller than a set threshold or i has reached a set maximum iteration count in step S1007, the process ends. If neither of the conditions is satisfied, the process returns to step S1004.
• position information two-dimensional coordinates of a mesh (each vertex)
• gradient information of the mesh each vertex
• color information RGB value
• color gradient information RGB value
• Fig. 12 is a view exemplifying a parametric surface obtained by performing the processing according to the flowchart shown in Fig. 10 for the parametric surface shown in Fig. 9.
• a parametric surface 1201 is generated by updating the two-dimensional coordinates of the adaptive initial mesh 905, the gradient of a mesh (each vertex) , and the color gradient to minimize the energy function.
• the embodiment employs the Levenberg- Marquardt method for iterative calculation, but may use another method such as a Gauss-Newton method.
• the embodiment adopts the preconditioned biconjugate gradient method to calculate simultaneous linear equations in optimization processing, but may use another method such as LU decomposition or Cholesky decomposition.
• the optimization unit 104 executes the above-described processes, codes and manages mesh data (calculation result) obtained by the aforementioned processes.
• the mesh data describes the contents of the following items as a text:
• the coding unit 105 zips and manages mesh data serving as text data describing the contents of these items.
• the coding unit 105 may adopt another coding method including lossy coding.
• the first embodiment can efficiently decide representative points in mesh generation and the shape and number of meshes in mesh control.
• the embodiment can achieve a higher image quality, smaller data amount, and shorter processing time .
• the respective units shown in Fig. 1 may be formed from hardware but may also be implemented as software (computer program) .
• the software is installed in the memory of a general computer such as a PC (Personal Computer) .
• the CPU of the computer executes the installed software, and the computer provides the functions of the above-described image processing apparatus (the functions of the respective units shown in Fig. 1) . That is, the computer is applicable to the foregoing image processing apparatus.
• Fig. 13 is a block diagram exemplifying the hardware configuration of a computer applicable as the image processing apparatus according to the first embodiment .
• a CPU 1301 controls the overall computer using computer programs and data stored in a RAM 1302 and ROM 1303. In addition, the CPU 1301 executes the processes described to be performed by the image processing apparatus. That is, the CPU 1301 executes the processes described to be executed by the respective units shown in Fig. 1.
• the RAM 1302 has an area for temporarily storing computer programs and data loaded from an external storage 1307 and storage medium drive 1308, data received from an external apparatus via an I/F (interface) 1309, and the like. Further, the RAM 1302 has a work area used to execute various processes by the CPU 1301. The RAM 1302 can appropriately provide various kinds of areas.
• the ROM 1303 stores computer set data, boot programs, and the like.
• the user of the computer can operate a keyboard 1304 and mouse 1305 to input a variety of instructions to the CPU 1301.
• a display device 1306 is formed from a CRT, liquid crystal display, or the like, and can display the result of processing by the CPU 1301 as an image, text, or the like.
• the display device 1306 can display an input image as described above, explicitly display an extracted object area, and display a cubic Bezier spline curve, parametric surface, and the like obtained during mesh coding.
• the external storage 1307 is a large- capacity information storage device typified by a hard disk drive. The external storage 1307 saves an OS (Operating System) , computer programs and data for causing the CPU 1301 to implement the functions of the respective units shown in Fig. 1, input image data as described above, information described as known information, and the like.
• OS Operating System
• the storage medium drive 1308 reads out computer programs and data stored in a storage medium such as a CD-ROM or DVD-ROM, and outputs the readout computer programs and data to the external storage 1307 or RAM 1302. Part of information described to be saved in the external storage 1307 may be stored in the storage medium and read by the storage medium drive
• the I/F 1309 is used to connect an external apparatus to the computer.
• an apparatus such as a digital camera for acquiring an input image as described above may be connected to the I/F 1309.
• an input image is acquired from the apparatus via the I/F 1309 into the RAM 1302 or external storage 1307.
• a bus 1310 connects these units.
• the first embodiment has described a method of dividing an object area into four and deciding four representative points in mesh generation.
• the four representative points are desirably arranged at positions containing the object area.
• four representative points are decided by calculating two straight lines each having a maximum distance in opposite regions. Generally when two straight lines have maximum distances and are perpendicular to each other, a generated mesh hardly distorts. However, in the four-division method capable of attaining maximum distances, two straight lines may be arranged almost parallel to each other.
• the second embodiment will explain a method of dividing an object area into eight and deciding four representative points.
• the eight-division method does not guarantee a maximum distance but can prevent the almost parallel arrangement of two straight lines.
• a difference of the second embodiment from the first embodiment will be mainly explained.
• the functional arrangement of an image processing apparatus according to the second embodiment is the same as that in the first embodiment, that is, the arrangement shown in Fig. 1.
• a mesh generation unit 102 executes processing according to a flowchart shown in Fig. 14.
• Fig. 14 is a flowchart of processing executed by the mesh generation unit 102.
• step S1401 the mesh generation unit 102 calculates a reference position in the object area using the coordinate positions of pixels which form the boundary of the object area.
• the barycentric position (center position) of the object area is attained as the reference position.
• the reference position in the object area is not limited to the center position, and the processing to calculate the center position is not limited to this.
• step S1402 the mesh generation unit 102 sets, in the object area, two axes (x- and y-axes) which are perpendicular to each other at the center position calculated in step S1401, and two axes obtained by rotating these two axes (x- and y-axes) by 45° about the center position.
• Fig. 15 is a view for explaining the processes in steps S1401 and S1402.
• a center position 1501 is one calculated in step S1401.
• Two axes 1502 and 1504 are perpendicular to each other at the center position 1501 and are along the x- and y- axes of an input image, respectively.
• Two axes 1503 and 1505 are set by rotating the axes 1502 and 1504 by 45° about the center position 1501 in a direction indicated by an arrow in Fig. 15.
• an object area 301 is divided into divided regions 1506, 1507, 1508, 1509, 1510, 1511, 1512, and 1513.
• the mesh generation unit 102 sets representative points on boundaries in the respective divided regions so that the distances between representative points in opposite divided regions are maximized. Then, the mesh generation unit 102 calculates the distances between representative points in opposite divided regions. The mesh generation unit 102 specifies two representative points exhibiting a maximum distance among the calculated distances. Further, the mesh generation unit 102 specifies two divided regions not adjacent to the two divided regions having the maximum distance, and specifies two representative points set in the two specified divided regions. Accordingly, four representative points can be specified.
• Fig. 16 is a view for explaining the processes in steps S1403 and S1404.
• reference numeral 1601 denotes a boundary in a divided region 1609; 1602, that in a divided region 1610; 1603, that in a divided region 1611; and 1604, that in a divided region 1612.
• Reference numeral 1605 denotes a boundary in a divided region 1613; 1606, that in a divided region 1614; 1607, that in a divided region 1615; and 1608, that in a divided region 1616.
• the mesh generation unit 102 sets a representative point on the boundary in each divided region.
• the mesh generation unit 102 decides the positions of the respective representative points to maximize the distance between the representative points in opposite divided regions.
• the processing for each pair of opposite divided regions is the same as that in the first embodiment except for the number of divided regions.
• the mesh generation unit 102 calculates the distance between the representative points for each pair of opposite divided regions.
• the mesh generation unit 102 specifies two representative points exhibiting a maximum distance among the calculated distances. [0108] In Fig. 16, two representative points having a maximum distance are two representative points 1619 and 1621. Reference numeral 1617 denotes a distance between these two representative points.
• the mesh generation unit 102 specifies the divided regions 1612 and 1616 not adjacent to the two divided regions 1610 and 1614 having the maximum distance.
• the mesh generation unit 102 specifies a representative point 1620 set in the divided region 1612 and a representative point 1622 set in the divided region 1616.
• Reference numeral 1618 denotes a distance between these two representative points .
• step S1405 the mesh generation unit 102 calculates a cubic Bezier spline curve (parametric curve) connecting adjacent representative points. The mesh generation unit 102 performs this processing P2009/067819
• the mesh generation unit 102 also calculates cubic Bezier spline curves connecting these four cubic Bezier spline curves. [0112] The shape of the boundary of the object area can be approximated using the cubic Bezier spline curves .
• step S1406 the mesh generation unit 102 generates Coons patches from the four cubic Bezier spline curves attained in step S1405, calculating one parametric surface from the four cubic Bezier spline curves .
• step S1407 the mesh generation unit 102 divides the parametric surface calculated in step S1406 equally in the u and v directions, generating an initial mesh. More specifically, points on the parametric surface are sampled two-dimensionally in the u and v directions, and the parametric surface (object area) is divided into meshes each defined by four sampled points. For each vertex which constructs a mesh, position information representing the position of the vertex, gradient information at the position, and color information at the position are attained upon the division processing.
• aspects of the present invention can also be realized by a computer of a system or apparatus (or devices such as a CPU or MPU) that reads out and executes a program recorded on a memory device to perform the functions of the above-described embodiment (s) , and by a method, the steps of which are performed by a computer of a system or apparatus by, for example, reading out and executing a program recorded on a memory device to perform the functions of the above-described embodiment (s) .
• the program is provided to the computer for example via a network or from a recording medium of various types serving as the memory device (e.g., computer-readable medium) .

Landscapes

• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• Processing Or Creating Images (AREA)
• Image Generation (AREA)
• Image Processing (AREA)

Priority Applications (1)

Applications Claiming Priority (2)

Publications (1)

Family

ID=42225572

Family Applications (1)

Country Status (3)

Families Citing this family (4)

Citations (1)

Family Cites Families (6)

• 2008
  • 2008-11-25 JP JP2008300183A patent/JP5230372B2/ja active Active
• 2009
  • 2009-10-07 WO PCT/JP2009/067819 patent/WO2010061692A1/en not_active Ceased
  • 2009-10-07 US US13/060,931 patent/US8564592B2/en not_active Expired - Fee Related

Patent Citations (1)

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events