JP2014106713A - Program, method, and information processor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a matching technique which is capable of suitably and automatically associating closed areas which should be collectively processed over a plurality of images.SOLUTION: A program which associates closed areas with each other among a plurality of images having the closed areas by an information processor 1 includes: a feature quantity extraction unit 21 which extracts feature quantities which are feature quantities of closed areas and are obtained by product sum operation of components at individual points in the closed areas or are obtained from distributions of distances between reference points and individual points in the closed areas; and a closed area association unit 23 which associates respective closed areas of a first image and those of a second image with each other on the basis of feature quantities of respective closed areas of images, which are extracted by the feature quantity extraction unit 21.

Description

  The present invention relates to a program, an information processing method, and an information processing apparatus for associating closed regions between a plurality of images having closed regions.

  Conventionally, in a coloring process of handwritten animation, digital painting is performed in which a line drawing drawn on paper is converted into digital data by a scanner, and each digitalized image is manually colored on a computer.

Japanese Patent No. 28355752

GARCIA TRIGO, P., JOHAN, H., IMAGIRE, T., AND NISHITA, T. "Interactive region matching for 2D animation coloring based on feature's variation." IEICE Transactions (E92-D), 2009, 6, p.1289 -1295. MADEIRA, J. S., STORK, A., AND GROSS, M. H. "An approach to computer-supported cartooning." The Visual Computer12, 1996, 1, p.1-17. QIU, J., SEAH, H. S. AND TIAN, F. "Auto coloring with character registration." In Proceedings of the 2006 internationalconference on Game research and development, CyberGames'06, 2006, p.25-32. SIVIC, J., AND ZISSERMAN, A. "Video Google: A text retrieval approach to object matching in videos." In Proceedings of the International Conference on Computer Vision, 2003, vol. 2, p.1470-1477. SYKORA, D., DINGLIANA, J., AND COLLINS, S. "As-rigid-as-possible image registration for hand-drawn cartoon animations." In Proceedings of the 7th International Symposium on Non-Photorealistic Animation and Rendering, NPAR ' 09, 2009, p. 25-33. SYKORA, D., DINGLIANA, J., AND COLLINS, S. "Lazy-Brush: Flexible painting tool for hand-drawn cartoons." Computer Graphics Forum 28, 2009, 2, p.599-608. SYKORA, D., BEN-CHEN, M., CADIK, M., WHITED, B., AND SIMMONS, M. "Textoons: practical texture mapping for hand-drawn cartoon animations." In Proceedings of the ACM SIGGRAPH / Eurographics Symposium on Non-Photorealistic Animation and Rendering, NPAR'11, 2011, p. 75-84.

  For example, in a line drawing coloring process of handwritten animation, line drawing is generally manually colored frame by frame. Line drawings of successive frames of handwritten animation often include corresponding closed regions that should be colored with the same color, such as the same character being drawn. However, there has conventionally been a problem that there is no method for suitably and automatically deriving the correspondence relationship between the closed regions, and the corresponding regions cannot be colored at once.

In view of the above-described problems, an object of the present invention is to provide a matching technique capable of suitably and automatically associating closed regions to be processed collectively over a plurality of images.

  The present invention employs the following means in order to solve the above-described problems. That is, the present invention is a program for associating the closed regions with each other between a plurality of images having closed regions by a computer, the feature amount of the closed region, and the component of the coordinates of each point in the closed region Extracted in the feature amount extraction step, the feature amount extraction step for extracting the feature amount obtained from the product-sum operation, or the feature amount obtained from the distance distribution of each point in the closed region with respect to the reference point, and the feature amount extraction step And a closed region association step for associating each closed region of the first image with each closed region of the second image based on the feature amount of each closed region of the image. .

  Here, in the present invention, the image may be a black-and-white line drawing, or may be an image having color shading or color. The image may be a raster image or a vector image. The closed region is, for example, a closed region surrounded by a line when the image is a line drawing.

  In the present invention, the feature amount includes not only one scalar value but also a vector value and a combination of a plurality of values. The closed region includes not only the inside of the closed region but also the case where the closed region is located on the boundary line of the closed region. Further, each point in the closed region may be an infinite number of points as well as a finite number of points in the closed region. Each point in the closed region in the raster image is, for example, a finite number of points corresponding to all the pixels in the closed region, or in the closed region where the distances in the x-axis direction and the y-axis direction are constant. A finite number of points. Further, each point of the closed region in the vector image is, for example, all points (infinite points) that can be taken in the closed region. Further, the coordinates are not limited to the orthogonal coordinate system, but also include coordinates in a polar coordinate system. In addition, the feature amount obtained by performing a product-sum operation on the coordinate components of each point in the closed region includes a feature amount obtained by performing an operation other than the product-sum operation. Further, the distance distribution of each point includes both a discrete value distribution and a continuous infinite value distribution.

  In the present invention, the association in the closed region association step includes association in which a part of the closed region included in the first image is not associated with the closed region included in the second image. It is. For example, when the number of closed areas of the first image is different from the number of closed areas of the second image, a correspondence that does not associate a part of the closed areas of the image having a large number of closed areas Includes attachments. In addition, the association in the closed region association step includes association in which a certain closed region included in the first image is associated with a plurality of closed regions included in the second image.

  According to the present invention, in order to associate the closed regions between the images based on the feature amount extracted from each point in each closed region, the closed regions to be processed collectively over a plurality of images are preferably and automatically processed. Can be associated with each other.

  Further, in the feature amount extraction step of the program according to the present invention, the angle of the inertial principal axis of the closed region may be extracted as a feature amount obtained by multiplying the coordinates of each point in the closed region. .

  Here, in the present invention, the angle of the inertial main axis includes that represented by a vector.

  Further, in the feature amount extraction step of the program according to the present invention, a feature amount obtained by multiplying the coordinates of each point in the closed region is calculated from each component of the coordinates of each point in the closed region. The eigenvalues and eigenvectors of the covariance matrix may be extracted.

  Further, in the feature amount extraction step of the program according to the present invention, a centroid in the closed region and a covariance matrix of the coordinates of each point in the closed region are obtained, and the obtained centroid and the covariance matrix are obtained as an expected value and In the two-dimensional normal distribution of the covariance matrix, an ellipse that passes through the coordinates having a predetermined probability density may be obtained, and the feature quantity of the closed region may be extracted from the components constituting the ellipse.

  Further, in the feature amount extraction step of the program according to the present invention, as the feature amount obtained from the distribution of the distance of each point in the closed region with respect to the reference point, the path from the starting point of the moving point moving on the boundary line of the closed region It is also possible to obtain a function of the distance from the reference point of the moving point, using as a variable, and extract a component having a frequency lower than a predetermined value from the result of Fourier transform of the obtained function.

  Here, in the present invention, the Fourier transform includes a discrete Fourier transform.

  Further, in the feature amount extraction step of the program according to the present invention, a plurality of points on the boundary line of the closed region as a feature amount obtained from the distribution of the distance of each point in the closed region with respect to a reference point based on a predetermined reference It is also possible to select and extract a vector whose component is the distance between each selected point and the reference point.

  Further, in the feature amount extraction step of the program according to the present invention, as the feature amount obtained from the distribution of the distance of each point in the closed region with respect to the reference point, polar coordinates of each point in the closed region with the reference point as the origin Alternatively, a bivariate histogram may be extracted with the radius and declination at.

  Also, in the feature amount extraction step of the program according to the present invention, as the feature amount of the closed region, an adjacent region centroid that is a coordinate calculated based on the centroid of each closed region adjacent to the closed region from which the feature amount is to be extracted The position may be further extracted.

  Here, in the present invention, adjoining means that the closed regions share a part of the boundary line.

  The program according to the present invention further includes a cost calculation step of calculating a cost indicating a degree of difference between two closed regions based on the feature amount extracted in the feature amount extraction step, and the closed region The association step may associate each closed region included in the first image with one or less of the closed regions included in the second image, based on the sum of the costs between the corresponding closed regions.

  Here, in the present invention, the association performed by the closed region association unit includes association in which a certain closed region included in the first image is not associated with a closed region included in the second image.

  Further, the present invention may be a program in which such a program is recorded on a recording medium readable by a computer, other devices, machines, or the like. Here, the computer-readable recording medium refers to a recording medium that stores information such as data and programs by electrical, magnetic, optical, mechanical, or chemical action and can be read from the computer or the like. Say.

  Further, the present invention is a method for associating the closed regions between a plurality of images having closed regions, wherein the computer is used to calculate a feature amount of the closed region and a coordinate component of each point of the closed region. A feature amount extraction step for extracting a feature amount obtained by product-sum operation or a feature amount obtained from a distance distribution of each point in the closed region with respect to a reference point; and extracted in the feature amount extraction step, A method of executing a closed region associating step of associating each closed region included in the first image with each closed region included in the second image based on the feature amount of each closed region included in the image. Also good.

  The present invention is also an information processing apparatus for associating the closed regions with each other between a plurality of images having the closed region, wherein the closed region is a feature quantity, and a coordinate component of each point in the closed region is obtained. An extraction unit that extracts a feature amount obtained by product-sum operation or a feature amount obtained from a distance distribution of each point in the closed region with respect to a reference point; and the image extracted by the extraction unit The information processing apparatus may include a closed region association unit that associates each closed region included in the first image with each closed region included in the second image based on the feature amount of each closed region.

  According to one aspect of the present invention, it is possible to provide a matching technique that can appropriately and automatically associate closed regions to be processed collectively over a plurality of images.

It is the schematic which shows the structure of the information processing apparatus with which the program which concerns on Embodiment 1 is performed. It is an example of the some image which the program concerning Embodiment 1 matches. It is a figure which shows the example of the closed area | region which the program concerning Embodiment 1 handles. It is a figure which shows the outline of a function structure of the information processing apparatus with which the program which concerns on Embodiment 1 is performed. It is a figure which shows the example of the inertia principal axis of a closed area | region. It is a figure which shows the example of the representative ellipse which concerns on each closed area | region. It is an image figure which shows the example of matching of the closed area | regions between two images. It is an image figure which shows the example of matching of the closed area | regions between several images. It is a flowchart which shows the flow of the process which matches closed areas between several images. It is a flowchart which shows the flow of the process which matches closed regions between two images. It is a figure which shows the example of the moving point which concerns on a closed area | region. It is a graph which shows the example of the function of the distance of a moving point and a gravity center. It is an image figure which shows the distance of each selected point and a gravity center.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the present embodiment, the plurality of images is implemented as a line drawing of each frame (frame) constituting the animation. The embodiment described below shows an example for carrying out the present invention, and the present invention is not limited to the specific configuration described below. In practicing the present invention, it is preferable to adopt a specific configuration according to the embodiment as appropriate.

Embodiment 1
<Configuration>
FIG. 1 is a schematic diagram illustrating a configuration of an information processing apparatus that executes a program according to the first embodiment.

  The information processing apparatus 1 includes a CPU (Central Processing Unit) 11, a RAM (Random Access Memory) 12, a ROM (Read Only Memory) 13, an auxiliary storage device 14 such as a HDD (Hard Disk Drive), and an external device that connects peripheral devices. A computer provided with an interface 15.

The CPU 11 is a central processing unit, and controls the RAM 12, the auxiliary storage device 14 and the like by processing instructions and data of various programs developed in the RAM 12 and the like. The RAM 12 is a main storage device and is controlled by the CPU 11 to write and read various commands and data. The auxiliary storage device 14 is a non-volatile auxiliary storage device, and is mainly used as a computer power source such as an OS (Operating System) loaded in the RAM 12, various programs such as the program according to the present embodiment, line drawings of animation frames, and the like. The information that you want to keep even if you drop is written and read. The scanner 2 is connected to the external interface 15. The scanner 2 reads a plurality of line drawings 3 drawn on a paper medium. The read line drawing 3 is taken into the information processing apparatus 1 as a digital image and constitutes each frame of the animation.

FIG. 2 is an example of a plurality of images associated with the program according to the first embodiment. FIG. 2 illustrates frames F1, F2, F3, and F4 that are sequentially displayed in the animation as a plurality of images. In the present embodiment, the line drawing of each frame is given as a raster image having a predetermined image size. In the present embodiment, each frame has a coordinate system with the upper left corner as the origin, the x axis in the horizontal right direction, and the y axis in the vertical lower direction. Note that the line drawing of each frame may be given as a vector image. When given as a vector image, the program may handle the given image by converting it to a raster image, or may treat it as a vector image.

  FIG. 3 is a diagram illustrating an example of a closed region handled by the program according to the first embodiment. FIG. 3 shows a closed region R101 (shaded portion), R102 (shaded portion), and R103 (check portion) as examples of the closed region included in the line drawing of the frame F1 in FIG. The closed region included in the line drawing of the frame F1 is not limited to these closed regions, but includes all the portions (white pixels) surrounded by other lines (black pixels). For an image having light and shade, a portion surrounded by pixels or lines having a darkness greater than a predetermined value may be treated as a closed region. In addition, for an image having a chromatic color, a portion surrounded by a black line may be treated as a closed region, and colors other than black may be ignored. In addition, for an image having a chromatic color, a portion surrounded by pixels having a predetermined color may be treated as a closed region.

  FIG. 4 is a diagram illustrating an outline of a functional configuration of the information processing apparatus 1 that executes the program according to the first embodiment. The program according to the present embodiment is read into the RAM 12 and executed by the CPU 11, thereby functioning as a feature amount extraction unit 21, a cost calculation unit 22, and a closed region association unit 23. In the present embodiment, each function of the information processing apparatus 1 is executed by the CPU 11 that is a general-purpose processor. However, part or all of these functions may be executed by one or a plurality of dedicated processors. .

  In the present embodiment, the feature amount extraction unit 21 is a feature amount of the closed region, and is a feature amount obtained by multiply-adding the components of the coordinates of each point in the closed region. Extract the angle. Specifically, the feature quantity extraction unit 21 calculates eigenvalues and eigenvectors of the covariance matrix calculated from the components of the coordinates of all white pixels (corresponding to each point) in the closed region, and calculates the eigenvectors from the calculated eigenvectors. Find the angle of the inertial spindle.

  In the present embodiment, the feature amount extraction unit 21 further uses the eigenvalue of the covariance matrix and the average coordinates of white pixels in the closed region (corresponding to the position of the center of gravity of the closed region) as the feature amount of the closed region. Extract. Further, the feature amount extraction unit 21 further extracts an adjacent region centroid position, which is a coordinate calculated based on the centroid of each closed region adjacent to the closed region from which the feature amount is to be extracted, as the feature amount of the closed region. . That is, the feature amount extraction unit 21 of the present embodiment uses the eigenvalues of the covariance matrix calculated from the components of the inertial principal axis of the closed region and the coordinates of the white pixels in the closed region as the feature amount of the closed region. The average coordinates of white pixels in the area and the barycentric position of the adjacent area are extracted. Details will be described below.

First, the feature quantity extraction unit 21 calculates an average coordinate t _i of all pixels in the closed region i. This average coordinate corresponds to the center of gravity of the closed region. Next, the feature quantity extraction unit 21 obtains a covariance matrix C _i of the coordinates of all the pixels in the closed region i. Here, in obtaining the covariance matrix C _i , the x component and the y component of the coordinates are treated as random variables. The covariance matrix C _i uses N as the total number of pixels in the closed region i, p _m (m = 1, 2,..., N) as the vertical vector indicating the coordinates of each pixel, and the average coordinate t _i as the vertical vector. It is expressed by Formula 1. The symbol T means vector transposition.

数式２より、ｘ座標成分の２乗の和（Σ）、ｙ座標成分の２乗の和、ｘ座標成分とｙ座標成分との積の和が演算されることがわかる。 In the present embodiment, obtaining the covariance matrix C _{i corresponds} to performing a product-sum operation on the coordinate components of each point in the closed region. Covariance matrix C _i, for example, the x component of t _i, a y-component t _ix, and t _iy, x component of p _m, the y component p _mx, as p _my, the equation 2.

From Equation 2, it can be seen that the sum of the squares of the x coordinate components (Σ), the sum of the squares of the y coordinate components, and the sum of the products of the x coordinate components and the y coordinate components are calculated.

Next, the feature quantity extraction unit 21 obtains large and small eigenvalues of the covariance matrix C _i , and sets the large eigenvalue to λ _i ^max and the small eigenvalue to λ _i ^min . Next, the feature extraction unit 21 obtains the eigenvector e _i ^min for the eigenvalue lambda _i eigenvector for ^max e _i ^max, and eigenvalue lambda _i ^min. Next, the feature quantity extraction unit 21 extracts the direction of the eigenvector e _i ^max as the angle of the inertia main axis, which is the feature quantity of the closed region. The feature quantity extraction unit 21 further extracts eigenvalues λ _i ^max , λ _i ^min , and average coordinates t _i as feature quantities.

また、楕円の中心は、重心ｔ_iで定義できる。特徴量抽出部２１が抽出する、慣性主軸の
角度、固有値、及び平均座標は、それぞれ、この楕円の傾き、大きさ、及び重心に対応する。なお、この楕円は、重心ｔ_i及び共分散行列Ｃ_iを、期待値及び共分散行列に持つ２次元正規分布において、所定の確率密度となる座標を通る楕円とみることもできる。 Note that eigenvalues λ _i ^max and λ _i ^min , eigenvectors e _i ^max and e _i ^min , and average coordinates t _i
Thus, an ellipse representing the distribution of coordinates of white pixels in the closed region can be formed. The major axis Li and minor axis Si of the ellipse representing the distribution of the coordinates of the closed region i can be defined by Equation 3.

The center of the ellipse may be defined by the center of gravity t _i. The inertia principal axis angle, eigenvalue, and average coordinate extracted by the feature quantity extraction unit 21 correspond to the inclination, size, and center of gravity of the ellipse, respectively. Note that this ellipse can also be regarded as an ellipse passing through coordinates having a predetermined probability density in the two-dimensional normal distribution having the centroid t _i and the covariance matrix C _i in the expected value and the covariance matrix.

FIG. 5 is a diagram illustrating an example of the inertial spindle in the closed region. In FIG. 5, an inertial principal axis I101 indicates an angle (direction) of the inertial principal axis with respect to the closed region R101. Ellipse E101 is closed region R10
1 is an ellipse representative of the distribution of coordinates of each pixel within 1. The center of gravity C101 is the center of gravity of the closed region R101 and the center of gravity (center) of the ellipse E101. The angle of the inertial main axis I101 is an ellipse E10
It coincides with the direction of the major axis of 1.

  FIG. 6 is a diagram illustrating an example of a representative ellipse relating to each closed region. FIG. 6 shows an ellipse representing the distribution of coordinates within each closed region of the frames F1 and F2. In FIG. 6, the feature quantities (inertial principal axis angle, eigenvalues, and average coordinates) extracted by the feature quantity extraction unit 21 are visually indicated by ellipses.

According to the present embodiment, the covariance matrix C _i is a 2-by-2 matrix, and the eigenvalues and eigenvectors thereof are relatively easy to calculate. Therefore, the feature amount reflecting the entire shape of the closed region is relatively small. It can be extracted with a calculation amount.

In the present embodiment, the feature quantity extraction unit 21 obtains the angle of the inertia main axis of the closed region from the eigenvector e _i ^max of the covariance matrix C _i. The angle θ may be extracted.

Note that the coordinates used when the feature quantity extraction unit 21 obtains the covariance matrix C _i may include the coordinates of the pixels on the boundary line of the closed region. The coordinates used when the feature amount extraction unit 21 obtains the covariance matrix C _i are predetermined for each of the x-axis direction (horizontal direction) and the y-axis direction (vertical direction) among the white pixels in the closed region. You may employ | adopt the coordinate about the thinned white pixel by the space | interval for every pixel. Further, as the coordinates used when the feature quantity extraction unit 21 obtains the covariance matrix C _i , the coordinates of a predetermined number of white pixels in the closed region selected based on random numbers may be adopted.

ここで、Ｏ_iは閉領域ｉに隣接する領域群、すなわち、境界線の一部を共有する領域群を
示す。ただし、Ｏ_iは、背景領域（境界線にフレームの端辺の一部または全部を含む領域
）も含む領域とする。また、α_kは、閉領域ｉの面積である。また、β_kは、閉領域ｉ境界線が閉領域ｋに接する程度を表す、０以上１以下の範囲の割合を示す。また、σ₀は、所
定の係数を示す。また、数式５及び数式６では、閉領域ｋが背景領域である場合、α_k＝
α_i、ｔ_k＝ｔ_i、として、隣接領域重心位置ｎ_iを算出する。なお、本実施形態では、Ｏ_i
が背景領域を含むが、Ｏ_iが背景領域を含まない実施形態を採用してもよい。 Further, in the present embodiment, the feature extraction unit 21 calculates the neighboring area centroid position n _i defined by the equation 5 and equation 6, further extracts it as a feature of the closed region i.

Here, O _i indicates a region group adjacent to the closed region i, that is, a region group sharing a part of the boundary line. However, O _i is a region including a background region (region including a part or all of the edge of the frame on the boundary line). Α _k is the area of the closed region i. Β _k represents a ratio in a range from 0 to 1 that represents the degree to which the closed region i boundary line is in contact with the closed region k. Σ ₀ represents a predetermined coefficient. In Equations 5 and 6, when the closed region k is a background region, α _k =
As α _i , t _k = t _i , the adjacent region centroid position n _i is calculated. In this embodiment, O _i
May include a background area, but O _i may not include a background area.

  There is a high possibility that the adjacent relationship in the closed region is similar between consecutive frames. According to the present embodiment, since the feature amount of the closed region is extracted in consideration of the feature of the adjacent closed region, it is possible to improve the accuracy of associating the closed regions.

In the present embodiment, the cost calculation unit 22 calculates a cost indicating the degree of difference between the two closed regions based on the feature amount. Specifically, the feature quantity extraction unit 21 calculates the cost a _ij between the closed region i and the closed region j using Equations 7 to 11.

In Equation 7, w _angle , w _scale , w _pos , and w _neighbor are predetermined constants for weighting each term. Equation 7, a _ij ^angle indicating the degree of angular difference of the principal axis of the feature extraction unit 21 has extracted (or slope of the differences between the ellipse representing the distribution of coordinates)
A _ij ^scale indicating the degree of eigenvalue difference (or size difference between ellipses representing the distribution of coordinates), a _ij ^pos indicating the degree of centroid position difference, and the degree of difference between adjacent area centroid positions The cost a _ij is defined as a weighted sum of a _ij ^neighbors indicating The cost a _ij is 0 when the feature amount of the closed region i and the feature amount of the closed region j are the same. The centroids t _i and t _j may be calculated as values standardized according to the size of the frame (image size). In this way, it is possible to make a suitable association between frames of different sizes.

The cost a _ij may be defined as a weighted average of a _ij ^angle , a _ij ^scale , and a _ij ^pos without considering a _ij ^neighbor .

  In the present embodiment, the closed region associating unit 23 assigns each closed region included in the line drawing of the first frame to the second based on the total cost calculated based on the feature amount of each closed region included in the line drawing. The frame is associated with a closed area of 1 or less. In the present embodiment, the first frame and the second frame are continuous frames such as frames F1 and F2, for example.

First, the closed region association unit 23 uses a cost a _ij between each closed region i included in the line drawing of the first frame f and each closed region j included in the line drawing of the second frame f + 1 as an element. A cost table relating to area association is calculated. This cost table shows the number of regions of the first frame f as N _f
If the number of areas of the second frame f + 1 is N _{f + 1} , the table becomes N _f rows N _{f + 1} columns. Next, the closed region association unit 23 refers to this cost table, and associates the closed regions with each other so that the sum of the costs between the closed regions to be associated is minimized. Here, the closed region associating unit 23 associates each closed region i with a maximum of one closed region j. Further, when N _f > N _{f + 1} , the closed region association unit 23 associates (N _f −N _{f + 1} ) closed regions among the closed regions of the line drawing of the first frame. Not performed.

  More specifically, the closed region association unit 23 solves a bipartite graph matching problem in which the closed region is a point, the association is a side, and each closed region i and each closed region j is a subset. . The closed region association unit 23 associates the regions with each other by obtaining the maximum matching using the Hungarian method based on the cost.

  FIG. 7 is an image diagram illustrating an example of associating closed regions between two images. FIG. 7 shows the correspondence between the closed regions between the frames F1 and F2. In FIG. 7, the association is expressed by connecting the centroids of the closed regions with line segments. For example, there are correspondences between the closed regions R101 and R201, between R102 and R202, and between R103 and R203.

  FIG. 8 is an image diagram illustrating an example of associating closed regions between a plurality of images. FIG. 8 shows the correspondence between the closed regions between the frames F1 and F2, between the frames F2 and F3, and between the frames F3 and F4. In FIG. 8, the association is expressed by connecting the centroids of the closed regions with line segments as in FIG.

  In FIG. 8, the number of closed regions of the frame F2 is larger than the number of closed regions of the frame F3. Therefore, for example, the closed region association unit 23 does not associate the closed region R202 of the frame F2 with the closed region of the frame F3. This is because, in the line drawing of the frame F2, the closed region R202 constituting the left ear of the drawn character exists, whereas in the line drawing of the frame F3, the left ear of the drawn character is hidden. This means that there is no closed region constituting the left ear, and no closed region to be processed in the same manner as the closed region R202 exists in the frame F3.

  In FIG. 8, the number of closed regions of the frame F3 is smaller than the number of closed regions of the frame F4. Therefore, for example, the closed region association unit 23 does not associate the closed region R404 of the frame F4 with the closed region of the frame F3.

  The program according to the present embodiment can derive a series of associations of closed regions (hereinafter referred to as “chains”) across a plurality of continuous frames from the association by the closed region association unit 23. The closed region association unit 23 associates the closed region R101 of the frame F1 with the closed region R201 of the frame F2, associates the closed region R201 with the closed region R301 of the frame F3, and closes the closed region R301 and the frame F4. R401 is associated. The program according to the present embodiment can derive a chain that associates the closed regions R101, R201, R301, and R401 from these associations.

  According to the present embodiment, for example, when the user specifies the closed region R101 to be colored and its color, the corresponding closed regions R101, R201, R301, and R401 are collectively specified by the above-described chain. It is possible to easily achieve coloring with the selected color, and it is possible to reduce labor for coloring the line drawing as compared with the case of coloring for each closed region.

<Process flow>
The processing flow of the program according to the present embodiment will be described using the flowcharts of FIGS. 9 and 10. Note that the specific contents and order of the processing shown in the flowchart are examples, and it is preferable that the processing contents and order that are suitable for the embodiment are appropriately adopted.

  FIG. 9 is a flowchart showing a flow of processing for associating closed regions between a plurality of images. The flow of this process is started when the user performs an operation of newly taking the line drawing of each frame constituting the animation into the information processing apparatus 1.

  In step S <b> 101, the program according to the present embodiment inputs a plurality of images as line images of continuous frames by capturing the line image 3 of the paper medium as a digital image via the scanner 2.

  In step S102, the program according to the present embodiment detects a closed region included in the line drawing of each frame. The program according to the present embodiment detects a closed region by scanning a pixel (black) that is a boundary between the inside and the outside of the region from each pixel in the line drawing. At this time, the program according to the present embodiment distinguishes between the already detected closed region and the undetected closed region by using a paint algorithm (seed fill or the like).

  Note that pixels other than black, such as blue and red, may be treated as pixels serving as boundaries. Alternatively, the closed region surrounded by blue may be detected as a closed region to be colored with a dark color separately from the closed region surrounded by black. Further, a closed region surrounded by red may be detected as a closed region to be colored with a bright color separately from the closed region surrounded by black. Then, the closed region association unit 23 may associate the closed regions surrounded by red separately detected in this way. In this way, it is possible to accurately associate the closed regions to be colored by batch processing with bright colors. Further, the closed region association unit 23 may associate the closed regions surrounded by blue separately detected in this way. In this way, it is possible to accurately associate the closed regions to be colored by batch processing with dark colors.

In step S103, the feature amount extraction unit 21 determines the feature amount of the closed region (inclination principal axis angle, eigenvalue of the covariance matrix calculated from the components of the coordinates of the white pixel in the closed region, the white pixel in the closed region). The average coordinates and the adjacent area centroid position) are extracted.

  In steps S104 to S107, the closed regions between all the frames are associated with each other. Steps S104 and S107 constitute an all-frame association loop. First, in step S104, the program according to the present embodiment determines whether or not the closed regions between all the frames have been associated with each other. If it is not determined in step S104 that the closed regions between all the frames have been associated with each other, the process proceeds to step S105. If it is determined in step S104 that the closed regions between all the frames have been associated with each other, the process ends. Next, in step S105, the program according to the present embodiment sets the next unprocessed two frames as a processing target. In step S107, the process returns to step S104.

  In step S105, the closed regions are associated with each other between the two frames to be processed. A specific processing flow of step S105 will be described later.

  FIG. 10 is a flowchart showing a flow of processing for associating closed regions between two images. The flow of this process shows the process of step S105 of FIG. 9 in detail.

  In steps S201 to S204, the cost calculation unit 22 calculates a cost table. Steps S201 and S204 constitute a cost table calculation loop. First, in step S201, the cost calculation unit 22 determines whether the cost between the fully closed regions has been calculated. If it is not determined in step S201 that the cost between the fully closed regions has been calculated, the process proceeds to step S202. If it is determined in step S201 that the cost between the fully closed regions has been calculated, the process proceeds to step S205. Next, in step S202, the cost calculation unit 22 sets the next unprocessed closed region as a processing target among a plurality of consecutive frames. In step S204, the process returns to step S201.

  In step S203, the cost calculation unit 22 calculates the cost (cost table) between the two closed regions to be processed based on the feature amount of each closed region.

  In step S205, the closed region association unit 23 refers to the cost table calculated by the cost calculation unit 22, and each closed region between the frames to be processed is minimized so that the total cost between the closed regions is minimized. Associate.

  As described above, according to the present embodiment, it is possible to appropriately and automatically associate closed regions to be processed for coloring and the like over a plurality of frames. Further, by associating the colored closed region with the achromatic closed region, it becomes easy to automatically color the achromatic closed region using the color of the colored closed region. Note that the processing to be performed collectively is not limited to coloring, but may be processing such as movement, deformation, addition of characters and figures, and the like.

  Further, according to the present embodiment, the feature amount obtained by multiplying the components of the coordinates of each point in each closed region, or the feature obtained from the distribution of the distance of each point in the closed region with respect to the reference point. Since the closed regions are associated with each other based on the amount, even if a plurality of images are line drawings without color shading information, suitable association can be performed. Further, according to the present embodiment, the closed regions can be more preferably associated with each other based on the feature amount considering the shape of the entire closed region. In addition, according to the present embodiment, there is a possibility that complicated features of the shape of the closed region, such as a change in unevenness in a direction not parallel to the coordinate axis, can be reflected in the feature amount.

Further, according to the present embodiment, compared to calculating the feature amount based only on the shape of each part when dividing the boundary line into several, based on the feature amount more reflecting the overall shape of the closed region,
Closed areas can be associated with each other. For example, a suitable association can be made even when the unevenness of the boundary line is large. In addition, according to the present embodiment, a plurality of rigid bodies are set in each frame, and compared with the case where the closed region (rigid body) between the frames is associated with the amount of deformation such as rotation or parallel movement of the set rigid bodies. It becomes easy to deal with closed areas where the boundary line changes complicatedly.

  Further, according to the present embodiment, each closed region included in the first image (frame) is associated with one or less closed regions included in the second image (frame) based on the total cost. As compared with the case where the closed regions are associated with each other simply based on the difference between the two closed regions between the images, the closed regions can be associated with each other with high accuracy as a whole.

<< Embodiment 2 >>
In the above-described first embodiment, the feature amount extraction unit 21 extracts the angle of the inertia main axis of the closed region as the feature amount obtained by multiplying the components of the coordinates of each point in the closed region. In the second embodiment to be described below, the feature amount extraction unit 21 uses the distance from the start point of the moving point that moves on the boundary line of the closed region as the feature amount obtained from the distance distribution of each point in the closed region with respect to the reference point. A function of the distance of the moving point from the reference point is obtained, and a component having a frequency lower than a predetermined value is extracted from the result of Fourier transform of the obtained function. The configuration of the information processing apparatus 1 that executes the program according to the present embodiment, the closed region association unit 23 that is a function of the information processing apparatus 1, and the flow of processing are the same as those in the first embodiment.

  In the present embodiment, the center of gravity of the closed region is adopted as the reference point. Further, the Euclidean distance is adopted as the distance. As the starting point of the moving point, a point located in the x-axis direction from the center of gravity and closest to the center of gravity among the points on the boundary line is employed. The moving point moves counterclockwise from the starting point and makes one round on the boundary line.

FIG. 11 is a diagram illustrating an example of a moving point related to the closed region. FIG. 11 shows a moving point M101 for the closed region R101. In FIG. 11, the point P100 is the starting point. First, the feature quantity extraction unit 21 obtains a function of the distance between the moving point M101 and the center of gravity C101 using the distance from the start point P100 as a variable.

FIG. 12 is a graph showing an example of a function of the distance between the moving point and the center of gravity. FIG. 12 shows a graph curve of the function c (l) in which the horizontal axis is the path l from the start point to the moving point, and the vertical axis is the distance between the center of gravity and the moving point. This graph corresponds to the example of the closed region R101 in FIG.

Next, the feature quantity extraction unit 21 performs a Fourier series expansion of the function c (l) to a lower order (for example, the 16th order) than a predetermined level, and uses a coefficient vector having the coefficient of each term described by the Fourier descriptor as the feature quantity. Extract as In other words, a component having a frequency lower than a predetermined value is extracted. Here, the coefficient f _n of the Fourier descriptor of the n-th term in the present embodiment indicates the degree N, the path l from the start point to the moving point, the boundary line length L, and the distance between the center of gravity and the moving point. Using the function c (l) and the imaginary unit i, it is defined by Equation 12.

In the present embodiment, the cost calculation unit 22 calculates a cost indicating a degree of difference between two closed regions based on a coefficient vector having a plurality of coefficients f _n as elements. In the present embodiment, the cost calculation unit 22 calculates the Euclidean distance between the coefficient vectors of the two closed regions as a cost. Note that other methods may be employed as the cost calculation method, for example, calculation is performed by weighting with the order of the coefficient. Further, the cost calculation unit 22 calculates a weighted sum of the cost calculated from the coefficient vector and a _ij ^neighbor indicating the degree of difference between the adjacent region centroid positions described in the first embodiment, and uses the calculated weighted sum as the cost. It may be adopted.

  According to the present embodiment, the boundary line of the closed region can be approximated with a coefficient vector, and the feature quantity that approximates the boundary line can be extracted with higher accuracy as the degree is increased.

<< Embodiment 3 >>
In the third embodiment, the feature amount extraction unit 21 selects a plurality of points on the boundary line of the closed region as a feature amount obtained from the distance distribution of each point in the closed region with respect to the reference point, and selects the selected point. A vector whose component is the distance between each point and the reference point is extracted. The configuration of the information processing apparatus 1 that executes the program according to the present embodiment, the closed region association unit 23 that is a function of the information processing apparatus 1, and the flow of processing are the same as those in the first embodiment.

  In the present embodiment, the center of gravity of the closed region is adopted as the reference point. In addition, the program according to the present embodiment defines a starting point for the closed region and handles a road along the boundary line from the starting point, as in the second embodiment. First, the feature quantity extraction unit 21 is a point on the boundary line when the length of the boundary line is L, and the path from the start point is 0, L / 16, 2L / 16, 3L / 16,. 16 points to be 15L / 16 are selected (corresponding to selecting a plurality of points on the boundary line with a predetermined reference). In other words, the feature amount extraction unit 21 selects points that divide the boundary line into 16 equal parts. Next, the feature quantity extraction unit 21 extracts a vector whose component is the distance between each selected point and the center of gravity as a feature quantity of the closed region.

FIG. 13 is an image diagram showing the distance between each selected point and the center of gravity. FIG. 13 shows an image of the distance between each selected point and the center of gravity corresponding to the example of the closed region R101 of FIG. In FIG. 13, the horizontal axis indicates the distance l from the point to the moving point, and the vertical axis indicates the distance between each selected point and the center of gravity. In FIG. 13, from the points L01 to L16, the path from the start point is 0, L / 16, 2L / 16, 3L / 16,..., 15L / 16, and the path l and the center of gravity of each point. It is a point which shows the relationship of distance. The feature amount extraction unit 21 extracts a vector having a distance from the center of gravity indicated by the ordinates from the points L01 to L16 as an element as a feature amount.

  In the present embodiment, the feature amount extraction unit 21 selects 16 points. However, the feature amount extraction unit 21 may select more points or may select fewer points.

  In the present embodiment, the cost calculation unit 22 calculates the Euclidean distance of the vector extracted as the feature amount as the cost, as in the second embodiment.

<< Embodiment 4 >>
In the fourth embodiment, the feature quantity extraction unit 21 calculates, as the feature quantity obtained from the distribution of the distance of each point in the closed area with respect to the reference point, the radius vector in the polar coordinates of each point in the closed area with the reference point as the origin. A bivariate histogram with the declination as a variable is extracted. The configuration of the information processing apparatus 1 that executes the program according to the present embodiment, the closed region association unit 23 that is a function of the information processing apparatus 1, and the flow of processing are the same as those in the first embodiment.

In the present embodiment, the center of gravity of the closed region is adopted as the reference point. As the bivariate histogram, Shape Context is adopted. The cost calculation unit 22 uses chi-square to calculate the similarity between the histograms that are feature quantities and the cost that indicates the degree of difference between the two closed regions.

<< Other Embodiments >>

そして、特徴量抽出部２１は、この共分散行列Ｃ_iの固有値及び固有ベクトルを算出し、
固有ベクトルから慣性主軸の角度を、閉領域内の各点の座標の成分を積和演算することで得られる特徴量として、抽出してもよい。また、この実施形態では、特徴量抽出部２１は、共分散行列Ｃ_iの固有ベクトルから慣性主軸の角度を求めたが、特徴量抽出部２１は、
閉領域i内の点をｘのｘ成分、ｙ成分をｘ_x、ｙ_y、ｔ_iのｘ成分、ｙ成分をｔ_ix、ｔ_iyとして、数式１４により、慣性主軸の角度θを抽出してもよい。

As another embodiment, the line drawing of each frame may be handled as a vector image. At this time, the feature quantity extraction unit 21 sets the area of the closed region i as A, the point in the closed region i as x, and the center of gravity t _i.
And the covariance matrix C _i may be calculated by Equation 13.

Then, the feature quantity extraction unit 21 calculates eigenvalues and eigenvectors of the covariance matrix C _i ,
You may extract the angle of an inertia principal axis from an eigenvector as a feature-value obtained by multiply-adding the component of the coordinate of each point in a closed region. In this embodiment, the feature quantity extraction unit 21 calculates the angle of the principal axis of inertia from the eigenvector of the covariance matrix C _i , but the feature quantity extraction unit 21
The point θ in the closed region i is x, the y component is x _x , y _y , the x component of t _i , the y component is t _ix , t _iy , and the angle θ of the inertial main axis is extracted using Equation 14 Also good.

  By doing so, it is possible to suitably associate the closed regions between the frames with respect to the vector image. The line drawing of each frame may be handled by converting a vector image into a raster image.

DESCRIPTION OF SYMBOLS 1 Information processing apparatus 2 Scanner 3 Line drawing 21 Feature-value extraction part 22 Cost calculation part 23 Closed area matching part

Claims (10)

A program for associating the closed areas between a plurality of images having closed areas by a computer,
The feature amount of the closed region, which is obtained from a product sum operation of the components of the coordinates of each point in the closed region, or obtained from the distribution of the distance of each point in the closed region with respect to a reference point. A feature amount extraction step for extracting a feature amount to be obtained;
Closed region correspondence that associates each closed region of the first image with each closed region of the second image based on the feature amount of each closed region of the image extracted in the feature amount extraction step. Attaching process,
A program comprising   The program for extracting the angle of the inertial principal axis of the closed region as a feature amount obtained by performing a product-sum operation on the coordinate components of each point in the closed region. In the feature quantity extraction step, a feature quantity obtained by multiply-adding the components of the coordinates of each point in the closed region is calculated from the covariance matrix calculated from the components of the coordinates of each point in the closed region Extract based on the eigenvalues and eigenvectors of
The program according to claim 1 or 2.   In the feature amount extraction step according to claim 1, as a feature amount obtained from a distribution of distances of each point in the closed region with respect to a reference point, a path from a starting point of a moving point moving on a boundary line of the closed region is used as a variable. A program for obtaining a function of a distance from the reference point of the moving point, and extracting a component having a frequency lower than a predetermined value from a result of performing Fourier transform on the obtained function.   In the feature amount extraction step according to claim 1, a plurality of points on a boundary line of the closed region are selected based on a predetermined reference as a feature amount obtained from a distribution of distances of the points in the closed region with respect to a reference point, A program that extracts a vector whose component is the distance between each selected point and a reference point.   In the feature amount extraction step according to claim 1, as a feature amount obtained from a distribution of distances of each point in the closed region with respect to a reference point, a radius vector in a polar coordinate of each point in the closed region with the reference point as an origin. A program that extracts a bivariate histogram with the angle and declination as variables. In the feature amount extraction step, as the feature amount of the closed region, an adjacent region centroid position, which is a coordinate calculated based on the centroid of each closed region adjacent to the closed region of the feature amount extraction target, is further extracted.
The program according to any one of claims 1 to 6. A cost calculation step of calculating a cost indicating a degree of difference between the two closed regions based on the feature amount extracted in the feature amount extraction step;
The closed region association step associates each closed region of the first image with one or less of the closed regions of the second image based on the sum of the costs between the closed regions to be associated.
The program according to any one of claims 1 to 7. A method of associating the closed regions between a plurality of images having closed regions,
By computer
A feature amount of the closed region, which is obtained from a feature amount obtained by multiplying the components of the coordinates of each point of the closed region, or a distribution of distances of the points in the closed region with respect to a reference point A feature extraction process for extracting features;
Closed region correspondence that associates each closed region of the first image with each closed region of the second image based on the feature amount of each closed region of the image extracted in the feature amount extraction step. Attaching process,
How is executed. An information processing apparatus for associating the closed areas between a plurality of images having closed areas,
The feature amount of the closed region, which is obtained from a product sum operation of the components of the coordinates of each point in the closed region, or obtained from the distribution of the distance of each point in the closed region with respect to a reference point. Extracting means for extracting a feature amount to be obtained;
Closed area association means for associating each closed area of the first image with each closed area of the second image based on the feature amount of each closed area of the image extracted by the extraction means. When,
An information processing apparatus comprising:

Images