JP2011128924A - Comic image analysis apparatus, program, and search apparatus and method for extracting text from comic image - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a comic image analysis apparatus that extracts described text by analyzing comic images on an image basis. <P>SOLUTION: The comic image analysis apparatus includes: an objective feature point extraction means for extracting many feature points from a comic image to be analyzed, a positional clustering means for clustering the many feature points according to the density of distribution on the image, a subregional image extraction means for extracting a subregional image on the image by a frame surrounding the many feature points contained in each positional cluster, a subregion vector calculation means for calculating a feature vector of the subregional image, a subregional image classification means for comparing the feature vector of the subregional image with a feature vector of character regions obtained from learning data to classify it according to whether it is a character region or not, and a text extraction means for extracting text from a subregional image determined to be a character region. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an image analysis technique for extracting text from an image. Further, the present invention relates to an image search technique for searching for the image by a keyword.

  In recent years, many sites for distributing and selling electronic books via the Internet have been established. In such a site, the search function is enhanced, and an electronic book can be searched based on various elements. Electronic books are not limited to books such as novels based on text, but also include books such as manga based on images. Meta information (title, author name, genre, etc.) is added to these electronic books, and the user can also search the meta information as a search element. For example, when an electronic book is based on a sentence, the text in the sentence can be searched by a keyword.

  On the other hand, as a different technique, conventionally, there is a technique for extracting a character region from an image (see, for example, Patent Documents 1 to 3). According to this technique, for example, an image in which characters are printed on paper can be used as an input, and a region where the characters appear can be automatically extracted. In order to detect a character, the edge characteristics of the font of the character, the interval between rectangular regions surrounding the outline, and the like are taken into consideration.

JP-A-8-293003 JP 2005-275854 A JP 2009-130899 A

D. Lowe, `` Distinctive image features fromscale-invariant keypoints '', IEEE Trans. Pattern Analysis Machine Intelligence, 20: 91-110, 2004. M. Ester, H.-P. Kriegel, J. Sander, and X.Xu, `` A density-based algorithm for discovering clusters in large spatial databases with noise '', in Proceedings from 2nd International Conference on Knowledge Discovery and Data Mining, 1996, pp. 226-230. Hironobu Fujiyoshi, “Gradient-based feature extraction”, Chubu University Faculty of Engineering, Department of Information Engineering, [online], [December 3, 2009 search], Internet <URL: http://ci.nii.ac.jp/naid / 110006423317 /> Daisuke Ishii, Satoshi Kawamura, Hiroshi Watanabe, Graduate School of International Information and Communication Studies, Waseda University, "A Study on Comic's Frame Division Processing", IEICE Transactions D Vol.J90-D No.7 pp.1667-1670, 2007, Image Encoding / Video Media Processing Letter Special Feature, [online], [Search December 9, 2009], Internet <URL: http://www.ams.giti.waseda.ac.jp/pdf- files / j90-d_7_1667.pdf>

  Among electronic books, comic content is important content desired by many users. However, since manga content is based on images, it can generally be searched only in units of meta information. This is because, in the manga image, text is described in a balloon, and these texts are also recognized as part of the image. Therefore, for example, a manga image cannot be searched using a specific line in the manga image as a keyword.

  FIG. 1 is an example of a manga image.

  According to FIG. 1, a manga image is often divided into frame image units, and text such as dialogue is included in a balloon of the image. In addition, the manga image is generally monochrome, and not only the image but also the text itself is often drawn with a hand-based pen or brush. Therefore, unlike an image such as a landscape photograph, a manga image has a characteristic that it has many edges as a whole.

  On the other hand, according to the techniques described in, for example, Patent Documents 1 to 3, an attempt is made to extract a character region from an image by using an edge feature of a character font. Therefore, it is extremely difficult to extract a character area from a character string drawn based on manpower.

  Therefore, the present invention provides a manga image analysis device, a program, a search device, and a method for appropriately identifying a character area by analyzing a manga image imagewise and extracting text described therein. Objective.

According to the present invention, there is provided a manga image analyzing apparatus for extracting text described in a manga image,
Feature point extraction means for extracting a large number of feature points from the manga image to be analyzed;
Positional clustering means for clustering a large number of feature points based on the distribution density on the image;
Sub-region image extraction means for extracting a sub-region image on an image from an outer frame surrounding a large number of feature points included in the cluster for each positional cluster;
Sub-region vector calculation means for calculating a feature vector in the sub-region image from all feature points included in the sub-region image;
Sub-region image classification means for comparing the feature vector of the sub-region image with the feature vector of the character region obtained from the learning data, and classifying according to whether or not it is a character region;
It has a text extraction means for extracting text from a sub-region image determined as a character region.

According to the manga image analysis apparatus of the present invention,
Learning feature point extracting means for extracting a large number of feature points from the learning image;
Elemental clustering means for clustering a large number of feature points into k learning clusters based on the feature values;
Learning cluster vector calculation means for calculating a feature vector in the cluster from all feature points included in the cluster for each elemental cluster;
For each elemental cluster, the image classification learning means further learning whether or not the feature point included in the cluster is a character region,
The learning data used by the sub-region image classification unit is preferably learning data learned by the image classification learning unit.

According to the manga image analysis apparatus of the present invention,
Elemental clustering means classify into k learning clusters by k-means clustering,
The sub-region vector calculating means and the learning cluster vector calculating means calculate k-dimensional feature vectors corresponding to k learning clusters,
It is also preferable that the image classification learning unit generates learning data of a k-dimensional feature vector determined as a character region.

  According to the manga image analyzing apparatus of the present invention, it is also preferable that the target feature point extracting unit or the learning feature point extracting unit detect the feature points by SIFT (Scale-Invariant Feature Transform).

  According to the manga image analysis apparatus of the present invention, it is also preferable that the image classification learning means uses a support vector machine.

  According to the manga image analysis apparatus of the present invention, the text extraction means is OCR (Optical Character Recognition) that specifies a character by collating a sub-region image determined to be a character region with a pre-stored character pattern. Is also preferable.

According to the manga image analysis apparatus of the present invention,
It further has a frame image extraction means for dividing the manga image to be analyzed into frame image units surrounded by straight lines on the image,
It is also preferable that the feature point extraction unit for object extracts a large number of feature points for each frame image.

According to the present invention, there is provided a manga image search device including all functions in the above-described manga image analysis device,
Index storage means for storing the text extracted by the text extraction means as an index in association with the identification information of the manga image;
A search keyword input means for inputting a search keyword;
Manga image search means for searching for identification information of a manga image including an index that matches the search keyword using the index storage means;
The apparatus further comprises search result output means for outputting identification information of the searched manga image.

According to the manga image retrieval apparatus of the present invention,
The text extraction means is an OCR that specifies a character by collating a sub-region image determined to be a character area with a character pattern stored in advance, and outputs the character size to the index storage means by the OCR.
The index storage means stores the priority for each text in association with each other. The text having a larger character size output from the text extraction means associates a higher priority,
It is also preferable that the manga search means outputs the search result with higher priority as the text stored in the index storage means has a higher priority.

According to the present invention, there is provided a program for causing a computer mounted on an apparatus for extracting text described in a manga image to function.
Feature point extraction means for extracting a large number of feature points from the manga image to be analyzed;
Positional clustering means for clustering a large number of feature points based on the distribution density on the image;
Sub-region image extraction means for extracting a sub-region image on an image from an outer frame surrounding a large number of feature points included in the cluster for each positional cluster;
Sub-region vector calculation means for calculating a feature vector in the sub-region image from all feature points included in the sub-region image;
Sub-region image classification means for comparing the feature vector of the sub-region image with the feature vector of the character region obtained from the learning data, and classifying according to whether or not it is a character region;
A computer is made to function as a text extraction means for extracting text from a sub-region image determined to be a character region.

According to the present invention, there is provided a manga image analysis method in an apparatus for extracting text described in a manga image,
A first step of extracting a large number of feature points from the manga image to be analyzed;
A second step of clustering a number of feature points based on the distribution density on the image;
For each positional cluster, a third step of extracting a sub-region image on the image from an outer frame surrounding a number of feature points included in the cluster;
A fourth step of calculating a feature vector in the sub-region image from all feature points included in the sub-region image;
A fifth step of comparing the feature vector of the sub-region image with the feature vector of the character region obtained from the learning data, and classifying the feature vector according to whether or not it is a character region;
And a sixth step of extracting text from the sub-region image determined as the character region.

  According to the manga image analysis device, program, search device, and method of the present invention, by characterically analyzing a manga image, a character region can be appropriately identified and text described therein can be extracted. In addition, by associating the extracted text with a manga image as an index, a manga image can be searched by a keyword.

It is an example of a manga image. It is a basic functional block diagram of the manga image analysis apparatus in this invention. It is an image figure showing the feature point of the image based on progress of manga image analysis. It is a functional block diagram of the learning process part in the manga image analysis apparatus of this invention. It is an image figure showing the feature point of the image based on progress of learning processing. It is a functional block diagram of the manga image search apparatus in this invention. It is a system block diagram provided with the manga search apparatus of this invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 2 is a basic functional configuration diagram of the manga image analyzing apparatus according to the present invention. FIG. 3 is an image diagram showing image feature points based on the progress of manga image analysis.

  The manga image analyzing apparatus 1 extracts text described in a manga image. For this purpose, for example, by associating the extracted text with the manga image as an index, the manga image can be searched by a keyword.

  As shown in FIG. 2, the manga image analysis apparatus 1 includes a manga image storage unit 10 and a manga image analysis unit 11.

  The manga image storage unit 10 stores manga images to be analyzed, and outputs the manga images to the manga image analysis unit 11. The manga image to be analyzed is, for example, the image as shown in FIG.

  The manga image analysis unit 11 extracts text described in the manga image to be analyzed, and includes a frame image extraction unit 111, a target feature point extraction unit 112, a positional clustering unit 113, and a sub-region. The image extracting unit 114 includes a sub region vector calculating unit 115, a sub region image classifying unit 116, and a text extracting unit 117. These functional configurations are realized by executing a manga image analysis program that causes a computer installed in the apparatus to function. Further, the processing flow comprising these functional configurations is understood as a manga image analysis method in the apparatus.

  The frame image extraction unit 111 is provided when a manga image is analyzed on a frame image basis. The frame image extraction unit 111 classifies the manga image to be analyzed into frame image units surrounded by straight lines on the image. FIG. 3A is an image diagram for specifying a frame image. In order to divide into frame images, there is a technique in which a dividing line candidate is detected by straight line detection using a band, and a dividing line is determined by dividing line matching inspection (see, for example, Non-Patent Document 4). According to this technique, the dividing line candidate detects a detection line pixel having a width of 1 pixel, and if the detection line angle is within ± 45 ° with respect to the horizontal axis, two detection lines adjacent in the vertical axis direction, Otherwise, a detection line adjacent in the horizontal axis direction is defined as a “detection band”. These detection bands are divided into frame images by density gradient direction inspection and frame inside / outside inspection. The divided frame image units are output to the positional clustering unit 113 as manga images to be analyzed.

  In addition, when the manga image is divided into frame images in advance, the frame image extraction unit 111 is not necessarily provided. This is because, for example, in the case of a manga image distributed for a mobile phone, the frame image is preliminarily divided manually. The frame image extraction unit 111 is optional only, and is drawn with a broken line in FIG.

The target feature point extraction unit 112 extracts a large number of feature points from the manga image to be analyzed. FIG. 3B is an image diagram showing feature points detected from a manga image. A “feature point” is a point representing a visual feature, and can be detected using, for example, SIFT (Scale-Invariant Feature Transform) (see, for example, Non-Patent Document 1 or 3). SIFT is a technique for describing a feature quantity that is invariant to scale change and rotation of an image by analyzing the image structure using a scale space. According to SIFT, the following two steps are required to detect feature points.
(S1) A key point and a scale are determined by an extreme value search of the scale space.
(S2) The key points determined are narrowed down to stable key points according to the main curvature and contrast.
A large number of feature points extracted in this way are output to the positional clustering unit 113.

  The positional clustering unit 113 clusters a large number of feature points based on the distribution density on the image. The clustering here is not clustering based on the features of the feature points but clustering based on the “positions” of the feature points on the image. For example, by applying the DBSCAN algorithm (see, for example, Non-Patent Document 2), feature points distributed on the image at a high density are extracted as clusters.

  For each positional cluster, the sub-region image extraction unit 114 extracts a sub-region image on the image from an outer frame (for example, a circumscribed polygon) surrounding many feature points included in the cluster. FIG. 3C is an image diagram showing sub-regions detected using feature points.

The sub-region vector calculation unit 115 calculates a feature vector in the sub-region image from all feature points included in the sub-region image. Here, the feature vector represents a k-dimensional vector (v ₁ , v ₂ ,..., V _k ) corresponding to k learning clusters. A feature amount of one feature point included in the sub-region image is represented as an index with respect to the feature amount of each learning cluster. Here, the k learning clusters may be generated in advance or may be described later with reference to FIG.

  The sub-region image classification unit 116 compares the feature vector of the sub-region image with the feature vector of the character region obtained from the learning data, and classifies it according to whether or not it is a character region. Here, “learning data” is a feature vector in an area image in which a character area is specified in advance. This learning data is not limited to a “positive example” that is a character area, but may be a “negative example” that is not a character area. The learning data may be generated in advance or may be described later with reference to FIG.

  The text extraction unit 117 extracts text from the sub-region image determined as the character region. The text extraction unit 117 may use, for example, an existing OCR (Optical Character Recognition) that identifies a character by matching a sub-region image determined as a character region with a previously stored character pattern. .

  FIG. 4 is a functional configuration diagram of the learning processing unit in the manga image analyzing apparatus of the present invention. FIG. 5 is an image diagram showing image feature points based on the progress of the learning process.

  According to the manga image analysis apparatus 1 of FIG. 4, in addition to the functional component of FIG. 2, a learning feature point extraction unit 121, an elemental clustering unit 122, a learning cluster vector calculation unit 123, and image classification learning And a portion 124. These functional configurations are also realized by executing a program that causes a computer mounted on the apparatus to function.

  The learning feature point extraction unit 121 extracts a large number of feature points from the learning image. FIG. 5A is an image diagram showing feature points detected from the learning image. Similar to the target feature point extraction unit 112 described above, detection can be performed using, for example, SIFT. The extracted many feature points are output to the elemental clustering unit 122.

  The elemental clustering unit 122 clusters a large number of feature points into k learning clusters based on the feature values. Here, the k-means method may be used for clustering. FIG. 5B is an image diagram showing k-means clustering of feature points. Then, for each of the k learning clusters, a representative feature value of the cluster is calculated. For example, for each cluster consisting of a large number of feature values, the center of the feature values may be regarded as the representative feature value.

The learning cluster vector calculation unit 123 calculates, for each elemental cluster, feature vectors (v ₁ , v ₂ ,..., V _k ) in the cluster from all feature points included in the cluster. The learning cluster vector calculation unit 123 calculates k-dimensional vectors corresponding to k learning clusters, similarly to the sub-region vector calculation unit 114 described above.

  The image classification learning unit 124 learns for each elemental cluster whether or not the feature point included in the cluster is a character region. Specifically, a k-dimensional vector is calculated for each learning sub-image for the learning image. Specifically, the distance between the feature points distributed in each sub-region image and the representative element points of k learning clusters obtained by the elemental clustering unit 122 is calculated, and the cluster to which each feature point belongs is determined. . FIG. 5C is an image diagram showing the correspondence of the vector of the sub-region image to each element clustering. As a result, it is possible to generate a k-dimensional vector whose feature value is the number (or ratio) of feature points belonging to each cluster.

  The image classification learning unit 124 may use a support vector machine. The support vector machine is an identification algorithm using supervised learning, and is applied to pattern recognition. The support vector machine constitutes two classes of pattern discriminators using linear input elements, and learns the parameters of the linear input elements by calculating the distance from each feature value from the learning sample. Specifically, learning data for image classification is generated by setting a vector of a sub-region designated as a character region in the learning image as a “positive example” and other vectors as “negative examples”.

Accordingly, the sub-region vector calculation unit 115 of the manga image analysis unit 11 generates k-dimensional vectors (v ₁ , v ₂ ,..., V _k ) corresponding to k learning clusters based on the image classification learning unit 124. Can be used. Further, the sub-region image classification unit 116 of the manga image analysis unit 11 uses the learning data for image classification generated by the image classification learning unit 124 to determine whether the sub-region image extracted from the manga image is a character region. Classify.

  FIG. 6 is a functional configuration diagram of the search processing unit in the manga image analysis apparatus of the present invention.

  FIG. 6 shows the manga image search apparatus 2 including all the functions in the manga image analysis apparatus of FIG. 2 or FIG. The manga image search device 2 further includes an index storage unit 13, an interface unit 14, a search keyword input unit 131, a manga image search unit 132, and a search result output unit 133. The search keyword input unit 131, the manga image search unit 132, and the search result output unit 133 are realized by executing a program that causes a computer installed in the apparatus to function.

  The index storage unit 13 stores the text extracted by the manga image analysis unit 11 as an index in association with the identification information of the manga image. When the comic image is divided into frame images, an index is associated with the identification information of the frame image. The identification information of the frame image is specified by, for example, the frame number or coordinates in the image in addition to the identification information of the manga image.

  The text may be directly associated with the identification number of the manga image, or may be associated with the morpheme unit by morphological analysis.

  The interface unit 14 may be a user interface or a communication interface. In the case of a user interface, for example, a search keyword may be input using a keyboard, and the search result (identification number or manga image itself) may be displayed on a display. In the case of the communication interface, the search keyword is received from the terminal via the network, and the search result is returned to the terminal.

  The search keyword input unit 131 inputs a search keyword (query) from the interface unit 14 and outputs the search keyword to the manga image search unit 132.

  The manga image search unit 132 uses the index storage unit 13 to search for identification information of a manga image including an index that matches (or resembles) the input search keyword. The search keyword and the index whose edit distance is equal to or smaller than a certain distance may be searched as being similar. The search result is output to the search result output unit 133.

  The search result output unit 133 outputs the manga image identification information or the manga image itself that is the search result to the interface unit 14.

  Here, as another embodiment, it is also preferable that the text extraction unit 117 of the manga image analysis unit 11 outputs the character size to the index storage unit 13 by OCR. As a result, the index storage unit 13 can store the priority in association with each text. Here, a higher priority is associated with a text having a larger character size.

  Further, the manga search unit 132 preferentially outputs the search result as the higher priority text stored in the index storage unit 13. As a result, a text with a larger character size in a manga image can be searched as a higher priority index.

  FIG. 7 is a system configuration diagram including the manga search apparatus of the present invention.

  According to FIG. 7, the manga image search device 2 is connected to the Internet and can communicate with the manga image database 3, the mobile phone 4 and the terminal 5. The manga image database 3 is a database storing manga images, and transmits manga content (manga images) to the manga image search device 2. The manga image search device 2 can generate an index for the manga content. On the other hand, the mobile phone 4 and the terminal 5 are connected to the Internet via the access network, and transmit a search keyword corresponding to the user's operation to the manga image search device 2. On the other hand, the manga image search device 2 returns the search result to the mobile phone 4 and the terminal 5.

  As described above in detail, according to the manga image analysis device, program, search device, and method of the present invention, a character region is appropriately identified by image analysis of a manga image and described therein. Text can be extracted. In addition, by associating the extracted text with a manga image as an index, a manga image can be searched by a keyword.

  Various changes, modifications, and omissions of the above-described various embodiments of the present invention can be easily made by those skilled in the art. The above description is merely an example, and is not intended to be restrictive. The invention is limited only as defined in the following claims and the equivalents thereto.

DESCRIPTION OF SYMBOLS 1 Manga image analysis apparatus 10 Manga image storage part 111 Frame image extraction part 112 Target feature point extraction part 113 Positional clustering part 114 Sub area image extraction part 115 Sub area vector calculation part 116 Sub area image classification part 117 Text extraction part 121 Feature point extraction unit for learning 122 Elemental clustering unit 123 Cluster vector calculation unit for learning 124 Image classification learning unit 13 Index storage unit 131 Search keyword input unit 132 Manga image search unit 133 Search result output unit 14 Interface unit 2 Manga image search device 3 Manga image database 4 Mobile phone 5 Terminal

Claims (11)

A manga image analyzer that extracts text described in a manga image,
Feature point extraction means for extracting a large number of feature points from the manga image to be analyzed;
Positional clustering means for clustering the multiple feature points based on distribution density on the image;
Sub-region image extraction means for extracting a sub-region image on an image from an outer frame surrounding a large number of feature points included in the cluster for each positional cluster;
Sub-region vector calculation means for calculating a feature vector in the sub-region image from all feature points included in the sub-region image;
Sub-region image classification means for comparing the feature vector of the sub-region image with the feature vector of the character region obtained from the learning data and classifying according to whether or not it is a character region;
A manga image analyzing apparatus, comprising: text extracting means for extracting text from the sub-region image determined to be a character region. Learning feature point extracting means for extracting a large number of feature points from the learning image;
Elemental clustering means for clustering the multiple feature points into k learning clusters based on the feature values;
Learning cluster vector calculation means for calculating a feature vector in the cluster from all feature points included in the cluster for each elemental cluster;
For each elemental cluster, the image classification learning means further learning whether or not the feature point included in the cluster is a character region,
The manga image analysis apparatus according to claim 1, wherein the learning data used by the sub-region image classification unit is learning data learned by the image classification learning unit. The elemental clustering means classifies into k learning clusters by k-means clustering,
The sub-region vector calculating means and the learning cluster vector calculating means calculate k-dimensional feature vectors corresponding to k learning clusters,
The manga image analysis apparatus according to claim 1, wherein the image classification learning unit generates learning data of a k-dimensional feature vector determined to be a character region. The target feature point extracting unit or the learning feature point extracting unit converts the feature points into SIFT (Scale-Invariant
The manga image analysis apparatus according to any one of claims 1 to 3, wherein the manga image analysis apparatus is detected by Feature Transform.   The manga image analysis apparatus according to claim 1, wherein the image classification learning unit uses a support vector machine.   6. The text extracting means is an OCR (Optical Character Recognition) that specifies a character by collating the sub-region image determined to be a character region with a pre-stored character pattern. The manga image analysis apparatus according to any one of the above. It further has a frame image extraction means for dividing the manga image to be analyzed into frame image units surrounded by straight lines on the image,
The manga image analyzing apparatus according to claim 1, wherein the target feature point extracting unit extracts a large number of feature points for each frame image. A manga image search device including all functions in the manga image analysis device according to any one of claims 1 to 7,
Index storage means for storing the text extracted by the text extraction means as an index in association with identification information of the manga image;
A search keyword input means for inputting a search keyword;
Manga image search means for searching for identification information of a manga image including an index that matches the search keyword using the index storage means;
A manga image search device further comprising search result output means for outputting identification information of a searched manga image. The text extraction means is an OCR that specifies a character by collating the sub-region image determined to be a character area with a pre-stored character pattern, and outputs the character size to the index storage means by the OCR And
The index storage means stores a priority in association with each text, the text having a larger character size output from the text extraction means associates a higher priority,
9. The manga image search apparatus according to claim 8, wherein the manga search means outputs a search result preferentially for a text having a higher priority stored in the index storage means. A program that allows a computer installed in a device that extracts text described in a manga image to function.
Feature point extraction means for extracting a large number of feature points from the manga image to be analyzed;
Positional clustering means for clustering the multiple feature points based on distribution density on the image;
Sub-region image extraction means for extracting a sub-region image on an image from an outer frame surrounding a large number of feature points included in the cluster for each positional cluster;
Sub-region vector calculation means for calculating a feature vector in the sub-region image from all feature points included in the sub-region image;
Sub-region image classification means for comparing the feature vector of the sub-region image with the feature vector of the character region obtained from the learning data and classifying according to whether or not it is a character region;
A manga image analysis program that causes a computer to function as text extraction means for extracting text from the sub-region image determined to be a character region. A method for analyzing a manga image in an apparatus for extracting text described in a manga image,
A first step of extracting a large number of feature points from the manga image to be analyzed;
A second step of clustering the multiple feature points based on a distribution density on the image;
For each positional cluster, a third step of extracting a sub-region image on the image from an outer frame surrounding a number of feature points included in the cluster;
A fourth step of calculating a feature vector in the sub-region image from all feature points included in the sub-region image;
Comparing the feature vector of the sub-region image with the feature vector of the character region obtained from the learning data, and classifying the feature vector according to whether or not it is a character region;
And a sixth step of extracting text from the sub-region image determined to be a character region.

Images