JP5480008B2 - Summary manga image generation apparatus, program and method for generating manga content summary - Google Patents

Description

  The present invention relates to an image analysis technique for a manga image.

  In recent years, many sites for distributing and selling electronic books via the Internet have been established. In such a site, the search function is enhanced, and an electronic book can be searched based on various elements. Electronic books are not limited to books such as novels based on text, but also include books such as manga based on images. Meta information (title, author name, genre, etc.) is added to these electronic books, and the user can also search the meta information as a search element. For example, when an electronic book is based on a sentence, the text in the sentence can be searched by a keyword.

  Users often read “summary explanations”, which are short summaries of the entire contents, in order to find more books to read from the searched electronic books. Some sites have a “browsing function” that reveals. Specifically, the “browsing function” clearly indicates to the user only images for the first few pages of each book. Thereby, the user can review a part of the contents of each book.

Daisuke Ishii, Satoshi Kawamura, Yutaka Watanabe, "A Study on (Letter) Comic's Frame-Splitting", IEICE Transactions, Vol. J90-D, No.7, pp.1667-1670, Jul. 2007. X. Hou, L. Zhang, “Saliency detection: aspectral residual approach”, Proc. Of CVPR’07. L. Itti, C. Koch, E. Niebur, `` A Model of Saliency-Based Visual Attention for Rapid Scene Analysis, IEEE Transactions on Pattern Analysis and Machine Intelligence '', Vol. 20, No. 11, pp.1254-1259, Nov1998. Q. Guo, K. Kato, N. Sato, Y. Hoshino, “AnAlgorithm for Extracting Text Strings from Comic Strips”, ACM SIGGRAPH 2006.

  Among electronic books, comic content is important content desired by many users. However, since manga content is based on images, it cannot be searched by text based on content other than meta information.

  On the other hand, the user can easily and intuitively understand the contents of each episode by using the “stand-by reading function” for the manga content. However, it is often the case that the user cannot understand even the outline of the contents depending on the published image page. On the other hand, the copyright owner would like to refrain from publishing as many image pages as possible so that the contents can be fully understood from the viewpoint of copyright.

  Further, in the case of a site that distributes and sells a large amount of manga content, it takes a lot of human power just to select an image page to be disclosed for the browsing function for each manga content.

  Accordingly, the present invention provides a summary manga image generation device, a program, and a program that can automatically extract frame images estimated to be as important as possible by analyzing manga content imagewise, and generate a summary manga image. It aims to provide a method.

According to the present invention, there is provided a summary manga image generating device for generating a summary manga image from a number of manga image pages including a plurality of frame images,
Image feature element extraction means for extracting a plurality of image feature elements for each frame image Fi;
Importance calculation means for calculating importance Score (Fi) based on a plurality of image feature elements for each frame image Fi,
Important frame image selection means for selecting a predetermined number of frame images in order from the highest importance frame image,
And a summary image generation unit that combines the selected predetermined number of frame images to generate a summary manga image.

According to another embodiment of the summary manga image generating apparatus of the present invention,
The importance calculation means uses the weighting factor w _k associated with each image feature element m _k (k = 1 to j, j: the number of image feature elements), and for each frame image Fi, the image feature element m. The sum of the _{product of ik} and the weighting factor w _k is the importance score (Fi).
Score (Fi) ＝ Σ _{k = 1} ^j w _k * m _ik
It is also preferable.

  According to another embodiment of the summary manga image generating apparatus of the present invention, it is also preferable to further comprise a frame image dividing means for dividing the entire manga image to be analyzed into frame image units surrounded by lines on the image.

  According to another embodiment of the summary manga image generating apparatus of the present invention, the image feature element extracting means preferably includes meta information extracting means, and meta information of the frame image is extracted as an image feature element.

  According to another embodiment of the summary manga image generating apparatus of the present invention, the image feature element extraction unit includes a high saliency region extraction unit, and the image feature element includes a high saliency in a portion of the frame image from the entire manga image. It is also preferable to extract sex regions.

According to another embodiment of the summary comic image generation apparatus of the present invention, the image feature element extraction means includes:
Blowing area extraction means for extracting a blowing area from the entire manga image;
A frame relevance extraction means for extracting a relevance representing which frame image is associated with which frame image by the area of the overlapping region of the balloon region in the frame image, and the frame image as an image feature element It is also preferable to extract meta information of the blowing area.

  According to another embodiment of the summary manga image generating apparatus of the present invention, character recognition means for extracting text from a balloon region by OCR (Optical Character Recognition) that specifies characters by collating with a previously stored character pattern It is also preferable to extract meta information of text of the frame image as an image feature element.

According to another embodiment of the summary manga image generating apparatus of the present invention, a large number of frame images selected as having high importance are used as learning images, and a support vector machine (Support Vector Machine) is used for each frame image Fi. It is also preferable to further include summary learning means for extracting a plurality of image feature elements in the entire frame image and calculating a weighting coefficient w _i from the tendency of the value for each image feature element.

According to the present invention, there is provided a summary manga image generation program for causing a computer mounted on an apparatus for generating a summary manga image to function from a large number of manga image pages including a plurality of frame images,
Image feature element extraction means for extracting a plurality of image feature elements for each frame image Fi;
Importance calculation means for calculating importance Score (Fi) based on a plurality of image feature elements for each frame image Fi,
Important frame image selection means for selecting a predetermined number of frame images in order from the highest importance frame image,
A computer is caused to function as a summary image generation unit that combines a predetermined number of selected frame images to generate a summary manga image.

According to the present invention, there is provided a summary manga image generation method for generating a summary manga image from a number of manga image pages including a plurality of frame images using an apparatus,
A first step of extracting a plurality of image feature elements for each frame image Fi;
A second step of calculating importance Score (Fi) based on a plurality of image feature elements for each frame image Fi;
A third step of selecting a predetermined number of frame images in order from the frame images having the highest importance;
And a fourth step of generating a summary manga image by combining a predetermined number of selected frame images.

  According to the summary manga image generation apparatus, program, and method of the present invention, a frame image presumed to be as important as possible is automatically extracted and a summary manga image is generated by analyzing manga content imagewise. Can do.

It is a system configuration diagram for distributing manga content. It is a functional block diagram of the summary manga image generation apparatus in this invention. It is explanatory drawing which has divided | segmented one manga image page into a frame image. It is explanatory drawing showing the high saliency area | region in the whole one manga image. It is explanatory drawing showing matching with a frame image and a blowing area | region. It is a functional block diagram of the summary manga image generation apparatus which further contains a summary learning part.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a system configuration diagram for distributing manga content.

  The manga content database 3 stores manga content as one work. One manga content comprises one episode with a plurality of manga image pages.

  The manga content distribution site 2 receives manga content to be distributed to the user from the manga content database 3 managed by the copyright holder. The manga content distribution site 2 accumulates the manga content so that it can be distributed to the user, and has a search function for the terminal and the mobile phone 4 based on the user operation. The terminal and the mobile phone 4 are connected to the Internet via an access network and connected to the comic content distribution site. The user can search for desired manga content by operating the terminal and the mobile phone 4.

  Here, according to the present invention, the summary manga image generation device 1 is further connected to the Internet and can communicate with the manga content distribution site 2. The summary manga image generation device 1 receives manga content that can be distributed to the user from the manga content distribution site 2 and generates the summary manga image. The generated summary comic image is returned to the comic content distribution site 2. Of course, the summary manga image generating apparatus 1 may receive manga content directly from the manga content database 3.

  The manga content distribution site 2 can publish a summary manga image as a part of the recommendation function of the manga content for the user. Thereby, the user can visually recognize the summary comic image of the desired comic content after searching by operating the terminal and the mobile phone 4. Thereby, the user can determine whether or not to purchase the manga content.

  FIG. 2 is a functional configuration diagram of the summary manga image generating apparatus according to the present invention.

  According to FIG. 2, summary manga image generation apparatus 1 inputs manga content for which a summary is to be generated from the outside. One manga content is composed of a plurality of manga image pages. The manga image page is composed of a number of frame images surrounded by lines on the image.

  According to FIG. 2, the summary manga image generation device 1 has a summary manga image generation unit 11, which is realized by executing a manga image analysis program that causes a computer installed in the device to function. Also, the processing flow in the functional component in the summary manga image generation unit 11 is understood as a summary manga image generation method in the apparatus.

  The summary manga image generation unit 11 includes an image feature element extraction unit 110, an importance level calculation unit 117, an important frame selection unit 118, and a summary image generation unit 119. The image feature element extraction unit 110 extracts a plurality of image feature elements for each frame image Fi, and includes a frame image division unit 111, a meta information extraction unit 112, a high saliency area extraction unit 113, A balloon region extraction unit 114, a balloon relevance extraction unit 115, and a character recognition unit 116 are included.

  The frame image dividing unit 111 divides the entire manga image to be analyzed into frame image units surrounded by lines on the image. A manga image is generally divided into frame image units.

  FIG. 3 is an explanatory diagram in which one manga image page is divided into frame images.

  According to FIG. 3, in order to divide into frame images, dividing line candidates are detected by straight line detection using bands, and dividing lines are determined by dividing line matching inspection (see, for example, Non-Patent Document 1). According to this technique, the dividing line candidate detects a detection line pixel having a width of 1 pixel, and if the detection line angle is within ± 45 ° with respect to the horizontal axis, two detection lines adjacent in the vertical axis direction, Otherwise, a detection line adjacent in the horizontal axis direction is defined as a “detection band”. These detection bands are divided into frame images by density gradient direction inspection and frame inside / outside inspection. The frame image dividing unit 111 outputs the frame image unit to the meta information extracting unit 112.

  In addition, when the manga image is previously divided into frame images, the frame image dividing unit 111 is not necessarily provided. This is because, for example, in the case of a manga image distributed for a mobile phone, it is preliminarily divided into frame images. The frame image dividing unit 111 is optional only, but is generally a necessary functional configuration unit.

  The meta information extraction unit 112 extracts meta information of the frame image as an “image feature element” for each frame image. The meta information is a characteristic for each frame image, and may be, for example, values of the width, height, and area of the frame image. The meta information as the image feature element is output to the importance calculation unit 117.

  The high saliency area extraction unit 113 extracts a high saliency area in the frame image portion from the entire manga image as an “image feature element”. This is to automatically extract a meaningful attention area (high saliency area) from the image by determining a prior probability as to which area has a high degree of attention. There is a technique for extracting a feature called “spectral residual” from the logarithmic spectrum information of an input image (for example, see Non-Patent Documents 2 and 3). According to this technique, the saliency at an arbitrary location in the image can be calculated. From the result, for example, a region exceeding the average value of the spectral residual of the entire image can be extracted as a “high saliency region”. The division from the image area can be formulated as a problem of binary segmentation that distinguishes between an important area (object) and other areas (background).

  FIG. 4 is an explanatory diagram showing a high saliency region in one entire manga image. According to FIG. 4, the high saliency area is displayed in highlight (white).

Further, the high saliency area extracting unit 113 associates the high saliency area obtained from the entire manga image page with a plurality of divided frame images. Specifically, the ratio of the area extracted as the high saliency area is calculated for each frame image. This ratio is used as an image feature element.
Image feature element = area of highly saliency area of frame image / area of entire frame image

  The balloon area extraction unit 114 extracts a balloon area from the entire manga image page. “Blowout area” refers to an area in which characters such as lines of characters appearing are written. The balloon area generally includes text such as dialogue. In addition, the manga image is generally monochrome, and not only the image but also the text itself is often drawn with a hand-based pen or brush. Therefore, unlike an image such as a landscape photograph, a manga image has a characteristic that it has many edges as a whole. In order to extract the blowing area, for example, the technique described in Non-Patent Document 4 can be applied.

  The blowing region extraction unit 114 outputs image feature elements (for example, height, width, and area) related to the size from the extracted blowing region to the blowing relationship extraction unit 115. The process of the frame image dividing unit 111 is executed separately and independently.

  The balloon relevance extraction unit 115 extracts the relevance with the balloon area for each frame image as an image feature element. The relevance here means to which frame image the balloon region is associated. Then, the balloon related extraction unit 115 outputs the identifier of the frame image corresponding to the balloon area to the importance calculating unit 117 together with the image feature elements (for example, height, width, and area) of the balloon area.

  FIG. 5 is an explanatory diagram showing the association between a frame image and a balloon area.

  In general, the balloon region in a manga image is often contained in one frame image. In this case, since the blowing area is completely overlapped with the frame image, it is associated with the frame image. On the other hand, there is a case where the blowing area is drawn so as to overlap a plurality of frames. According to FIG. 5, one blowing area is superimposed on two or more frame images.

  In such a case, by comparing the overlapping areas of the balloon region and each frame image, it is determined which frame image is associated with the balloon region. According to FIG. 5, the blowing area A is superimposed on the frame images 4, 5, and 7, and is associated with the frame image 4 having the widest overlapping area. The blowing area B is superimposed on the frame images 3, 6 and 7, and is associated with the frame image 46 having the widest overlapping area.

  The character recognition unit 116 extracts text such as dialogue by applying character recognition processing to the extracted image of the balloon region. For example, the character is specified by collating the image of the blowing area with a character pattern stored in advance. For example, an existing OCR (Optical Character Recognition) technique may be used. In order to detect a character, the edge characteristics of the font of the character, the interval between rectangular regions surrounding the outline, and the like are taken into consideration. From the extracted text, for example, the number of characters in the balloon area can be obtained as an image feature element. The number of characters as the image feature element is also output to the importance calculation unit 117.

The importance calculation unit 117 inputs, for example, the following image feature elements from all manga image pages in one manga content.
・ Height / width / area for each frame image (meta information extraction part)
-Ratio of high saliency area for each frame image in the entire manga image (high saliency area extraction unit)
・ Height / width / area for each blowing area associated with the frame image (blowing relationship extraction unit)
-Number of characters for each blowing area (character recognition part)
Note that elements other than the image feature elements described above may be included as long as they can be extracted from the manga image.

For each frame image Fi, the image feature element j (= 1 to j) is expressed as follows.
M _Fi = {m _i1 , m _i2 ,..., M _ij }
Using the weight coefficient w _i associated with each image feature elements m _i, the entire frame image Fi, and the sum of the multiplied value of the image feature elements m _i and the weighting coefficient w _i, severity Score (Fi) And
Score (Fi) ＝ Σ _{k = 1} ^j w _k * m _ik
m _ix : x-th image information element in the frame image Fi
j: Total number of image information elements w _k: weighting coefficient for the k-th image feature elements

  The important frame selection unit 118 selects a predetermined number of frame images (the top N) in order from the frame images having the highest importance from among the many frame images.

  Here, the predetermined number represents the number of frame images included in the summary manga image. The predetermined number represents, for example, the number of frame images of the summary manga image published by the manga content distribution site 2 in FIG. This predetermined number may be set by the operation manager of the manga content distribution site 2 or may be set by the copyright owner who manages the manga content database 3.

  The summary image generation unit 119 generates a summary manga image that combines the selected frame images. As a method for combining frame images, for example, the frame images may be rearranged into each image, or may be displayed in a slide show in which the frame images are displayed while being sequentially transmitted. The generated summary manga image is output from the summary manga image generation device 1.

  FIG. 6 is a functional configuration diagram of a summary manga image generation apparatus further including a summary learning unit.

  The summary manga image generating apparatus 1 of FIG. 6 further includes a summary learning unit 12 in addition to the summary manga image generation unit 11 of FIG. The summary learning unit 12 inputs a number of frame images selected as having high importance as learning images. A summary manga image is generated based on the tendency of the image feature elements in the learning image.

  The summary learning unit 12 includes an image feature element extraction unit 120 and a learning information output unit 121. The image feature element extraction unit 120 is exactly the same as the image feature element extraction unit 110 of the summary manga image generation unit 11. The learning information output unit 121 extracts a plurality of image feature elements in the entire frame image for each frame image Fi using a large number of frame images selected as having a high importance as a learning image, and serves as an important frame image selection unit. In response, learning information is output.

  It is also preferable to use a support vector machine for the learning information output unit 121. The “support vector machine” is an identification algorithm using supervised learning and is applied to pattern recognition. The support vector machine constitutes two classes of pattern discriminators using linear input elements, and learns the parameters of the linear input elements by calculating the distance from each feature value from the learning sample.

Specifically, the image feature element M _Fi (= {m _i1 , m _i2 ,..., M _ij }) is handled as a vector. A weighting coefficient is generated by regression analysis with a vector of an image selected as a frame having high importance for a learning image as a “positive example” and other vectors as “negative examples”. However, in the case of a support vector machine, since only the identification result is output, depending on the input learning image, the important frame image may be detected excessively, or the number of cases detected as the important frame image may be insufficient. is assumed. In this case, for example, it is also preferable to sort the frame images based on the distance from the separation hyperplane and select the top N frame images.

  As described above in detail, according to the summary manga image generation apparatus, program, and method of the present invention, frame images that are estimated to be as important as possible are automatically extracted by analyzing manga content imagewise. And a summary manga image can be generated. The manga content distribution site publishes the generated summary manga image to the user, so that the user can visually recognize the summary manga image of the desired manga content. Thereby, the user can determine whether or not to purchase the manga content.

  Various changes, modifications, and omissions of the above-described various embodiments of the present invention can be easily made by those skilled in the art. The above description is merely an example, and is not intended to be restrictive. The invention is limited only as defined in the following claims and the equivalents thereto.

DESCRIPTION OF SYMBOLS 1 Summary manga image generation apparatus 11 Summary manga image generation part 12 Summary learning part 110,120 Image feature element extraction part 111 Frame image division part 112 Meta information extraction part 113 High saliency area extraction part 114 Blowout area extraction part 115 Blowout relevance Extraction unit 116 Character recognition unit 117 Importance calculation unit 118 Important frame selection unit 119 Summary image generation unit 121 Learning information output unit 2 Manga content distribution site 3 Manga content database 4 Terminal, mobile phone

Claims (10)

A summary manga image generation device for generating a summary manga image from a number of manga image pages including a plurality of frame images,
Image feature element extraction means for extracting a plurality of image feature elements for each frame image Fi;
Importance calculation means for calculating importance Score (Fi) based on a plurality of image feature elements for each frame image Fi,
Important frame image selection means for selecting a predetermined number of frame images in order from the frame images having the highest importance,
A summary manga image generation device comprising summary image generation means for combining a predetermined number of selected frame images to generate a summary manga image. The importance calculation means uses the weighting factor w _k associated with each of the image feature elements m _k (k = 1 to j, j: the number of image feature elements) for each frame image Fi. The sum of the multiplication values of the element m _ik and the weighting factor w _k is defined as the importance score (Fi).
Score (Fi) ＝ Σ _{k = 1} ^j w _k * m _ik
The summary comic generation apparatus according to claim 1, wherein:   3. The summary comic image generating apparatus according to claim 1, further comprising a frame image dividing unit for dividing the entire comic image to be analyzed into frame image units surrounded by lines on the image. The image feature element extraction means includes meta information extraction means,
The summary manga image generation apparatus according to claim 1, wherein meta information of the frame image is extracted as the image feature element. The image feature element extraction means includes high saliency area extraction means,
The summary manga image generating apparatus according to any one of claims 1 to 4, wherein a high saliency area in the frame image portion is extracted from the entire manga image as the image feature element. The image feature element extraction means includes
Blowing area extracting means for extracting a blowing area from the entire manga image;
A balloon relevance extraction means for extracting a relevance representing which frame image is associated with which frame image by the area of the overlapping region of the balloon region in the frame image, and 6. The summary comic image generation apparatus according to claim 1, wherein meta information of a frame image blowing area is extracted. Character recognition means for extracting text by OCR (Optical Character Recognition) that specifies characters by collating with a prestored character pattern from the blowing area,
The summary comic image generating apparatus according to claim 6, wherein meta information of text of the frame image is extracted as the image feature element.   Using a support vector machine (Support Vector Machine) as a learning image for a number of frame images selected as having a high degree of importance, a plurality of image feature elements in the entire frame image are extracted for each frame image Fi. The summary comic image generating apparatus according to claim 1, further comprising summary learning means for outputting learning information to the important frame image selection means. A summary manga image generation program for causing a computer mounted on a device for generating a summary manga image to function from a large number of manga image pages including a plurality of frame images,
Image feature element extraction means for extracting a plurality of image feature elements for each frame image Fi;
Importance calculation means for calculating importance Score (Fi) based on a plurality of image feature elements for each frame image Fi,
Important frame image selection means for selecting a predetermined number of frame images in order from the frame images having the highest importance,
A program for generating a manga image, comprising: combining a predetermined number of selected frame images to cause a computer to function as summary image generation means for generating a summary manga image. A summary manga image generation method for generating a summary manga image from a number of manga image pages including a plurality of frame images using an apparatus,
A first step of extracting a plurality of image feature elements for each frame image Fi;
A second step of calculating importance Score (Fi) based on a plurality of image feature elements for each frame image Fi;
A third step of selecting a predetermined number of frame images in order from the frame image having the highest importance;
And a fourth step of generating a summary manga image by combining a predetermined number of selected frame images.

Images