JP2020024555A - Cartoon data set generation apparatus and cartoon data set generation program - Google Patents

Abstract

To automatically generate a cartoon data set to which an annotation is added.SOLUTION: A frame arranging unit 24 arranges a frame 40 in an area within a page image, and an image allocating processing unit 28 allocates a cartoon image 42 converted from a natural image 16 by an image converting unit 26 to each frame 40. A balloon 44, a character 46, and an effect line 48 are provided to each frame 40 by a balloon providing unit 30, a character providing unit 32, and an effect line providing unit 34, and a page image as cartoon data is generated. An annotation adding unit 36 refers to processing results of the above-described units, acquires information on each element of the cartoon included in the page image, and adds an annotation to the page image.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a cartoon data set generation device and a cartoon data set generation program.

  Conventionally, there has been proposed a technique of displaying a plurality of images in a comic style by dividing an output page into a plurality of frames and pasting a plurality of images to each frame (for example, Patent Document 1).

Japanese Patent No. 5073612

  By the way, it is conceivable that an AI agent (learning device) is trained using a manga as learning data, and a sufficiently learned AI agent analyzes the manga. By letting the AI agent analyze the manga, for example, a service that suitably extracts the character portion from the manga and translates the character portion, or a service that specifies the number of appearances of a character or the scene of the first appearance, etc. Can be provided.

  In order for the AI agent to perform learning by using comics as learning data, a large amount of annotated comic data (cartoon data set) is required. The annotation indicates the meaning or characteristic of the learning data. Examples of the annotation of the comic dataset include, for example, what indicates the object drawn in each frame or the position of the object in the frame, the position or shape of the balloon, and the position or shape of the character area. And so on.

  Conventionally, a cartoon data set provided with an annotation has been provided (for example, Manga 109 (http://www.manga109.org/ja/annotations)). However, in order to sufficiently learn the AI agent, a large amount of cartoon data sets is required, and at present, the provided cartoon data sets with annotations are insufficient.

  In addition, as a secondary problem, it can be pointed out that the conventionally provided manga data set lacks the added annotations. For example, in a conventional cartoon data set, the shape of one frame is represented by a rectangle circumscribing the frame instead of the shape of the frame itself, and the shape of the balloon is not shown. Therefore, a problem can be pointed out that even if learning is performed using the comic data set, the shape of the frame itself cannot be analyzed or the shape of the speech balloon cannot be analyzed.

  An object of the present invention is to automatically generate a cartoon data set to which an annotation is added.

  According to the present invention, a frame arrangement unit for arranging a frame in an area within a page image, an image allocation processing unit for allocating a cartoon image to the frame, the frame arranged by the frame arrangement unit, and the image allocation processing unit A cartoon data set generating apparatus, comprising: an annotation giving unit that gives an annotation related to the frame and the cartoon image to the page image based on the assigned cartoon image.

  Preferably, the annotation indicates a shape of the frame.

  Desirably, further comprising: a balloon giving unit for giving a balloon to the frame, the annotation providing unit, based on the balloon provided by the balloon providing unit, provides an annotation related to the balloon to the page image, It is characterized by the following.

  Preferably, the annotation indicates the shape of the balloon.

  Desirably, further comprising: a character providing unit that provides a character to the frame, the annotation providing unit, based on the character provided by the character providing unit, provides an annotation related to the character to the page image, It is characterized by the following.

  Desirably, the frame further comprises an effect line imparting unit for imparting an effect line to the frame, wherein the annotation imparting unit, based on the effect line imparted by the effect line imparting unit, the annotation on the effect line the page. Is added to an image.

  Preferably, the image processing apparatus further includes an image conversion unit that converts a natural image into the comic image.

  Preferably, the annotation providing unit provides an annotation related to the comic image based on the annotation provided to the natural image.

  Further, the present invention provides a computer, a frame arranging unit for arranging a frame in an area within a page image, an image allocation processing unit for allocating a cartoon image to the frame, the frame arranged by the frame arranging unit, and A cartoon data set generation program, which functions as an annotation providing unit that gives an annotation related to the frame and the cartoon image to the page image based on the cartoon image assigned by the image assignment processing unit.

  According to the present invention, a cartoon data set to which an annotation has been added can be automatically generated.

It is a schematic diagram of a configuration of a comic data set generation device according to the present embodiment. It is a figure showing the example of frame division. It is a figure showing a process in which a cartoon image is generated from a natural image. FIG. 11 is a diagram illustrating a page image in which a comic image is assigned to each frame and a balloon is provided. It is a figure showing a page image to which a character and an effect line were further given. It is a figure which shows the circumscribed rectangle of a frame and the shape of a frame. It is a figure showing a person field and a face field. It is a figure which shows the shape of a speech balloon. It is a figure showing a character field and an effect line field. It is a flowchart which shows the flow of a comics data set generation process.

  Hereinafter, embodiments of the present invention will be described.

  FIG. 1 is a schematic configuration diagram of a comic data set generation device 10 according to the present embodiment. The cartoon data set generation device 10 in the present embodiment may be a general computer (for example, a server computer) or the like, and may be any device as long as it has the following functions.

  The communication unit 12 includes, for example, a network adapter and the like. The communication unit 12 has a function of communicating with another device via a communication line such as a LAN or the Internet.

  The storage unit 14 is configured to include, for example, a ROM, a RAM, or a hard disk. The storage unit 14 stores a cartoon data set generation program for operating each unit of the cartoon data set generation device 10. In addition, as shown in FIG. 1, a natural image 16 is stored in the storage unit 14.

  The natural image 16 is an image obtained by capturing a natural scene including a person, an object, or a building with a camera. The storage unit 14 may store a large number of natural images 16. In the present embodiment, the natural image 16 is stored in the storage unit 14 in the comic data set generation device 10, but the natural image 16 is stored in another device accessible from the comic data set generation device 10. It may be.

  Various annotations are added to the natural image 16 in advance. For example, a circumscribed rectangle (person area) of a person included in the natural image 16, a face area of a person included in the natural image 16, a circumscribed rectangle of each object included in the natural image 16, or a circumscribed rectangle of each object included in the natural image 16 Annotation such as name is attached.

  The input unit 18 includes, for example, a mouse, a keyboard, a touch panel, and the like. The input unit 18 is used to input an instruction from a user to the comic data set generation device 10.

  The display unit 20 is configured to include, for example, a liquid crystal panel. Various screens are displayed on the display unit 20.

  When the comics data set generation device 10 is a server computer, the input unit 18 and the display unit 20 may not be provided.

  The control unit 22 includes, for example, a CPU. The control unit 22 controls each unit of the cartoon data set generation device 10 according to the cartoon data set generation program stored in the storage unit 14. Also, as shown in FIG. 1, the control unit 22 includes a frame arrangement unit 24, an image conversion unit 26, an image assignment processing unit 28, a balloon giving unit 30, a character giving unit 32, an effect line giving unit 34, and an annotation giving unit. Also functions as 36. When the control unit 22 performs these functions, a cartoon data set to which an annotation is added is generated. Hereinafter, details of these functions will be described.

  The frame arranging unit 24 performs a process of arranging one or a plurality of comic frames in a region within a page image serving as a basis of comic data (the page image may be initially blank). For the arrangement of the frames, for example, the method described in Patent Document 1 can be used. More specifically, the page image is divided into a plurality of frames by repeating a process of vertically dividing a region in the page image and a process of dividing the region horizontally. For example, a process in which a page image is divided horizontally into a plurality of frames, and then the divided frames are vertically divided and further divided into frames is repeated until a predetermined stop criterion is satisfied.

  A plurality of frames divided in this way can be represented by a tree structure as shown in FIGS. In other words, in the tree structure, the number of branches of the branch extending downward from the horizontal (horizontal) division node “H” indicates the number of frames obtained by dividing horizontally, and the vertical (vertical) division node “V” The number of branches of a branch extending downward from, and the number of frames obtained by dividing vertically are shown.

  As the stop criterion, for example, the area of the divided area (frame) becomes equal to or smaller than a predetermined area, the area stops at a probability proportional to the reciprocal of the area of the divided frame, or the depth of the tree structure reaches a predetermined value. For example, it is possible to employ a method of stopping at a probability proportional to the depth of the tree structure or a method of stopping at random.

  2A to 2C show examples of a plurality of frames 40 arranged on a page image by the frame arrangement unit 24. FIG. The frame 40 may be provided with a frame to visualize the area, or as shown in FIG. 2A, a frame 40 without a frame may be arranged. A margin may be set between the frames 40. Further, the width of the margin between the frames 40 may not be constant.

  In the present embodiment, the frame arrangement unit 24 arranges the frames 40 by dividing the page image. However, the method of arranging one or a plurality of frames 40 on the page image employs another method. Is also good.

  The frame arranging unit 24 stores information indicating the layout of each frame 40 in the page image, that is, information indicating the shape and position of each frame in the storage unit 14.

  The image conversion unit 26 performs a process of converting the natural image 16 into a cartoon image (cartoon-touch image). The comic image is used as an image (image) in each frame 40. The cartoon data set generated by the cartoon data set generation device 10 is used to train an AI agent that analyzes cartoons. If the image in each frame 40 is the natural image 16, the AI agent for comic analysis may not be properly learned. Therefore, the image converter 26 converts the natural image 16 into a cartoon image so that the AI agent for cartoon analysis can be more appropriately learned.

  In the present embodiment, the image conversion unit 26 converts the natural image 16 into a cartoon image according to the following procedure. First, the image conversion unit 26 converts the natural image 16 into a gray scale. FIG. 3A shows a grayscale natural image 16a.

  Next, the image conversion unit 26 applies a bilateral filter to the grayscale natural image 16a. A bilateral filter is an averaging of pixel values of pixels in a filter based on a weight corresponding to a distance from a target pixel (for example, the center of the filter) and a weight corresponding to a difference in pixel value from the target pixel. Is a filter that performs This makes it possible to smooth the pixel values while preserving the edges of the image. FIG. 3B shows a natural image 16b after the application of the bilateral filter.

  Next, the image conversion unit 26 performs an unsharp mask process for sharpening the natural image 16b. The unsharp mask process is a process in which the original image is sharpened by weighting the original image and applying a low-pass filter to the smoothed image and obtaining the difference between the original image and the original image. FIG. 3C shows a natural image 16c after the unsharp mask processing.

  Finally, the image conversion unit 26 ternarizes the natural image 16c. Specifically, the pixel value of each pixel of the natural image 16c is set to a value corresponding to one of black, white, and an intermediate color (gray), and the intermediate color area is represented by a screen tone (dot or the like). Through the above processing, the natural image 16 is converted into a cartoon image 42 as shown in FIG.

  In the present embodiment, the cartoon image 42 is generated by converting the natural image 16. However, a cartoon image may be prepared in advance and stored in the storage unit 14 instead of the natural image 16.

  The image assignment processing unit 28 performs a process of assigning the cartoon image 42 generated by the image conversion unit 26 to each frame 40 arranged by the frame arrangement unit 24. Specifically, the image assignment processing unit 28 performs a process of fitting the comic image 42 into each frame 40.

  Depending on the relationship between the shape of the top 40 and the outer shape of the comic image 42, a part of the comic image 42 is cut out and fitted into the top 40. In this case, the image allocation processing unit 28 performs the annotation based on the annotation attached to the natural image 16 from which the comic image 42 is generated, or the image analysis on the comic image 42 (or the natural image 16 as the generating source). It is desirable to cut out a region including as many objects (including a person) as possible from the comic image 42 and fit the frame 40 therein.

  Note that the plurality of comic images 42 fitted into the plurality of frames 40 may not have any semantical relevance.

  The balloon giving unit 30 performs a process of giving a balloon to each frame 40 in which the comic image 42 is fitted. Speech balloons are frames of various shapes in which words of characters in a manga are described. In the present embodiment, balloons of various shapes are stored in the storage unit 14 in advance, and the balloon giving unit 30 randomly selects balloons from balloons of various shapes stored in the storage unit 14 and selects each frame 40. To be given. Note that the balloon does not need to be applied to all the frames 40 arranged by the frame arrangement unit 24, and may be applied to some of the frames 40 (the same applies to characters and effect lines described below). .

  FIG. 4 shows a page image in which a balloon 44 is added to each frame 40 in which a cartoon image 42 is fitted.

  The speech balloon providing unit 30 performs an annotation attached to the natural image 16 that is the generation source of the comic image 42 fitted into each frame 40, or an image analysis of the comic image 42 (or the natural image 16 of the generation source). , It is desirable to provide the balloon 44 at a position where the area overlapping the object included in the cartoon image 42 is as small as possible. Preferably, the balloon 44 is provided so as not to overlap the object included in the cartoon image 42. In particular, when a person is included in the cartoon image 42, it is preferable to provide the balloon 44 so as not to overlap the face area of the person. Note that the balloon 44 does not need to be accommodated in the frame 40, and a part thereof may protrude from the frame 40 as shown in FIG.

  The balloon giving unit 30 stores information indicating the shape and position of the balloon 44 given to each frame 40 in the storage unit 14.

  The character assigning unit 32 performs a process of assigning a character to each frame 40 in which the cartoon image 42 is fitted. In the present embodiment, the character assigning unit 32 assigns a character in the balloon 44 assigned by the balloon assigning unit 30. In addition, in the case where characters are written outside the speech bubble 44 in the manga (for example, characters expressing sound effects), the character assignment unit 32 may assign a character to an area outside the speech bubble 44. Good. It should be noted that the plurality of characters (sentences) added need not be meaningful. Further, the size of the character and the number of characters may be determined at random.

  FIG. 5 shows a page image in which a character 46 is provided in a balloon 44.

  The character assigning unit 32 stores information indicating the shape and position of the character area, which is the area of the character assigned to each frame 40, in the storage unit 14.

  The effect line giving unit 34 performs a process of giving an effect line to each frame 40 in which the comic image 42 is fitted. The effect line is a line that is often used in comics and is used for giving an expression effect or the like to an image. In the present embodiment, various effect lines are stored in the storage unit 14 in advance, and the effect line giving unit 34 randomly selects an effect line from the various effect lines stored in the storage unit 14 and To be given.

  FIG. 5 shows an effect line 48 provided in the frame 40.

  The effect line giving unit 34 holds information indicating the shape (including the line extending direction) and the position of the effect line area, which is the area of the effect line given to each frame 40, in the storage unit 14.

  Cartoon data is automatically generated by the processing of the frame arrangement unit 24, the image conversion unit 26, the image allocation processing unit 28, the balloon giving unit 30, the character giving unit 32, and the effect line giving unit 34.

  The annotation providing unit 36 adds an annotation to the comic data (page image) generated by each unit described above. Since the page image is generated by each of the above-described units, the annotation providing unit 36 refers to the processing result of each of the above-described units stored in the storage unit 14 so that each element of the manga included in the page image can be referred to. Information can be acquired, and various annotations can be given based on the information. Hereinafter, each annotation will be specifically described.

  The annotation providing unit 36 provides an annotation for each frame 40 based on the layout of the frames 40 (the shape and position of each frame 40) arranged by the frame arranging unit 24. FIG. 6A shows an example of the frames 40 arranged by the frame arrangement unit 24. Since the shape of the top 40 is known, the annotation imparting unit 36 displays not only the circumscribed rectangle 50 of the top 40 as shown in FIG. Such a shape 52 of the top 40 itself can be included. The circumscribed rectangle 50 and the shape 52 of the top 40 indicate not only the shape of the top 40 but also its position.

  In addition, the annotation giving unit 36 gives an annotation related to the cartoon image 42 based on the cartoon image 42 assigned to the frame 40 by the image assignment processing unit 28. As shown in FIG. 7, the annotations related to the cartoon image 42 include a person area 54 indicating a person area included in the cartoon image 42, and a face area 56 indicating a person face area included in the cartoon image 42. included. The person area 54 or the face area 56 may include information for identifying who the corresponding person or face is. In addition, areas indicating various objects (for example, buildings, animals, objects, and the like) included in the cartoon image 42 are included. The area indicating the object may also include information identifying what the corresponding object is. Each of the areas indicates the size and position of the corresponding person, face, or object.

  The annotation related to the cartoon image 42 is given based on the annotation given to the natural image 16 from which the cartoon image 42 is generated. For example, as the annotation indicating the above-described human area 54, face area 56, and other object areas related to the cartoon image 42, the annotation given to the natural image 16 can be used as it is.

  The annotation providing unit 36 performs image analysis on the cartoon image 42 to identify the human region 54, the face region 56, and other regions, and annotates the cartoon image 42 based on the identification result. It may be.

  In addition, the annotation providing unit 36 provides an annotation related to the balloon 44 based on (the shape and position of) the balloon 44 provided by the balloon providing unit 30. FIG. 8A shows an example of the balloon 44 provided by the balloon providing unit 30. Since the shape of the balloon 44 is known, the annotation providing unit 36 can include the shape 58 of the balloon 44 itself as shown in FIG. The shape 58 of the balloon 44 indicates not only the shape of the balloon 44 but also its position.

  Further, the annotation providing unit 36 provides an annotation related to the character 46 based on the character 46 provided by the character providing unit 32. Specifically, the annotation giving unit 36 gives an annotation indicating the shape and position of the character area 60 as shown in FIG. 9 based on the known shape and position of the character area.

  Further, the annotation giving unit 36 gives an annotation related to the effect line 48 based on the effect line 48 given by the effect line giving unit 34. Specifically, the annotation giving unit 36 gives an annotation indicating the shape and position of the effect line region 62 as shown in FIG. 9 based on the known shape and position of the effect line region.

  The type of annotation added to the page image by the annotation adding unit 36 is not limited to the above. For example, the annotation imparting unit 36 analyzes the expression of a person or a character included in the cartoon image 42 or the corresponding natural image 16 and expresses the emotion of the person or the character with respect to the frame 40 including the person or the character. May be added.

  A plurality of page images are automatically generated by the frame arrangement unit 24, the image conversion unit 26, the image allocation processing unit 28, the balloon giving unit 30, the character giving unit 32, and the effect line giving unit 34. An annotation is automatically added to the image. As a result, a cartoon data set including a plurality of page images to which the annotation has been added is automatically generated.

  According to the generated cartoon data set with annotation, various learning devices for comic analysis can be suitably trained. For example, by learning a learning device such as R-CNN, Fast R-CNN, Faster R-CNN, SSD, and YOLO / YOLOv2 / YOLOv3 from a cartoon data set with annotations, a frame region, a character region, and a face region are obtained from the cartoon data. Or detection of a human area. Further, by learning Mask R-CNN using a cartoon data set with annotations, it becomes possible to extract the shape of a frame and the shape of a speech bubble in a cartoon.

  The outline of the configuration of the cartoon data set generation device 10 according to the present embodiment is as described above. Hereinafter, the flow of the comic data set generation process will be described with reference to the flowchart shown in FIG.

  In step S10, the frame arrangement unit 24 arranges one or a plurality of cartoon frames in an area in the page image.

  In step S12, the image conversion unit 26 performs grayscale processing, application of a bilateral filter, unsharp mask processing, and ternarization processing on the natural image 16 to convert the natural image 16 into the cartoon image 42. Convert to

  In step S14, the image assignment processing unit 28 assigns the comic image 42 generated in step S12 to each frame 40 arranged in step S10.

  In step S16, the balloon giving unit 30 gives a balloon 44 to the frame 40 to which the cartoon image 42 has been assigned in step S12.

  In step S18, the character assigning unit 32 assigns the character 46 assigned in step S16.

  In step S20, the effect line giving unit 34 gives an effect line 48 to the frame 40 to which the cartoon image 42 has been assigned in step S12.

  In step S22, the annotation imparting unit 36 imparts various annotations to the page images generated in steps S10 to S20.

  By repeating steps S10 to S22, a cartoon data set including a plurality of annotated page images is generated.

  Although the embodiment according to the present invention has been described above, the present invention is not limited to the above embodiment, and various modifications can be made without departing from the spirit of the present invention.

  Reference Signs List 10 cartoon data set generation device, 12 communication unit, 14 storage unit, 16 natural image, 18 input unit, 20 display unit, 22 control unit, 24 frame arrangement unit, 26 image conversion unit, 28 image allocation processing unit, 30 speech balloon application Section, 32 character giving section, 34 effect line giving section, 36 annotation giving section.

Claims (9)

A frame arrangement unit for arranging the frames in an area within the page image;
An image assignment processing unit that assigns a cartoon image to the frame,
The frame arranged by the frame arrangement unit, and, based on the comic image allocated by the image allocation processing unit, based on the comic image, an annotation providing unit that provides an annotation on the page image and the comic image,
A cartoon data set generation device comprising: The annotation indicates the shape of the top,
The cartoon data set generation device according to claim 1, wherein: A balloon providing unit for providing a balloon to the frame,
Further comprising
The annotation providing unit, based on the balloon provided by the balloon providing unit, provides an annotation related to the balloon to the page image,
The comic data set generation device according to claim 1 or 2, wherein: The annotation indicates the shape of the balloon,
The cartoon data set generation device according to claim 3, wherein: A character providing unit for providing a character to the frame,
Further comprising
The annotation providing unit, based on the character provided by the character providing unit, to provide an annotation on the character to the page image,
The cartoon data set generation device according to any one of claims 1 to 4, wherein: An effect line giving unit that gives an effect line to the frame,
Further comprising
The annotation imparting unit, based on the effect line imparted by the effect line imparting unit, imparts an annotation related to the effect line to the page image,
The comic data set generation device according to any one of claims 1 to 5, wherein: An image conversion unit that converts a natural image into the cartoon image,
The cartoon data set generation device according to claim 1, further comprising: The annotation providing unit is configured to provide an annotation related to the cartoon image based on the annotation provided to the natural image,
The cartoon data set generation device according to claim 7, wherein: Computer
A frame arrangement unit for arranging the frames in an area within the page image;
An image assignment processing unit that assigns a cartoon image to the frame,
The frame arranged by the frame arrangement unit, and, based on the comic image allocated by the image allocation processing unit, based on the comic image, an annotation providing unit that provides an annotation related to the comic image and the page image,
A manga data set generation program characterized by functioning as a.