JP2021060875A - Frame extraction method and program - Google Patents

Abstract

To provide a frame extraction method that can extract frames from a cartoon image simply and accurately.SOLUTION: A frame extraction method includes: a binarization step of binarizing a cartoon image into black and white; a first extraction step of extracting an outline of the cartoon image and extracting areas surrounded by frames in the cartoon image as frames; a clustering step of clustering black areas with respect to a remaining area not extracted as the frames by the first extraction step in the cartoon image; and a second extraction step of setting each of the clustered areas as a rectangular area and extracting the rectangular area as a frame.SELECTED DRAWING: Figure 1

Description

The present invention relates to a frame extraction method and a program.

In recent years, there have been an increasing number of opportunities to read manga on mobile terminals such as smartphones. However, when reading a manga on a mobile terminal such as a smartphone, it is often more difficult to read than when reading a manga printed on paper. For example, a plurality of frames are drawn on one page of a manga, and when one page is displayed and read as one image, one frame becomes small and difficult to read. In addition, when the target frame on the page is enlarged and read, the operation becomes complicated and difficult to read.

On the other hand, it is conceivable to extract frames from the cartoon image and display each frame separately to make it easier to read even on a mobile terminal. Conventionally, a technique for extracting frames has been proposed, but it is required to improve the accuracy of frame extraction. For example, Patent Document 1 discloses that frame extraction can be performed even when the frame line of a frame is interrupted. Further, in Patent Document 2, after a frame defined by a boundary line is detected from the entire cartoon image, the detection area of the frame is narrowed and the frame or a frame defined by the boundary line is included. It is disclosed to detect coma.

However, in Patent Documents 1 and 2, it is considered that the accuracy of frame extraction is improved for a cartoon image having a specific frame, but the frame can be extracted with high accuracy for a cartoon image having other frames. Absent. On the other hand, Patent Document 3 proposes a technique for recognizing a frame or the like by using a machine learning technique. In Patent Document 3, the recognition accuracy is improved by using the detection means in the state of being machine-learned to detect the frame portion of the cartoon.

JP-A-2017-45219 Japanese Unexamined Patent Publication No. 2014-212476 Japanese Unexamined Patent Publication No. 2019-46253

However, the machine learning technique for detecting the frame portion of the cartoon proposed in Patent Document 3 requires a large amount of teacher data, and has a problem that it takes a lot of time and effort.

Therefore, an object of the present invention is to provide a frame extraction method capable of easily and accurately extracting frames from a cartoon image.

In order to solve the above problems, the frame extraction method of the present invention performs a binarization step of binarizing a cartoon image into black and white, and contour extraction of the cartoon image to form a border in the cartoon image. A first extraction step of extracting the region surrounded by the frame as a frame, and a clustering step of clustering a black region with respect to the remaining region not extracted as the frame by the first extraction step in the cartoon image. It is characterized by having a second extraction step in which each of the clustered regions is a rectangular region and the rectangular region is extracted as a frame.

According to the present invention, frame extraction can be easily and accurately performed from a cartoon image.

It is a flowchart in an example of the frame extraction method which concerns on this invention. It is a figure for demonstrating the cartoon image in an example of the frame extraction method which concerns on this invention. It is a figure for demonstrating the clustering process. It is a figure for demonstrating another cartoon image in an example of the frame extraction method which concerns on this invention. It is a flowchart in another example of the frame extraction method which concerns on this invention. It is a figure for demonstrating the sorting process. It is another figure for demonstrating the sorting process. It is another figure for demonstrating the sorting process.

Hereinafter, the frame extraction method and the program according to the present invention will be described with reference to the drawings. The present invention is not limited to the embodiments shown below, and can be modified within the range conceivable by those skilled in the art, such as other embodiments, additions, modifications, and deletions. However, as long as the action and effect of the present invention are exhibited, it is included in the scope of the present invention.

(First Embodiment)
An embodiment of the frame extraction method according to the present invention will be described.
The frame extraction method of the present embodiment has a binarization step of binarizing the cartoon image into black and white, and contour extraction of the cartoon image, and the area surrounded by the frame in the cartoon image is framed. A first extraction step of extracting as, a clustering step of clustering a black region with respect to the remaining region not extracted as the frame by the first extraction step in the cartoon image, and each of the clustered regions. It is characterized by having a second extraction step in which the region is a rectangular region and the rectangular region is extracted as a frame.

According to this embodiment, frame extraction can be easily and accurately performed from a cartoon image. According to the present embodiment, it is possible to extract not only frames without a border but also frames surrounded by wavy lines instead of straight lines such as hand-drawn lines. Here, "performing frame extraction with high accuracy" means extracting what a human recognizes as a frame, and means that it is closer to human recognition than the conventional technology. It is preferable to match human perception as much as possible, but it is not necessary to match it exactly.

Further, in the present embodiment, frame extraction can be performed by using the machine learning technique by unsupervised learning, and the frame extraction can be performed more easily than the conventional technique. In the conventional technique, it is necessary to prepare teacher data, and when performing machine learning by supervised learning with a correct answer, labor and cost are required. On the other hand, according to the present embodiment, frame extraction can be performed by using the machine learning technique without preparing teacher data, so that frame extraction can be performed easily and accurately. In addition, by combining machine learning without teacher data and area extraction, it is possible to accurately extract frames even if there is no frame frame.

FIG. 1 shows a flowchart of the frame extraction method of the present embodiment.
In the present embodiment, the binarization step (S101), the first extraction step (S102), the clustering step (S103), and the second extraction step (S104) are performed, and other steps are performed as needed.

<Binarization process>
In the binarization step, the cartoon image is binarized into black and white (S101).
The binarization is not particularly limited, but for example, a cartoon image is converted into an HSV color space and binarized. When converting to the HSV color space and binarizing, for example, the region having a brightness of 80% or less and a hue of 70% or less is made black, and the other regions are made white.
The binarization step is not limited to the HSV color space, and may be binarized using another color space.

In the present embodiment, in order to improve the accuracy of frame extraction in the first extraction step described later, it is preferable to increase the accuracy of binarization. In order to improve the accuracy of binarization in the present embodiment, for example, edge detection is used to complement binarization by gray scale.

<First extraction step>
In the first extraction step, contour extraction is performed on the cartoon image, and the area surrounded by the frame line in the cartoon image is extracted as a frame (S102).

The first extraction step of this embodiment will be described with reference to FIG. FIG. 2A shows an example of a cartoon image, and frames A to E are drawn in the cartoon image 10. When contour extraction is performed on the cartoon image 10, frames A to C and E are extracted as shown in FIG. 2 (B).

In contour extraction, in the target image, the area surrounded by the frame line is recognized, and the area surrounded by the frame line is cut out and output. Therefore, when contour extraction is performed on the cartoon image 10, frames A to C and E can be extracted and output, respectively.

In the first extraction step of this example, the frame D in the cartoon image 10 is excluded from the extraction target.
Since the frame D has a break in the frame line, it is not recognized as an area surrounded by the frame line in the contour extraction. Such a frame D is extracted through the clustering and the second extraction step described later.

The frame line of the frame E is a wavy line by handwriting or the like, but in this example, it can be extracted as a frame. Even when the frame E is not extracted, it can be an extraction target by going through the clustering and the second extraction step described later.

As the contour extraction, a region growth method (watershed algorithm), a region integration method, or the like can be used.

<Clustering process>
In the clustering step, a black region is clustered with respect to the remaining region of the cartoon image that was not extracted as the frame by the first extraction step (S103).

By clustering the remaining regions that were not extracted as coma by the first extraction step, it is possible to extract the coma that were not extracted by the first extraction step. Actually, the coma is extracted in the second extraction step described later, but the clustering step corresponds to the preparation.

The clustering step of this embodiment will be described with reference to FIG.
FIG. 3 (A) shows the frame D in FIG. 2, and FIG. 3 (B) shows an enlarged view of the frame line 20 in the frame D. Further, FIG. 3B shows a state after clustering, and by performing clustering, even if there is a break in the frame line 20, it is treated as one unit (clustering area 30). be able to.

As a result, as shown in FIG. 3C, the broken frame line 20 of the frame D can be treated as the clustering region 30. If there is no continuous border, such as when the border is interrupted, fine areas will be detected separately. Therefore, by performing clustering, close objects are combined in the fine areas.

In the clustering step in the present embodiment, not only the frame line in the above example but also various regions are clustered to form a cluster. By forming a cluster, a rectangular area for extracting coma can be formed in the second extraction step described later.

Clustering is preferably performed by DBSCAN (Density based spatial clustering of applications with noise). DBSCAN has an advantage that clustering can be performed with high accuracy, and frame extraction can be performed with high accuracy in the second frame extraction.

Parameters for clustering include, for example, a parameter indicating how far the dots are treated as a cluster, a parameter indicating how many dots around a specific dot are treated as a cluster, and the like. Can be mentioned. For example, when DBSCAN is used, ε and minPts can be mentioned. Since DBSCAN is unsupervised learning, it is not necessary to create teacher data, and labor and cost can be reduced.

<Second extraction process>
In the second extraction step, each clustered region is defined as a rectangular region, and the rectangular region is extracted as a frame (S104).

As shown in FIG. 3C, since the clustering region 30 of the frame D is a rectangular region, the frame D is extracted as a frame by the second extraction step.

In the second extraction step, the “each region” refers to the above-mentioned cluster, and the clustered region is defined as a rectangular region. The method of creating the rectangular area can be changed as appropriate. For example, four points may be selected from the cluster and the arrangement of the four points may be taken into consideration and arranged so as to be a rectangle as a rectangular area, or a plurality of straight lines may be selected from the cluster and the straight lines become rectangular. The case where they are arranged in such a manner may be a rectangular area.

In this way, in the second extraction step in the present embodiment, for example, a rectangular area can be created from each clustered area, and the created rectangular area can be extracted as a frame. As a result, frame extraction can be performed more accurately than in the conventional technique.

In the present embodiment, the black region is clustered with respect to the remaining region not extracted as a frame by the first extraction step, and then the second extraction step is performed. However, the binarized cartoon After clustering the images, a second extraction step may be performed.

<Other examples>
Here, other examples in this embodiment will be described. According to this embodiment, frame extraction can be performed even in a cartoon image having no frame border. An example is shown in FIG. FIG. 4A is a diagram showing an example of a cartoon image, and FIG. 4B is a diagram showing a frame extracted from frames. According to the present embodiment, it is possible to easily and accurately extract frames not only for a cartoon image as shown in FIG. 2 but also for a cartoon image without a border as shown in FIG.

<Program>
The program of the present embodiment performs binarization processing for binarizing a cartoon image into black and white, contour extraction for the cartoon image, and extracting an area surrounded by a frame in the cartoon image as a frame. In the first extraction process to be performed, a clustering process for clustering black regions with respect to the remaining regions not extracted as the frames by the first extraction process in the cartoon image, and the clustered cartoon image. A second extraction process of determining a rectangular area and extracting the rectangular area as a frame is performed by a computer.

(Second embodiment)
Other embodiments of the frame extraction method according to the present invention will be described.
The flowchart of the frame extraction method of this embodiment is shown in FIG. The binarization step (S202), the first extraction step (S205), the clustering step (S206), and the second extraction step (S207) in the present embodiment are the same steps as those in the above embodiment. Therefore, the description is omitted.

<Normalization process>
In the normalization step, the width size of the cartoon image is normalized before the binarization step (S201). By performing the normalization, the accuracy of the first extraction step and the second extraction step can be improved. The width size of the cartoon image can be changed as appropriate, and can be, for example, 300pixel to 500pixel.

Normalization is not essential and can be omitted, but in that case, it is preferable that the width of the cartoon image is a predetermined size.

<Straight line judgment process>
In the straight line determination step, Hough transform is performed on the cartoon image after the binarization step and before the first extraction step, and it is determined whether or not a straight line is detected (S203). ..

When it is determined whether or not a straight line is determined (S205) and it is determined that a straight line is detected in the straight line determination step (YES in S205), the first extraction step (S205), the clustering step (S206), and the first The extraction step (S207) of 2 is performed. As a result, it is possible to prevent unnecessary steps from being performed, and it is possible to speed up the processing.

Among the cartoon images, there are a considerable number of single-frame images without the frame itself. In the case of a one-frame image, there are often no straight lines, so if there is no straight line in the cartoon image, it is determined that it is a one-frame image, and it is output as one frame without performing the steps after the first extraction step. (S209). As a result, the steps to be performed can be omitted, and the time required for frame extraction can be omitted.

For the detection of straight lines by Hough transform, a commonly used method can be used. For example, a straight line passing through a dot
ρ = x · cos θ + y · sin θ
It can be expressed as. Curves on the coordinate axes of ρ and θ can be drawn as many as the number of dots, and the intersection of the curves becomes a straight line passing through a plurality of dots.

In the straight line determination step, if a horizontal straight line having a length of 50% or more with respect to the width of the cartoon image is detected after the Hough transform, it can be determined that the cartoon image has a straight line. In the case of normalization, if a horizontal straight line having a length of 50% or more with respect to the width of the normalized cartoon image is detected, it can be determined that the cartoon image has a straight line.

<Sort process>
In the rearrangement step, the frames extracted by the first extraction step and the second extraction step are rearranged in the order in which the frames in the cartoon image are read (S208). By performing the sorting process, it becomes easier to read the frames extracted by a mobile terminal such as a smartphone.

The sorting process in this embodiment will be described with reference to FIGS. 6 and 7.
Here, a case where the cartoon image 10 is read from the upper left to the lower right will be described as an example. In the example shown in FIG. 6, A, B, C, D, and E are read in this order. The origin (0,0) is set, the horizontal direction is the x-axis, and the vertical direction is the y-axis. Further, the upper left points in each frame are a to e, and the respective values are x1 <x2 <x3 and y1 <y2 <y3.

At the stage of the first extraction step and the second extraction step, the sorting is not performed and the numbering (also referred to as numbering) is not performed. Numbering may be performed when the frames are extracted in the first extraction step and the second extraction step, but the order in which they are extracted may not correspond to the order in which the frames are read, so a sorting step is performed. Is preferable.

FIG. 7A is an example in which the sorting step is performed. The sorting process can be changed as appropriate, but when reading from the upper left to the lower right, first, the frames with the smallest y-axis values are arranged in order. Next, when there are frames having the same y-axis value, the frames having the same y-axis value are sorted in ascending order of the x-axis values.

By rearranging in this way, the order can be as shown in FIG. 7 (B). Here, as a result of sorting, numbers 1 to 5 are assigned as display numbers. In the sorting step, the display numbers may or may not be assigned as described above. The display number can be used as the display order when displaying the frames extracted by a mobile terminal such as a smartphone.

In the rearrangement step in the present embodiment, a step of adjusting the width of the extracted frames may be performed. By adjusting the width, the extracted frames can be easily read when displayed on a mobile terminal such as a smartphone. FIG. 7B shows an example of adjusting the width of each frame with reference to the width of frame B. For example, since the width of the frame A and the frame E is smaller than the width of the frame B, the width is widened to be the same width as the frame B. At this time, it is preferable that the aspect ratio of the frames A and E remains constant and the width is set to the reference size. By doing so, the picture in the frame can be displayed without being destroyed. Similarly, the widths of the frames C and D are narrowed to the reference size.

In the present embodiment, the width adjustment is not essential and may be performed as appropriate. If necessary, instead of adjusting the width, the part that is not the standard size may be painted black. For example, in the frame E whose width is smaller than the reference width (width of frame B), the portion insufficient in the reference width may be painted black without increasing the width. An example of this case is shown in FIG. 7 (C). By doing so, it is possible to prevent a small frame from being enlarged too much and becoming difficult to see. It should be noted that other colors such as white paint may be used instead of black paint.

The present embodiment is not limited to the cartoon image that is read from the upper left to the lower right, and may be a cartoon image that is read from the upper right to the lower left. An example in this case will be described with reference to FIG. FIG. 8 is a diagram similar to that of FIG. 6, in which frame extraction is performed on the cartoon image 10, rearrangement is performed, and the width is adjusted.

Here, in FIG. 8, the upper right points in each frame are a to e, and the respective values are x1 <x2 <x3 and y1 <y2 <y3. The sorting process can be changed as appropriate, but when reading from the upper right to the lower left, first, the frames with the smallest y-axis values are arranged in order. Next, when there are frames having the same y-axis value, the frames having the same y-axis value are sorted in descending order of the x-axis value.
By rearranging in this way, the order can be set along the direction in which the cartoon images are read, as in FIG. 6C.
As described above, in the present embodiment, even a cartoon image that is read from the upper right to the lower left can be frame-extracted and rearranged.

In the rearrangement process, frames that are too small may be removed as noise. As the "too small frame", for example, a case where a rectangular area is drawn in a part of the picture and this rectangular area is extracted is assumed.

The size of the frame to be removed as noise can be changed as appropriate. For example, a frame having a width of 1/5 or less of the width of the cartoon image may be removed. When normalized, frames whose width is 1/5 or less of the width of the normalized cartoon image may be removed. By removing it as noise, unnecessary images are removed and the cartoon image becomes easier to read.

10 cartoon image 20 border 30 clustering area

Claims (9)

The binarization process that binarizes the cartoon image into black and white,
The first extraction step of extracting the outline of the cartoon image and extracting the area surrounded by the frame in the cartoon image as a frame, and
A clustering step of clustering black regions with respect to the remaining regions not extracted as the frames by the first extraction step in the cartoon image.
A frame extraction method comprising: a second extraction step of extracting each of the clustered regions as a rectangular region and extracting the rectangular region as a coma. The frame extraction method according to claim 1, wherein the clustering step is performed by DBSCAN (Density based spatial clustering of applications with noise). The frame extraction method according to claim 1 or 2, wherein the binarization step is converted into an HSV color space and binarized. The frame extraction method according to any one of claims 1 to 3, wherein the contour extraction is performed by a region growth method or a region integration method. It has a straight line determination step of determining whether or not a straight line is detected by performing a Hough transform on the cartoon image after the binarization step and before the first extraction step.
The method according to any one of claims 1 to 4, wherein when it is determined that a straight line is detected in the straight line determination step, the first extraction step, the clustering step, and the second extraction step are performed. Frame extraction method. The frame extraction method according to any one of claims 1 to 5, further comprising a normalization step of normalizing the width size of the cartoon image before the binarization step. Claims 1 to 6 include a rearrangement step of rearranging the frames extracted by the first extraction step and the second extraction step so as to read the frames in the cartoon image. The frame extraction method described in any of the above. The sorting step is characterized in that when the frames extracted in the first extraction step and the second extraction step are 1/5 or less of the width of the cartoon image, they are excluded from the sorting. The frame extraction method according to claim 7. Binarization process that binarizes cartoon images into black and white,
The first extraction process of performing contour extraction on the cartoon image and extracting the area surrounded by the frame in the cartoon image as a frame, and
A clustering process for clustering black regions with respect to the remaining regions not extracted as the frames by the first extraction process in the cartoon image.
A program characterized by having a computer execute a second extraction process in which each clustered area is a rectangular area and the rectangular area is extracted as a frame.

Images