JP7333526B2 - Comic machine translation device, comic parallel database generation device, comic machine translation method and program - Google Patents

Description

Application of Article 30, Paragraph 2 of the Patent Act Aug. 28, 2018, Innovative Technologies 2018 Adopted Technology Announcement https://www. dcaj. or. jp/news/2018/08/innovative-technologies-2018. html

The present invention relates to a machine translation device for comics, a parallel translation database generation device for comics, a machine translation method for comics, and a program.

In recent years, with the improvement of computer processing power, methods for machine-translating sentences written in one natural language into sentences in another natural language have attracted attention, and various machine translation methods and devices have been proposed ( For example, see Patent Document 1).

JP 2019-96303 A

By the way, currently, the translation of manga is done by translators. In translation by a translator, translation is performed flexibly according to the situation such as the story of the manga, so highly accurate translation is possible. On the other hand, translations by translators are relatively expensive, so even if they are translated and published in other regions or countries, copyright holders and publishers may not be able to obtain sufficient profits. Also, due to the high cost of translation, there are cases where overseas publishers refuse to purchase translation and publishing rights, and official translations are not published overseas. Translation by a translator takes a relatively long time for translation work. Therefore, before the official copyright holder translates the manga and publishes it in another region or country, or while the official translation cannot be published, low-quality pirated copies are sold in the market. It is possible.

On the other hand, when machine translation is used, relatively inexpensive and rapid translation is possible. Therefore, when machine translation is used, it is possible to eliminate the shortcomings that occur in translation by translators. However, as a result of investigation by the present inventors, it has been found that highly accurate machine translation of comics cannot be performed even if existing machine translation methods are employed.

The present invention has been made in order to solve the above-mentioned problems, and is a machine translation apparatus for comics, a machine translation method and program for comics, and parallel translation of comics therefor, capable of machine-translating comics with high accuracy. An object of the present invention is to provide a database generation device.

The gist of the present invention provided to solve the above problems is mainly as follows.
(1) a character area detection unit that detects a character area from a first natural language image that constitutes a cartoon created using a first natural language;
a character information estimation unit for estimating character information of the first natural language from the character area;
a machine translation unit that translates the character information in the first natural language into character information in the second natural language by machine translation using a bilingual database;
The bilingual database includes a first natural language character region included in the first natural language reference image of the reference cartoon and a second natural language character region included in the second natural language reference image of the reference cartoon and corresponding to the first natural language character region. bilingual information automatically generated by detecting a language character region and extracting first natural language character information existing in the first natural language character region and second natural language character information existing in the second natural language character region A machine translation device for cartoons, including
(2) the character information estimation unit estimates the character information from the character region using a trained character recognition model;
The learned character recognition model is machine-learned using teacher data including one or more font images of the first natural language and an image obtained by adding deformation, tilt and/or noise to the font image, The cartoon machine translation device according to (1), which is generated.
(3) extracting the first natural language character information from the first natural language character region is performed by estimating the first natural language character information using a trained character recognition model;
The learned character recognition model is machine-learned using teacher data including one or more font images of the first natural language and an image obtained by adding deformation, tilt and/or noise to the font image, The cartoon machine translation device according to (1) or (2), which is generated.
(4) extraction of the second natural language character information from the second natural language character region is performed by estimating the second natural language character information using a trained second natural language character recognition model;
The trained second natural language character recognition model includes one or more second natural language font images of the second natural language, and an image obtained by adding deformation, tilt and / or noise to the second natural language font image. The machine translation device for comics according to any one of (1) to (3), which is generated by machine learning using teacher data including the training data.
(5) The comic machine translation device according to any one of (1) to (4), wherein the first natural language is Japanese.
(6) Furthermore, an image generation unit that generates a second natural language image by adding the character information in the second natural language translated by the machine translation unit as an image to the first natural language image, The machine translation device for comics according to any one of (1) to (5).
(7) a first natural language character region included in the first natural language reference image of the reference cartoon and a second natural language character included in the second natural language reference image of the reference cartoon and corresponding to the first natural language character region; a character area detection unit for detecting an area;
a parallel translation information extraction unit for extracting first natural language character information existing in the first natural language character area and second natural language character information existing in the second natural language character area;
A parallel translation database generation apparatus for comics, comprising a storage unit for storing at least part of the first language character information and at least part of the second language character information as parallel translation information.
(8) extracting the first natural language character information from the first language character region by estimating the first natural language character information using a trained character recognition model;
The learned character recognition model is machine-learned using teacher data including one or more font images of the first natural language and an image obtained by adding deformation, tilt and/or noise to the font image, The bilingual database generation device for comics according to (7), which is generated.
(9) extraction of the second natural language character information from the second language character region is performed by estimating the second natural language character information using a trained second natural language character recognition model;
The trained second natural language character recognition model includes one or more second natural language font images of the second natural language, and an image obtained by adding deformation, tilt and / or noise to the second natural language font image. The bilingual comics database generation device according to (8), which is generated by machine learning using teacher data including the data.
(10) by the processor,
Detecting a character region from a first natural language image that constitutes a cartoon created using a first natural language;
estimating character information in the first natural language from the character area; and translating the character information in the first natural language into character information in a second natural language by machine translation using a bilingual database. , and
The bilingual database includes a first natural language character region included in the first natural language reference image of the reference cartoon and a second natural language character region included in the second natural language reference image of the reference cartoon and corresponding to the first natural language character region. bilingual information automatically generated by detecting a language character region and extracting first natural language character information existing in the first natural language character region and second natural language character information existing in the second natural language character region Machine translation methods for cartoons, including
(11) a computer;
a character area detection unit that detects a character area from a first natural language image that constitutes a cartoon created using a first natural language;
a character information estimation unit for estimating character information of the first natural language from the character area;
a machine translation unit that translates the character information in the first natural language into character information in the second natural language by machine translation using a bilingual database;
The bilingual database includes a first natural language character region included in the first natural language reference image of the reference cartoon and a second natural language character region included in the second natural language reference image of the reference cartoon and corresponding to the first natural language character region. bilingual information automatically generated by detecting a language character region and extracting first natural language character information existing in the first natural language character region and second natural language character information existing in the second natural language character region A program for functioning as a machine translation device for comics, including

As described above, according to the present invention, it is possible to provide a comics machine translation device, a comics machine translation method and program, and a comics bilingual database generation device for these that are capable of highly accurate comics machine translation.

1 is a block diagram illustrating the functional configuration of a machine translation device for comics according to an embodiment of the present invention; FIG. 2 is a block diagram illustrating the functional configuration of a character recognition model generation device that generates a learned character recognition model used in the cartoon machine translation device shown in FIG. 1; FIG. 1 is a block diagram illustrating the functional configuration of a bilingual comic database generation device according to an embodiment of the present invention; FIG. 3 is a diagram for explaining a process of generating teacher data by the character recognition model generation device shown in FIG. 2; FIG. FIG. 4 is a diagram for explaining how the bilingual comic database generation device shown in FIG. 3 captures and associates images of reference comics; FIG. 4 is a diagram for explaining an example of a method of detecting a character region by the bilingual comic database generation device shown in FIG. 3; FIG. 4 is a diagram for explaining an example of a method of detecting a character region by the bilingual comic database generation device shown in FIG. 3; FIG. 4 is a diagram for explaining an example of a method of specifying a character string by the comic bilingual database generation device shown in FIG. 3; FIG. 4 is a diagram for explaining an example of a method of specifying a character string by the comic bilingual database generation device shown in FIG. 3; 2 is a diagram for explaining an example of a method of extracting character information by the comic machine translation apparatus shown in FIG. 1; FIG. 2 is a diagram for explaining an example of a method for generating a comic image in a second natural language by the comic machine translation device shown in FIG. 1; FIG. 4 is a flowchart illustrating a character recognition model generation method according to an embodiment of the present invention; 4 is a flow chart illustrating a method for generating a parallel translation database of comics according to an embodiment of the present invention; 4 is a flow chart illustrating a method for machine translation of comics according to an embodiment of the present invention; 2 is a block diagram showing an example of the hardware configuration of the machine translation apparatus for comics shown in FIG. 1; FIG.

Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.
<1. Study by the present inventors>
First, prior to describing embodiments of the present invention, studies by the inventors will be described. As described above, when the inventors of the present invention have studied, even if the existing machine translation method is adopted, it is not possible to perform highly accurate machine translation of comics.

The inventors considered the following possibilities for this reason. A bilingual database (corpus) is necessary for accurate machine translation, but existing bilingual databases may not be suitable for machine translation of comics. In particular, comics are mainly written as sentences, unlike ordinary sentences, in which the lines of characters in the comics are written. Such lines often have no correspondence between subjects and predicates, or have broken sentences. Furthermore, the use of symbols such as question marks, exclamation marks, long notes, and arrogances also varies greatly depending on the author.

In view of such possibilities, the present inventors conducted extensive research and found that highly accurate machine translation is possible by using a specific bilingual database containing bilingual information on comics, leading to the present invention. Ta.

<2. System Overview>
First, a cartoon machine translation device according to the present embodiment (hereinafter also simply referred to as a "machine translation device") and a manga bilingual database generation device (hereinafter simply referred to as a "bilingual database generation device"). An overview of the machine translation system will be explained. FIG. 1 is a block diagram illustrating the functional configuration of a machine translation device according to one embodiment of the present invention, and FIG. 2 is a character recognition system that generates a learned character recognition model used in the machine translation device described in FIG. FIG. 3 is a block diagram for explaining the functional configuration of the model generation device, and FIG. 3 is a block diagram for explaining the functional configuration of the bilingual database generation device according to one embodiment of the present invention.

The machine translation device 100 shown in FIG. 1 is a device for machine-translating Japanese character information in a comic created using Japanese as a first natural language into English character information as a second natural language.

Also, the character recognition model generation device 200 shown in FIG. 2 is a device for generating a trained character recognition model for estimating natural language character information existing in a cartoon image. The bilingual database generation device 300 shown in FIG. 3 is a device that automatically generates a bilingual database by extracting bilingual information in Japanese as the first natural language and English as the second natural language from the translated reference comics. is.

As shown in FIG. 1, the machine translation device 100, the character recognition model generation device 200, and the bilingual database generation device 300 can communicate with each other via a network 400 to form a machine translation system for comics. are doing. Here, the learned character recognition model generated by the character recognition model generation device 200 is used in the machine translation device 100 and the bilingual database generation device 300. FIG. Also, the parallel translation database generated by the parallel translation database generation device 300 is used in the machine translation performed by the machine translation device 100 . Therefore, for ease of explanation, the character recognition model generation device 200, the bilingual database generation device 300, and then the machine translation device 100 will be described in this order.

In this embodiment, as an example, a case where the first natural language is Japanese and the second natural language is English will be described. However, the first natural language and the second natural language are not limited to these natural languages, and can be any natural language written using characters.

(2.1. Character recognition model generation device 200)
As shown in FIG. 2 , the character recognition model generation device 200 has a teacher data generation section 210 , a machine learning section 230 and a storage section 250 .

The teacher data generation unit 210 generates teacher data used by the machine learning unit 230, which will be described later. Specifically, as shown in FIG. 4, the training data generation unit 210 prepares an image including character images 411 for a plurality of types of fonts, that is, a font image group 410. A processed character image group 420 including a processed character image 421 to which Here, the font image group 410 may include a unit composed of a plurality of character images 411, such as a word or sentence composed of character images 411, or a character string or line unit. In this case, it is also possible to generate a character recognition model for each of these units. Next, the processed character images 421 in the processed font image group 420 and the character images 411 in the font image group 410 are combined to generate a learning character image group 430 including a plurality of learning character images 431 . For the font image group 410, data saved in the storage unit 250 can be used. Various data such as the generated learning character image group 430 may be transmitted to and stored in the storage unit 250 as necessary.

Note that in the present disclosure, the learning character image group 430 is not limited to the embodiment illustrated in FIG. may further include a character image, such as a character image of the character recognition data set.

Further, the teacher data generation unit 210 prepares character information of the learning character image 431 corresponding to each learning character image 431 in the learning character image group 430 as correct data. When the learning character image group 430 includes character images other than the processed character image 421 and the character image 411, the character information corresponding to the correct data is automatically or manually added to the character image as necessary. may Then, the teacher data generation unit 210 generates teacher data including the learning character image group 430, which is the example data, and the correct answer data. The generated teacher data may be directly output to the machine learning section 230 or may be stored in the storage section 250 .

The machine learning unit 230 performs machine learning using the teacher data generated by the teacher data generation unit 210 to generate a learned character recognition model. Techniques (algorithms) used in machine learning are not particularly limited, and various techniques available to those skilled in the art can be used alone or in combination. Examples of the method include a ResNet (residual network) image recognition module, CRNN (convolutional recurrent neural network), convolutional neural networks such as LSTM including Bi-Directional LSTM (Long short-term memory), recursive neural networks, etc. and combinations thereof. The trained character recognition model may recognize character information in units of characters, or may recognize character information in units composed of multiple characters such as words, sentences, character strings, or lines. It may be something to do. The machine learning unit 230 stores the generated learned character recognition model in the storage unit 250, and transmits it to the machine translation device 100 and the bilingual database generation device 300 as necessary.

The storage unit 250 stores various information necessary for generating teacher data and learned character recognition models, and stores the generated teacher data and learned character recognition models. Various types of information necessary for generating teacher data and trained character recognition models include font information (character images and character information), deformation, inclination, information for processing necessary for adding noise, and information for machine learning. Various types of information related to the method can be mentioned.

(2.2. Bilingual database generation device 300)
The bilingual database generation device 300 extracts bilingual information in Japanese and English from translated reference comics, and automatically generates a bilingual database. As shown in FIG. 3 , the translation database generation device 300 has a reference image acquisition section 310 , a character area detection section 330 , a translation information extraction section 350 and a storage section 370 .

The reference image acquisition unit 310 acquires images in the translated reference comics, and associates the Japanese reference images with the English reference images. Here, as reference cartoons, there are a Japanese reference cartoon 500J created in Japanese as the first natural language and an English reference cartoon 500E created in English as the second natural language. If there is, it is not particularly limited, and any one can be used. The Japanese reference comics 500J and the English reference comics 500E do not have to be accompanied by electronic text data. In this embodiment, the character area detection unit 330 and the parallel translation information extraction unit 350, which will be described later, can accurately extract Japanese character information and English character information. At least one of the Japanese reference cartoon 500J and the English reference cartoon 500E is preferably translated by a translator. This enables more accurate machine translation.

As shown in FIG. 5, the reference image acquisition unit 310 first captures the images of each page of the Japanese reference cartoon 500J and the English reference cartoon 500E as Japanese reference images _510Jn and English reference images _510Em , respectively. where n and m are natural numbers). Here, comics are generally represented by pictures and characters, and even when translated, the original comics and the translated comics correspond to each other in terms of the contents of each page. However, depending on the composition of the cover and table of contents, etc., the corresponding position (the number of pages from the cover) may differ between the original manga and the translated manga.

Therefore, the reference image acquisition unit 310 associates the captured Japanese reference image _510Jn of the captured Japanese reference cartoon 500J with the English reference image 510Em of the English reference cartoon _500E . Although the method for matching is not particularly limited, it can be performed, for example, by calculating local feature amounts. Specifically, a detector such as the AKAZE detector detects the local descriptors of the Japanese reference image 510J _n and the English reference image 510E _m . The homography matrices of the two pages of the Japanese reference image _510Jn and the English reference image _510Em are then calculated. The similarity between the Japanese reference image 510J _n and the English reference image 510E _m is then determined by counting the pairs of local features that are determined to be inferior.

Based on this result, the reference image acquisition unit 310 associates the Japanese reference image _510Jn with the English reference image _510Em . For example, in FIG. 5, Japanese reference image _510J1 maps to English reference image _510E1 , Japanese reference image _510J2 maps to English reference image _510E3 , and Japanese reference image _510J3 maps to English reference image _510E4 . be done. The reference image acquisition unit 310 outputs the pair of the associated Japanese reference image 510J _n and English reference image 510E _m to the character area detection unit 330 and/or stores them in the storage unit 370 . Note that the reference image acquiring unit 310 may perform image correction on the Japanese reference image _510Jn and the English reference image _510Em to be stored or output, as necessary, so that their shapes match. For example, the reference image acquisition unit 310 may unify the sizes of the Japanese reference image _510Jn and the English reference image _510Em , or may adjust the distortion or tilt present in the Japanese reference image _510Jn and the English reference image _510Em . may be corrected.

As shown in FIGS. 6 and 7, the character region detection unit 330 detects the Japanese character regions 520J and 530J included in the Japanese reference image _510Jn and the Japanese character regions included in the English reference image _510Em . English character areas 520E and 530E are detected.

The Japanese character areas 520J and 530J and the English character areas 520E and 530E may be detected by any method, but can be detected by, for example, an object detector. The object detector is not particularly limited. For example, R-CNN (Regions with Convolutional Neural Networks) system object detector, SSD (Single Shot MultiBox Detector), YOLO (You Look Only Once), M2Det, etc. can be used.

In addition, only the character area of one of the Japanese character areas 520J and 530J and the English character areas 520E and 530E is detected, and based on the position of the detected character area, the character area existing in the other natural language is specified. You may Since the character information of a comic is normally arranged at the same position in the picture of the comic even after translation, it is possible to detect the character region with high accuracy. Furthermore, the time required for the object detection process is saved. For example, in FIG. 7, Japanese character areas 520J and 530J are detected by object detection processing, and then portions corresponding to the Japanese character areas 520J and 530J in the English reference image _510Em are identified as English character areas 520E and 530E. can do.

The parallel translation information extraction unit 350 extracts Japanese character information 521J and 531J and English character information 521E and 531E from the Japanese character areas 520J and 530J and the English character areas 520E and 530E detected by the character area detection unit 330, respectively. . Specifically, parallel translation information extraction unit 350 identifies character strings that exist in Japanese character regions 520J and 530J and English character regions 520E and 530E, and extracts character information that exists in the character strings.

Character strings can be specified as follows, for example. Character strings in cartoons are usually written in black and surrounded by white. Therefore, when detecting a vertically written character string, as shown in FIG. Determine if pixel sites are present and identify columns 543 and column groups 545 where black pixel sites are present. After that, the parallel translation information extraction unit 350 removes the column 543 that is small as a character string, and identifies the column group 545 as the character string 545 . After that, the parallel translation information extraction unit 350 divides the character string 545 as necessary to obtain character images 547 and 549 .

Similarly, when detecting a horizontally written character string, as shown in FIG. Determine if pixel sites exist and identify rows and groups of rows 553 where black pixel sites exist. After that, the parallel translation information extraction unit 350 removes lines that are small as character strings, and identifies the line group 553 as the character strings 553 .

It should be noted that the present disclosure is not limited to the embodiments illustrated in FIGS. 8 and 9, and the parallel translation information extraction unit 350 directly detects the Japanese character regions 520J and 530J by performing object detection in units of character strings. and English character regions 520E and 530E. In this case, the character area detection section 330 can be omitted. Even in this case, the object detector is not particularly limited. -CNN (Regions with Convolutional Neural Networks) system object detector, SSD (Single Shot MultiBox Detector), YOLO (You Look Only Once), M2Det, etc. can be used.

Next, parallel translation information extraction unit 350 extracts Japanese character information 521J and 531J and English character information 521E and 531E from the character strings specified in Japanese character areas 520J and 530J and English character areas 520E and 530E. The Japanese character information 521J, 531J and the English character information 521E, 531E can be extracted using the trained Japanese character recognition model and the trained English character recognition model generated by the character recognition model generating device 200. FIG.

Further, the parallel translation information extraction unit 350 converts the extracted Japanese character information 521J and the corresponding English character information 521E, and the Japanese character information 531J and the corresponding English character information 531E into a pair of sentence parallel translation information. , and output to the bilingual database 371, which will be described later.

The storage unit 370 stores various kinds of information necessary for processing performed in the parallel translation information extraction unit 350 and also stores a parallel translation database 371 . Various types of information necessary for the processing performed in the parallel translation information extraction unit 350 include, for example, a trained Japanese character recognition model and a trained English character recognition model.

Also, the parallel translation database 371 includes at least the parallel translation information extracted by the parallel translation information extraction unit 350 . The parallel translation database 371 may also have parallel translation information obtained without using the parallel translation information extraction unit 350 .

(2.3. Machine translation device 100)
As shown in FIG. 1, the machine translation apparatus 100 includes a machine translation learning unit 110, a cartoon image acquisition unit 120, a character area detection unit 130, a character information estimation unit 140, a machine translation unit 150, and an image generation unit. 160 and a storage unit 170 .

The machine translation learning unit 110 uses the parallel translation database 371 generated by the parallel translation information extraction unit 350 to learn a machine translation model. In this embodiment, a neural machine translator is used as the machine translation model. The neural machine translator is not particularly limited, but for example, Attention-based Encoder-Decoder, Convolutional Sequence to Sequence, Transformer, etc. can be used. Machine translation learning section 110 outputs the learned machine translation model to storage section 170 .

As shown in FIG. 10, the comic image acquisition unit 120 acquires each page of the comic to be translated as a comic image _600Jn (n is a natural number). The comic image acquisition section 120 may perform image correction on the comic image _600Jn as necessary. For example, the comic image acquisition unit 120 may unify the sizes of the comic images _600Jn between pages, or may correct distortions and tilts present in the comic images _600Jn . The method of obtaining the comic image 600Jn is not particularly limited, and it may be provided as electronic data or may be obtained by scanning a paper medium comic. It is assumed here that the manga to be translated is created in Japanese and is scheduled to be translated into English.

Character area detection section 130 detects character area 610 from comic image _600Jn . The character area 610 can be detected by any method, like the character area detection unit 330. For example, the character area 610 can be detected by an object detector. The object detector is not particularly limited. For example, R-CNN (Regions with Convolutional Neural Networks) system object detector, SSD (Single Shot MultiBox Detector), YOLO (You Look Only Once), M2Det, etc. can be used. Note that the character region detection unit 130 may directly detect and identify a character string existing in the comic image 600J _n as the character region 610 .

Character information estimation unit 140 estimates Japanese character information 620J from detected character area 610 . Specifically, character information estimation unit 140 identifies a character string that exists in character region 610, and extracts character information 620J that exists in the character string. Character strings can be identified by the same technique as the technique used by the parallel translation information extraction unit 350 . Note that when the character region detection unit 130 directly detects the character string as the character region 610, the character string identification by the character information estimation unit 140 can be omitted.

Next, character information estimation unit 140 extracts character information 620J from the character string identified in character area 610 . The character information 620J can be extracted using the learned Japanese character recognition model generated by the character recognition model generation device 200. FIG.

The machine translation unit 150 translates the Japanese character information 620J estimated by the character information estimation unit 140 into English character information 620E by machine translation. Machine translation in the machine translation unit 150 is performed by a machine translation model learned in the machine translation learning unit 110 . This machine translation model is learned using the parallel translation information of the parallel translation database 371, and is capable of accurate translation.

The image generation unit 160 adds the English character information 620E translated by the machine translation unit 150 to the comic image _600Jn created in Japanese as an image to generate an English comic image _600En . Specifically, as shown in FIG. 11, the image generation unit 160 makes the character area 610 in the comic image _600Jn a white area, and then adds the character information 620E as an image. Note that the area to which the character information 620 is added may correspond to the area where the character information 620J exists, and does not have to match the area where the character information 620J exists.

The storage unit 170 can input/output to/from each unit, and stores information necessary for processing of each unit. For example, the storage unit 170 stores a comic image 600J _n to be processed by the comic image acquiring unit 120, a learned character recognition model used in the character information estimating unit 140, a neural machine translation generated in the machine translation learning unit 110. Store the model, etc.

<3. Manga Machine Translation Method, Character Recognition Model Generating Method, and Bilingual Database Generating Method>
Next, the operations of the machine translation apparatus 100, the character recognition model generation apparatus 200, and the bilingual database generation apparatus 300 described above will be described together with the cartoon machine translation method, character recognition model generation method, and bilingual database generation method. As in the explanation of the apparatus configuration above, after explaining the character recognition model generation method and the bilingual database generation method, the machine translation method for comics will be explained.

(3.1 Character recognition model generation method)
In the character recognition model generation method according to the present embodiment, a processor performs machine learning using teacher data including one or more font images of a natural language and an image obtained by adding deformation, tilt, and/or noise to the font image. generating a trained character recognition model by doing.
FIG. 12 is a flow chart of the character recognition model generation method according to this embodiment.

In this embodiment, first, teacher data is created by the teacher data generator 210 . Specifically, the training data generation unit 210 acquires a font image group 410 including character images 411 for multiple types of fonts from the storage unit 250 (S101).
Next, the teacher data generation unit 210 generates a processed character image group 420 including processed character images 421 obtained by adding deformation, tilt and/or noise to the character images 411 in the font image group 410 (S103).

Next, the teacher data generation unit 210 combines the processed character images 421 in the processed font image group 420 and the character images 411 in the font image group 410 to create a learning character image group including a plurality of learning character images 431. 430 is generated (S105).

Finally, the machine learning unit 230 performs machine learning using the training data generated by the training data generation unit 210 to generate a learned character recognition model (S107).

When the trained character recognition model obtained as described above is used, it is possible to recognize characters in cartoon images corresponding to a wide variety of fonts and their modified images. In general, comics differ from ordinary documents in that texts are written together with pictures, and since comics have large visual elements, a wide variety of fonts are used even on the same page. In addition, a wide variety of fonts may be further modified in order to realistically describe the conversations and situations in the manga. As a result of studies by the present inventors, it is difficult to accurately recognize sentences written in such cartoons by general optical character recognition means (OCR: Optical Character Recognition). On the other hand, when the learned character recognition model obtained above is used, character information can be accurately recognized even in comics.

(3.2 Bilingual database generation method)
In the bilingual database generation method according to the present embodiment, the first natural language character region included in the first natural language reference image of the reference cartoon and the second natural language reference image of the reference cartoon are included in the first natural language character region by the processor. detecting a second natural language character region corresponding to the language character region;
and extracting first natural language character information present in the first natural language character region and second natural language character information present in the second natural language character region.
FIG. 13 is a flow chart for explaining a method for generating a parallel translation database of comics according to this embodiment. A case where the first natural language is Japanese and the second natural language is English will be described below.

First, prior to each step, the reference image acquisition unit 310 first captures images of each page of the Japanese reference cartoon 500J and the English reference cartoon 500E as a Japanese reference image _510Jn and an English reference image _510Em , respectively. where n and m are natural numbers) (S201).
Next, the reference image acquisition unit 310 associates the captured Japanese reference image _510Jn of the captured Japanese reference comic 500J with the English reference image _510Em of the English reference comic 500E (S203).

Next, the character area detection unit 330 detects Japanese character areas 520J and 530J included in the Japanese reference image _510Jn and English character areas _520E and 530E included in the English reference image 510Em and corresponding to the Japanese character areas. Detect (S205). Here, the character region detection unit 330 may directly detect the character strings included in the Japanese reference image _510Jn and the English reference image 510Em as the Japanese character regions 520J and _530J and the English character regions 520E and 530E. .

Next, from the Japanese character areas 520J and 530J and the English character areas 520E and 530E detected by the character area detection section 330, the parallel translation information extraction section 350 extracts the Japanese character information 521J and 531J and the English character information 521E and 531E, respectively. Extract (S207). Specifically, parallel translation information extraction unit 350 identifies character strings existing in Japanese character areas 520J and 530J and English character areas 520E and 530E, and extracts character information existing in the character strings. When the character region detection unit 330 directly identifies the character strings as the Japanese character regions 520J and 530J and the English character regions 520E and 530E, the character string identification by the parallel translation information extraction unit 350 can be omitted. The Japanese character information 521J, 531J and the English character information 521E, 531E can be extracted using the trained Japanese character recognition model and the trained English character recognition model generated in the character recognition model generating method.

Finally, the bilingual information extraction unit 350 translates the extracted Japanese character information 521J and the corresponding English character information 521E, and the Japanese character information 531J and the corresponding English character information 531E into a pair of sentences. It is specified as information (S209) and output to the bilingual database 371, which will be described later. By performing the above, the bilingual database 371 can be generated.

According to the above, it is possible to automatically generate a parallel translation database of comics accurately and quickly. In particular, the above-described bilingual database generation method does not require text data of the bilingual comics, and only requires the existence of comics with bilingual translations. It is easy to generate a large number of bilingual databases at low cost. In particular, when the trained character recognition model generated by the character recognition model generating method is used, it is possible to recognize character information with high accuracy in response to a wide variety of fonts peculiar to comics and variations thereof. . As a result, it is possible to automatically generate a highly accurate bilingual database.

(3.3 Machine translation method for manga)
The machine translation method for comics according to the present embodiment is performed by a processor,
Detecting a character region from a first natural language image that constitutes a cartoon created using a first natural language;
estimating character information in the first natural language from the character area; and translating the character information in the first natural language into character information in a second natural language by machine translation using a bilingual database. , including executing
FIG. 14 is a flow chart for explaining the machine translation method for comics according to the present embodiment. A case where the first natural language is Japanese and the second natural language is English will be described below.

First, the comic image acquiring unit 120 acquires each page of the comic to be translated as a Japanese comic image _600Jn (n is a natural number) (S301). Next, the character area detection unit 130 detects the character area 610 from the comic image _600Jn (S303).

Next, the character information estimation unit 140 estimates Japanese character information 620J from the detected character area 610 (S305). The character information 620J can be extracted using the learned Japanese character recognition model generated by the character recognition model generation device 200. FIG.

Next, the machine translation unit 150 translates the Japanese character information 620J estimated by the character information estimation unit 140 into English character information 620E by machine translation (S307). Machine translation is performed by a machine translation model learned by the machine translation learning unit 110 .

Finally, the image generation unit 160 adds the English character information 620E translated by the machine translation unit 150 to the comic image 600J _n created in Japanese as an image to generate the English comic image 600E _n ( S307).

According to the above, by performing machine translation using the bilingual database of comics described above, it is possible to perform machine translation of comics accurately and quickly. In particular, when the trained character recognition model generated by the character recognition model generating method is used, it is possible to recognize character information with high accuracy in response to a wide variety of fonts peculiar to comics and variations thereof. . As a result, the accuracy of machine translation is further improved.

<4. Hardware configuration example>
Finally, with reference to FIG. 15, the hardware configuration of the machine translation device 100 for comics according to the present embodiment will be described. FIG. 15 is a block diagram showing an example of the hardware configuration of the machine translation device for comics according to this embodiment. Note that the information processing apparatus (computer) 900 shown in FIG. 15 can implement the machine translation apparatus 100 shown in FIG. 1, for example. Information processing by the machine translation apparatus 100 according to this embodiment is realized by cooperation between software and hardware described below. The same applies to the bilingual comic database generation device 300 and the character recognition model generation device 200 .

As shown in FIG. 15, an information processing apparatus 900 includes a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, a RAM (Random Access Memory) 903, and a host bus 904a. The information processing device 900 also includes a bridge 904 , an external bus 904 b , an interface 905 , an input device 906 , an output device 907 , a storage device 908 , a drive 909 , a connection port 911 and a communication device 913 . The information processing apparatus 900 may have a processing circuit such as an electric circuit, a DSP (Digital Signal Processor), or an ASIC (Application Specific Integrated Circuit) instead of or together with the CPU 901 .

The CPU 901 is an example of a processor, functions as an arithmetic processing device and a control device, and controls general operations within the information processing device 900 according to various programs. Alternatively, the CPU 901 may be a microprocessor. The ROM 902 stores programs, calculation parameters, and the like used by the CPU 901 . The RAM 903 temporarily stores programs used in the execution of the CPU 901, parameters that change as appropriate during the execution, and the like.

The CPU 901, ROM 902 and RAM 903 are interconnected by a host bus 904a including a CPU bus. The host bus 904a is connected via a bridge 904 to an external bus 904b such as a PCI (Peripheral Component Interconnect/Interface) bus. Note that the host bus 904a, the bridge 904 and the external bus 904b do not necessarily have to be configured separately, and these functions may be implemented in one bus.

The input device 906 is realized by a device such as a mouse, keyboard, touch panel, button, microphone, switch, lever, etc., through which information is input by the user. Further, the input device 906 may be, for example, a remote control device using infrared rays or other radio waves, or an external connection device such as a mobile phone or PDA (Personal Digital Assistant) compatible with the operation of the information processing device 900. may be Furthermore, the input device 906 may include, for example, an input control circuit that generates an input signal based on information input by the user using the above input means and outputs the signal to the CPU 901 . A user of the information processing apparatus 900 can input various data to the information processing apparatus 900 and instruct processing operations by operating the input device 906 .

The output device 907 is formed by a device capable of visually or audibly notifying the user of the acquired information. Examples of such devices include CRT (Cathode Ray Tube) display devices, liquid crystal display devices, plasma display devices, EL (Electroluminescence display) display devices, laser projectors, LED projectors, lamps and other display devices, and speakers and headphones for sound. There is an output device, etc. The output device 907 outputs, for example, results obtained by various processes performed by the information processing device 900 . Specifically, the output device 907 visually displays the results obtained by various processes performed by the information processing device 900 in various formats such as text, image, table, and graph. On the other hand, when an audio output device is used, an audio signal composed of reproduced audio data, acoustic data, etc. is converted into an analog signal and audibly output.

The storage device 908 is a data storage device formed as an example of the storage unit of the information processing device 900 . The storage device 908 is implemented by, for example, a magnetic storage device such as a HDD (Hard Disk Drive), a semiconductor storage device, an optical storage device, a magneto-optical storage device, or the like. The storage device 908 may include a storage medium, a recording device that records data on the storage medium, a reading device that reads data from the storage medium, a deletion device that deletes data recorded on the storage medium, and the like. The storage device 908 stores programs executed by the CPU 901, various data, and various data acquired from the outside. Storage device 908 may perform the functions of storage unit 170 shown in FIG. 1, for example.

The drive 909 is a reader/writer for storage media, and is built in or externally attached to the information processing apparatus 900 . The drive 909 reads out information recorded on a removable storage medium such as a mounted magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, and outputs the information to the RAM 903 . Drive 909 can also write information to a removable storage medium.

The connection port 911 is an interface connected to an external device, and is a connection port with an external device capable of data transmission by, for example, USB (Universal Serial Bus).

The communication device 913 is, for example, a communication interface formed by a communication device or the like for connecting to the network 920 . The communication device 913 is, for example, a communication card for wired or wireless LAN (Local Area Network), LTE (Long Term Evolution), Bluetooth (registered trademark), or WUSB (Wireless USB). Further, the communication device 913 may be a router for optical communication, a router for ADSL (Asymmetric Digital Subscriber Line), a modem for various types of communication, or the like. This communication device 913 can transmit and receive signals and the like to and from the Internet and other communication devices, for example, according to a predetermined protocol such as TCP/IP.

Note that the network 920 is a wired or wireless transmission path for information transmitted from devices connected to the network 920 . For example, the network 920 may include a public network such as the Internet, a telephone network, a satellite communication network, various LANs (Local Area Networks) including Ethernet (registered trademark), WANs (Wide Area Networks), and the like. Network 920 may also include a dedicated line network such as IP-VPN (Internet Protocol-Virtual Private Network).

<5. Program and recording medium>
The machine translation device 100, the bilingual comic database generation device 300, the character recognition model generation device 200, and various methods using them according to the present embodiment have been described above.
Therefore, the present invention, in another aspect, also relates to a program for causing a computer to function as these devices. A recording medium storing the program is also provided.

<6. Summary>
An embodiment of the present invention has been described above with reference to FIGS. As described above, according to the present invention, it is possible to provide a machine translation device for comics, a machine translation method and program for comics, and a parallel translation database generation device for comics, which are capable of machine-translating comics with high accuracy. can be done. In particular, when the bilingual database of comics described above is used, the accuracy of machine translation of comics is improved. Furthermore, by using the above-described trained character recognition model, it becomes possible to accurately recognize character information from cartoon images, which has been difficult in the past.

In addition, this invention is not limited to said embodiment. For example, in the above-described embodiments, machine translation was explained as neural machine translation, but it is not limited to this. Any statistical machine translation or neural machine translation can be used as the machine translation as long as the bilingual database of comics described above is used.

Further, for example, although the machine translation device for comics and the bilingual database generation device for comics according to the above-described embodiments use the above-described learned character recognition model, the invention is not limited to this. The machine translation device and the parallel translation database generation device for comics need not use the learned character recognition model.

Further, for example, in the above-described embodiment, the learning character image group 430 is obtained by combining the processed character image 421 in the processed font image group 420 and the character image 411 in the font image group 410. However, the present disclosure is not so limited. For example, the learning character image group may not include the processed character image 421 in the processed font image group 420 and the character image 411 in the font image group 410 . In this case, the learning character image group can include other character images such as character images described in cartoons and other known character recognition data sets.

Also, for example, the detection of the first natural language character region and the second natural language character region in the generation of the bilingual database is not limited to the object detector described above, for example, the first natural language reference image and the second Corresponding pages of the natural language reference images may be overlaid and different portions detected as first and second natural language text regions.

In the above description, the first natural language is Japanese and the second natural language is English. The language and second natural language can be any natural language written using characters.

In the above description, the machine translation device 100 for comics, the parallel translation database generation device 300 for comics, and the character recognition model generation device 200 are each configured by one information processing device. Not limited. For example, the machine translation device for comics, the parallel translation database generation device for comics, and the character recognition model generation device may each be composed of a plurality of information processing devices. Further, two or more of the machine translation device for comics, the parallel translation database generation device for comics, and the character recognition model generation device may be implemented in one information processing device.

Although the preferred embodiments of the present invention have been described in detail above with reference to the accompanying drawings, the present invention is not limited to such examples. It is obvious that a person having ordinary knowledge in the technical field to which the present invention belongs can conceive of various modifications or modifications within the scope of the technical idea described in the claims. It is understood that these also naturally belong to the technical scope of the present invention.

100 machine translation device 110 machine translation learning unit 120 comic image acquisition unit 130 character area detection unit 140 character information estimation unit 150 machine translation unit 160 image generation unit 170 storage unit 200 character recognition model generation device 210 teacher data generation unit 230 machine learning unit 250 Storage unit 300 Translation database generation device 310 Reference image acquisition unit 330 Character region detection unit 350 Translation information extraction unit 370 Storage unit 371 Translation database 400 Network

Claims (9)

a character area detection unit that detects a character area from a first natural language image that constitutes a cartoon created using a first natural language;
a character information estimation unit for estimating character information of the first natural language from the character area;
a machine translation unit that translates the character information in the first natural language into character information in the second natural language by neural machine translation using a bilingual database;
The bilingual database includes a first natural language character region included in the first natural language reference image of the reference cartoon and a second natural language character region included in the second natural language reference image of the reference cartoon and corresponding to the first natural language character region. A sentence automatically generated by detecting a language character region and extracting first natural language character information existing in the first natural language character region and second natural language character information existing in the second natural language character region including bilingual information,
The character information estimation unit estimates the character information from the character region using a learned character recognition model,
The trained character recognition model uses teacher data including a plurality of font images of the first natural language for a plurality of types of fonts, and images obtained by adding deformation, tilt, and/or noise to the font images. A cartoon machine translation device generated by machine learning using an image recognition module. Extraction of the first natural language character information from the first natural language character region is performed by estimating the first natural language character information using a trained character recognition model,
The trained character recognition model is a machine using teacher data including a plurality of font images of the first natural language for a plurality of types of fonts and an image obtained by adding deformation, tilt and / or noise to the font image. 2. The cartoon machine translation device according to claim 1, which is generated by learning. Extraction of the second natural language character information from the second natural language character region is performed by estimating the second natural language character information using a trained second natural language character recognition model,
The trained second natural language character recognition model includes a plurality of second natural language font images of the second natural language for a plurality of types of fonts, and deformation, tilt, and/or noise on the second natural language font images. 3. The machine translation apparatus for comics according to claim 1, which is generated by machine learning with a Resnet image recognition module using teacher data including attached images. 4. The machine translation device for comics according to claim 1, wherein said first natural language is Japanese. 2. Further comprising an image generation unit that generates a second natural language image by adding the character information in the second natural language translated by the machine translation unit as an image to the first natural language image. 5. The machine translation device for comics according to any one of -4. A first natural language character region included in the first natural language reference image of the reference cartoon and a second natural language character region included in the second natural language reference image of the reference cartoon and corresponding to the first natural language character region a character area detection unit to detect;
a parallel translation information extraction unit for extracting first natural language character information existing in the first natural language character area and second natural language character information existing in the second natural language character area;
a storage unit that stores at least part of the first natural language character information and at least part of the second natural language character information as parallel translation information of a sentence;
Extraction of the first natural language character information from the first natural language character region is performed by estimating the first natural language character information using a trained character recognition model,
The trained character recognition model is a Resnet image using teacher data including a plurality of font images of a first natural language for a plurality of types of fonts, and images obtained by adding deformation, tilt and/or noise to the font images. A manga bilingual database generation device that is generated by machine learning using a recognition module. Extraction of the second natural language character information from the second natural language character region is performed by estimating the second natural language character information using a trained second natural language character recognition model,
The trained second natural language character recognition model provides a plurality of second natural language font images of a second natural language for a plurality of types of fonts, and deformation, tilt, and/or noise to the second natural language font images. 7. The comic bilingual database generation apparatus according to claim 6, wherein the database is generated by machine learning with a Resnet image recognition module using training data including the images. by the processor
Detecting a character region from a first natural language image that constitutes a cartoon created using a first natural language;
estimating character information in the first natural language from the character area; and translating the character information in the first natural language into character information in a second natural language by neural machine translation using a bilingual database. and run
The bilingual database includes a first natural language character region included in the first natural language reference image of the reference cartoon and a second natural language character region included in the second natural language reference image of the reference cartoon and corresponding to the first natural language character region. A sentence automatically generated by detecting a language character region and extracting first natural language character information existing in the first natural language character region and second natural language character information existing in the second natural language character region including bilingual information,
Extraction of the first natural language character information from the first natural language character region is performed by estimating the first natural language character information using a trained character recognition model,
The trained character recognition model uses teacher data including a plurality of font images of the first natural language for a plurality of types of fonts, and images obtained by adding deformation, tilt, and/or noise to the font images. A cartoon machine translation method generated by machine learning using an image recognition module. the computer,
a character area detection unit that detects a character area from a first natural language image that constitutes a cartoon created using a first natural language;
a character information estimation unit for estimating character information of the first natural language from the character area;
a machine translation unit that translates the character information in the first natural language into character information in the second natural language by neural machine translation using a bilingual database;
The bilingual database includes a first natural language character region included in the first natural language reference image of the reference cartoon and a second natural language character region included in the second natural language reference image of the reference cartoon and corresponding to the first natural language character region. A sentence automatically generated by detecting a language character region and extracting first natural language character information existing in the first natural language character region and second natural language character information existing in the second natural language character region including bilingual information,
Extraction of the first natural language character information from the first natural language character region is performed by estimating the first natural language character information using a trained character recognition model,
The trained character recognition model uses teacher data including a plurality of font images of the first natural language for a plurality of types of fonts, and images obtained by adding deformation, tilt, and/or noise to the font images. A program for functioning as a machine translation device for comics, generated by machine learning using an image recognition module.

Images