JP7365845B2 - Learning devices, learning methods, and programs - Google Patents

Description

The present invention relates to a learning device, a learning method, and a program.

There is a technique for extracting character information in an image from a scanned image created by reading a document such as a form using a scanner or the like (see, for example, Patent Documents 1 and 2). The technique of Patent Document 1 discloses a technique that makes it easier to correct errors in character information by structuring characters in an image based on their positions. Structuring in Patent Document 1 describes that text information is grouped into groups of information, and the hierarchical relationship of the grouped information is specified and expressed. For example, when character information such as title, document creator, and document creation date are extracted from an image, in structured data, the title is shown at the top level, and the document creator and The document creation date is indicated. The technique disclosed in Patent Document 2 extracts character information in an image and feature information indicating features of ruled lines. Thereby, when searching for a document, in addition to the characters written in the document, the characteristics of the ruled lines written in the document can be specified, making it possible to search efficiently.

When digitizing a form, its layout is often changed. The aspect ratio of the paper on which the form is printed and the screen of an electronic device such as a smartphone is different, and if you display the paper form as it is on the screen of the electronic device without changing the layout, the aspect ratio may differ depending on the display scale. This is because part of the document may not be displayed, or if you try to display the entire image, it may be reduced considerably and the text may become difficult to read. When changing the layout, it is necessary to reflect exactly what was written in the form before conversion on the form after conversion. As a countermeasure to this problem, for example, it is possible to apply the technology of Patent Documents 1-2 to change the layout of the form. By using the techniques of Patent Documents 1 and 2, it is possible to change the layout while maintaining the character structure and ruled line characteristics written on the form.

JP2019-82814A Japanese Patent Application Publication No. 2008-40834

However, even if the layout is changed while maintaining the character structure and the characteristics of the ruled lines, the content written in the form before conversion cannot be reflected in the form after conversion without excess or deficiency. Many forms have entry frames for entering necessary items. Most of these entry frames are shown as simple rectangles that do not contain any text. Character information cannot be extracted from such an entry frame itself. Therefore, with the technique of Patent Document 1, it is difficult to determine the hierarchical structure of all items written in a form including rectangular shapes such as entry frames. Furthermore, even if the characteristics of the ruled lines before the change are maintained in the form after the layout has been changed using Patent Document 2, it is still difficult to decide in which areas divided by the ruled lines characters should be written, or in which areas they should not be written and should be used as entry frames. If this is not known, layout conversion cannot be performed appropriately. As described above, it is difficult to change the layout while maintaining the semantic connection (structure) of the items (including rectangles) written on the form by simply using the conventional technology as is.

The present invention was made in view of this situation, and provides a learning device, a learning method, and a program that can generate a trained model that estimates information necessary for converting the layout of a document including rectangles. I will provide a.

In order to solve the above-mentioned problems of the present invention, the present invention includes an area information acquisition unit that acquires area information indicating respective areas of characters and rectangles included in a learning image; a structure information acquisition unit that acquires structural information indicating a hierarchical structure of rectangles; and a structure information acquisition unit that acquires structural information indicating a hierarchical structure of rectangles; A learning dataset generation unit that generates a learning dataset using a rectangular hierarchical structure as training data, and a learning dataset that generates a rectangle included in an input image as a learning result of training a learning model using the training dataset. A trained model generation unit that generates a trained model that outputs the structural information in the learning device.

The present invention also provides the above-mentioned learning device, in which the learning data set generation unit generates a set of data from the rectangle of interest, a character located within a predetermined first range from the position of the rectangle of interest, and a character located within a predetermined first range from the position of the rectangle of interest. The area information of each rectangle located within the second range is used as the input data.

Further, in the above learning device, the present invention generates semantic tag information including a specific second character corresponding to a first character shown in a character area included in the learning image based on the area information. The learning data set generating section further includes a semantic tag information generating section that uses the semantic tag information of the character indicated in the region information as input data.

Further, in the present invention, the area information acquisition unit acquires area information indicating respective areas of characters and rectangles included in the learning image, and the structure information acquisition unit acquires area information indicating the respective areas of characters and rectangles included in the learning image. The learning dataset generation unit obtains structural information indicating the structure, and generates a plurality of pieces of information regarding a rectangle of interest selected from rectangles included in the learning image as input data based on the region information and the structural information. Then, a training dataset is generated using the hierarchical structure of the rectangle of interest as training data, and the trained model generation unit generates the input image as a learning result of training the learning model using the training dataset. This is a learning method that generates a trained model that outputs the structural information on a rectangle included in the rectangle.

Further, the present invention is a program for causing a computer to operate as the learning device described above, and a program for causing the computer to function as each part included in the learning device.

According to the present invention, it is possible to generate a trained model that estimates information necessary for converting the layout of a document including rectangles.

FIG. 1 is a block diagram showing a configuration example of a learning device 10 according to an embodiment. It is a figure explaining area data concerning an embodiment. FIG. 2 is a diagram illustrating structured data according to an embodiment. It is a figure showing an example of composition of conversion table 170 concerning an embodiment. It is a figure showing an example of composition of rectangle information 171 concerning an embodiment. It is a figure showing an example of composition of semantic tag information 172 concerning an embodiment. It is a diagram showing an example of the configuration of a learning data set 173 according to the embodiment. FIG. 2 is a diagram illustrating processing performed by the learning device 10 according to the embodiment. It is a flowchart which shows the flow of processing which creates a data set for learning performed by learning device 10 concerning an embodiment. It is a flowchart which shows the flow of the process which makes a learning model learn, which is performed by the learning device 10 according to the embodiment.

Embodiments of the invention will be described below with reference to the drawings.

The learning device 10 generates a trained model that estimates information necessary for converting the layout of a document including rectangles.

In the following explanation, a case will be explained in which the document whose layout is to be converted is a form, but the present invention is not limited to this. The target for layout conversion may be any document as long as it includes at least characters and rectangles, and may be any document such as a questionnaire, medical questionnaire, test question, fixed phrase template, or idea sheet. A rectangle included in a document refers to an area surrounded by a quadrilateral shape such as a rectangle or a square in the document. A rectangle includes not only an area surrounded by a solid line, but also a rectangular area surrounded by a dotted line, a specific symbol, or a figure, or a rectangular area divided by the shade of a background color. Further, characters included in a document include not only a single character but also a string of characters and a group of characters.

The information necessary to convert the layout is information indicating the hierarchical structure of characters and rectangles included in the form (hereinafter referred to as structured data). If the hierarchical structure of characters and rectangles included in a form is known, the layout can be converted while maintaining that structure. Therefore, it is possible to maintain the characters, entry fields, etc. shown on the form and their relative positional relationships before and after the layout conversion. That is, in order to change the layout while maintaining the contents shown in the form, it is necessary to extract structured data of characters and rectangles included in the form.

An example of structured data will be explained. As shown in FIG. 2, consider a case where a form includes rectangular areas K1 to K5. In this case, as shown in FIG. 3, the hierarchical structure of the regions K1 to K3 is such that the region K1 is at the upper level, and the regions K2 and K3 are subordinated therebelow. Structured data is information that shows such a hierarchical structure. For example, the structured data of areas K4 and K5 is information indicating a hierarchical structure in which area K4 is at the upper level and area K5 is subordinated below it.

Below, a case where the learning device 10 specifies the hierarchical structure of a "rectangle" shown in a form will be described as an example. A similar method can be applied to specifying the hierarchical structure of "characters" shown on a form.

Further, as a hierarchical structure, a case will be described below in which identification information of a rectangle or a character (hereinafter referred to as a parent ID) that is a dependent source of a rectangle included in a form is determined as an example. In this case, the structured data is information that associates a rectangle with a parent ID of that rectangle. By using a method of determining the parent ID as a hierarchical structure, it is possible to uniquely specify the rectangular structure while suppressing an increase in data capacity, which is preferable. However, it is not limited to this. As a method of specifying the hierarchical structure of rectangles, it is also possible to determine the identification information (hereinafter referred to as child ID) of a rectangle or a character to which the rectangle is dependent. In this case, since there may be a structure in which multiple characters or rectangles are subordinate to one rectangle, it is necessary to adopt a configuration that allows multiple child IDs to be associated with a rectangle, which may lead to an increase in data capacity. The method for specifying the rectangular hierarchical structure may be any method as long as at least the hierarchical structure can be specified. The method for specifying the hierarchical structure of rectangles may be a method of associating a parent ID with a rectangle, a method of associating a child ID with a rectangle, or a method of associating a rectangle with both a parent ID and a child ID. Of course, other methods may also be used.

The configuration of the learning device 10 will be explained using FIG. 1. FIG. 1 is a block diagram showing a configuration example of a learning device 10 according to an embodiment. As shown in FIG. 1, the learning device 10 includes, for example, a region information acquisition section 11, a rectangle information generation section 12, a semantic tag information generation section 13, a structure information acquisition section 14, and a learning dataset generation section 15. , a trained model generation section 16 , and a storage section 17 .

The area information acquisition unit 11 acquires area information indicating a rectangular or character area in the learning image. The learning image is an image for learning by a learning model, which will be described later. The learning image is an image that includes a rectangle and characters, and the hierarchical structure of the rectangles is known. The region information is information in which pixel coordinates in the learning image are associated with identification information indicating that the pixel specified by the pixel coordinates is included in a rectangular or character region. Of the acquired area information, the area information acquisition unit 11 outputs area information indicating a rectangular area to the rectangular information generation unit 12. The area information acquisition unit 11 outputs area information indicating a character area out of the acquired area information to the semantic tag information generation unit 13.

The rectangle information generation unit 12 generates rectangle information based on area information indicating a rectangular area. Rectangle information is information in which coordinates indicating a position in a rectangular area are associated with identification information indicating that the area is a rectangular area. Here, the coordinates indicating the position in the region are, for example, when the shape of the region is a quadrangle, the coordinates of two vertices located diagonally among the four vertices of the quadrangle. Alternatively, the coordinates indicating the position in the region may be information indicating the coordinates of a predetermined specific vertex (for example, the lower left vertex) among the four vertices of the quadrangle, and the length and width of the rectangle. The rectangle information generation unit 12 outputs the generated rectangle information to the structure information acquisition unit 14. The rectangle information generation unit 12 stores the generated rectangle information as rectangle information 171 in the storage unit 17.

The semantic tag information generation unit 13 generates semantic tag information based on area information indicating a character area. The meaning tag information is information to which a tag (meaning tag) is attached according to the meaning of the character shown in the area. The semantic tag may be any information that indicates that the words are semantically equivalent. Semantic tags are, for example, words that represent semantically equivalent words, and more specifically, words such as "home", "address", "place", "address", etc. This information indicates that the address is "address". By generating semantic tag information, the semantic tag information generation unit 13 can unify semantically equivalent sentences into one sentence. Therefore, compared to the case where the wording is not unified, subsequent processing can be simplified, and the learning model described later can more easily estimate the hierarchical structure.

The semantic tag information generation unit 13 uses existing technology such as optical character recognition (OCR) to recognize the characters shown in the character area. The semantic tag information generation unit 13 generates semantic tag information by converting characters set according to the meaning of the characters into predetermined characters based on the result of character recognition. The semantic tag information generation unit 13 uses the conversion table 170 (see FIG. 4) for the conversion. The conversion table 170 is information stored in the storage unit 17, and is information (table) in which characters before conversion and characters after conversion are associated with each other. For example, the character string before conversion in the conversion table 170 shows characters that frequently appear in forms and whose notation may vary. The character string before conversion is an address, place, address, etc. The converted character string shows one character set according to the meaning, for example, the word "address" corresponding to "address, place, address".

The semantic tag information generation unit 13 refers to the conversion table 170 based on the character recognition result. If the same character as the recognized character exists in the characters shown before conversion in the conversion table 170, the meaning tag information generation unit 13 acquires the converted character that is associated with the character before conversion. do. The semantic tag information generation unit 13 converts the recognized characters into converted characters shown in the conversion table 170. The semantic tag information generation unit 13 generates semantic tag information by associating the converted characters with area information indicating a character area. The semantic tag information generation unit 13 outputs the generated semantic tag information to the learning data set generation unit 15. The semantic tag information generation unit 13 stores the generated semantic tag information in the storage unit 17 as semantic tag information 172.

Note that if the same character as the recognized character does not exist in the characters shown before conversion in the conversion table 170, the semantic tag information generation unit 13 converts the character into area information indicating the area of the character without converting the character. , generates semantic tag information by associating the recognized characters.

The structural information acquisition unit 14 acquires structural information in the learning image. The structure information is information in which a rectangle and a hierarchical structure (parent ID) of the rectangle are associated for each rectangle in the image. The structural information acquisition unit 14 outputs the acquired structural information to the learning dataset generation unit 15.

The learning dataset generation unit 15 generates a learning dataset using the rectangular area data and semantic tag information. The learning data set is data that is a set of input data and teacher data for making the learning model learn. The learning model is trained using the learning data set until it can accurately output (estimate) the parent ID of the rectangle in the input image.

The learning model is, for example, an RNN (Recurrent Neural Network). However, it is not limited to this. As the learning model, for example, a model based on a method such as a DCNN (Deep Convolutional Neural Network), a CNN, a decision tree, a hierarchical Bayesian, an SVM (Support Vector Machine), or a method that combines these appropriately may be used.

The learning data set generation unit 15 extracts a rectangle of interest in the learning image. The rectangle of interest is a rectangle to which a hierarchical structure as teacher data is associated in the learning data set. The learning data set generation unit 15 extracts rectangle information (hereinafter referred to as a group of neighboring rectangles) of rectangles within a predetermined range (hereinafter referred to as the first range) from the rectangle of interest in the learning image. The learning data set generation unit 15 extracts semantic tag information (hereinafter referred to as a neighboring semantic tag group) of characters within a predetermined range (hereinafter referred to as a second range) from the rectangle of interest in the learning image. The learning data set generation unit 15 uses the extracted rectangle information of the rectangle of interest, a group of neighboring rectangles, and a group of neighboring semantic tags as input data for the rectangle of interest. The rectangle information of the rectangle of interest, a group of neighboring rectangles, and a group of neighboring semantic tags extracted by the learning data set generation unit 15 are examples of "information regarding the rectangle of interest." The learning data set generation unit 15 outputs the generated learning data set to the trained model generation unit 16. The learning data set generation unit 15 stores the generated learning data set in the storage unit 17 as a learning data set 173.

The trained model generation unit 16 generates a trained model. The trained model is a learning result obtained by training a learning model using a training data set, and is a model trained to output structural information on a rectangle included in an input image.

The trained model generation unit 16 adjusts the parameters of the learning model so that the output obtained by inputting the input data to the learning model approaches the teacher data associated with the input data in the learning dataset. Repeat. This allows the learning model to accurately output structural information on rectangles included in the input image. The learned model generation unit 16 sets a learned model that has been trained until a predetermined termination condition is satisfied as a learned model. The predetermined termination condition is, for example, that all the training datasets created by the training dataset generation unit 15 have been trained, or that the accuracy of estimating the structural information in the rectangle included in the input image is high. For example, the threshold value has exceeded a predetermined threshold value.

The trained model generation unit 16 determines the order in which input data sets are input to the learning model during the learning process. In particular, when an RNN is used as a learning model, the order of data input to the learning model has information. That is, the RNN has a configuration that performs estimation based on the order of input data. Therefore, it is thought that by specifying the order in which the inputs are input to the learning model, it becomes possible to estimate the parent ID with high accuracy.

The trained model generation unit 16 uses data obtained by sorting representative coordinates (for example, center coordinates) of each of the rectangle of interest, a group of neighboring rectangles, and a group of neighboring semantic tags in raster order as input data. The raster order here refers to the reading (or writing) order along a predetermined direction when reading (or writing) pixels arranged two-dimensionally. For example, the raster order is an order along the horizontal direction from the left to the right in the image, and a vertical direction from the top to the bottom. However, the predetermined direction in the raster order may be any direction, and may be an order from the right side to the left side, or an order from the bottom side to the top side.

The storage unit 17 stores a conversion table 170, rectangle information 171, meaning tag information 172, a learning data set 173, and a trained model 174.

FIG. 2 is a diagram illustrating area information according to the embodiment. As shown in FIG. 2, character regions M1 to M6 and rectangular regions K1 to K5 are extracted from the learning image. Area M1 is an area where the characters "Application Form" are shown. Area M2 is an area where the characters "address" are shown. Area M3 is an area where the characters "prefecture" are shown. Area M4 is an area where the characters "name" are shown. Area M5 is an area where the characters "Date of Entry" are shown. Area M6 is an area where the characters "Year Month Day" are shown. As shown in this example, in this embodiment, a character area is extracted as a rectangular (square) shaped area.

Region K1 is a region in which a rectangle surrounding region M2 is shown. Area K2 is an area in which a rectangle surrounding area M3 is shown so that area M3 is placed at the right end of the frame. Area K3 is an area in which a rectangle placed on the right side of area K2 is shown. Region K3 is a region in which a rectangle surrounding region M4 is shown. Area K5 is an area in which a rectangle placed on the right side of area K4 is shown.

FIG. 3 is a diagram illustrating structure information according to the embodiment. As shown in FIG. 3, area M1# indicates an area after the characters shown in character area M1 are converted based on conversion table 170. Similarly, the regions M2# to M6# indicate the regions after the characters shown in the character regions M2 to M6 have been converted based on the conversion table 170.

FIG. 4 is a diagram showing a configuration example of the conversion table 170 according to the embodiment. The conversion table 170 includes items such as, for example, meaning tag ID, after conversion, and before conversion. The meaning tag ID indicates identification information that uniquely identifies the meaning tag. After conversion, the converted characters are shown. Before conversion, the character string before conversion is shown. In this example, the meaning tag ID (E0001) shows "name" as characters after conversion, and "name", "name", and "name" as characters before conversion.

FIG. 5 is a diagram illustrating a configuration example of rectangle information 171 according to the embodiment. The rectangle information 171 includes items such as rectangle ID, position coordinates 1, position coordinates 2, and representative position coordinates, for example. The rectangle ID is identification information that uniquely identifies a rectangular area included in the learning image. Position coordinate 1 and position coordinate 2 are the position coordinates of two points for specifying the rectangular area specified by the rectangle ID, and for example, among the four vertices corresponding to the four corners of the rectangle, the position coordinates are the coordinates of the two vertices. The representative position coordinates are the coordinates of a position that represents the position of the rectangular area specified by the rectangle ID, and are, for example, the center coordinates of the rectangular area. The representative position coordinates are used as representative coordinates to be sorted in raster order when the learned model generation unit 16 determines the order of input data.

FIG. 6 is a diagram illustrating a configuration example of the semantic tag information 172 according to the embodiment. The meaning tag information 172 includes items such as, for example, character ID, character, meaning group ID, position coordinate 1, position coordinate 2, and representative position coordinate. The character ID is identification information that uniquely identifies a character area included in the learning image. The characters are shown as characters recognized in the character area specified by the character ID. The meaning group ID indicates which meaning group in the conversion table 170 the character corresponds to. Position coordinate 1 and position coordinate 2 are the position coordinates of two points for specifying the area of the character specified by the character ID. The representative position coordinates are the coordinates of a position that represents the position of the character area specified by the character ID. As shown in FIG. 2, in this embodiment, the character area is extracted as a rectangular (square) shaped area.

FIG. 7 is a diagram showing a configuration example of the learning data set 173 according to the embodiment. The learning data set 173 includes, for example, a rectangle ID, input data, and teacher data. The rectangle ID is identification information that uniquely identifies a rectangular area included in the learning image. The input data is input data when the rectangle specified by the rectangle ID is the rectangle of interest. The input data includes position coordinates, neighboring character IDs, and neighboring rectangle IDs. The position coordinates indicate the position coordinates that are used as a reference when calculating the neighborhood of the rectangle of interest. The neighborhood character ID indicates the character ID of the neighborhood meaning tag group in the rectangle of interest. As in this example, a plurality of character IDs may be indicated in the neighboring character ID. The neighboring rectangle ID indicates the rectangle ID of a group of neighboring rectangles in the rectangle of interest. As in this example, a plurality of rectangle IDs may be indicated in the neighboring rectangle ID. The teacher data indicates the parent ID of the rectangle of interest.

FIG. 8 is a diagram illustrating processing performed by the learning device 10 according to the embodiment.
In step S1, a form prepared as a learning image is input to a form splitter, and a form prepared as a learning image is input to OCR. The OCR here assumes that it is optical character recognition as part of the function of the semantic tag information generation unit 13 of the learning device 10, but is not limited to this, and the OCR is It may also be an external device located outside.

In step S2, the form divider is a device that divides the input image into regions for rectangles and characters, and outputs region information for each rectangle and characters. The form divider outputs rectangular area information in the input learning image to the learning device 10. Although not shown in this figure, the form divider may output character area information in the input learning image to the OCR.

In step S3, the rectangle information generation unit 12 of the learning device 10 generates rectangle information using the rectangle area information. The rectangle information generation unit 12 outputs all the rectangle information generated in the learning image to the learning data set generation unit 15.

In step S4, the semantic tag information generation unit 13 performs character recognition on the character area in the learning image, outputs information (described as character information) indicating the result of character recognition in step S5, and in step S6, the semantic tag information generation unit 13 performs character recognition on the character area in the learning image. generate. In step S7, the semantic tag information generation unit 13 outputs all the semantic tag information generated in the learning image to the learning data set generation unit 15.

In step S8, the learning data set generation unit 15 generates input data of a learning data set (described as learning data). The learned model generation unit 16 determines the order in which the input data is input to the learning model by sorting the center points of the rectangle of interest, the neighborhood semantic tag group, and the neighborhood rectangle group in the input data in raster order. .

In step S9, the trained model generation unit 16 inputs the input data to the learning model. In step S10, the learned model generation unit 16 obtains the output obtained from the learning model as a prediction result of the parent ID of the rectangle of interest. In step S11, the trained model generation unit 16 acquires the teacher data of the learning data set, that is, the parent ID of the rectangle of interest. In step S12, the trained model generation unit 16 generates a loss function using the prediction result of the parent ID of the rectangle of interest and the teacher data of the learning data set, and reflects the result in the learning model.

FIG. 9 is a diagram illustrating a flow of processing for creating a learning data set performed by the learning device 10 according to the embodiment. In step S20, the learning data set generation unit 15 acquires the position coordinates of the rectangle of interest. In step S21, the learning data set generation unit 15 acquires rectangle information of nearby rectangles. In step S22, the learning data set generation unit 15 acquires semantic tag information of nearby characters. In step S23, the learning data set generation unit 15 acquires the parent ID of the rectangle of interest. In step S24, the learning dataset generation unit 15 creates a learning dataset by combining the rectangle of interest, neighborhood semantic tag group, and neighborhood rectangle group as input data with the parent ID as teacher data. .

FIG. 10 is a diagram illustrating the flow of learning performed by the learning device 10 according to the embodiment. In step S30, the trained model generation unit 16 inputs the input data to the learning model. At this time, the trained model generation unit 16 determines in advance the order in which the input data is input to the trained model according to a predetermined rule. In step S31, the trained model generation unit 16 performs forward propagation calculation using the learning model and obtains an output from the learning model. In step S32, the trained model generation unit 16 derives a loss function based on the error between the output obtained from the learning model and the teacher data, and performs error backpropagation based on the loss function. In step S33, the trained model generation unit 16 stores the parameters of the learning model updated by error backpropagation based on the loss function. In step S34, the trained model generation unit 16 determines whether a predetermined termination condition is satisfied. If the predetermined termination condition is met, the learning for the learning model is completed and the learning model is set as a trained model. If the predetermined termination condition is not met, the process returns to step S30 and learning is repeated.

As described above, the learning device 10 of the embodiment includes the region information acquisition section 11, the structure information acquisition section 14, the learning dataset generation section 15, and the learned model generation section 16. The area information acquisition unit 11 acquires area information indicating respective areas of characters and rectangles included in the learning image. The structural information acquisition unit 14 acquires structural information indicating a hierarchical structure of rectangles included in the learning image. The learning data set generation unit 15 generates learning data based on the region information and structure information, using information regarding a rectangle of interest among the rectangles included in the learning image as input data, and using the hierarchical structure of the rectangle of interest as training data. Generate a set. The trained model generation unit 16 generates a trained model that outputs structural information on a rectangle included in the input image as a learning result of training the learning model using the learning data set. Thereby, the learning device 10 of the embodiment can generate information necessary for converting the layout of a document including a rectangle, that is, a trained model that estimates the parent ID of the rectangle.

Furthermore, in the learning device 10 of the embodiment, the learning data set generation unit 15 generates a rectangle of interest, a character located within a predetermined first range from the position of the rectangle of interest, and a character located within a predetermined second range from the position of the rectangle of interest. Rectangle information (an example of "area information") and semantic tag information (an example of "area information") of each rectangle located in are used as the input data. As a result, the learning device 10 of the embodiment can make the learning model learn to estimate the parent ID based on the rectangle of interest and the information of the rectangles and characters in its vicinity, so that the learning device 10 can more accurately estimate the parent ID. A trained model for estimation can be generated.

Further, in the learning device 10 of the embodiment, the learning data set generation unit 15 determines the order in which the input data is input to the learning model according to the position of each character or rectangle indicated in the area information used for the input data. decide. As a result, the learning device 10 of the embodiment can have information on the order in which the input data is input to the learning model, and uses a learning model that takes the order as information, such as an RNN, to consider the input order. It becomes possible to perform learning, and it is possible to generate a trained model that estimates the parent ID with higher accuracy.

In addition, in the learning device 10 of the embodiment, semantic tag information that generates semantic tag information including a specific second character corresponding to the first character shown in the character area included in the learning image based on the area information. The learning data set generating section 15 further includes a generating section 13, and the learning data set generating section 15 uses the meaning tag information of the character indicated in the region information as input data. Thereby, the learning device 10 of the embodiment can tag characters shown in the learning image according to their meanings, and compare learning to the learning model with a case where no tagging is performed. By doing so, it is possible to simplify the process and reduce the processing load.

All or part of the learning device 10 in the embodiment described above may be realized by a computer. In that case, a program for realizing this function may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read into a computer system and executed. Note that the "computer system" herein includes hardware such as an OS and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, magneto-optical disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into computer systems. Furthermore, a "computer-readable recording medium" refers to a storage medium that dynamically stores a program for a short period of time, such as a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. It may also include a device that retains a program for a certain period of time, such as a volatile memory inside a computer system that is a server or client in that case. Further, the above-mentioned program may be one for realizing a part of the above-mentioned functions, or may be one that can realize the above-mentioned functions in combination with a program already recorded in the computer system. It may also be realized using a programmable logic device such as an FPGA.

Although the embodiments of the present invention have been described above in detail with reference to the drawings, the specific configuration is not limited to these embodiments, and includes designs within the scope of the gist of the present invention.

10...Learning device 11...Region information acquisition unit 12...Rectangle information generation unit 13...Semantic tag information generation unit 14...Structure information acquisition unit 15...Learning dataset generation unit 16...Learned model generation unit 17...Storage unit 170... Conversion table 171...Rectangle information 172...Semantic tag information 173...Learning dataset 174...Learned model

Claims (5)

an area information acquisition unit that acquires area information indicating respective areas of characters and rectangles included in the learning image;
a structural information acquisition unit that acquires structural information indicating a hierarchical structure of rectangles included in the learning image;
Learning to generate a learning data set based on the area information and the structure information, using information regarding a rectangle of interest among the rectangles included in the learning image as input data, and using a hierarchical structure of the rectangle of interest as training data. a data set generation unit for
a trained model generation unit that generates a trained model that outputs the structural information in a rectangle included in the input image as a learning result of training the learning model using the training data set;
A learning device equipped with. The learning data set generation unit generates each of the rectangle of interest, characters located within a predetermined first range from the position of the rectangle of interest, and rectangles located within a second predetermined range from the position of the rectangle of interest. using the area information as the input data;
The learning device according to claim 1. Further comprising a semantic tag information generation unit that generates semantic tag information including a specific second character corresponding to the first character shown in the character area included in the learning image based on the area information,
The learning data set generation unit uses the meaning tag information of the character indicated in the region information as input data;
The learning device according to claim 1 or claim 2 . The area information acquisition unit acquires area information indicating respective areas of characters and rectangles included in the learning image,
a structural information acquisition unit acquires structural information indicating a hierarchical structure of rectangles included in the learning image;
A learning data set generation unit uses a plurality of pieces of information regarding a rectangle of interest selected from rectangles included in the learning image as input data, based on the area information and the structure information, and generates a hierarchical structure of the rectangle of interest. Generate a training dataset to be used as training data,
a trained model generation unit generates a trained model that outputs the structural information in a rectangle included in the input image as a learning result of training the learning model using the training data set;
How to learn. A program for causing a computer to operate as the learning device according to any one of claims 1 to 3 , the program causing the computer to function as each part included in the learning device.

Images