JP7365835B2 - Structure recognition system, structure recognition device, structure recognition method, and program - Google Patents

Description

The present invention relates to a structure recognition system, a structure recognition device, a structure recognition method, and a program.

There is a technique for extracting character information in an image from a scanned image created by reading a document such as a form using a scanner or the like (see, for example, Patent Documents 1 and 2). The technique of Patent Document 1 discloses a technique that makes it easier to correct errors in character information by structuring characters in an image based on their positions. Structuring in Patent Document 1 describes that text information is grouped into groups of information, and the hierarchical relationship of the grouped information is specified and expressed. For example, when character information such as title, document creator, and document creation date are extracted from an image, in structured data, the title is shown at the top level, and the document creator and The document creation date is indicated. The technique disclosed in Patent Document 2 extracts character information in an image and feature information indicating features of ruled lines. Thereby, when searching for a document, in addition to the characters written in the document, the characteristics of the ruled lines written in the document can be specified, making it possible to search efficiently.

On the other hand, with the recent development of computer and communication network technology, there is a tendency for paper-based information to be replaced with electronic data. By digitizing forms, it is possible to achieve paperless work efficiency and resource conservation, and it is also possible to fill in fields on documents using electronic devices such as smartphones. Therefore, user convenience can be improved.

When digitizing a form, its layout is often changed. The aspect ratio of the paper on which the form is printed and the screen of an electronic device such as a smartphone is different, and if you display the paper form as it is on the screen of the electronic device without changing the layout, the aspect ratio may differ depending on the display scale. This is because part of the document may not be displayed, or if you try to display the entire image, it may be reduced considerably and the text may become difficult to read. When changing the layout, it is necessary to reflect exactly what was written in the form before conversion on the form after conversion. As a countermeasure to this problem, for example, it is possible to apply the technology of Patent Documents 1-2 to change the layout of the form. By using the techniques of Patent Documents 1 and 2, it is possible to change the layout while maintaining the character structure and ruled line characteristics written on the form.

JP2019-82814A Japanese Patent Application Publication No. 2008-40834

However, even if the layout is changed while maintaining the character structure and the characteristics of the ruled lines, the content written in the form before conversion cannot be reflected in the form after conversion without excess or deficiency. Many forms have entry frames for entering necessary items. Most of these entry frames are shown as simple rectangles that do not contain any text. Character information cannot be extracted from such an entry frame itself. Therefore, with the technique of Patent Document 1, it is difficult to determine the hierarchical structure of all items written in a form including rectangular shapes such as entry frames. Furthermore, even if the characteristics of the ruled lines before the change are maintained in the form after the layout has been changed using Patent Document 2, it is still difficult to decide in which areas divided by the ruled lines characters should be written, or in which areas they should not be written and should be used as entry frames. If this is not known, layout conversion cannot be performed appropriately. As described above, it is difficult to change the layout while maintaining the semantic connection (structure) of the items (including rectangles) written on the form by simply using the conventional technology as is.

The present invention has been made in view of this situation, and provides a structure recognition system, a structure recognition device, a structure recognition method, and a program that can extract information necessary for converting the layout of a document including rectangles. I will provide a.

In order to solve the above-mentioned problems of the present invention, the present invention includes an image data acquisition unit that acquires image data in a target image including characters and rectangles, and determines respective regions of characters and rectangles in the target image. an area determination unit; a structure determination unit that determines a hierarchical structure of rectangles included in the target image based on area data regarding the area determined by the area determination unit; This is a structure recognition system that determines, among rectangles included in a target image, a rectangle or a character that is a dependent source of a rectangle of interest whose hierarchical structure is to be determined.
The present invention also provides an image data acquisition unit that acquires image data in a target image including characters and rectangles, an area determination unit that determines areas of each of the characters and rectangles in the target image, and an area determination unit that a structure determining unit that determines a hierarchical structure of rectangles included in the target image based on the determined area data regarding the area; a preprocessing unit that generates semantic tag information including a specific second character corresponding to a character, the structure determination unit determines the hierarchical structure using a learned model, and the learned model An input image using a learning data set that associates the semantic tag information and the area data indicating the rectangular area in the learning image containing a rectangle with the hierarchical structure of the rectangles included in the learning image. This is a structure recognition system, which is a model trained to output the hierarchical structure of rectangles contained in .

Furthermore, in the structure recognition system described above, the present invention uses the area data indicating a character area to generate semantic tag information including a specific second character corresponding to the first character shown in the area. The apparatus further includes a processing section, and the structure determining section determines the hierarchical structure based on the semantic tag information and the area data indicating a rectangular area.

Further, in the structure recognition system of the present invention, the structure determination unit determines the hierarchical structure using a trained model, and the trained model is configured to identify the meaning tags in the learning image including characters and rectangles. Outputting a hierarchical structure of rectangles included in the input image using a learning data set in which information and the area data indicating a rectangular area are associated with the hierarchical structure of rectangles included in the learning image. This is a model trained as follows.

Further, in the structure recognition system of the present invention, the structure determination unit selects a rectangle of interest for determining the hierarchical structure in the target image, and within a predetermined first range from the position of the selected rectangle of interest. A group of neighboring semantic tags, which is the semantic tag information located in the selected rectangle of interest, is obtained, and a group of neighboring rectangles, which is the region data of a rectangle located within a predetermined second range from the position of the selected rectangle of interest, is obtained. By performing rearrangement according to the positions of the rectangle of interest, the group of neighborhood meaning tags, and the group of neighborhood rectangles, the order of input data to be input to the trained model is determined.

The present invention also provides a region data acquisition unit that acquires region data regarding each region of characters and rectangles in a target image, and a hierarchical structure of rectangles included in the target image based on the region data , The structure recognition apparatus includes a structure determination unit that determines, among rectangles included in the target image, a rectangle or a character that is a dependent source of a rectangle of interest whose hierarchical structure is to be determined.

Further, in the present invention, the area data acquisition unit acquires area data regarding the respective areas of characters and rectangles in the target image, and the structure determination unit determines the area of the rectangle included in the target image based on the area data. This is a structure recognition method that determines a hierarchical structure and determines, among rectangles included in the target image, a rectangle or a character that is a dependent source of a rectangle of interest whose hierarchical structure is to be determined.

Further, the present invention is a program for causing a computer to operate as the structure recognition device described above, and a program for causing the computer to function as each part included in the structure recognition device.

According to the present invention, information necessary for converting the layout of a document including rectangles can be extracted.

1 is a diagram showing a configuration example of a structure recognition system 1 according to an embodiment. FIG. 2 is a diagram illustrating processing performed by the structure recognition system 1 according to the embodiment. 1 is a block diagram showing a configuration example of a region dividing device 10 according to an embodiment. FIG. It is a block diagram showing an example of composition of structure recognition device 30 concerning an embodiment. It is a figure showing an example of composition of conversion table 360 concerning an embodiment. FIG. 3 is a diagram illustrating processing performed by the region dividing device 10 according to the embodiment. It is a figure explaining the processing which the structure recognition device 30 concerning an embodiment performs. It is a figure showing an example of layout conversion to which structure recognition system 1 concerning an embodiment is applied. It is a sequence diagram showing the flow of processing performed by the structure recognition system 1 according to the embodiment.

Embodiments of the invention will be described below with reference to the drawings.

The structure recognition system 1 of this embodiment is a system that extracts information necessary for converting the layout of a document including rectangles.

In the following explanation, a case will be explained in which the document whose layout is to be converted is a form, but the present invention is not limited to this. The target for layout conversion may be any document as long as it includes at least characters and rectangles, and may be any document such as a questionnaire, medical questionnaire, test question, fixed phrase template, or idea sheet. A rectangle included in a document refers to an area surrounded by a quadrilateral shape such as a rectangle or a square in the document. A rectangle includes not only an area surrounded by a solid line, but also a rectangular area surrounded by a dotted line, a specific symbol, or a figure, or a rectangular area divided by the shade of a background color. Further, characters included in a document include not only a single character but also a string of characters and a group of characters.

The information necessary to convert the layout is information indicating the hierarchical structure of characters and rectangles included in the form (hereinafter referred to as structured data). If the hierarchical structure of characters and rectangles included in a form is known, the layout can be converted while maintaining that structure. Therefore, it is possible to maintain the characters, entry fields, etc. shown on the form and their relative positional relationships before and after the layout conversion. That is, in order to change the layout while maintaining the contents shown in the form, it is necessary to extract structured data of characters and rectangles included in the form.

An example of structured data will be explained. As shown in FIG. 6, consider a case where a form includes rectangular areas K1 to K5. As shown in FIG. 7, the structured data of regions K1 to K3 is information indicating a hierarchical structure in which region K1 is at the upper level and regions K2 and K3 are subordinated therebelow. The structured data of areas K4 and K5 is information indicating a hierarchical structure in which area K4 is at the upper level and area K5 is subordinated below it.

The overall configuration of the structure recognition system 1 will be explained using FIG. 1. FIG. 1 is a block diagram showing a configuration example of a structure recognition system 1 according to an embodiment. As shown in FIG. 1, the structure recognition system 1 includes, for example, a region dividing device 10, an OCR device 20, and a structure recognition device 30. These components (area segmentation device 10, OCR device 20, and structure recognition device 30) in the structure recognition system 1 are communicably connected.
Although FIG. 1 illustrates a case where the OCR device 20 performs character recognition, the function of character recognition processing may be provided in the area dividing device 10 or the structure recognition device 30. In this case, the OCR device 20 can be omitted.

The region dividing device 10 is a device that divides a document into regions such as characters and rectangles shown on the document. The OCR device 20 is a device that performs character recognition processing to recognize characters shown in an input image. The structure recognition device 30 is a device that determines the hierarchical structure of rectangles shown in a form.

In the following, a case will be explained in which the structure recognition device 30 specifies the hierarchical structure of a "rectangle" shown in a form. A similar method can be applied to specifying the hierarchical structure of "characters" shown on a form.

Further, as a hierarchical structure, a case will be described below in which identification information of a rectangle or a character (hereinafter referred to as a parent ID) that is a dependent source of a rectangle included in a form is determined as an example. In this case, the structured data is information that associates a rectangle with a parent ID of that rectangle. By using a method of determining the parent ID as a hierarchical structure, it is possible to uniquely specify the rectangular structure while suppressing an increase in data capacity, which is preferable. However, it is not limited to this. As a method of specifying the hierarchical structure of rectangles, it is also possible to determine the identification information (hereinafter referred to as child ID) of a rectangle or a character to which the rectangle is dependent. In this case, since there may be a structure in which multiple characters or rectangles are subordinate to one rectangle, it is necessary to adopt a configuration that allows multiple child IDs to be associated with a rectangle, which may lead to an increase in data capacity. The method for specifying the rectangular hierarchical structure may be any method as long as at least the hierarchical structure can be specified. The method for specifying the hierarchical structure of rectangles may be a method of associating a parent ID with a rectangle, a method of associating a child ID with a rectangle, or a method of associating a rectangle with both a parent ID and a child ID. Of course, other methods may also be used.

Here, the processing performed by the structure recognition system 1 will be explained using FIG. 2. FIG. 2 is a diagram illustrating processing performed by the structure recognition system 1 according to the embodiment. As shown in FIG. 2, information (scan image data SD) representing a scanned image of the form T (scan image) is created by the process of reading the form T with a scanner (scan processing SC). The scanned image is an image to be processed by the area segmentation device 10. That is, the scanned image is an example of a "target image." The scan image data SD is input to the region dividing device 10. Thereby, the region dividing device 10 obtains scan image data SD.

The region dividing device 10 divides the form T into regions for each element such as a character or a rectangle based on the scan image data SD. The area dividing device 10 outputs information indicating a character area (character area data MD) and information indicating a rectangular area (rectangular area data KD) in the scanned image. The area dividing device 10 outputs the character area data MD to the OCR device 20. The character area data MD is an example of "area data." The rectangular area data KD is an example of "area data."

The OCR device 20 performs character recognition processing to recognize characters shown in the character region data acquired from the region dividing device 10. The OCR device 20 notifies the structure recognition device 30 of information indicating the content of the recognized characters (character recognition data MND). The character recognition data MND is input to the structure recognition device 30. Thereby, the structure recognition device 30 acquires character recognition data MND.

On the other hand, the character region data MD and rectangular region data KD output by the region dividing device 10 are input to the structure recognition device 30. Thereby, the structure recognition device 30 obtains the character area data MD and the rectangular area data KD.

The structure recognition device 30 determines the hierarchical structure of rectangles shown in the form based on the character area data MD and rectangular area data KD acquired from the area dividing device 10 and the character recognition data MND acquired from the OCR device 20. . The structure recognition device 30 uses the conversion table 360 to classify the character recognition data MND according to the meaning of the character. In the structure recognition device 30, the structure determination unit 34 determines the hierarchical structure of a rectangle using information in which the character area data MD, rectangular area data KD, and character recognition data MND are classified according to the meanings of the characters. The method by which the structure recognition device 30 determines the hierarchical structure of rectangles will be described in detail later. The structure recognition device 30 outputs information (structured data KZD) indicating a rectangular hierarchical structure.

Here, the configuration of the region dividing device 10 will be explained using FIG. 3. FIG. 3 is a block diagram showing a configuration example of the region dividing apparatus 10 according to the embodiment. As shown in FIG. 3, the region dividing device 10 includes, for example, an image data acquisition section 11, a modulated image generation section 12, a region determination section 13, a region data output section 14, and a storage section 15.

The image data acquisition unit 11 acquires scan image data SD. The scan image data SD is, for example, information in which image-related information is associated with each pixel, such as information indicating a gray scale value for each pixel, or information indicating an RGB value for each pixel. be. The image data acquisition unit 11 outputs the acquired scan image data SD to the modulated image generation unit 12 and the area determination unit 13.

The modulated image generation unit 12 generates an enhanced image based on the scan image data SD acquired from the image data acquisition unit 11. The enhanced image is an image in which the pixel value (grayscale value or RGB value) of each pixel in the scanned image is changed based on predetermined modulation conditions.

The modulated image generation unit 12 generates, as an enhanced image, an image that has been subjected to enhancement processing to enhance the edges of the scanned image, for example. In this case, the modulated image generation unit 12 detects edges in the scanned image and performs processing to emphasize the detected edges. The modulated image generation unit 12 detects edges in the scanned image using, for example, any conventional method. The arbitrary method is, for example, a method of detecting edges by detecting a difference between a scanned image subjected to median filter processing and an image obtained after smoothing processing using a Gaussian filter or the like. Alternatively, a method of detecting edges may be used by applying a Laplacian filter or a Sobel filter to the scanned image. The modulated image generation unit 12 sets the detected edge to a certain specific pixel value (for example, a gray scale value indicating "black" or an RGB value), and sets the pixel value of the pixel that is not detected as an edge to another specific pixel value. An enhanced image is generated by converting into pixel values (for example, grayscale values indicating "white" or RGB values). The modulated image generation unit 12 outputs image data of the generated enhanced image to the area determination unit 13.

The area determination unit 13 determines areas of characters, rectangles, and backgrounds (elements that are not characters and are not rectangles) in each of the scanned image and the emphasized image. The area determination unit 13 uses, for example, the area determination model 150 to determine characters and rectangular areas in the image. The area determination model 150 is information written in the storage unit 15, and is a learning data set that associates image data with the results of determining each area of characters, rectangles, and background in the image. This is the learning result obtained by training the robot. By learning such a training data set, the learning model is trained to accurately output (predict) characters and rectangular areas in the input image data. The learning model is, for example, a DCNN (Deep Convolutional Neural Network), but is not limited to this. As the learning model, for example, methods such as CNN, decision tree, hierarchical Bayes, SVM (Support Vector Machine), and methods combining these appropriately may be used.

The area determination unit 13 determines the respective areas of the text, rectangle, and background in the scan image based on the determination results of the scan image and the emphasized image. For example, for a region where the scan image determination result and the enhanced image determination result match, the area determination unit 13 determines the scan image determination result as is.

On the other hand, for areas where the scan image determination result and the enhanced image determination result do not match, the area determination unit 13 determines the respective areas of characters, rectangles, and background based on predetermined regulations. For example, the area determining unit 13 determines that an area in which at least one of the scanned image and the emphasized image is a text and the other is a background is determined to be a text area in the scanned image. The area determining unit 13 determines, for example, an area in which at least one of the scan image and the emphasized image is a rectangle and the other is a background, as a rectangular area in the scan image. The area determination unit 13 outputs information indicating a character area in the scanned image (character area data MD) and information indicating a rectangular area (rectangular area data KD) to the area data output unit 14.

The area data output unit 14 outputs the character area data MD to the OCR device 20. The area data output unit 14 outputs the character area data MD and the rectangular area data KD to the structure recognition device 30. The storage unit 15 stores an area determination model 150.

Here, the configuration of the structure recognition device 30 will be explained using FIG. 4. FIG. 4 is a block diagram showing a configuration example of the structure recognition device 30 according to the embodiment. As shown in FIG. 4, the structure recognition device 30 includes, for example, a region data acquisition section 31, a character recognition data acquisition section 32, a preprocessing section 33, a structure determination section 34, a structure data output section 35, and a memory. 36.

The area data acquisition unit 31 acquires area data (character area data MD and rectangular area data KD) from the area dividing device 10. The character area data MD is, for example, information in which coordinates indicating a position in a character area are associated with identification information indicating that the area is a character area. The rectangular area data KD is, for example, information in which coordinates indicating a position in a rectangular area are associated with identification information indicating that the area is a rectangular area. Here, the coordinates indicating the position in the region are, for example, when the shape of the region is a quadrangle, the coordinates of two vertices located diagonally among the four vertices of the quadrangle. Alternatively, the coordinates indicating the position in the region may be information indicating the coordinates of a predetermined specific vertex (for example, the lower left vertex) among the four vertices of the quadrangle, and the length and width of the rectangle. The area data acquisition unit 31 outputs the acquired area data to the structure determination unit 34.

The character recognition data acquisition unit 32 acquires character recognition data MND from the OCR device 20. The character recognition data MND is, for example, information in which character area data is associated with character recognition results indicating characters recognized in that area. The character recognition data acquisition unit 32 outputs the acquired character recognition data MND to the preprocessing unit 33.

The preprocessing unit 33 performs preliminary processing (preprocessing) prior to the determination process for the purpose of making it easier for the structure determination unit 34 (described later) to determine the hierarchical structure. Specifically, the preprocessing unit 33 uses the character recognition data MND acquired from the character recognition data acquisition unit 32 to generate semantic tag information.

The semantic tag information is information in which a tag (meaning tag) corresponding to the meaning of the character shown in the area is added to the character area data. The semantic tag may be any information that indicates that the words are semantically equivalent. Semantic tags are, for example, words that represent semantically equivalent words, and more specifically, words such as "home", "address", "place", "address", etc. This information indicates that the address is "address". By generating semantic tag information, the preprocessing unit 33 can unify semantically equivalent phrases into one phrase. Therefore, compared to the case where the wording is not unified, subsequent processing can be simplified, and the subsequent structure determination unit 34 can easily determine the hierarchical structure.

The preprocessing unit 33 generates semantic tag information by converting the character recognition results in the character recognition data MND into predetermined characters using the conversion table 360 (see FIG. 5). The conversion table 360 is information stored in the storage unit 36, and is information (table) in which characters before conversion and characters after conversion are associated with each other. For example, the character string before conversion in the conversion table 360 shows characters that frequently appear in forms and whose notation may vary. The character string before conversion is an address, place, address, etc. The converted character string shows one character set according to the meaning, for example, the word "address" corresponding to "address, place, address".

The preprocessing unit 33 refers to the conversion table 360 based on the character recognition result in the character recognition data MND. If a character recognition result exists for the character shown before conversion in the conversion table 360, the preprocessing unit 33 acquires the converted character associated with the pre-conversion character. The preprocessing unit 33 converts the character recognition results into converted characters shown in the conversion table 360. The preprocessing unit 33 generates semantic tag information by associating the converted characters with the character area data. The preprocessing unit 33 outputs the generated semantic tag information to the structure determining unit 34. Note that if there is no character recognition result for the character shown before conversion in the conversion table 360, the preprocessing unit 33 associates the character of the character recognition result with the character area data without converting the character recognition result. By this, semantic tag information is generated.

The structure determining unit 34 determines the hierarchical structure of a rectangle using the rectangular area data and semantic tag information. The structure determination unit 34 determines the hierarchical structure of the rectangle using the structure determination model 361. The structure determination model 361 is a learning result obtained by causing a learning model to learn a learning dataset in which rectangular area data, semantic tag information, and rectangular parent IDs are associated with each other. By learning such a training data set, the learning model is trained to accurately output (predict) the parent ID of a rectangle based on the input rectangular area data and semantic tag information. The learning model is, for example, an RNN (Recurrent Neural Network). By using RNN, learning based on ordered sequence information can be performed.

When using an RNN as a learning model, the structure determination unit 34 generates input data such that the order of data input to the structure determination model 361 (hereinafter referred to as input data) has information. The structure determination unit 34 selects a rectangle of interest in the scan image. The rectangle of interest is a rectangle whose hierarchical structure is to be determined. The structure determination unit 34 extracts rectangular area data (hereinafter referred to as a group of neighboring rectangles) within a predetermined range (hereinafter referred to as a first range) from the rectangle of interest. The structure determination unit 34 extracts semantic tag information (hereinafter referred to as a group of neighboring semantic tags) within a predetermined range (hereinafter referred to as a second range) from the rectangle of interest. The predetermined range here may be set arbitrarily. The first range and the second range may be different ranges from each other, or may be the same range. Further, the first range and the second range may be predetermined fixed values, or the first range and the second range may vary depending on the size of the scanned image or the size of the rectangle of interest. It's okay.

The structure determination unit 34 uses, as input data, data obtained by sorting the representative coordinates (for example, center coordinates) of each of the rectangle of interest, a group of neighboring rectangles, and a group of neighboring semantic tags in raster order. The raster order here refers to the reading (or writing) order along a predetermined direction when reading (or writing) pixels arranged two-dimensionally. For example, the raster order is an order along the horizontal direction from the left to the right in the image, and a vertical direction from the top to the bottom. However, the predetermined direction in the raster order may be any direction, and may be an order from the right side to the left side, or an order from the bottom side to the top side.

The structure determination unit 34 determines the parent ID of the rectangle of interest based on the output obtained by inputting the generated input data to the structure determination model 361. The structure determination unit 34 selects all the rectangles in the scanned image one by one as the rectangle of interest, and repeatedly performs the above-described method to determine the parent IDs of all the rectangles. Thereby, the structure determination unit 34 determines the hierarchical structure of the rectangle. The structure determination unit 34 outputs information indicating the determined hierarchical structure of the rectangle, that is, structured data, to the structure data output unit 35. The structured data output unit 35 outputs structured data. The storage unit 36 stores a conversion table 360 and a structure determination model 361.

Note that when inputting the above-mentioned input data and causing the structure determination model 361 to output a rectangular hierarchical structure, it is necessary to generate the input data in the learning dataset using a similar method in the learning stage as well. That is, a rectangle of interest is selected from the learning image, and a group of neighboring rectangles and a group of neighboring semantic tags for the selected rectangle of interest are extracted. Then, input data is data obtained by sorting the representative coordinates (for example, center coordinates) of each of the rectangle of interest, a group of neighboring rectangles, and a group of neighboring semantic tags in raster order. A structure determination model 361 is generated by learning such that the output obtained by inputting input data to the learning model becomes the parent ID of the rectangle of interest.

FIG. 5 is a diagram showing a configuration example of the conversion table 360 according to the embodiment. The conversion table 360 includes items such as, for example, meaning tag ID, after conversion, and before conversion. The meaning tag ID indicates identification information that uniquely identifies the meaning tag. After conversion, the converted characters are shown. Before conversion, the character string before conversion is shown. In this example, the meaning tag ID (E0001) shows "name" as characters after conversion, and "name", "name", and "name" as characters before conversion.

FIG. 6 is a diagram illustrating processing performed by the region dividing apparatus 10 according to the embodiment. FIG. 6 shows examples of character and rectangular regions determined by the region dividing device 10. The region dividing device 10 extracts character regions M1 to M6 and rectangular regions K1 to K5 from the scanned image shown in FIG. Area M1 is an area where the characters "Application Form" are shown. Area M2 is an area where the characters "address" are shown. Area M3 is an area where the characters "prefecture" are shown. Area M4 is an area where the characters "name" are shown. Area M5 is an area where the characters "Date of Entry" are shown. Area M6 is an area where the characters "Year Month Day" are shown. In this way, the region dividing device 10 may extract, for example, a character region as a rectangular (square) shaped region.

Region K1 is a region in which a rectangle surrounding region M2 is shown. Area K2 is an area in which a rectangle surrounding area M3 is shown so that area M3 is placed at the right end of the frame. Area K3 is an area in which a rectangle placed on the right side of area K2 is shown. Region K3 is a region in which a rectangle surrounding region M4 is shown. Area K5 is an area in which a rectangle placed on the right side of area K4 is shown.

FIG. 7 is a diagram illustrating processing performed by the structure recognition device 30 according to the embodiment. FIG. 7 shows an example in which structured data determined by the structure recognition device 30 is visualized in a tree structure. In FIG. 7, area M1# indicates an area after the characters shown in character area M1 have been converted by the preprocessing unit 33. Similarly, the regions M2# to M6# indicate the regions after the characters shown in the character regions M2 to M6 have been converted by the preprocessing unit 33.

The structure recognition device 30 determines the hierarchical structure of a rectangle based on the semantic tag information and rectangular area data in the scanned image shown in FIG. 6, for example. The structure recognition device 30 determines that the parent (dependent source) of the area K1 is the area K2. The structure recognition device 30 determines that the parent of the area K4 is the area K2. The structure recognition device 30 determines that the parent of the region K5 is the region K3.

FIG. 8 is a diagram illustrating an example of layout conversion according to the embodiment. As shown in FIG. 8, consider converting the vertically long form shown in FIG. 6 into a horizontally long layout. In this case, the layout is changed while maintaining the rectangular hierarchical structure determined by the structure recognition device 30. That is, the layout is converted so that the parent of area K1 becomes area K2 and the parent of area K4 becomes area K2. By doing this, it becomes possible to convert the layout so that the necessary items written in the original form can be written in just the right amount and in the same way as the original form. Note that, as shown in this example, regions K6 and K7 may be supplemented as necessary. Area K6 is a rectangular area containing characters of "date". The area K7 is a rectangular area containing the characters "Year Month Day". For example, if it is determined that the parent of area K7 is area K6, by using the determination result, it becomes possible to perform appropriate conversion as shown in FIG. 8.

FIG. 9 is a sequence diagram showing the flow of processing performed by the structure recognition system 1 according to the embodiment. The area dividing device 10 acquires scanned image data (step S10), and generates area data for each of the characters and rectangles by determining the areas of the characters and rectangles in the scanned image (step S11). The structure recognition device 30 generates semantic tag information using the character area data and the character recognition data obtained by character recognition by the OCR device 20 (step S12).

The structure recognition device 30 selects a rectangle of interest from the scanned image (step S13). The structure recognition device 30 obtains a group of neighboring semantic tags for the rectangle of interest (step S14), and obtains a group of neighboring rectangles (step S15). The structure recognition device 30 generates input data by sorting the representative coordinates of the rectangle of interest, the neighboring semantic tag group, and the neighboring rectangular group in raster order (step S16). The structure recognition device 30 determines the parent ID of the rectangle of interest based on the output obtained by inputting the input data to the structure determination model 361 (step S17). The structure recognition device 30 determines whether parent IDs have been determined for all rectangles in the scanned image (step S18), and if there is a rectangle for which parent IDs have not been determined, the process returns to step S13 and the parent IDs are determined for all rectangles in the scanned image. The process of determining is repeated.

As described above, the structure recognition system 1 of the embodiment includes the image data acquisition section 11, the area determination section 13, and the structure determination section 34. The image data acquisition unit 11 acquires image data in a scanned image (an example of a "target image") including characters and rectangles. The area determining unit 13 determines each character and rectangular area in the scanned image. The structure determination unit 34 determines the hierarchical structure of rectangles included in the target image based on the area data. Thereby, the structure recognition system 1 of the embodiment can determine the hierarchical structure of rectangles. Therefore, it is possible to obtain information necessary for changing the layout.

Further, the structure recognition system 1 of the embodiment further includes a preprocessing section 33. The preprocessing unit 33 uses the character area data to generate a character (a ``specific second character'') set according to the meaning corresponding to the character recognition result (an example of a ``first character'') shown in the area. Generate semantic tag information including (an example). As a result, in the structure recognition system 1 of the embodiment, it is possible to tag characters shown in the character area data according to their meanings, and the determination processing by the structure determination unit 34 can be performed by tagging. It is possible to simplify it compared to the case without it.

Further, in the structure recognition system 1 of the embodiment, the structure determination unit 34 determines the hierarchical structure of a rectangle using the structure determination model 361 (an example of a "trained model"). The structure determination model 361 is input using a learning dataset that associates semantic tag information and rectangular area data in a learning image containing characters and rectangles with structured rectangular data included in the learning image. This model is trained to output rectangular structured data included in the image. Thereby, in the structure recognition system 1 of the embodiment, it is possible to recognize a rectangular hierarchical structure by a simple method of inputting data to a trained model.

In the structure recognition system 1 of the embodiment, the structure determination unit 34 selects a rectangle of interest in the scan image, acquires a group of neighboring semantic tags for the rectangle of interest, obtains a group of neighboring rectangles of the rectangle of interest, and The order of input data to be input to the structure determination model 361 is determined by sorting according to the positions of the rectangle of interest, the group of meaning tags, and the group of neighboring rectangles. As a result, in the structure recognition system 1 of the embodiment, it is possible to give meaning (information) to the input data, and use a trained model based on an RNN-based learning model to perform predictions that take into account the order of the input data, i.e. The parent ID can be predicted from the relationship with nearby characters and rectangles, and an improvement in prediction accuracy can be expected.

Further, the structure recognition device 30 of the embodiment includes a region data acquisition section 31 and a structure determination section 34. The area data acquisition unit 31 acquires area data regarding respective areas of characters and rectangles in the scanned image. The structure determination unit 34 determines the hierarchical structure of rectangles included in the scanned image based on the area data. This produces effects similar to those described above.

All or part of the structure recognition system 1 and structure recognition device 30 in the embodiments described above may be realized by a computer. In that case, a program for realizing this function may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read into a computer system and executed. Note that the "computer system" herein includes hardware such as an OS and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, magneto-optical disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into computer systems. Furthermore, a "computer-readable recording medium" refers to a storage medium that dynamically stores a program for a short period of time, such as a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. It may also include a device that retains a program for a certain period of time, such as a volatile memory inside a computer system that is a server or client in that case. Further, the above-mentioned program may be one for realizing a part of the above-mentioned functions, or may be one that can realize the above-mentioned functions in combination with a program already recorded in the computer system. It may also be realized using a programmable logic device such as an FPGA.

Although the embodiments of the present invention have been described above in detail with reference to the drawings, the specific configuration is not limited to these embodiments, and includes designs within the scope of the gist of the present invention.

DESCRIPTION OF SYMBOLS 1... Structure recognition system 10... Area division device 11... Image data acquisition part 12... Modulated image generation part 13... Area determination part 14... Area data output part 15... Storage part 150... Area determination model 20... OCR device 30... Structure recognition Device 31...Region data acquisition unit 32...Character recognition data acquisition unit 33...Preprocessing unit 34...Structure determination unit 35...Structure data output unit 36...Storage unit 360...Conversion table 361...Structure determination model

Claims (8)

an image data acquisition unit that acquires image data in a target image including characters and rectangles;
an area determination unit that determines respective areas of characters and rectangles in the target image;
A hierarchical structure of rectangles included in the target image is determined based on area data regarding the area determined by the area determining unit , and the hierarchical structure of the rectangles included in the target image is to be determined. a structure determining unit that determines a rectangle or character that is a dependent source of the rectangle of interest ;
A structure recognition system equipped with an image data acquisition unit that acquires image data in a target image including characters and rectangles;
an area determination unit that determines respective areas of characters and rectangles in the target image;
a structure determining unit that determines a hierarchical structure of rectangles included in the target image based on area data regarding the area determined by the area determining unit;
a preprocessing unit that uses the area data indicating a character area to generate semantic tag information including a specific second character corresponding to the first character shown in the area;
Equipped with
The structure determining unit determines the hierarchical structure included in the target image using a learned model,
The trained model includes a training dataset in which the semantic tag information in the learning image containing characters and rectangles, the area data indicating the rectangular area, and the hierarchical structure of the rectangles included in the learning image are associated. This is a model trained to output a hierarchical structure of rectangles included in the input image using
Structure recognition system. further comprising a preprocessing unit that uses the area data indicating a character area to generate semantic tag information including a specific second character corresponding to the first character shown in the area;
The structure determining unit determines the hierarchical structure based on the semantic tag information and the area data indicating a rectangular area.
The structure recognition system according to claim 1. The structure determining unit determines the hierarchical structure using a learned model,
The trained model is a training data set in which the semantic tag information and the area data indicating a rectangular area in a learning image including characters and rectangles are associated with the hierarchical structure of rectangles included in the learning image. This is a model trained to output a hierarchical structure of rectangles included in the input image using
The structure recognition system according to claim 3 . The structure determining unit selects a rectangle of interest whose hierarchical structure is to be determined in the target image, and selects a group of neighboring semantic tags that are the semantic tag information located within a predetermined first range from the position of the selected rectangle of interest. , obtain a group of neighboring rectangles that are the area data of a rectangle located within a predetermined second range from the position of the selected rectangle of interest, and obtain the obtained rectangle of interest, the group of neighboring semantic tags, and the neighborhood. determining the order of input data to be input to the trained model by rearranging the rectangle groups according to their positions;
The structure recognition system according to claim 2 or claim 4 . an area data acquisition unit that acquires area data regarding each area of characters and rectangles in the target image;
A hierarchical structure of rectangles included in the target image is determined based on the area data, and among the rectangles included in the target image, a rectangle or a character is a dependent source of the rectangle of interest whose hierarchical structure is to be determined. a structure determination unit that determines the
A structure recognition device comprising: The area data acquisition unit acquires area data regarding each area of characters and rectangles in the target image,
A structure determination unit determines a hierarchical structure of rectangles included in the target image based on the area data , and determines a dependent element of a rectangle of interest whose hierarchical structure is to be determined among the rectangles included in the target image. Determine the rectangle or character that is
Structure recognition method. A program for causing a computer to operate as the structure recognition device according to claim 6 , the program for causing the computer to function as each section included in the structure recognition device.

Images