JP2021056796A - Structure recognition system, structure recognition device, structure recognition method, and program - Google Patents

Abstract

To provide a structure recognition system that can extract information necessary to transform a layout of a document containing a rectangle.SOLUTION: A structure recognition system includes: an image data acquisition unit that acquires image data in a target image including characters and rectangles; an area determination unit that determines each area of the characters and the rectangles in the target image; and a structure determination unit that determines, based on area data related to the area determined by the area determination unit, a hierarchical structure of the rectangles included in the target image.SELECTED DRAWING: Figure 1

Description

The present invention relates to a structure recognition system, a structure recognition device, a structure recognition method, and a program.

There is a technique for extracting character information in an image from a scanned image created by reading a document such as a form with a scanner or the like (see, for example, Patent Document 1-2). The technique of Patent Document 1 discloses a technique of structuring characters in an image based on their positions to facilitate correction of errors in character information. The structuring of Patent Document 1 is described as summarizing character information for each group of information, specifying and expressing the hierarchical relationship of the summarized information. For example, when textual information such as title, document creator, and document creation date is extracted from an image, in the structured data, the title is shown in the highest layer, and the document creator and the document creator are in the lower layer. The document creation date is shown. In the technique of Patent Document 2, character information in an image and feature information indicating the features of ruled lines are extracted. As a result, when searching a document, in addition to the characters described in the document, the characteristics of the ruled lines described in the document can be specified, and the search can be performed efficiently.

On the other hand, with the recent development of computer and communication network technology, information on paper media tends to be replaced with electronic data. By digitizing the form, it is possible to realize efficiency and resource saving of work by making it paperless, and it is also possible to fill in the entry field of the document via an electronic device such as a smartphone. Therefore, the convenience of the user can be improved.

When digitizing forms, the layout is often changed. The aspect ratios of the paper on which the form is printed and the screen of an electronic device such as a smartphone are different from each other, and if the paper form is displayed on the screen of the electronic device as it is without changing the layout, depending on the scale of the display. This is because if a part of the document cannot be displayed or if the entire image is displayed, the image may be reduced considerably and the characters may become difficult to read. When changing the layout, it is necessary to reflect the contents described in the form before conversion in the form after conversion without excess or deficiency. As a countermeasure for this, for example, it is conceivable to apply the technique of Patent Document 1-2 to change the layout of the form. By using the technique of Patent Document 1-2, it is possible to change the layout while maintaining the character structure and the characteristics of the ruled lines described in the form.

Japanese Unexamined Patent Publication No. 2019-82814 Japanese Unexamined Patent Publication No. 2008-40834

However, even if the layout is changed while maintaining the character structure and the characteristics of the ruled lines, the contents described in the form before conversion cannot be reflected in the form after conversion without excess or deficiency. Many forms have an entry frame for entering necessary items. Most of these boxes are represented by simple rectangles that do not contain letters. Character information cannot be extracted from such an entry frame itself. Therefore, in the technique of Patent Document 1, it is difficult to determine the hierarchical structure for all the items described in the form including the rectangle such as the entry frame. Further, even if the characteristics of the ruled line before the change are maintained in the form after the layout change using Patent Document 2, the character is described in any area divided by the ruled line, or is used as an entry frame without being described. If you do not know, you cannot properly perform layout conversion. As described above, it has been difficult to change the layout while maintaining the semantic connection (structure) of the items (including the rectangle) described in the form only by using the conventional technique as it is.

The present invention has been made in view of such a situation, and is a structure recognition system, a structure recognition device, a structure recognition method, and a program capable of extracting information necessary for converting the layout of a document including a rectangle. I will provide a.

In order to solve the above-mentioned problems of the present invention, the present invention determines an image data acquisition unit that acquires image data in a target image including characters and a rectangle, and each region of the characters and the rectangle in the target image. It is a structure recognition system including a region determination unit and a structure determination unit that determines a rectangular hierarchical structure included in the target image based on region data related to the region determined by the region determination unit.

Further, in the above-mentioned structure recognition system, the present invention uses the area data indicating a character area before generating semantic tag information including a specific second character corresponding to the first character shown in the area. The structure determination unit further includes a processing unit, and the structure determination unit determines the hierarchical structure based on the semantic tag information and the area data indicating a rectangular area.

Further, according to the present invention, in the above-mentioned structure recognition system, the structure determination unit determines the hierarchical structure using a learned model, and the learned model is the semantic tag in a learning image including characters and rectangles. The learning data set in which the area data indicating the information and the rectangular area and the hierarchical structure of the rectangle included in the learning image are associated with each other is used to output the hierarchical structure of the rectangle included in the input image. It is a model learned as.

Further, in the above-described structure recognition system, the structure determination unit selects a rectangle of interest for determining the hierarchical structure in the target image, and is within a predetermined first range from the position of the selected rectangle of interest. The neighborhood semantic tag group which is the meaning tag information located in is acquired, and the neighborhood rectangle group which is the area data of the rectangle located within a predetermined second range from the position of the selected rectangle of interest is acquired and acquired. The order of the input data to be input to the trained model is determined by rearranging according to the positions of the rectangle of interest, the neighborhood semantic tag group, and the neighborhood rectangle group.

Further, the present invention has an area data acquisition unit that acquires area data relating to each area of a character and a rectangle in the target image, and a structure that determines a hierarchical structure of a rectangle included in the target image based on the area data. It is a structure recognition device including a determination unit.

Further, in the present invention, the area data acquisition unit acquires area data relating to each area of the character and the rectangle in the target image, and the structure determination unit acquires the area data of the rectangle included in the target image based on the area data. This is a structure recognition method for determining a hierarchical structure.

Further, the present invention is a program for operating the computer as the structure recognition device described above, and is a program for operating the computer as each part included in the structure recognition device.

According to the present invention, it is possible to extract information necessary for converting the layout of a document including a rectangle.

It is a figure which shows the structural example of the structure recognition system 1 which concerns on embodiment. It is a figure explaining the process performed by the structure recognition system 1 which concerns on embodiment. It is a block diagram which shows the structural example of the area division apparatus 10 which concerns on embodiment. It is a block diagram which shows the structural example of the structure recognition apparatus 30 which concerns on embodiment. It is a figure which shows the structural example of the conversion table 360 which concerns on embodiment. It is a figure explaining the process performed by the area division apparatus 10 which concerns on embodiment. It is a figure explaining the process performed by the structure recognition apparatus 30 which concerns on embodiment. It is a figure which shows the example of the layout conversion which applied the structure recognition system 1 which concerns on embodiment. It is a sequence diagram which shows the flow of the process performed by the structure recognition system 1 which concerns on embodiment.

Hereinafter, embodiments of the invention will be described with reference to the drawings.

The structure recognition system 1 of the present embodiment is a system that extracts information necessary for converting the layout of a document including a rectangle.

In the following description, the case where the document to be converted the layout is a form will be described as an example, but the present invention is not limited to this. The target for converting the layout may be any document including at least characters and rectangles, and may be any document such as a questionnaire, a questionnaire, a test question, a fixed phrase template, and an idea sheet. The rectangle included in the document indicates an area surrounded by a rectangle such as a rectangle or a square in the document. The rectangle includes not only an area surrounded by a solid line, but also a rectangular area surrounded by a dotted line or a specific symbol or figure, or a rectangular area divided by the shade of the background color or the like. In addition, the characters included in the document include not only a single character but also a character string composed of a plurality of characters and a character group.

The information required to convert the layout is information indicating the hierarchical structure of characters and rectangles included in the form (hereinafter referred to as structured data). If the hierarchical structure of characters and rectangles included in the form is known, the layout can be converted while maintaining that structure. Therefore, it is possible to maintain the relative positional relationship between the characters and entry fields shown on the form before and after the layout conversion. That is, in order to change the layout while maintaining the contents indicated by the form, it is necessary to extract the structured data of characters and rectangles included in the form.

An example of structured data will be described. As shown in FIG. 6, consider a case where the form includes rectangular areas K1 to K5. As shown in FIG. 7, the structured data of the areas K1 to K3 is information indicating a hierarchical structure in which the area K1 is subordinated to the upper layer and the areas K2 and K3 are subordinated to the upper layer. The structured data of the areas K4 and K5 is information indicating a hierarchical structure in which the area K4 is subordinated to the upper layer and the area K5 is subordinate to the area K5 below the area K4.

The overall configuration of the structure recognition system 1 will be described with reference to FIG. FIG. 1 is a block diagram showing a configuration example of the structure recognition system 1 according to the embodiment. As shown in FIG. 1, the structure recognition system 1 includes, for example, an area division device 10, an OCR device 20, and a structure recognition device 30. These components (area dividing device 10, OCR device 20, and structure recognition device 30) in the structure recognition system 1 are communicably connected.
Although the case where the OCR device 20 performs character recognition is illustrated in FIG. 1, the character recognition processing function may be provided in the area dividing device 10 or the structure recognition device 30, and in this case, the OCR device 20 may be provided. Can be omitted.

The area division device 10 is a device that divides a form into areas such as characters and rectangles shown on the form. The OCR device 20 is a device that performs character recognition processing for recognizing the characters shown in the input image. The structure recognition device 30 is a device that determines a rectangular hierarchical structure shown on a form.

In the following, a case where the structure recognition device 30 specifies the hierarchical structure of the “rectangle” shown in the form will be described as an example. The same method can be applied when specifying the hierarchical structure of "characters" shown in the form.

Further, in the following, as a hierarchical structure, a case where identification information (hereinafter, referred to as a parent ID) of a rectangle or a character that is a subordinate source of a rectangle included in a form is determined will be described as an example. In this case, the structured data is information in which the rectangle and the parent ID of the rectangle are associated with each other. By using the method of determining the parent ID as the hierarchical structure, it is possible to uniquely identify the rectangular structure while suppressing the increase in the data capacity, which is preferable. However, it is not limited to this. As a method of specifying the hierarchical structure of the rectangle, it is also conceivable to determine the identification information (hereinafter, referred to as a child ID) of the rectangle or the character to which the rectangle depends. In this case, since there may be a structure in which a plurality of characters or a rectangle are subordinate to one rectangle, it is necessary to have a configuration in which a plurality of child IDs can be associated with the rectangle, which may cause an increase in data capacity. The method for specifying the hierarchical structure of the rectangle may be any method as long as the hierarchical structure can be specified at least. The method of specifying the hierarchical structure of the rectangle may be a method of associating the parent ID with the rectangle, a method of associating the child ID with the rectangle, or a method of associating both the parent ID and the child ID with the rectangle. Of course, it may be another method.

Here, the processing performed by the structure recognition system 1 will be described with reference to FIG. FIG. 2 is a diagram illustrating a process performed by the structure recognition system 1 according to the embodiment. As shown in FIG. 2, information (scanned image data SD) indicating an image (scanned image) of the form T scanned by a process of reading the form T by a scanner (scan processing SC) is created. The scanned image is an image to be processed by the area dividing device 10. That is, the scanned image is an example of the "target image". The scanned image data SD is input to the area dividing device 10. As a result, the area dividing device 10 acquires the scanned image data SD.

The area division device 10 divides into areas for each element such as characters and rectangles in the form T based on the scanned image data SD. The area dividing device 10 outputs information indicating a character area (character area data MD) and information indicating a rectangular area (rectangular area data KD) in the scanned image. The area division device 10 outputs the character area data MD to the OCR device 20. The character area data MD is an example of "area data". The rectangular area data KD is an example of "area data".

The OCR device 20 performs character recognition processing for recognizing the characters shown in the character area data acquired from the area dividing device 10. The OCR device 20 notifies the structure recognition device 30 of information (character recognition data MND) indicating the content of the recognized character. The character recognition data MND is input to the structure recognition device 30. As a result, the structure recognition device 30 acquires the character recognition data MND.

On the other hand, the character area data MD and the rectangular area data KD output by the area dividing device 10 are input to the structure recognition device 30. As a result, the structure recognition device 30 acquires the character area data MD and the rectangular area data KD.

The structure recognition device 30 determines the rectangular hierarchical structure shown in the form based on the character area data MD acquired from the area dividing device 10, the rectangular area data KD, and the character recognition data MND acquired from the OCR device 20. .. The structure recognition device 30 uses the conversion table 360 to classify the character recognition data MND according to the meaning of the characters. The structure recognition device 30 determines the hierarchical structure of the rectangle by using the information obtained by dividing the character area data MD, the rectangular area data KD, and the character recognition data MND according to the meaning of the character by the structure determination unit 34. The method by which the structure recognition device 30 determines the rectangular hierarchical structure will be described in detail later. The structure recognition device 30 outputs information (structured data KZD) indicating a rectangular hierarchical structure.

Here, the configuration of the region dividing device 10 will be described with reference to FIG. FIG. 3 is a block diagram showing a configuration example of the area dividing device 10 according to the embodiment. As shown in FIG. 3, the area dividing device 10 includes, for example, an image data acquisition unit 11, a modulated image generation unit 12, an area determination unit 13, an area data output unit 14, and a storage unit 15.

The image data acquisition unit 11 acquires the scanned image data SD. The scanned image data SD is, for example, information in which information about an image is associated with each pixel, and is information indicating a grayscale value for each pixel, information indicating an RGB value for each pixel, or the like. is there. The image data acquisition unit 11 outputs the acquired scanned image data SD to the modulation image generation unit 12 and the area determination unit 13.

The modulated image generation unit 12 generates an emphasized image based on the scanned image data SD acquired from the image data acquisition unit 11. The emphasized image is an image in which the pixel value (grayscale value or RGB value) for each pixel in the scanned image is changed based on a predetermined modulation condition.

The modulated image generation unit 12 generates, for example, an image that has been subjected to an enhancement process for emphasizing the edges of the scanned image as an enhanced image. In this case, the modulated image generation unit 12 detects an edge in the scanned image and performs a process of emphasizing the detected edge. The modulated image generation unit 12 detects an edge in the scanned image by, for example, an arbitrary conventional method. The arbitrary method is, for example, a method of detecting an edge by detecting a difference between a scanned image subjected to a median filter process and a smoothed image processed by a Gaussian filter or the like. Alternatively, a method of detecting edges by applying a Laplacian filter or a Sobel filter to the scanned image may be used. The modulated image generation unit 12 sets the detected edge as a specific pixel value (for example, a grayscale value indicating “black” or an RGB value), and sets the pixel value of a pixel that is not detected as an edge to another specific pixel value. A highlighted image is generated by converting to a pixel value (for example, a grayscale value indicating "white" or an RGB value). The modulated image generation unit 12 outputs the image data of the generated emphasized image to the area determination unit 13.

The area determination unit 13 determines areas of characters, rectangles, and backgrounds (elements that are not characters and are not rectangles) in each of the scanned image and the emphasized image. The area determination unit 13 determines characters and rectangular areas in an image using, for example, the area determination model 150. The area determination model 150 is information described in the storage unit 15, and is a learning data set in which an image data is associated with a result of determining each area of a character, a rectangle, and a background in the image. It is a learning result that was made to learn. By training such a learning data set, the learning model is trained so that characters and rectangular areas in the input image can be accurately output (predicted) with respect to the input image data. The learning model is, for example, DCNN (Deep Convolutional Neural Network), but is not limited to this. As the learning model, for example, a method such as CNN, decision tree, hierarchical Bayes, SVM (Support Vector Machine), or a method in which these are appropriately combined may be used.

The area determination unit 13 determines the respective areas of the character, the rectangle, and the background in the scanned image based on the respective determination results in the scanned image and the emphasized image. For example, the area determination unit 13 determines the determination result of the scanned image as it is for the region where the determination result of the scanned image and the determination result of the emphasized image match.

On the other hand, for the region where the determination result of the scanned image and the determination result of the emphasized image do not match, the region determination unit 13 determines the respective regions of the character, the rectangle, and the background based on a predetermined predetermined rule. For example, the area determination unit 13 determines an area determined that at least one of the scanned image and the emphasized image is a character and the other is a background as a character area in the scanned image. The area determination unit 13 determines, for example, an area in which at least one of the scanned image and the emphasized image is rectangular and the other is determined to be the background as a rectangular area in the scanned image. The area determination unit 13 outputs information indicating a character area (character area data MD) in the scanned image and information indicating a rectangular area (rectangular area data KD) to the area data output unit 14.

The area data output unit 14 outputs the character area data MD to the OCR device 20. The area data output unit 14 outputs the character area data MD and the rectangular area data KD to the structure recognition device 30. The storage unit 15 stores the area determination model 150.

Here, the configuration of the structure recognition device 30 will be described with reference to FIG. FIG. 4 is a block diagram showing a configuration example of the structure recognition device 30 according to the embodiment. As shown in FIG. 4, the structure recognition device 30 stores, for example, an area data acquisition unit 31, a character recognition data acquisition unit 32, a preprocessing unit 33, a structure determination unit 34, a structure data output unit 35, and storage. A unit 36 is provided.

The area data acquisition unit 31 acquires area data (character area data MD and rectangular area data KD) from the area dividing device 10. The character area data MD is, for example, information in which coordinates indicating a position in a character area and identification information indicating that the character area is a character area are associated with each other. The rectangular area data KD is, for example, information in which coordinates indicating a position in a rectangular area and identification information indicating that the area is a rectangular area are associated with each other. Here, the coordinates indicating the position in the area are, for example, the coordinates of two vertices located on the diagonal line among the four vertices of the quadrangle when the shape of the area is a quadrangle. Alternatively, the coordinates indicating the position in the region may be the coordinates of a predetermined specific vertex (for example, the lower left vertex) among the four vertices of the quadrangle, and the information indicating the length of each of the vertical and horizontal directions. The area data acquisition unit 31 outputs the acquired area data to the structure determination unit 34.

The character recognition data acquisition unit 32 acquires the character recognition data MND from the OCR device 20. The character recognition data MND is, for example, information in which character recognition results indicating characters recognized in the area are associated with character area data. The character recognition data acquisition unit 32 outputs the acquired character recognition data MND to the preprocessing unit 33.

The pre-processing unit 33 performs pre-processing (pre-processing) prior to the determination processing for the purpose of facilitating the determination of the hierarchical structure by the structure determination unit 34, which will be described later. Specifically, the preprocessing unit 33 generates semantic tag information using the character recognition data MND acquired from the character recognition data acquisition unit 32.

The semantic tag information is information in which a tag (semantic tag) corresponding to the meaning of the character shown in the area is added to the character area data. The semantic tag may be any information indicating that the wording is semantically equivalent. The meaning tag is, for example, a word that represents a word that is semantically equivalent, and more specifically, a word such as "house", "address", "place", or "address" is "address". It is information indicating that it is an "address". By generating the semantic tag information, the preprocessing unit 33 can unify the semantically equivalent words into one word. Therefore, as compared with the case where the wording is not unified, the processing in the subsequent stage can be simplified, and the structure determination unit 34 in the subsequent stage can easily determine the hierarchical structure.

The preprocessing unit 33 generates semantic tag information by converting the character recognition result in the character recognition data MND into a predetermined character using the conversion table 360 (see FIG. 5). The conversion table 360 is information stored in the storage unit 36, and is information (table) in which the characters before conversion and the characters after conversion are associated with each other. For example, the character string before conversion in the conversion table 360 indicates characters that frequently appear in the form and that may have variations in notation. The character string before conversion is an address, a place, an address, and the like. In the converted character string, one character set according to the meaning, for example, the wording "address" corresponding to "address, place, address" is shown.

The preprocessing unit 33 refers to the conversion table 360 based on the character recognition result in the character recognition data MND. When the character recognition result exists in the characters shown before the conversion in the conversion table 360, the preprocessing unit 33 acquires the converted characters associated with the characters before the conversion. The preprocessing unit 33 converts the character recognition result into the converted characters shown in the conversion table 360. The preprocessing unit 33 generates semantic tag information by associating the converted characters with the character area data. The preprocessing unit 33 outputs the generated semantic tag information to the structure determination unit 34. In addition, when the character recognition result does not exist in the character shown before the conversion of the conversion table 360, the preprocessing unit 33 associates the character of the character recognition result with the character area data without converting the character recognition result. This generates semantic tag information.

The structure determination unit 34 determines the hierarchical structure of the rectangle by using the rectangular area data and the semantic tag information. The structure determination unit 34 determines the rectangular hierarchical structure using the structure determination model 361. The structure determination model 361 is a learning result in which a learning model is trained with a learning data set in which rectangular area data, semantic tag information, and a rectangular parent ID are associated with each other. By training such a learning data set, the learning model is trained so that the parent ID of the rectangle can be accurately output (predicted) with respect to the input rectangular area data and the semantic tag information. The learning model is, for example, an RNN (Recurrent Neural Network). By using RNN, learning based on ordered sequence information can be performed.

When RNN is used as the learning model, the structure determination unit 34 generates input data so that the order of the data to be input to the structure determination model 361 (hereinafter referred to as input data) has information. The structure determination unit 34 selects the rectangle of interest in the scanned image. The rectangle of interest is a rectangle whose hierarchical structure is to be determined. The structure determination unit 34 extracts rectangular area data (hereinafter, referred to as a neighborhood rectangle group) in a predetermined range (hereinafter, referred to as a first range) from the rectangle of interest. The structure determination unit 34 extracts semantic tag information (hereinafter, referred to as a neighborhood semantic tag group) within a predetermined range (hereinafter, referred to as a second range) from the rectangle of interest. The predetermined range here may be arbitrarily set. The first range and the second range may be different ranges from each other, or may be the same range. Further, the first range and the second range may be predetermined fixed values, and the first range and the second range may be changed according to the size of the scanned image and the size of the rectangle of interest. You may.

The structure determination unit 34 uses data obtained by sorting the representative coordinates (for example, center coordinates) of the rectangle of interest, the neighborhood rectangle group, and the neighborhood meaning tag group in raster order as input data. The raster order here is a read (write) order along a predetermined direction when reading (or writing) pixels arranged in two dimensions. For example, the raster order is the order along the horizontal direction from the left side to the right side in the image, and the vertical direction from the upper side to the lower side. However, the predetermined direction in the raster order may be any direction, may be an order along the direction from the right side to the left side, or may be an order along the direction from the lower side to the upper side.

The structure determination unit 34 determines the parent ID of the rectangle of interest based on the output obtained by inputting the generated input data to the structure determination model 361. The structure determination unit 34 selects all the rectangles in the scanned image one by one as the rectangles of interest, and repeats the above-mentioned method to determine the parent IDs of all the rectangles. As a result, the structure determination unit 34 determines the rectangular hierarchical structure. The structure determination unit 34 outputs information indicating the determined hierarchical structure of the rectangle, that is, structured data to the structure data output unit 35. The structural data output unit 35 outputs structured data. The storage unit 36 stores the conversion table 360 and the structure determination model 361.

When the above-mentioned input data is input and the structure determination model 361 outputs a rectangular hierarchical structure, it is necessary to generate the input data in the learning data set by the same method also in the learning stage. That is, the rectangle of interest is selected from the image for learning, and the neighborhood rectangle group and the neighborhood meaning tag group for the selected rectangle of interest are extracted. Then, the data obtained by sorting the representative coordinates (for example, the center coordinates) of the rectangle of interest, the neighborhood rectangle group, and the neighborhood meaning tag group in raster order is used as the input data. The structure determination model 361 is generated by learning so that the output obtained by inputting the input data to the learning model becomes the parent ID of the rectangle of interest.

FIG. 5 is a diagram showing a configuration example of the conversion table 360 according to the embodiment. The conversion table 360 includes items such as a meaning tag ID, after conversion, and before conversion. The meaning tag ID indicates identification information that uniquely identifies the meaning tag. After conversion, the converted characters are shown. Before conversion, the character string before conversion is shown. In this example, the meaning tag ID (E0001) indicates "name" as the characters after conversion, and "name", "name", and "name" as the characters before conversion.

FIG. 6 is a diagram illustrating a process performed by the area dividing device 10 according to the embodiment. FIG. 6 shows an example of each area of the character and the rectangle determined by the area dividing device 10. The area dividing device 10 extracts each of the character areas M1 to M6 and the rectangular areas K1 to K5 from the scanned image shown in FIG. The area M1 is an area in which the characters "application form" are shown. The area M2 is an area in which the characters "address" are shown. The area M3 is an area in which the characters "prefecture" are shown. The area M4 is an area in which the characters "name" are shown. The area M5 is an area in which the characters "entry date" are shown. The area M6 is an area in which the characters "year / month / day" are shown. In this way, the area dividing device 10 may, for example, extract a character area as a rectangular (quadrilateral) shaped area.

The area K1 is an area in which a rectangle surrounding the area M2 is shown. The area K2 is an area in which a rectangle surrounding the area M3 is shown so that the area M3 is arranged at the right end in the frame. The area K3 is an area where a rectangle arranged on the right side of the area K2 is shown. The area K3 is an area in which a rectangle surrounding the area M4 is shown. The area K5 is an area where a rectangle arranged on the right side of the area K4 is shown.

FIG. 7 is a diagram illustrating a process performed by the structure recognition device 30 according to the embodiment. FIG. 7 shows an example in which the structured data determined by the structure recognition device 30 is visualized by a tree structure. In FIG. 7, the area M1 # indicates an area after the characters shown in the character area M1 have been converted by the preprocessing unit 33. Similarly, for the areas M2 # to M6 #, the characters shown in the character areas M2 to M6 indicate the area after being converted by the preprocessing unit 33.

The structure recognition device 30 determines the hierarchical structure of the rectangle based on, for example, the semantic tag information in the scanned image shown in FIG. 6 and the rectangular area data. The structure recognition device 30 determines that the parent (subordinate source) of the area K1 is the area K2. The structure recognition device 30 determines that the parent of the area K4 is the area K2. The structure recognition device 30 determines that the parent of the area K5 is the area K3.

FIG. 8 is a diagram showing an example of layout conversion according to the embodiment. As shown in FIG. 8, consider converting the vertically long form shown in FIG. 6 into a horizontally long layout. In this case, the layout is changed while maintaining the rectangular hierarchical structure determined by the structure recognition device 30. That is, the layout is transformed so that the parent of the area K1 becomes the area K2 and the parent of the area K4 becomes the area K2. By doing so, it is possible to convert the layout so that the necessary items described in the original form can be described without excess or deficiency and with the same feeling as the original form. As shown in this example, the regions K6 and K7 may be supplemented as needed. The area K6 is a rectangular area containing the characters "date". The area K7 is a rectangular area containing the characters "year / month / day". For example, when it is determined that the parent of the region K7 is the region K6, it is possible to perform an appropriate conversion as shown in FIG. 8 by using the determination result.

FIG. 9 is a sequence diagram showing a flow of processing performed by the structure recognition system 1 according to the embodiment. The area division device 10 acquires the scanned image data (step S10) and determines the area of the character and the rectangle in the scanned image to generate the area data of each character and the rectangle (step S11). The structure recognition device 30 generates semantic tag information using the character area data and the character recognition data recognized by the OCR device 20 (step S12).

The structure recognition device 30 selects the rectangle of interest from the scanned image (step S13). The structure recognition device 30 acquires the neighborhood meaning tag group in the rectangle of interest (step S14), and acquires the neighborhood rectangle group (step S15). The structure recognition device 30 generates input data by sorting the representative coordinates of the rectangle of interest, the neighborhood meaning tag group, and the neighborhood rectangle group in raster order (step S16). The structure recognition device 30 determines the parent ID of the rectangle of interest based on the output obtained by inputting the input data into the structure determination model 361 (step S17). The structure recognition device 30 determines whether or not the parent ID has been determined for all the rectangles in the scanned image (step S18), and if there is a rectangle for which the parent ID has not been determined, returns to step S13 and returns to the parent ID. The process of determining is repeated.

As described above, the structure recognition system 1 of the embodiment includes an image data acquisition unit 11, an area determination unit 13, and a structure determination unit 34. The image data acquisition unit 11 acquires image data in a scanned image (an example of a “target image”) including characters and rectangles. The area determination unit 13 determines each area of the character and the rectangle in the scanned image. The structure determination unit 34 determines the rectangular hierarchical structure included in the target image based on the area data. Thereby, in the structure recognition system 1 of the embodiment, the rectangular hierarchical structure can be determined. Therefore, it is possible to acquire the information necessary for changing the layout.

Further, the structure recognition system 1 of the embodiment further includes a preprocessing unit 33. The preprocessing unit 33 uses the character area data to set characters (“specific second character”” according to the meaning corresponding to the character recognition result (an example of “first character”) shown in the area. Generate semantic tag information including one example). As a result, in the structure recognition system 1 of the embodiment, the characters shown in the character area data can be tagged according to their meanings, and the determination process by the structure determination unit 34 is tagged. It can be easier than it would be without it.

Further, in the structure recognition system 1 of the embodiment, the structure determination unit 34 determines the rectangular hierarchical structure by using the structure determination model 361 (an example of the “learned model”). The structure determination model 361 inputs using a training data set in which semantic tag information and rectangular area data in a training image including characters and rectangles and structured data of a rectangle included in the training image are associated with each other. It is a model trained to output the structured data of the rectangle included in the image. As a result, in the structure recognition system 1 of the embodiment, it is possible to recognize the rectangular hierarchical structure by a simple method of inputting data into the trained model.

Further, in the structure recognition system 1 of the embodiment, the structure determination unit 34 selects the rectangle of interest in the scanned image, acquires the neighborhood meaning tag group in the rectangle of interest, and acquires and acquires the neighborhood rectangle group in the rectangle of interest. By sorting (sorting) according to the positions of the rectangle of interest, the semantic tag group, and the neighboring rectangle group, the order of the input data to be input to the structure determination model 361 is determined. As a result, in the structure recognition system 1 of the embodiment, the input data can be given meaning (information), and the prediction considering the order of the input data, that is, using the trained model based on the learning model of the RNN system, that is, The parent ID can be predicted from the relationship with nearby characters and rectangles, and improvement in prediction accuracy can be expected.

Further, the structure recognition device 30 of the embodiment includes an area data acquisition unit 31 and a structure determination unit 34. The area data acquisition unit 31 acquires area data relating to each area of the character and the rectangle in the scanned image. The structure determination unit 34 determines the rectangular hierarchical structure included in the scanned image based on the area data. As a result, the same effect as the above-mentioned effect is obtained.

The structure recognition system 1 and the structure recognition device 30 in the above-described embodiment may be realized by a computer in whole or in part. In that case, the program for realizing this function may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read by the computer system and executed. The term "computer system" as used herein includes hardware such as an OS and peripheral devices. Further, the "computer-readable recording medium" refers to a portable medium such as a flexible disk, a magneto-optical disk, a ROM, or a CD-ROM, or a storage device such as a hard disk built in a computer system. Further, a "computer-readable recording medium" is a communication line for transmitting a program via a network such as the Internet or a communication line such as a telephone line, and dynamically holds the program for a short period of time. It may also include a program that holds a program for a certain period of time, such as a volatile memory inside a computer system that serves as a server or a client in that case. Further, the above program may be for realizing a part of the above-mentioned functions, and may be further realized for realizing the above-mentioned functions in combination with a program already recorded in the computer system. It may be realized by using a programmable logic device such as FPGA.

Although the embodiments of the present invention have been described in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs and the like within a range that does not deviate from the gist of the present invention.

1 ... Structure recognition system 10 ... Area division device 11 ... Image data acquisition unit 12 ... Modulated image generation unit 13 ... Area determination unit 14 ... Area data output unit 15 ... Storage unit 150 ... Area determination model 20 ... OCR device 30 ... Structure recognition Device 31 ... Area data acquisition unit 32 ... Character recognition data acquisition unit 33 ... Preprocessing unit 34 ... Structural judgment unit 35 ... Structural data output unit 36 ... Storage unit 360 ... Conversion table 361 ... Structural judgment model

Claims (7)

An image data acquisition unit that acquires image data in a target image including characters and rectangles,
An area determination unit that determines each area of a character and a rectangle in the target image,
A structure determination unit that determines a rectangular hierarchical structure included in the target image based on the area data related to the area determined by the area determination unit.
Structure recognition system equipped with. A preprocessing unit that generates semantic tag information including a specific second character corresponding to the first character shown in the area by using the area data indicating the area of the character is further provided.
The structure determination unit determines the hierarchical structure based on the semantic tag information and the area data indicating a rectangular area.
The structure recognition system according to claim 1. The structure determination unit determines the hierarchical structure using the trained model, and determines the hierarchical structure.
The trained model is a learning data set in which the semantic tag information in a learning image including characters and a rectangle and the area data indicating a rectangular area are associated with the hierarchical structure of the rectangle included in the learning image. Is a model trained to output the hierarchical structure of the rectangles contained in the input image using.
The structure recognition system according to claim 2. The structure determination unit selects a rectangle of interest for determining the hierarchical structure in the target image, and a neighborhood semantic tag group which is the semantic tag information located within a predetermined first range from the position of the selected rectangle of interest. Is acquired, and the neighborhood rectangle group which is the area data of the rectangle located within the predetermined second range from the position of the selected rectangle of interest is acquired, and the acquired rectangle of interest, the neighborhood meaning tag group, and the neighborhood are obtained. By rearranging according to the position of the rectangle group, the order of the input data to be input to the trained model is determined.
The structure recognition system according to claim 3. An area data acquisition unit that acquires area data related to each area of characters and rectangles in the target image,
A structure determination unit that determines the hierarchical structure of rectangles included in the target image based on the area data, and
A structure recognition device comprising. The area data acquisition unit acquires area data related to each area of the character and the rectangle in the target image, and obtains the area data.
The structure determination unit determines the rectangular hierarchical structure included in the target image based on the area data.
Structure recognition method. A program for operating a computer as the structure recognition device according to claim 5, wherein the computer functions as each part included in the structure recognition device.

Images