JP2023165062A - Information reading device, information reading method, and program - Google Patents

Abstract

To provide an information reading device, an information reading method, and a program for reading information such as various tables having various formats.SOLUTION: In an information reading system in which one or a plurality of user terminals can be connected to one or a plurality of management servers via a network, a management server 101 of the information reading system as an information reading device includes: an image management module 212 for acquiring image data on information to be read; a frame recognition module 213 for recognizing a plurality of frame units constituting a table in the image data; and output means for outputting information on the plurality of recognized frame units.SELECTED DRAWING: Figure 2

Description

The present invention relates to an information reading device, an information reading method, and a program.

As background technology in this technical field, there is Japanese Patent Application Publication No. 2006-252575 (Patent Document 1). This bulletin states, ``The results of inputting the statements as an image using a scanner are processed for recognition by subject column and amount column, and the characters in the image information are digitized by a processing means, and the digitized recognized characters are converted into strings and numerical values. and a matching processing means for checking the character string against an account dictionary; a processing means for replacing the misread characters and the determined character string with the correct character string; , including processing to automatically frame account items and amount ranges by selecting the most similar reading frame pattern from the reading frame patterns displayed on the screen (see summary).

Japanese Patent Application Publication No. 2006-252575

Patent Document 1 describes that after scanning is completed, when a tabulation pattern is selected while comparing it with the image displayed on the screen, a frame indicating the reading range is automatically displayed on the image. . However, Patent Document 1 does not consider a mechanism for reading information such as tables having various formats.
Therefore, the present invention provides a mechanism for reading information such as tables having various formats.

In order to solve the above problems, for example, the configurations described in the claims are adopted.
The present application includes a plurality of means for solving the above-mentioned problems, and one example thereof is an image management means for acquiring image data generated by image recognition, and a plurality of means for configuring a table in the image data. The present invention is characterized in that it provides an information reading system comprising: frame recognition means for recognizing the frame portions; and output means for outputting information regarding the plurality of recognized frame portions.

According to the present invention, it is possible to provide a mechanism for reading information such as tables having various formats.
Problems, configurations, and effects other than those described above will be made clear by the following description of the embodiments.

FIG. 1 is an example of the configuration of the entire information reading system 100. FIG. 2 is an example of the hardware configuration of the management server 101. FIG. 3 is an example of the hardware configuration of the user terminal 102. FIG. 4 is an example of user information 400. FIG. 5 is an example of dictionary information 500. FIG. 6 is an example of balance sheet dictionary information 600. FIG. 7 is an example of an information reading flow 700. FIG. 8 is an example of a frame recognition flow 800. FIG. 9 is an example of a frame correction flow 900. FIG. 10 is an example of a linking flow 1000. FIG. 11 is an example of a hierarchy acquisition flow 1100. FIG. 12 is another example of the frame recognition flow 1200. FIG. 13 is another example of the frame correction flow 900. FIG. 14 is an example of a balance sheet 1400. FIG. 15 is an example of a ruled line recognition result 1500. FIGS. 16A and 16B are examples of classification 1600 of frames. FIG. 17 is an example of classification 1700 of frames. FIG. 18 is an example of read information 1800. FIG. 19 is an example of a reading result display screen 1900. FIGS. 20A to 20C are examples of frame recognition results 2000. FIG. 21 is an example of a frame correction screen 2100. FIG. 22 is an example of an annotation confirmation screen 2200. FIG. 23 is another example of the annotation confirmation screen 2300. FIG. 24 is another example of the annotation confirmation screen 2400. FIG. 25 is another example of the annotation confirmation screen 2500. FIG. 26 is another example of the reading result display screen 2600. FIG. 27 is an example of real estate information material 2700. FIG. 28 is another example of the ruled line recognition result 2800. FIG. 29 is an example of an annotation recognition result 2900.

Examples will be described below with reference to the drawings. Note that in each drawing, reference numerals and redundant explanations may be omitted for structures having the same functions.

FIG. 1 is an example of a configuration diagram of an information reading system 100 according to an embodiment.
The information reading system 100 includes one or more management servers 101 and one or more user terminals 102. The management server 101 is an example of an information reading device in the present technology. One or more user terminals 102 are configured to be connectable to one or more management servers 101 via a network. Note that each terminal can send and receive information via the network, regardless of whether the network is wired or wireless.

The user terminal 102 is a terminal used by a user to confirm or correct recognized information using information confirmation content output from the management server 101. Furthermore, in the information reading system 100, the user terminal 102 may be provided separately from the management server 101, or may be provided in the same server as the management server 101. Further, the management server 101 can be used alone as the information reading system 100.

Each terminal 102 and management server 101 of the information reading system 100 may be a mobile terminal such as a smartphone, a tablet, a mobile phone, or a personal digital assistant (PDA), or may be a glasses-type, wristwatch-type, or clothing-type terminal. It may also be a wearable terminal such as. Furthermore, these may be stationary or portable computers, or servers located on a cloud or network. In addition, the function may be a VR (Virtual Reality) terminal, an AR (Augmented Reality) terminal, or an MR (Mixed Reality) terminal. Alternatively, it may be a combination of these multiple terminals. For example, a combination of one smartphone and one wearable terminal can logically function as one terminal. Further, information processing terminals other than these may also be used.

Each terminal of the information reading system 100 and the management server 101 each include a processor that executes an operating system, an application, a program, etc., a main storage device such as a RAM (Random Access Memory), an IC card, a hard disk drive, and an SSD (Solid State Drive). (state drive), auxiliary storage devices such as flash memory, communication control units such as network cards, wireless communication modules, and mobile communication modules, and input devices that detect movement by capturing images of touch panels, keyboards, mice, voice input devices, and cameras. It is equipped with an input device such as, and an output device such as a monitor, display, printer, audio output device, oscillator, etc. Note that it may be a device or terminal that transmits information to be output to an output device, an external monitor, display, printer, device, or the like.

The main storage device stores various programs and applications (software modules), and when a processor executes these programs and applications, each functional element of the overall system is realized. Note that each module may be an independent program or application, or may be implemented as a part of a subprogram or function within one integrated program or application. Furthermore, each of these modules may be implemented as hardware (hardware module) by integrating circuits or employing a macrocomputer.

In this specification, each module is described as an entity (subject) that performs processing, but in reality, a processor that processes various programs, applications, etc. (modules) executes processing.

Various databases (DB) are stored in the auxiliary storage device. A "database" is a collection of data that is organized and collected in a manner that allows for arbitrary data manipulation (eg, extraction, addition, deletion, overwriting, etc.) from a processor or an external computer. The auxiliary storage device is a functional element (storage unit) that stores one or more data sets. The implementation method of the database is not limited, and may be, for example, a database management system, spreadsheet software, or a text file such as XML or JSON.

The information read by the information reading system 100 is not particularly limited. The information reading system 100 according to the present technology may convert the information to be read into image data, for example, from a document in which various information is written on a paper medium. The information to be read is typically, but not limited to, a balance sheet, a profit and loss statement, a profit appropriation plan, a cash flow statement, a statement of changes in shareholders' equity, and a securities report. Financial statements such as books, financial statements, accounting books, and invoices may be used. In addition, the information to be read includes real estate appraisal reports, real estate information materials (for example, materials that compile real estate summaries, floor plans, contract information, etc., and one example is a document called "Maisoku") ) and other real estate information, various contracts, medical records, flow charts, and other various materials.

These materials contain a lot of information, and the information is typically in tabular form, with the contents described for each item. A table may include one or more items (subitems) in one column (item). When such information is converted from paper material into image data, the reading angle may shift. OCR technology has become highly accurate, and if the document is written in a certain format, it may not be very difficult to read if a template or training data is prepared. However, when reading materials written in an unspecified number of formats without templates or training data, when reading many sheets of materials automatically, or when the reading angle is shifted, etc. Can be difficult to read. This technology allows information to be read without selecting a template. Additionally, with this mechanism, if there is some kind of relationship between character strings in the document, such a relationship is associated with the character strings to be recognized. Therefore, the present technology may be particularly useful for reading materials such as those described above.

In the following, the present technology will be described using as an example a case where a balance sheet is the main object to be read by the information reading system 100, and information is read about the balance sheet.

FIG. 2 illustrates the hardware configuration of the management server 101.
The management server 101 is an element that manages the information reading system 100 of this embodiment. The management server 101 is configured by a server placed on a cloud, for example. The management server 101 includes a main storage device 201 and an auxiliary storage device 202. The management server 101 also includes a processor 203 as described above, an input device 204, an output device 205 (an example of an output means), and a communication control unit 206.

The main storage device 201 includes a user terminal management module 211, an image management module 212 (an example of image management means), a frame recognition module 213, a frame correction module 214, a frame selection module 215, a character string recognition module 216, a classification module 217, etc. programs and applications are stored. Each functional element of the management server 101 is realized by the processor 203 executing these programs and applications stored in the main storage device 201.

The auxiliary storage device 202 stores information necessary for the operation of the information reading system 100. The auxiliary storage device 202 stores, for example, user information 400, dictionary information 500, balance sheet dictionary information 600, and the like. The auxiliary storage device 202 also stores, for example, original image data 221, processed image data 222, and the like. Details of this information will be described later.

First, each functional element of the management server 101 will be explained.
The user terminal management module 211 manages the operation of the user terminal 102. The user terminal management module 211 cooperates with the user execution module 311 of the user terminal 102 to control basic operations for reading information, which are executed in the user terminal 102 using the information reading system 100.

For example, the user terminal management module 211 cooperates with the user execution module 311 to display a login page, user information management page, etc. used for information reading executed by the management server 101 on the output device 305 such as the display of the user terminal 102. Output (display). Further, the user terminal management module 211 cooperates with the user execution module 311 of the user terminal 102 to obtain input information input from the user terminal 102 via these pages. For example, the user terminal management module 211 operates based on the acquired information (various information, instructions, etc.), or outputs (stores) the acquired information to the auxiliary storage device 202 as user information 400 or the like.

The image management module 212 acquires image data (hereinafter sometimes simply referred to as "original image data") regarding information to be read. The image management module 212 may obtain original image data via the input device 204, or may obtain original image data from an external device or the like via the communication control unit 206, for example. As an example, the image management module 212 cooperates with the user execution module 311 of the user terminal 102 to obtain original image data via the user terminal 102 . The original image data may be stored in the user terminal 102, for example, or may be generated by the camera 306 or image scanner (an example of the input device 304) of the user terminal 102 by operating the user terminal 102. Good too. Image management module 212 may also obtain original image data stored at a storage location that is different from user terminal 102, for example, by accessing the storage location.

The image management module 212 outputs (records) the acquired original image data 221 to, for example, the auxiliary storage device 202.

The frame recognition module 213 recognizes a plurality of frames forming a table structure in the original image data 221. Information described in materials is often organized and arranged in a tabular structure, for example. In the information reading system 100, a frame recognition module 213 recognizes a frame portion before character recognition is performed for a character string.

The table structure can be regarded as a set of frames that partition each column (cell). Here, when the original image data 221 includes a table, the frame portion may be a set of ruled lines that partition each column of the table. In addition, if the original image data 221 does not include a table, a virtual table that can be provided to partition each character string in accordance with the orderliness of the arrangement of character strings included in the original image data 221. It can be a set of ruled lines that partition each column. The frame generally has a quadrangular shape, and may typically be rectangular. The specific operation of the frame recognition module 213 will be described later.

The frame recognition module 213 includes a character recognition function represented by, for example, an optical character reader (OCR) function and an AI-equipped OCR (Intelligent Character Recognition: ICR). The character recognition function is a function that identifies character strings from image data. Here, the "character string" in this technology is one or more characters or symbols, and at least one of the characters (kana, kanji, alphabets, numbers, etc.) and symbols is arranged singly, or multiple characters are arranged. They are arranged in series. The character recognition function can, for example, recognize a character string from image data as text data.

The character recognition function also includes a ruled line recognition function that recognizes, from the original image data 221, ruled lines that have linearity and are distinguished from character strings. For example, the ruled line is based on the characteristics of lines and points (for example, any one or a combination of two or more of arbitrary point coordinates, starting point, ending point, direction vector, curvature (including linearity), slope, etc.) can be identified. The character recognition function can acquire information on the starting point and end point of a ruled line, as well as information on the intersection point where two different ruled lines intersect.

The character recognition function further includes an annotation creation function that creates an "annotation" that indicates the area to which the character string belongs. The annotation creation function is, for example, a function that adds annotation information (typically an outline or a rectangular bounding box) indicating the area to an area detected as a character string in an image. . The annotation is added while recognizing that the string is a string before the individual characters that make up the string are recognized as text. The character recognition function can also grasp the positional information of pixels in the original image data 221, character strings recognized based on this, ruled lines, intersections, annotations, etc., based on, for example, two-dimensional XY coordinates. ing.

The frame correction module 214 is an additional element of the management server 101, and receives an instruction to correct the frame portion recognized by the frame recognition module 213. The frame correction module 214 cooperates with the user frame correction module 312 of the user terminal 102, for example, and corrects information regarding the frame or annotation based on a frame or annotation correction instruction sent from the user terminal 102. The specific operation of the frame correction module 214 will be described later. Information regarding the corrected frame portion is output (stored) to the auxiliary storage device 202 in association with the original image data 221, for example, as processed image data 222. The information regarding the corrected frame may be recorded (overwritten and saved) instead of the information regarding the original frame, or may be recorded in association with the information regarding the original frame.

The frame selection module 215 is an additional element of the management server 101, and selects one of the plurality of frames recognized by the frame recognition module 213 that satisfies predetermined frame setting conditions. The frame setting conditions will be described later. Information regarding the selected frame portion is outputted (stored) as processed image data 222 to the auxiliary storage device 202 in association with the original image data 221, for example. For example, the frame selection module 215
Only information regarding the selected frame may be stored in the auxiliary storage device 202, or information regarding the selected frame may be stored in the auxiliary storage device 202 together with information (such as a flag) indicating that the frame setting conditions are met. It may also be stored.

The character string recognition module 216 is an additional element of the management server 101, and recognizes, for each of the plurality of frames recognized by the frame recognition module 213, the character string arranged inside the frame. The character string recognition module 216 includes, for example, a character recognition function. This character recognition function may be realized by a common configuration with the character recognition function included in the frame recognition module 213. Alternatively, the frame recognition module 213 may use the character recognition function of the character string recognition module 216.

The classification module 217 is an additional element of the management server 101, and classifies the character string recognition results into, for example, items and values (contents).

FIG. 3 is an example of the hardware configuration of the user terminal 102. The user terminal 102 is a terminal operated by a user who uses this information reading system, and is configured of a terminal such as a smartphone, a tablet, a notebook PC, or a desktop PC. The user terminal 102 includes a main storage device 301 and an auxiliary storage device 302. The user terminal 102 also includes a processor 303 as described above, an input device 304, an output device 305, a camera 306, and a communication control unit 307.

The main storage device 301 stores programs and applications such as a user execution module 311 and a user frame correction module 312, and when the processor 303 executes these programs and applications, each functional element of the user terminal 102 is Realized.

The user execution module 311 controls the basic operations of the user terminal 102. The user execution module 311 controls basic operations for reading information executed using the information reading system 100, for example, in cooperation with the user terminal management module 211 of the management server 101.

The user frame correction module 312 controls operations for correcting information regarding frames read by the management server 101. For example, the user frame correction module 312 cooperates with the frame correction module 214 of the management server 101 to send the original image data 221 and the processed image data including the frame read by the management server 101 to the user terminal 102. Output (display). The user frame correction module 312 also outputs (displays) a screen and user interface (UI) to the user terminal 102, for example, for correcting a frame in the processed image data. The user frame correction module 312 further transmits to the management server 101, for example, an instruction to correct a frame in the processed image data, which is input by the user into the user terminal 102.

The auxiliary storage device 302 stores information necessary for the operation of the user terminal 102. The auxiliary storage device 302 stores, for example, user information 400, original image data 221, processed image data 222, and the like. These pieces of information may be part or all of the information related to the user terminal 102 among the user information 400, original image data 221, processed image data 222, etc. stored in the management server 101.

4 to 6 show various types of information stored in the management server 101. Although not limited thereto, it is assumed that some or all of this information is stored in a JSON format file. Part or all of this information may be stored in a relational database or a non-relational database.

FIG. 4 is an example of user information 400.
User information 400 is information regarding a user who uses information reading system 100. The user information 400 includes information such as a user ID, user display ID, user name, industry ID, industry type, capital, person in charge, address, etc., and a sample value 420 for each field name (item name) 410. The values shown in the example are entered.

The user ID is a symbol attached to each user to identify the user, and is automatically generated as a hash value or the like. Information regarding each user is basically linked by this user ID. The user ID is a primary key that is referenced from other information. The user display ID is a user display ID displayed on the screen of the management server 101, user terminal 102, etc., and can be set arbitrarily. The user display ID can be arbitrarily set by the user, for example, using predetermined symbols or characters within a predetermined condition (for example, 10 characters or less). The user name, capital, person in charge, and address are information regarding the user (including natural persons and corporations; the same shall apply hereinafter), including the name, capital, person in charge of using the information reading system 100, and address or location. It is. The industry ID and industry are information regarding the industry for the user and a symbol for identifying the industry. For the industry type and industry ID, for example, a classification and a classification code according to the Japanese Standard Industrial Classification can be adopted. The user information 400 may also include various other procedural information, user registration information, and the like.

FIG. 5 is an example of dictionary information 500.
The dictionary information 500 is dictionary information regarding character strings (for example, terms) that the information reading system 100 reads. Depending on the purpose, the dictionary information 500 may include, for example, (A) balance sheet information used to read a balance sheet, (B) real estate information used to read real estate information materials, etc. can. The dictionary information 500 includes, for example, information such as a dictionary ID, a dictionary name, a word storage URL, and the like, and values such as those exemplified by sample values 520 are input for each field name 510.

The dictionary ID is an ID for identifying terminology information (dictionary) related to the material to be read. The dictionary name is information indicating a dictionary related to the material to be read. The word storage location URL indicates the storage location of dictionary information. For example, when the dictionary information 500 includes term dictionary information regarding a plurality of materials or technical fields, different storage locations can be prepared for each piece of term dictionary information.

FIG. 6 is an example of balance sheet dictionary information 600.
Balance sheet dictionary information 600 is dictionary information used when the information reading system 100 reads a balance sheet. The balance sheet dictionary information 600 includes, for example, information such as an item 610, a hierarchy 620, a content attribute 630, and the like, and information on items as exemplified by the item 610 is stored.

Item 610 is information regarding terms used as item names on the balance sheet. The item 610 typically includes a "subject" indicating that the item name or content of a row or column of a table in the image data to be recognized is an account, and an item name or content of the row or column of the table. It includes information about "amount" indicating that is an amount, and specific account item names (for example, asset section, current assets, etc.).

In the example of FIG. 6, the item 610 stores information about one word in one column, but information about a plurality of different words indicating the same content (account item) may be stored. For example, the word "asset department" and the word "assets" used with the same meaning may be stored in the same item field.

Hierarchy 620 shows the hierarchy of the words shown in item 610. For example, the account items of a balance sheet are broadly comprised (in other words, as a first layer) of assets, liabilities, and net assets. Further, the assets in the account items further include current assets, fixed assets, and deferred assets, for example (in other words, as a second layer). Current assets in the account may further include, for example (in other words, as a third tier) cash and deposits, accounts receivable, and work in progress. The hierarchy 620 stores information (here, number information from 1 to 8, for example) indicating the hierarchy of such account items (terms). Note that among the words shown in item 610, "subject" and "amount" indicate the item name or content of the row or column of the table, and do not indicate the hierarchy of account items. Information (here, number information of 9) is attached for identification.

The content attribute 630 indicates the content attribute corresponding to the item shown in the balance sheet item 610. For example, the content corresponding to the item "Assets" on a balance sheet includes items and amounts at lower levels, so its attributes are "character string" and "numeric value." Since the contents corresponding to the item "subject" are various account subjects, its attribute is "character string". Since the content corresponding to each account item is the amount of money for each account item, its attribute is "numeric value."

FIG. 14 is an example of a balance sheet 1400. The original image data 221 includes image information corresponding to the balance sheet 1400.
Hereinafter, a method of reading information written in the balance sheet 1400 from the original image data 221 of the balance sheet 1400 using the information reading system 100 will be described.

First, the user execution module 311 of the user terminal 102 is executed by a user operation, and the user terminal 102 and the management server 101 are connected. The user terminal management module 211 of the management server 101 cooperates with the user execution module 311 of the user terminal 102 and the like to execute processes such as user login.

FIG. 7 is an example of an information reading flow 700. The information reading system 100 typically reads the information described in the balance sheet 1400 according to each step (S710 to S770) of the information reading flow 700.
First, the image management module 212 acquires the original image data 221 from which information is to be read (S710). For example, the image management module 212 acquires image data by scanning materials with a scanner, which is the input device 204 of the management server 101. The image management module 212 also acquires image data transmitted from the user terminal 102.

The image management module 212 can also output (display) on the display of the user terminal 102 a screen for specifying image data from which information is to be read, in cooperation with the user execution module 311 of the user terminal 102 . This allows the user to specify the image data from which information is to be read. At this time, the user may be configured to be able to specify or select the type of material to be read via the user terminal 102.

The image data designation method may be, for example, to select the original image data 221 to be read from among the image data already stored in the auxiliary storage device 302 of the user terminal 102. Alternatively, the method of specifying the image data may be, for example, image data generated by scanning or imaging the balance sheet 1400 made of a paper medium using the scanner (an example of the input device 304) or the camera 306 of the user terminal 102. may be the original image data 221 to be read.

The specified original image data 221 is sent from the user terminal 102 to the management server 101. Image management module 212 receives original image data 221 and stores it, for example, in auxiliary storage device 202.

Next, the frame recognition module 213 recognizes the frame in the original image data 221 (S720).
FIG. 8 is an example of a frame recognition flow 800.
The frame recognition module 213 uses the ruled line recognition function to acquire information about ruled lines included in the original image data 221 and their intersections (S810). Although the positional information of the intersection point is not limited to this, for example, XY coordinate information where the lower side of the character string in the vertical direction is the Y-axis direction, and the right side of the character string in the left-right direction is the X-axis direction. It can be expressed by

FIG. 15 is an example of a ruled line recognition result 1500.
The ruled line recognition result 1500 corresponds to an image from which ruled lines and intersection points are extracted, which are included in the original image data 221. The frame recognition module 213 sets the upper leftmost intersection among the ruled lines and intersections for which information has been acquired as the first intersection P1 (S820). In other words, the intersection point with the minimum distance from the upper left corner (origin) of the image is set as the first intersection point P1.

The frame recognition module 213 sequentially sets the intersections that exist when proceeding in the X-axis direction (rightward) on the ruled line (latitude line) from the first intersection P1 as intersections X1, X2..., Xn (n is a natural number). . The frame recognition module 213 sequentially sets the intersections that exist when proceeding in the Y-axis direction (downward) on the ruled line (meridian) from the first intersection P1 as intersections Y1, Y2..., Ym (m is a natural number) ( S830). Here, in the frame recognition module 213, the intersection point adjacent to the first intersection point P1 in the X-axis direction (to the right) is the intersection point X1. In the frame recognition module 213, an intersection point adjacent to the first intersection point P1 in the Y-axis direction (downward) is an intersection point Y1.

Then, the frame recognition module 213 detects when there are line segments connecting the first intersection point P1 and the intersection points X1 and Y1, and line segments orthogonal to these line segments, and the orthogonal line segments intersect at the second intersection point P2. , the rectangle P1-X1-P2-Y1 is recognized as a frame (S840). The frame recognition module 213 detects the first intersection point P1 even if the intersection points X1, Y1, and P2 exist, and when any one of the line segments P1-X1, X1-P2, P2-Y1, and Y1-P1 does not exist. The frame is not recognized. Furthermore, even if there are intersection points X1, Y1, line segments P1-X1, Y1-P1, and line segments orthogonal to these line segments, if the orthogonal line segments do not intersect at the second intersection point P2, , the frame portion is not recognized for the first intersection point P1.

From the above, the frame recognition module 213 recognizes two adjacent ruled lines along the X-axis direction (an example of the first direction) and the Y-axis direction (an example of the second direction intersecting the first direction). When two adjacent ruled lines intersect at four intersections, the line segment arranged between these four intersections is recognized as a frame.

If the frame is not recognized between the first intersection P1 and the intersections X1 and Y1, the frame recognition module 213, for example, replaces the intersection X1 with the intersection X2 and determines whether or not to recognize the frame as described above. do. The frame recognition module 213 changes the intersection point Xn one by one until it recognizes the frame portion between the first intersection point P1 and the intersection points Xn and Y1. Note that if the frame is not recognized, the frame recognition module 213 replaces the intersection Y1 with the intersection Y2, and determines whether or not the frame is recognized between the first intersection P1 and the intersections X1 and Y2. This process is repeated until the frame is recognized between the first intersection P1 and the intersections Xn and Ym. If no frame is recognized between the first intersection P1 and the intersections Xn and Ym, the frame recognition module 213 determines that no frame is recognized for the first intersection P1.

As described above, the frame recognition module 213 determines whether or not to recognize a frame with respect to the first intersection point P1, and then replaces the first intersection point P1 with the next intersection point X1 on the same latitude line, and continues with the above. Similarly, it is determined whether or not the frame portion is recognized for the replaced first intersection point P1. This process is repeated until the first intersection point P1 is replaced with the intersection point Xn-1. This makes it possible to recognize all the frame parts that exist in the left-right direction along the first latitude line.

The frame recognition module 213 subsequently sets the upper leftmost intersection as the first intersection point P1, excluding the intersection that has already been set as the first intersection point P1. Then, similarly to S830 and S840 above, it is determined whether or not a frame portion is recognized for the first intersection P1. This process is repeated until the first intersection P1 is replaced with the intersection Xn-1 on the same latitude. Thereby, all the frame parts that exist in the left-right direction along the second latitude line can be recognized.

The frame recognition module 213 repeats the above process exhaustively for all latitude lines. Thereby, all the frames included in the original image data 221 can be recognized. Note that those skilled in the art will understand that the frame portion may be recognized by scanning vertically along meridians instead of latitude lines.

The information regarding the frame recognized by the frame recognition module 213 is output (stored) to the auxiliary storage device 202 in association with the original image data 221, for example, as processed image data 222.

Furthermore, as shown in FIG. 14, the balance sheet 1400 may include a plurality of character strings arranged within one frame. Therefore, the frame recognition module 213 may be configured to recognize one frame for one character string. For example, the frame recognition module 213 is configured to recognize a character string from the original image data 221, create an annotation corresponding to the part where the character string exists, and recognize the created annotation as a frame. Good too. Note that if the annotation has a frame shape surrounding the area occupied by the character string, the frame recognition module 213 can use the annotation as it is as a frame, for example. On the other hand, if the annotation has a planar shape corresponding to the area occupied by the character string, the frame recognition module 213 can recognize the outline of the annotation as a frame.

For example, the frame recognition module 213 can recognize a frame based on information regarding annotations in addition to recognizing a frame based on information on ruled lines and intersections.
FIG. 12 is another example of the frame recognition flow 1200. Note that the frame recognition module 213 can recognize a frame based on at least one of information regarding the annotation and information on ruled lines and intersections (at least one of the frame recognition flows 800 and 1200). In the following, an embodiment in which the frame recognition flow 800 is executed first and then the frame recognition flow 1200 is added will be described as an example. That is, when the recognition target area of the image data includes a frame made of ruled lines, the frame recognition module 213 generates an annotation for the inside of this frame as a new recognition target area, and generates a new annotation based on the multiple annotations. The frame can be recognized. Note that in the frame recognition flow 1200, S1220 and S1230 are not essential steps, but are steps that can be executed additionally.

Specifically, the frame recognition module 213 creates an annotation for the character string arranged inside each frame portion recognized based on the information of ruled lines and intersections (S1210). Although the frame recognition module 213 is not limited to this, the annotation is shaped into a frame by a line along a first direction along the character string and a line along a second direction that intersects the first direction. It is recommended to form the Although the annotation is not limited to this, it may be formed to have the minimum circumscribed rectangle of the recognized character string or a similar shape thereof. Thereby, information regarding the shape of the annotation can be simplified.

The frame recognition module 213 enlarges the created annotation by a predetermined size (S1220). At this time, the "predetermined size" can be defined as, for example, a relative value to the size of one character in a character string or the pitch between characters (interline pitch). The "predetermined size" is not limited to this, but for example, based on the size of one character in a character string, 0.1 character or more and 2 characters or less (in other words, the size of one character 1.1 times or more and 3 times or less). As a result, annotations can be recognized from character-based blocks to word-based or sentence-based blocks, or even as phrase-based blocks consisting of multiple sentences. The above-mentioned "predetermined size" may be determined for each piece of information to be read, for example, or may be set as appropriate by machine learning or the like.

Here, if the enlarged plurality of annotations do not overlap, the frame recognition module 213 determines that each character string is independent of each other, and recognizes the annotation of each character string as a frame (S1250). On the other hand, when annotations are enlarged in this way, a plurality of enlarged annotations may overlap. It can be considered that the multiple superimposed annotations were originally one annotation that should be added to one character string. Therefore, when the enlarged annotations are superimposed, the frame recognition module 213 connects the superimposed annotations to generate one new annotation (S1230). Thereby, it is possible to reduce the occurrence of a situation in which a single character string is mistakenly recognized as multiple character strings due to the pitch between characters or the like.

For example, when the width of an annotation is expanded horizontally by 0.1 character, if the interval between consecutive adjacent characters is smaller than 0.1 character, the annotation will be connected to the annotation of the adjacent character part. On the other hand, two character strings that are separated by, for example, one character or more will not be combined.
Further, for example, when the width of an annotation is expanded horizontally by two characters, if the distance between the adjacent character string part is two characters or less, it will be connected to the annotation. On the other hand, two character strings that are separated by, for example, three characters or more will not be combined.

The frame recognition module 213 acquires information regarding the upper left coordinate and the lower right coordinate for each annotation, when the area occupied by each annotation is expressed in two-dimensional XY coordinates (S1240). More specifically, the frame recognition module 213 selects a first coordinate A1 that includes the annotation and is represented by the smallest X coordinate and the smallest Y coordinate, and the largest one among the XY coordinates occupied by the annotation. Information regarding the second coordinate A2 represented by the X coordinate and the largest Y coordinate is acquired. Then, the frame recognition module 213 recognizes a rectangle (frame shape) whose diagonal points are the first coordinate A1 and the second coordinate A2 as a frame (S1250). The frame recognition module 213 outputs (stores) information regarding the recognized frame to the auxiliary storage device 202 as, for example, processed image data 222 (S1260). In this embodiment, a frame recognized as one frame in the frame part recognition flow 800 is subdivided into a plurality of frames for recognition.

FIG. 29 is an example of an annotation recognition result 2900 in each step of the frame recognition flow 1200.
As shown in Figure 29, in various information materials, etc., when it comes to combinations of items and values, the character strings for items have a small number of characters, while the character strings for values have a large number of characters compared to the items. It could be more. For example, a string of values may include multiple sentences spanning multiple lines.

When creating an annotation for a character string arranged in one frame (S1210), if the character string has a large number of characters and especially includes punctuation marks, the character string is The whole string is not recognized as a single block (value), but is easily recognized as being divided into multiple character strings (values). On the other hand, as already explained, the frame recognition module 213 enlarges the plurality of recognized annotations according to the frame recognition flow 1200 (S1220), and as for the annotations to be superimposed, as shown in the recognition result (C). can be connected to each other (S1230). Furthermore, the frame recognition module 213 acquires information regarding the first intersection point P1 (top left coordinates) and second intersection point P2 (bottom right coordinate acquisition) for the connected annotations, and based on these first intersection point P1 and second intersection point P2. The frame portion is recognized (S1240). As a result, as shown in the recognition result (D), even a character string with an extremely large number of characters can be recognized as one character string (that is, one value).

For each frame recognized by the frame recognition module 213, the character string recognition module 216 uses a character recognition function to recognize each character constituting the character string arranged inside the frame (S730). For example, the character string recognition module 216 outputs (stores) information regarding the recognized character string to the auxiliary storage device 202 as processed image data 222 or the like. The character string recognition module 216 may be configured, for example, to pass (output) information regarding the recognized character string to the classification module 217.

For example, as a characteristic of materials created by people, character strings are configured to include combinations of items and values, and these may generally be strings in which one item is associated with one or more values. Therefore, the classification module 217 classifies the character string recognized by the character string recognition module 216 into, for example, items and values (S740). The classification module 217 can classify a character string into items and values based on, for example, but not limited to, a dictionary that includes information about items.

(Classification method 1)
For example, the classification module 217 may determine that a character string is a value if it consists only of numbers, and may determine that it is an item if the character string includes characters. In addition, if a predetermined character string such as year, fiscal year, yen, ¥, dollar, $, etc. is used consecutively with a numerical value, the character string will be recognized as a numerical unit, and the numerical value with the relevant unit attached will be changed. It can be determined as one value. These predetermined character strings may be stored in a dictionary as one piece of information regarding values, for example, and the classification module 217 may determine that a numerical value to which the unit is attached is a value based on such a dictionary. good.

(Classification method 2)
A characteristic of materials created by people is that headings may be added at the beginning of items using specific characters or symbols (hereinafter simply referred to as "specific characters"). In this case, as specific characters, characters generally attached to items in the same hierarchy can be considered, such as ordinal numbers (for example, 1., (1), 1), i, I, round (numbers, etc.), alphabets (e.g., A, B, C), parentheses (e.g., round brackets, square brackets, corner brackets, etc.), middle dots, bullet points (e.g., center circle, double circle, triangle symbol, (square symbol, cross mark, etc.), section mark, etc., or a combination of two or more. These specific characters have the characteristic (regularity) that two or more (for example, three or more) of these specific characters are repeatedly arranged in the same layer in the material.

Therefore, in the classification module 217, for a character string extending in one direction (for example, the left-right direction), if a specific character is placed at the left end more than once, and in another direction (for example, For example, for character strings extending in the vertical direction, if a specific character is placed at the top twice or more, it can be determined that these character strings are items. In addition, if a character string containing these specific characters is arranged more than once in one direction (for example, left/right or up/down direction), the classification module 217 will All character strings arranged in other directions (for example, on the bottom or right side) can be classified as values corresponding to the item, and can be correlated with each other.

Further, the classification module 217 can also consider, for example, a character string written in bold (an example of specific character information) relative to other character strings as an item. Further, a character string that uses a specific font or color (an example of specific character information) that is different from other characters can also be regarded as an item.
The above-mentioned specific characters and specific character information may be stored in a dictionary as one of the information regarding items, for example, and the classification module 217 determines that the character string is an item based on the dictionary. Good too.
Further, rules for associating items and values based on the arrangement of specific characters may be stored in a dictionary as one piece of information regarding the correspondence between items and values, and the classification module 217 may The item and the value may be associated with each other.

Note that when classifying character strings into items and values using classification method 2, the classification module 217 can perform linking (S750) and structuring (S760) at the same time as classifying these character strings.

(Classification method 3)
Alternatively, the classification module 217 may classify the character strings recognized by the character string recognition module 216 into items and values based on dictionary information prepared in advance. For example, based on the balance sheet dictionary information 600 of the dictionary information 500, the classification module 217 determines that among the recognized character strings, a character string that corresponds to item 610 of the balance sheet dictionary information 600 is an item. , a configuration may be provided in which a character string that does not correspond to item 610 is determined to be a value.

Note that when classifying character strings into items and values using classification method 3, the classification module 217 uses information regarding the hierarchy 620 and content attributes 630 of the balance sheet dictionary information 600 for the character strings that correspond to the items 610. Based on this, hierarchical information of the character string and attribute information of the string to be linked may also be acquired. This makes it possible to improve the accuracy of the classification by the management server 101, as well as the linking and hierarchization (structuring) described below.

The classification module 217 associates the classified character strings with items and values (S750). Furthermore, the classification module 217 obtains hierarchical information of character strings classified into items and values (S760). Either of the character string linking (S750) and the hierarchical information acquisition (S760) may be performed first, or both may be performed simultaneously (in parallel).

FIG. 10 is an example of a linking flow 1000.
FIG. 11 is an example of a hierarchy acquisition flow 1100.
The classification module 217 acquires the recognition result of the character string (S1010), associates the item and value of the character string (S1020), and outputs the result to the auxiliary storage device 202. The classification module 217 also obtains the recognition results of the character strings (S1110), obtains information regarding the hierarchy for each character string (S1120), and outputs the results to the auxiliary storage device 202.

(Linking method 1)
The classification module 217 is configured to, for example, link character strings or create a hierarchical structure based on the characteristics of the arrangement of frames in the ruled line recognition result 1500, although the classification module 217 is not limited thereto. Good too. For example, in the ruled line recognition result 1500, there are a plurality of (two in this case) frame portions W1 and W2 arranged at the uppermost position. Furthermore, a ruled line L1 arranged between these frame portions W1 and W2 extends from the top of the table to the bottom. Further, for example, one or more frame portions are arranged below the frame portion W1 so as to have the same width as the frame portion W1. One or more frame portions are arranged below the frame portion W2 so that the width thereof is equal to that of the frame portion W2.

FIGS. 16A and 16B are examples showing how to classify frames 1600.
The fact that the ruled line L1 extends from the top to the bottom means that the ruled line L1 has the same length as the outer frame parallel to the ruled line. In such a case, the classification module 217 determines that the table can be divided into two by a ruled line L1 having the same length as the outer frame, as shown in FIG. 16A, for example. The classification module 217 divides the table consisting of the ruled line recognition results 1500 into left and right parts, a table part including the frame part W1 and a table part including the frame part W2.

Note that the ruled line forming the lower sides of the frame portions W1 and W2 and two ruled lines that are parallel to the ruled line and have the same length can also satisfy the same conditions as the ruled line L1. However, for example, if the table is divided into two vertically by the ruled line forming the lower sides of the frames W1 and W2, the upper divided table will have a frame (cell: the smallest unit of a partitioned frame; also referred to as a cell). There are only W1 and W2, which is significantly smaller than the number of cells included in the divided lower table, and can be determined to be inappropriate as a table dividing line. Therefore, for example, the classification module 217 can determine the priority of a ruled line to be adopted as a dividing line for dividing a table based on the difference in the number of cells included in each table after being divided based on the ruled line. . For example, the classification module 217 can give higher priority as a dividing line to a ruled line with a smaller difference in the number of cells in the table after division. Therefore, in the case of the table in FIG. 15, the classification module 217 lowers the priority of the lower sides of the frames W1 and W2 as dividing lines, and reduces the number of frames (cells) included in the divided table. A ruled line L1 with a smaller difference is adopted as a dividing line.

Further, the right portion of the table including the divided frame portion W2 shown on the right side of FIG. 16(A) includes a frame portion W3 having the same dimensions as the frame portion W2. The frame portions W2 and W3 have the same length as the outer frame (in this case, the outer frame of the right portion of the table after division). In such a case, the classification module 217 classifies the table portion including the frame portion W2 into the portion above the frame portion W3 (the table portion including the frame portion W2) and the frame portion W2, as shown in FIG. 16(B), for example. It is determined that the image can be divided into an upper and a lower portion, a portion below the portion W3 (a front portion including the frame portion W3).

The classification module 217 examines whether the three tables after division can be further divided. In each of the three tables, the ruled lines forming the lower sides of the frames W1, W2, and W3 extend from the left end to the right end of the outer frame without being crossed by other ruled lines. Note that two ruled lines parallel to and of the same length as the ruled lines forming the lower sides of the frame portions W1, W2, and W3 may also satisfy the same conditions. Here, when each table portion is divided by the ruled line forming the lower side of the frame portions W1, W2, and W3, each of the frame portions W1, W2, and W3 becomes a single cell after division. Such frame portions W1, W2, and W3 are often item cells in which character strings at a relatively higher level than other divided cells are written. Therefore, the classification module 217 can determine the priority of the ruled line to be adopted as the dividing line for partitioning the item cells, based on the number of cells included in each table after being divided based on the ruled line. For example, the classification module 217 can raise the priority as a dividing line for a ruled line where the number of cells in the table after division is 1, and can recognize the single cell after division as an item cell. Note that the dividing line that partitions the item cell may also have the characteristic that although it intersects with other ruled lines, it does not intersect with other ruled lines. Therefore, the classification module 217 may determine that a ruled line that is not crossed by other ruled lines and has the same length as one side of the outer frame is a dividing line that partitions item cells.

As a result of the division, the classification module 217 determines that the frame parts W1, W2, and W3, which are independent cells that do not include other frame parts, are the highest hierarchy of the three tables after the division, and divides them into the frame part W1. , W2, and W3 are determined to be frames in a lower hierarchy relative to the frames W1, W2, and W3. Then, the classification module 217 ends the table division process when a single cell (single frame) appears as a result of the division. The classification module 217 can hierarchize and associate the frame portions W1, W2, and W3 with the frame portions arranged below the frame portions W1, W2, and W3 (first hierarchization and ).

FIG. 17 is an example of classification 1700 of frames.
As shown in FIG. 16(B), in the table portion made up of frame sections lower in hierarchy than frame sections W1, W2, and W3, ruled lines L2, L3, and L4 extend from the top to the bottom, respectively. In such a case, the classification module 217 determines that the table portions lower than the frame portions W1, W2, and W3 can be partitioned into left and right by ruled lines L2, L3, and L4, respectively, as shown in FIG. 17, for example. . That is, the lower front part of the frame W1 is divided into the frame W11 and the frame parts W13 and W15 arranged below it, and the frame part W12 and the frame parts W14 and W16 arranged below it ( classification). The lower front part of the frame part W2 can be divided into a frame part W21 and frame parts W23 and W25 arranged below it, and a frame part W22 and frame parts W24 and W26 arranged below it. can. Regarding the lower front part of the frame part W3, the frame part W31 and the frame parts W33, W35, W37 arranged below it, and the frame part W32 and the frame parts W34, W36, W38 arranged below it. It can be partitioned. The classification module 217 can hierarchize and associate the frame parts W11 to W32 and the frame parts arranged below the frame parts W11 to W32 (second hierarchization and linking).

However, the frame portion W11 and the frame portions W13 and W15 disposed below the frame portion W12 and the frame portions W14 and W16 disposed below the frame portion W12 are inseparably associated with each other by the frame portion W1. . Therefore, the character strings in the frame W11 and the frames W13 and W15 arranged below it, and the character strings in the frame W12 and the frames W14 and W16 arranged below it, are They can be associated with each other along the same lines (third layering and linking). As for the frame portions W13 and W14, a plurality of frame portions recognized based on the annotation exist inside each of the frame portions W13 and W14. The classification module 217 can also associate the plurality of frames in the frame part W13 and the plurality of frames in the frame part W14 with each other along the left-right direction.

The classification module 217 classifies the frames W21 and W22, the frames W23 and W24, the frames W25 and W26, the frames W31 and W32, the frames W33 and W34, and the frames W35 and W35. The portion W36, the frame portion W37, and the frame portion W38 can be similarly associated (third hierarchization and linkage).

In addition, the classification module 217 similarly performs the hierarchization and linking of character strings when the characteristics of the arrangement of the frames in the ruled line recognition result 1500 differ by 90 degrees (when the matrix structure is an inverted table structure). You can do the following.

(Linking method 2)
Alternatively, the classification module 217 may associate the character strings placed in each frame based on a predetermined table rule.

The classification module 217 first classifies the character strings placed in each frame into items and values using the method described above (S740). At this time, it is confirmed whether or not a predetermined regularity (rule) is observed in the appearance of items and values in order along at least one of the X-axis direction and the Y-axis direction. For example, the classification module 217 may check whether a predetermined rule is observed for character strings that are determined to be in the same hierarchy in the above-mentioned "linking method 1". The classification module 217 attaches a flag to each rule when a predetermined rule is seen, and when the same rule appears multiple times (for example, three or more times) in reading one material, the classification module 217 adds that rule to the table. A common rule is set, and items and contents are linked according to this rule (S760). Examples of how items and values appear and how each character string is linked (rules) include the following.
Note that the classification module 217 may use the same rule as a common rule when the same rule appears multiple times in one table, or may use the same rule as a common rule when the same rule appears multiple times for all materials having multiple tables. Often, it can be both.

(1) For example, if the attributes of character strings in a certain array are "letter", "number", "letter", and "number" in order, the first "letter" and "number" are stringed as one combination. and link the following "letters" and "numbers" as a single combination.
(2) For example, if the attributes of character strings in a certain array are "letter", "number", "number", "letter", and "number" in order, then the first "letter", "number", ""Numbers" are linked as one combination, and the following "letters" and "numbers" are linked as one combination.
(3) For example, if the attributes of character strings in a certain array are "letter", "number", "number", and "number" in order, "letter" is the item, and the following three "numbers" are the values. Assigned to the item in question.

Through such processing, the classification module 217 can easily classify the character strings inside the frame into items and values and link them. Although the character strings are not limited to this, for example, they can be linked in a key-value format. As a result, even when reading information from a large amount of materials, for example, the information can be stored without having to define a schema in advance.
For example, after identifying an item or a hierarchical structure of items, the classification module 217 can identify and link values that exist near the item. For example, values having the same hierarchical structure as the item's hierarchical structure can be associated with each item. Additionally, a character string that exists immediately to the right of an item can be associated as a value. Furthermore, when the structure of items and values exists in a vertical positional relationship, the character string that exists immediately below the item can be associated as a value.

FIG. 18 is an example of read information 1800.
The classification module 217 outputs (records) the read information 1800 to, for example, a JSON format file (S770). The classification module 217 outputs (stores) the read information 1800 as key-value structured items and values. However, the data structure of read information 1800 is not limited to this example. The read information 1800 may be of a type other than the key-value type, for example, a non-relational type represented by a sorted column-oriented type, a document-oriented type, a graph-oriented type, etc., a hierarchical type, a network type, etc. good.

FIG. 19 is an example of a reading result display screen 1900.
The user terminal management module 211 cooperates with the user execution module 311 of the user terminal 102, for example, and outputs (displays) the reading result display screen 1900 on the display of the user terminal 102 (an example of the output device 305).

The user terminal management module 211 displays an image 1901 based on the original image data 221 and an image 1902 (frame recognition result) based on the processed image data 222 side by side vertically or horizontally on the reading result display screen 1900. . This makes it possible to easily compare and confirm whether the management server 101 has properly read the material. Note that in the image 1902, the display of character strings is omitted to make it clear that the recognition result of the frame is displayed.

The user terminal management module 211 displays, for example, a slide bar icon 1903, a previous page button 1904, a next page button 1905, and a mode switching button 1906 on the reading result display screen 1900. Previous page button 1904 and next page button 1905 are used to display the reading results for the previous material and the next material, respectively, when information is being read for multiple materials. , is a button for turning the page.

The user terminal management module 211 may display, on the reading result display screen 1900, a move button or the like that jumps to a page displaying the target material reading result. The mode switching button 1906 is a button for switching to the correction mode, for example, when the recognition result image 1902 requires correction, and is also a button for confirming the correction contents when the correction is completed in the correction mode. This button is used to return to normal mode (confirmation mode). The slide bar icon 1903 is, for example, a UI used in correction mode to make various corrections to recognition results and the like.

FIGS. 20A to 20C are other examples of frame recognition results 2000.
Recognition result (A) shows that character strings are read with high precision using the character recognition function, so when recognizing frames based on ruled lines and intersections, the characters (in this case, the kanji ``bu'', ``koku'', and ``go'') are recognized. ”) is mistakenly recognized as a frame. Although the recognition result (B) is not as good as the recognition result (A), it still erroneously recognizes a part of the character (in this case, the kanji "go") as a frame. Recognition result (C) shows that part of the character is not recognized as a frame, but instead there are parts that should be recognized as multiple frames but are recognized as one frame in an excessively large size. .

In order to reduce such poor frame recognition, the frame selection module 215 selects one of the plurality of frames recognized by the frame recognition module 213 that satisfies predetermined frame setting conditions. The user terminal management module 211 is configured to output information regarding the plurality of selected frames.

(Correction method 1)
FIG. 9 is an example of a frame correction flow 900.
In the frame correction flow 900, the frame selection module 215 first obtains frame information for each recognized frame (S910). Examples of the information on the frame include information regarding coordinates and dimensions. The coordinates may be, for example, the coordinates of the upper left vertex of the frame (corresponding to the first intersection P1) as the representative coordinates.

Next, the frame selection module 215 acquires frame setting conditions (S920). Examples of frame setting conditions include setting a lower threshold for an index (eg, area, side dimension) indicating the size of the frame. The threshold value can be appropriately set, for example, so that the size of the frame is larger than "one character" in the character string. In addition, the threshold value may be set, for example, as a ratio of the size of the frame to the overall size of the image (that is, equivalent to the size of the paper).

Then, the frame selection module 215 selects a frame that satisfies the frame setting conditions from among the plurality of recognized frames, and outputs (stores) information regarding the selected frame to the auxiliary storage device 202, for example. Thereby, character strings can be recognized in a subsequent process based on the frame selected by the frame selection module 215. As a result, information on the material can be read appropriately and with high precision. Further, the user terminal management module 211 can display, for example, only the frames that satisfy the frame setting conditions selected by the frame selection module 215 on the reading result display screen 1900.

(Correction method 2)
FIG. 13 is another example of the frame correction flow 1300.
The frame selection module 215 may select an appropriate frame according to the frame correction flow 1300. Specifically, the frame selection module 215 acquires information about the frame recognized by the frame recognition module 213 and frame setting conditions (S1310, S1320). Here, it is assumed that the frame setting condition is a lower limit threshold of an index (for example, area) indicating the size of the frame portion. The frame selection module 215 determines whether the area of the recognized frame is equal to or greater than the lower limit threshold of the area in the frame setting conditions (that is, whether the frame setting conditions are satisfied) based on the information of each recognized frame. (S1330).

Then, if the area of the recognized frame is smaller than the lower limit threshold of area in the frame setting conditions (No in S1330), the frame selection module 215 deletes information about the recognized frame (S1340). ). On the other hand, if the area of the recognized frame is equal to or greater than the lower limit threshold of the area in the frame setting conditions (Yes in S1330), the frame selection module 215 outputs (memorizes) information about the recognized frame as is. (S1350).

When the frame selection module 215 deletes the information about the frame (S1340), the frame selection module 215 asks the frame recognition module 213 to select the frame based on the ruled line and the intersection (for example, the first intersection P1) corresponding to the frame. The computer may also include a configuration that instructs the user to re-recognize the information.
The frame selection module 215 may select a frame according to the frame correction flow 1300, for example, after the frame recognition module 213 has recognized all frames, or after the frame recognition module 213 has recognized a frame. It may be executed sequentially as needed.
With such a configuration as well, the information on the material can be read appropriately and with high precision.

(Correction method 3)
The frame correction module 214 is configured to accept changes in frame setting conditions. The instruction to change the frame setting conditions may be input directly to the management server 101 or may be input to the management server 101 via the user terminal 102.

(Correction method 3-1)
FIG. 21 is an example of a frame correction screen 2100.
The frame correction module 214 is configured to cooperate with the user frame correction module 312, for example, and receive an instruction to change the frame setting conditions input by the user into the user terminal 102. For example, when a frame portion that has been misrecognized is displayed in the reading result displayed on the reading result display screen 1900, the frame correction module 214 receives a correction mode transition instruction from the user terminal 102 (for example, a mode The frame correction screen 2100 is output (displayed) on the display of the user terminal 102 (an example of the output device 305).

Then, when the frame correction screen 2100 is clicked (selected) on the frame correction screen 2100 based on a user operation, the frame correction module 214 creates a graphic 2101 (here, , a circle with a center point) is displayed on the frame correction screen 2100. When the frame correction screen 2100 is clicked and dragged based on a user operation, the frame correction module 214 changes the size of the figure (here, the radius of the circle) according to the drag width.

The frame correction module 214 further changes the size of the threshold value in the frame setting condition according to the drag width. The frame selection module 215 selects a frame portion that satisfies the frame setting condition based on the changed frame setting condition, and the user terminal management module 211 displays only the selected frame portion on the display of the user terminal 102. The display of the selected frame based on the changed frame setting conditions is reflected in real time together with the change in the frame setting conditions. Since the drag operation continuously moves the cursor, the threshold value can be changed continuously. The user can confirm the threshold value (and thus the recognition result of the frame) by continuously moving the cursor and releasing the click when the threshold value is changed to the optimum value.

The frame correction module 214 may be configured to change frame setting conditions for all recognized frame sections, or may be configured to change frame setting conditions for one or more preselected frame sections. may be configured. When changing frame setting conditions only for one or more frames, the frame correction module 214 changes the frame setting conditions from among the plurality of frames displayed on the frame correction screen 2100 via the user terminal 102. A frame selection instruction is received in advance. Thereafter, the frame correction module 214 changes the size of the figure and the frame setting conditions according to the drag width according to the above procedure.

(Correction method 3-2)
Note that the figure 2101 corresponding to the size of the threshold value is not limited to a circle, and may be, for example, another geometric figure, a scale, a slide bar (which may be the slide bar icon 2103), or a combination thereof. . The frame correction module 214 may be configured to accept, for example, an instruction to change the rate of change of the frame setting conditions in a range of about 0.1 to 10 times by operating the slide bar icon 210, for example.

Furthermore, the correction using the radius of the circle shown in (correction method 3-1) and the correction using the slide bar icon 2103 may be combined. For example, instead of changing the size of the threshold in the frame setting condition according to the drag width, the frame correction module 214 is configured to change the size of the threshold by sliding the slide bar icon 2103. Good too. In this case, the user does not need to click and drag the cursor continuously (while holding the cursor).

(Correction method 3-3)
Further, the frame correction module 214 may be configured to display one or more slide bar icons 2103 on the frame correction screen 2100 and to receive correction instructions by operating the slide bar icons 2103, for example. For example, the frame correction module 214 displays four slide bar icons 2103 with different functions on the frame correction screen 2100.

Although the content of the correction instruction is not limited to this, the first slide bar icon 2103 from the top is configured to accept, for example, a change in the upper limit size of the noise to be removed. The second slide bar icon 2103 from the top is configured to accept, for example, a change in the correction angle (rotation angle) in tilt correction, which will be described later. The third slide bar icon 2103 from the top has a configuration that can accept, for example, a change in the correction angle in keystone correction that corrects distortion when reading materials (distortion caused by deviation in the normal direction of the paper surface). It has become. The fourth slide bar icon 2103 from the top is configured to be able to accept changes in the threshold value in binarization, for example.

(Correction method 3-4)
Furthermore, the frame correction module 214 may include a ruled line editing function. The ruled line editing function includes, for example, functions such as adding ruled lines, deleting ruled lines, and transforming ruled lines. The frame correction module 214 has a configuration that allows a ruled line to be added to an arbitrary position on the frame correction screen 2100, an arbitrary ruled line to be deleted, and an arbitrary frame to be transformed using a ruled line editing function. ing.

For example, when a click operation is performed on the frame correction screen 2100 with the ruled line addition mode selected, the frame correction module 214 adds a ruled line to the frame correction screen 2100. For example, the frame correction module 214 can change the position of a ruled line and the position of one end or the other end of the ruled line independently by accepting an operation using a pointing device or the like while the ruled line deformation mode is selected. It is composed of

The frame correction module 214 is configured to be able to delete a ruled line, for example, when the ruled line is selected by an operation using a pointing device or the like while the ruled line deletion mode is selected. Thereby, for example, in the recognition result (C) of FIG. 20, the frame portion that is recognized as being excessively large can be divided into two by adding a ruled line. The frame correction module 214 changes information regarding the frame according to the ruled line correction processing performed by the ruled line editing function.

(Correction method 3-5)
In addition, the frame correction module 214 displays, for example, a plurality of different thresholds (typically 2 to 6 types) of frame setting conditions, such as recognition results (A) to (C), on the frame correction screen 2100. , for example, three types of recognition results may be displayed side by side at the same time. In this case, the frame correction module 214 can determine the threshold values for the frame setting condition so that the threshold values include a threshold value that is slightly smaller than the appropriate value and a threshold value that is slightly larger than the appropriate value. Then, if an appropriate recognition result is included in the displayed plurality of recognition results, by selecting the appropriate recognition result, the frame correction module 214 adjusts the threshold value of the frame setting condition to the selected recognition result. The threshold value may be changed to

If the displayed recognition results do not include a suitable one, between the thresholds corresponding to any two recognition results (for example, recognition results (A) and (B)), A threshold will be included that will produce an appropriate recognition result. In such a case, the user selects one of the recognition results (A) and (B) and drags it toward the other. Then, the frame correction module 214 changes the threshold value of the one recognition result being dragged in accordance with the drag width so that it approaches the threshold value of the other recognition result, and also changes the threshold value of the frame part in one recognition result. The shape is changed according to the threshold value. Thereby, the user can intuitively and appropriately correct the frame setting conditions by a simple operation of dragging one of the recognition results.

Note that the frame correction module 214 displays a slide bar icon 2103 and a confirmation icon 2106 on the frame correction screen 2100. Then, when one recognition result is selected, the frame correction module 214 changes the threshold value by moving the slide bar according to the movement width, instead of changing the threshold by dragging the one recognition result. A configuration may be adopted in which the threshold value is changed. Furthermore, the frame correction module 214 may be configured to finalize the threshold value by selecting the confirmation icon 2106 without changing the threshold value or after changing the threshold value.

That is, the frame correction module 214 displays, on the screen, a figure corresponding to the size of the threshold value under the current frame setting conditions, and also changes the size of the threshold value according to the operation input, and adjusts the size of the threshold value to the changed threshold value. At least one of the position, size, and shape of the figure can be changed in accordance with the size of the figure.
According to the above configuration, the user can easily change the size of the threshold value of the frame setting condition, and can intuitively understand the degree of change. Furthermore, the degree of change in frame setting conditions can be adjusted while visually checking in real time.

Note that the frame correction module 214 displays, in addition to the confirmation icon 2106, a leftward triangular icon 2104 and a rightward triangular icon 2105 at the bottom of the frame correction screen 2100. The frame correction module 214 allows these icons on the frame correction screen 2100 to function as, for example, an icon 2104 for canceling a correction operation and an icon 2105 for repeating a correction operation, rather than buttons for moving pages. I can do it.

When the left-pointing triangular icon 2104 is selected on the user terminal 102, the frame correction module 214 is configured to undo the user's frame setting condition changing operation by one step. By selecting the right-left triangular icon 2105 on the user terminal 102, for example, the frame correction module 214 is configured to advance the frame setting condition changing operation returned by the user by one step. When the confirmation icon 2106 is selected on the user terminal 102, the frame correction module 214 is configured to confirm the frame setting condition changing operation changed by the user.

FIG. 22 is an example of an annotation confirmation screen 2200.
For example, when an annotation is added to a character string, the user terminal management module 211 cooperates with the user execution module 311 of the user terminal 102 to display an annotation confirmation screen on the display of the user terminal 102 (an example of the output device 305). Output (display) 2200.
On the annotation confirmation screen 2200, the user terminal management module 211 displays an image 2201 based on the original image data 221 and an image 2202 based on the processed image data 222 reflecting the annotation recognition results side by side vertically or horizontally. This makes it possible to easily compare and confirm whether the management server 101 has properly read the material.

FIG. 23 is another example of the annotation confirmation screen 2300. When reading paper-based materials with a reading device, if the reading is performed while the paper is tilted, the character string itself in the original image data 221 will be tilted, and the character string itself will be tilted relative to the area of the image 2302 based on the processed image data 222. The recognition result of the annotation will also be tilted. At this time, variations tend to occur in the angles of annotations created for character strings. In such a case, for example, with AI, it is easy to misjudge which annotation should be rotated to be horizontal or vertical.

The frame correction module 214 displays an angle icon 2303 consisting of a horizontal line or a vertical line and a line along the inclination angle of the annotation on the annotation confirmation screen 2300 for a plurality of large (that is, long) annotations (for example, 2 to 3 annotations). Display above. Furthermore, when the cursor approaches or overlaps the angle icon 2303 on the annotation confirmation screen 2300, the frame correction module 214 selects whether to "correct" the tilted annotation horizontally or vertically (rotation correction) or to "ignore" it. Possible selection icons 2304 are displayed on the annotation confirmation screen 2300. The frame correction module 214 does not perform correction when "ignore" is selected. When “correct” is selected, the frame correction module 214 performs rotation correction on the processed image data 222 by the angle formed by the angle icon 2303 (that is, so that the annotation is horizontal or vertical).

In addition, for an annotation that is tilted by a predetermined angle or more in an image based on the processed image data 222 after rotation correction, an angle icon 2303 consisting of a horizontal line or a vertical line and a line along the tilt angle of the annotation is displayed. A selection icon 2304 is now displayed. The frame correction module 214 performs no correction for each annotation for which the selection icon 2304 is displayed when "ignore" is selected, and performs rotation correction when "correct" is selected. As a result, the inclination of the character string can be corrected before character string recognition is performed for each frame, and the accuracy of character string recognition can be improved. Note that the rotation correction angle is not limited to that determined by the angle icon 2303, and the frame correction module 214 is configured to perform rotation correction at an angle adjusted by the slide bar icon, etc., as described above. Good too.

FIG. 24 is another example of the annotation confirmation screen 2400.
When creating an annotation, the frame recognition module 213 may be configured not to annotate a predetermined specific character 2403, as shown in FIG. 24. For example, when recognizing a character string when creating an annotation, if the frame recognition module 213 detects a specific character registered in advance, the frame recognition module 213 recognizes the specific character as an icon and annotates the remaining part of the character string. create. The specific characters can be the same as those described in (Classification method 2). As a result, the character string recognition module 216 does not perform character recognition on, for example, a specific character. As a result, in the character string, it is possible to prevent the specific characters and the item from being erroneously recognized as forming one meaningful word together.

FIG. 25 is another example of the annotation confirmation screen 2500.
The process of extracting specific characters from a character string with high precision can be relatively difficult. For example, as shown in annotation 2503 in FIG. 25, there is a possibility that a specific character cannot be detected and an annotation is created for the entire character string including the specific character. Furthermore, among the specific characters, there is a possibility that midpoints 2504 and the like having a small area may not be recognized as noise.

When the user finds such a misrecognition, for example, by double-clicking the misrecognition location on the annotation confirmation screen 2500, the user can instruct re-recognition of the location. The frame recognition module 213 receives an instruction from the user terminal 102 (here, for example, a double click), and re-recognizes the presence or absence of a specific character in the relevant location (which may be a character string) and re-creates the annotation. . At this time, the frame recognition module 213 may display a specific character selection button (not shown) on the annotation confirmation screen 2500, for example, so that the specific character can be forcibly recognized. In addition, the frame recognition module 213 displays one or more slide bar icons 2505 on the annotation confirmation screen 2500, and changes various thresholds using the one or more slide bars to make it easier for the frame recognition module 213 to recognize specific characters. The recognition result may be changed so that the recognition result is changed.

FIG. 26 is another example of the reading result display screen 2600.
When displaying the reading results, the frame recognition module 213 may decorate a predetermined frame portion with coloring, shading, etc., as shown in FIG. 26, for example. For example, the frame recognition module 213 may set the background of items in a bright color (e.g., fluorescent color) so as not to interfere with confirmation of the character string recognition results for items in higher levels (e.g., hierarchies 1 and 2). It may also be colored. Thereby, when the user checks the reading result on the reading result display screen 2600, the visibility of the information can be improved.

(Linking method 3)
FIG. 27 is an example of real estate information material 2700.
FIG. 28 is an example of a ruled line recognition result 2800 for the real estate information material 2700.
In real estate information materials 2700 and the like, for example, a format may be adopted in which item columns are decorated with coloring, shading, etc., and value columns are not decorated. In this case, the decoration content may also be reflected in the ruled line recognition result 2800. Here, the combination of the decorated frame part and the undecorated frame part is arranged in one direction (for example, When two or more sets are arranged in the axial direction or the Y-axis direction, the classification module 217 determines that one of the combinations of decorative frame portions and non-decorative frame portions is an item, the other is a value, and these are linked to each other. It can be determined that

Specifically, for example, as shown in the upper left of the recognition result 2800, there are two or more combinations (in this case, three (1) When they are lined up, the classification module 217 uses the classification module 217 to determine that each decorated frame is an item, each non-decorated frame is a value, and that the decorated and non-decorated frames that are lined up above and below each other are It can be determined that they are linked. That is, "floor plan" and "1LDK", "rent" and "215,000 yen", and "management fee, etc." and "15,000 yen" are respectively linked. Furthermore, as shown on the right side of the recognition result 2800, there are two or more combinations in the vertical direction (here, two or three , 16), the classification module 217 determines that each decorated frame is an item, each non-decorated frame is a value, and the decorated and non-decorated frames are arranged on the left and right. It can be determined that the two are linked to each other. That is, "security deposit" and "215,000 yen", "key money" and "215,000 yen", "property type" and "apartment", etc. are linked, respectively. Note that the classification module 217 may be configured to determine that the decorated frame portion is a value and the non-decorated frame portion is an item.

Furthermore, when displaying the reading results, the frame recognition module 213 may decorate and display a decorated frame portion that has been decorated, as shown in FIG. 28, for example. This makes it easy to compare the original image data 221 and the processed image data 222 on the reading result display screen, for example.

In addition, when displaying the reading result, the frame recognition module 213 uses any of the methods described in Example 1 in addition to the decorated frame portion that has been decorated, as shown in FIG. 28, for example. A frame portion determined to be an item (for example, a frame portion corresponding to a column such as exterior appearance, floor plan, other transaction mode, etc.) may be displayed in a decorated manner. This makes it even easier to compare the original image data 221 and the processed image data 222 on the reading result display screen, for example. In addition, when checking the reading results, linking results, etc. manually, the frame parts that should be noted and the frame parts that serve as landmarks are decorated and displayed, which reduces the burden on the operator. is also preferable.

Note that the present invention is not limited to the above-described embodiments, and includes various modifications. For example, the embodiments described above are described in detail to explain the present invention in an easy-to-understand manner, and the present invention is not necessarily limited to having all the configurations described. Furthermore, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Further, it is possible to add, delete, or replace a part of the configuration of each embodiment with other configurations.

Further, each of the above-mentioned configurations, functions, processing units, processing means, etc. may be partially or entirely realized in hardware by designing, for example, an integrated circuit. Furthermore, each of the above configurations, functions, etc. may be realized by software by a processor interpreting and executing a program for realizing each function. Information such as programs, tables, files, etc. that implement each function can be stored in a memory, a recording device such as a hard disk, an SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD.

Further, the control lines and information lines are shown to be necessary for explanation purposes, and not all control lines and information lines are necessarily shown in the product. In reality, almost all components may be considered to be interconnected.
Note that the above embodiments disclose at least the configuration described in the claims.

DESCRIPTION OF SYMBOLS 100... Information reading system, 101... Management server, 102... User terminal, 211... User terminal management module, 212... Image management module, 213... Frame recognition module, 214... Frame correction module, 215... Frame selection module, 216... Character Column recognition module, 217...Classification module, 311...User execution module, 312...User frame correction module, 400...User information, 500...Dictionary information, 600...Balance sheet dictionary information

Claims (33)

an image management means for acquiring image data regarding information to be read;
frame recognition means for recognizing a plurality of frames constituting a table in the image data;
output means for outputting information regarding the plurality of recognized frames;
Information reading system equipped with. The frame recognition means may be arranged such that two ruled lines that are adjacent to each other along a first direction intersect with two ruled lines that are adjacent to each other along a second direction that intersects the first direction at four intersection points. Recognizing a line segment placed between the four intersection points as the frame portion,
The information reading system according to claim 1. The frame recognition means includes:
Generate multiple annotations corresponding to existing parts of the string,
When two-dimensional coordinates are displayed for each of the plurality of annotations, a first coordinate that includes the plurality of annotations and is represented by a minimum X coordinate and a minimum Y coordinate, and a first coordinate represented by a maximum X coordinate and a maximum Y coordinate. Recognizing a rectangle whose diagonal points are the second coordinates and as the frame portion,
The information reading system according to claim 1 or 2. The frame recognition means includes:
Generate multiple annotations corresponding to existing parts of the string,
The plurality of annotations are enlarged by a predetermined size, and if the enlarged plurality of annotations overlap, the superimposed annotations are concatenated to generate a new annotation, and based on the concatenated new annotations, recognizing the frame;
The information reading system according to claim 1 or 2. The predetermined size is determined as a relative value to the size of one character in the character string,
The information reading system according to claim 4. The frame recognition means forms the plurality of annotations by a line along a first direction in which the character string extends and a line along a second direction intersecting the first direction.
The information reading system according to any one of claims 3 to 5. The frame recognition means includes:
when the recognition target area of the image data includes a ruled line, generating the plurality of annotations with the inside of the frame part as a new recognition target area;
recognizing the frame based on the plurality of annotations;
The information reading system according to any one of claims 3 to 6. comprising frame selection means for selecting one of the plurality of frame parts that satisfies a predetermined frame setting condition;
the output means outputs information regarding the plurality of selected frames;
The information reading system according to any one of claims 1 to 7. The frame selection means determines that the frame satisfies a predetermined frame setting condition when at least one of the area and side length of the frame is equal to or greater than a predetermined threshold.
The information reading system according to claim 8. comprising frame correction means for accepting changes in the frame setting conditions;
The information reading system according to claim 8 or 9. The frame correction means displays on the screen a figure corresponding to the size of the threshold value under the current frame setting conditions.
The information reading system according to claim 10. The figure is at least one of a circle, a scale, and a slide bar.
The information reading system according to claim 11. The frame correction means displays, on the screen, a figure corresponding to the size of the threshold value under the current frame setting conditions, and also changes the size of the threshold value in accordance with an operation input, and adjusts the size of the changed threshold value. changing at least one of the position, size, and shape of the figure in response to the
The information reading system according to claim 11 or 12. The figure is a circle, and the frame correction means changes the radius of the circle according to the operation input.
The information reading system according to claim 13. Further comprising a character string recognition means for recognizing a character string arranged inside the frame for each of the plurality of recognized frames,
The output means outputs the recognized character string.
The information reading system according to any one of claims 1 to 14. Further comprising a classification means for classifying the recognized character string into items and values and making them correspond to each other,
the output means outputs the associated character string;
The information reading system according to claim 15. an image acquisition step of acquiring image data about the information to be read;
a frame recognition step of recognizing a plurality of frames constituting a table in the image data;
an output step of outputting information regarding the plurality of recognized frames;
An information reading method comprising: In the frame recognition step, two ruled lines adjacent to each other along a first direction and two ruled lines adjacent to each other along a second direction intersecting the first direction intersect at four intersection points. Recognizing a line segment placed between the four intersection points as the frame portion,
The information reading method according to claim 17. The frame recognition step includes:
Generate multiple annotations corresponding to existing parts of the string,
When two-dimensional coordinates are displayed for each of the plurality of annotations, a first coordinate that includes the plurality of annotations and is represented by a minimum X coordinate and a minimum Y coordinate, and a first coordinate represented by a maximum X coordinate and a maximum Y coordinate. Recognizing a rectangle whose diagonal points are the second coordinates and as the frame portion,
The information reading method according to claim 17 or 18. The frame recognition step includes:
Generate multiple annotations corresponding to existing parts of the string,
The plurality of annotations are enlarged by a predetermined size, and if the enlarged plurality of annotations overlap, the superimposed annotations are concatenated to generate a new annotation, and based on the concatenated new annotations, recognizing the frame;
The information reading method according to claim 19. The predetermined size is determined as a relative value to the size of one character in the character string,
The information reading method according to claim 20. In the frame recognition step, the plurality of annotations are formed by a line along a first direction in which the character string extends and a line along a second direction that intersects the first direction.
The information reading method according to any one of claims 19 to 21. The frame recognition step includes:
when the recognition target area of the image data includes a ruled line, generating the plurality of annotations with the inside of the frame part as a new recognition target area;
recognizing the frame based on the plurality of annotations;
The information reading method according to any one of claims 19 to 22. a frame selection step of selecting one of the plurality of frames that satisfies a predetermined frame setting condition;
The output step outputs information regarding the selected plurality of frames.
The information reading method according to any one of claims 17 to 23. The frame selection step determines that the frame satisfies a predetermined frame setting condition when at least one of the area and side length of the frame is equal to or greater than a predetermined threshold.
The information reading method according to claim 24. comprising a frame correction step that accepts a change in the frame setting conditions;
The information reading method according to claim 24 or 25. The frame correction step displays on the screen a figure corresponding to the size of the threshold value under the current frame setting conditions.
The information reading method according to claim 26. The figure is at least one of a circle, a scale, and a slide bar.
The information reading method according to claim 27. In the frame correction step, a figure corresponding to the size of the threshold value under the current frame setting conditions is displayed on the screen, and the size of the threshold value is changed in accordance with an operation input, and the size of the changed threshold value is changed. changing at least one of the position, size, and shape of the figure in response to the
The information reading method according to claim 27 or 28. The figure is a circle, and the radius of the circle is changed according to the operation input.
The information reading method according to claim 29. Further comprising a character string recognition step of recognizing a character string arranged inside the frame for each of the plurality of recognized frames,
The output step outputs the recognized character string.
The information reading method according to any one of claims 17 to 30. further comprising a classification step of classifying the recognized character string into items and values, and making them correspond to each other;
The output step outputs the associated character string.
The information reading method according to claim 31. A program for causing a computer to execute each step of the information reading method according to any one of claims 17 to 32.

Images