JP2023081132A - Information processing system, form recognition method, program, form recognition system - Google Patents

Abstract

To provide an information processing system, a form recognition system, a form recognition method, and a program for more appropriately extracting item values written in a plurality of lines.SOLUTION: In a form recognition system 100, a server apparatus 200 provides a terminal apparatus 400 with recognition results obtained by recognizing character strings from image data in a tabular format having rows and columns. The server apparatus 200 comprises a form recognition processing unit 220 for concatenating item values of a first row that contains values in the preset column heading column and the item values of a second row that does not contain preset exclusion terms and is not blank, and a display screen generation unit for providing the terminal apparatus 400 with the item values of the first row and the second row that are concatenated by the form recognition processing unit 220.SELECTED DRAWING: Figure 1

Description

The present invention relates to an information processing system, a form recognition method, a program, and a form recognition system.

A technique is known that scans a form in which item names and their item values are arranged in a tabular format and converts it into text data. Since each company creates a form according to its business, the format of the form varies depending on the company. Therefore, when the form is not recognized correctly, the user directly designates the area from which the item value is to be extracted.

A technique for setting an area corresponding to a predetermined item value with a simple operation is known (see Patent Document 1, for example). Patent Literature 1 discloses an information processing system that designates an extraction region by displaying an extraction guide in a table-format image and accepting the setting of the position of the extraction guide.

However, the conventional technique has a problem that item values described in multiple lines are not properly extracted. For example, when the written content spans a plurality of lines, the written content will be divided and extracted if line delimiters are not properly recognized.

SUMMARY OF THE INVENTION It is an object of the present invention to provide an information processing system that more appropriately extracts item values described in multiple lines.

In view of the above problems, the present invention provides an information processing system for providing a terminal device with a recognition result obtained by recognizing characters from table-format image data having rows and columns, wherein predetermined columns are obtained from the recognition result. a form recognition unit that concatenates the item values of the first row that have values in the heading column and the item values of the second row that does not contain preset exclusion terms and is not blank, and the form recognition unit concatenates and a display screen generator that provides the terminal device with the item values of the first row and the second row that have been generated.

It is possible to provide an information processing system that more appropriately extracts item values described in multiple lines.

It is a figure which shows an example of the system configuration|structure of a form recognition system. It is a figure which shows an example of the hardware constitutions of a server apparatus. It is a figure which shows an example of the hardware constitutions of a terminal device. It is a figure which shows an example of an industry determination database. It is a figure which shows the kind of the specification definition information stored in a specification definition database. It is a figure which shows an example of general-purpose specification definition information. It is a figure which shows an example of industry specification definition information. It is a figure which shows an example of individual company specification definition information. It is a figure explaining the function of a server apparatus. It is a figure explaining the function of a terminal device. FIG. 11 is an example of a sequence diagram explaining the operation of the form recognition system (No. 1); FIG. 11 is an example of a sequence diagram explaining the operation of the form recognition system (No. 2); FIG. 10 is a diagram showing an example of a job list; FIG. FIG. 10 is an example of a first flowchart for explaining processing of a form recognition unit; FIG. 10 is an example of a second flowchart for explaining the processing of the form recognition unit; FIG. 11 is an example of a third flowchart illustrating processing of the form recognition unit; FIG. 4 is a first diagram for explaining form image data; FIG. 10 is a second diagram for explaining extraction of detailed information; FIG. 11 is a third diagram for explaining detailed information of a detailed part generated based on reading result data; It is a figure which shows an example of a recognition result confirmation screen. FIG. 10 is a diagram illustrating an example of adjusting line breaks in a recognition result; FIG. 10 is an example of a flowchart illustrating a procedure for a form recognition unit to concatenate a plurality of lines; FIG. 10 is a diagram schematically illustrating connection of rows of labels A and D; FIG. 10 is a diagram showing an example of concatenation of rows labeled A and D; FIG. 22 is a diagram showing labels A to D set in each row of form image data by the linking process of FIG. 21; 21. It is a figure explaining the product name connected by the connection process of FIG. FIG. 4 is a diagram illustrating a method of recognizing line breaks to be compared; FIG. 4 is a diagram showing an outline of a method for recognizing line breaks; FIG. 11 is an example of a sequence diagram for explaining the operation of the form recognition system (second embodiment); FIG. 29 is a diagram illustrating product names linked by the linking process of FIG. 28 (Example 2); FIG. 12 is a diagram showing a part of a multi-column form in which a consumption tax is also displayed in the amount column (Example 2). FIG. 12 is a diagram showing a rendering example of logical ruled lines in a multi-column form when the user designates the column of “money” and skips one line as the line break of the logical ruled lines (Embodiment 2).

An information processing system and a form recognition method performed by the information processing system will be described below as an example of embodiments of the present invention with reference to the drawings.

<Terms>
The user in the present embodiment is, for example, a business office, company, organization, etc., and its employees, employees, etc. who have concluded a contract for using the service provided by the form recognition system.

A user obtains a form from a form issuer, who is a business partner, and uses a form recognition system for accounting processing or the like to recognize the form.

A row refers to a horizontal arrangement of characters and the like, and a column refers to a vertical arrangement of characters and the like. A table is at least partially arranged in rows and columns so that things can be easily viewed. A display method in which items are arranged in this way is called a tabular format. If the rows are separated by ruled lines, one row is the space between the ruled lines; if not separated by ruled lines, one row is the edge-to-edge of the table that contains the horizontal text area. be. A blank line is a line approximately one character high. A blank line may contain a simple list of symbols (eg ----, ****, etc.).

If the columns are separated by vertical ruled lines, one column is the space between the vertical ruled lines. The width is the length of the column heading), and the height is the bottommost character string that exists vertically under this character string.

A correct line break in a form means a line break intended by the issuer of the form. In other words, correct line delimiting in a form means determining from which line to which line should be concatenated when the description content of the form extends over multiple lines. Of course, the line breaks intended by the publisher are also the correct line breaks for the user.
A character string in this embodiment refers to alphanumeric characters, kana characters, kanji characters, various symbols (!$%, etc.), etc., and it is assumed that an item value for an item name is a character string. Depending on the form, the character string may include marks and codes that are used only by the company.

<System configuration example>
FIG. 1 is a diagram showing an example of the system configuration of a form recognition system. A form recognition system 100 of this embodiment has a server device 200 , an image forming device 300 and a terminal device 400 . In the form recognition system 100 of this embodiment, the server device 200, the image forming device 300, and the terminal device 400 are connected via a network. The server device 200 of this embodiment is an example of an information processing system.

Also, the server device 200 is connected to the core system 500 via a network, for example. The core system 500 is, for example, a system that performs specific processing using the result of recognition of a form by the form recognition system 100 of this embodiment. Specifically, for example, the core system 500 may be an accounting system, an estimate creation system, or the like.

In the form recognition system 100 of the present embodiment, the server device 200 extracts items and item values included in the form from the image data of the form read by the scanner function of the image forming apparatus 300, and extracts the items and item values. to text data associated with

The server device 200 may exist on the Internet or may exist on-premises. If it exists on the Internet, it should be compatible with cloud computing. Cloud is a term used when not referring to specific hardware resources.

In the following description, a form image or image data read by the scanner function of the image forming apparatus 300 is referred to as a form image or form image data.

In the following description, extracting items included in a form image and their item values, converting the items and item values into text data, and associating them with each other will be referred to as form recognition.

The server device 200 of this embodiment has a storage unit 210 and a form recognition processing unit 220 . The storage unit 210 is provided with a form image database 230 , an industry determination database 240 , a statement definition database 250 and a recognition result database 260 .

The form image database 230 stores form image data. The industry determination database 240 stores information referred to when determining the industry of the issuer of the form. The statement definition database 250 stores statement definition information that is referred to when the form recognition processing unit 220 performs form recognition. The recognition result database 260 stores recognition result data, which is information indicating the result of form recognition by the form recognition processing unit 220 .

The form recognition processing unit 220 of the present embodiment acquires the form image data stored in the form image database 230, refers to the business type determination database 240, determines the business type of the issuer of the form, and specifies the specification definition information to be referred to. identification. Then, the form recognition processing unit 220 performs form recognition using the specified detailed definition information in the detailed definition database 250 and stores the result in the recognition result database 260 . The detail definition information includes the item names of the detail items included in the form and information indicating the description positions of the item names.

Here, the form recognition processing unit 220 of the present embodiment recognizes the individual company statement definition information, the industry statement definition information, and the general statement definition information stored in the statement definition database 250 when performing the form recognition. refer.

The general-purpose specification definition information of this embodiment is information defining words corresponding to specification item names in general forms. The industry detail definition information is information defining words corresponding to the detail item names in the form for each industry. Individual company statement definition information is information that defines information extracted from a word corresponding to a statement item name and a form image in a form specific to a user who uses the form recognition system 100 . Details of the business category determination database 240 and the specification definition database 250 will be described later.

In this embodiment, by using three types of detail definition information when recognizing a form, the detailed part of the form can be recognized in addition to the general-purpose form. can be improved.

A brief description will be given of the detail part of the form. The detail part of the form is a part showing the result of totaling data and the like, and is a part in which data of the same format is repeatedly printed in the vertical direction for each detail item. A detail item is an item of data that is aggregated in the detail part. Common detail items include product name, quantity, unit price, price, and the like. A specific example of the specification part will be described later.

In the form recognition system 100 of this embodiment, the image forming apparatus 300 is a multifunction machine having a scanner function. The image forming apparatus 300 is equipped with applications for realizing a copy function, a FAX function, a scanner function, etc. By selecting an application corresponding to each function, these functions are realized. The image forming apparatus 300 may be an information processing apparatus including a camera, such as a smartphone, document camera, tablet terminal, or the like. An OCR application operates in these information processing apparatuses, and the same processing as in the present embodiment can be performed independently or in cooperation with the server apparatus 200 . Alternatively, a form in which a standalone reading function such as a scanner device and an information processing device equipped with an OCR application are connected via a network, or a form in which the OCR function is provided in a server device may be employed.

The terminal device 400 of this embodiment is used by a user who uses the form recognition system 100 . Further, the terminal device 400 may display the result of form recognition by the server device 200 .

In addition, in the example of FIG. 1, the server device 200 is configured to have four databases in the storage unit 210, but the present invention is not limited to this. A part of each database may be provided in an external device of the server device 200, or all of the databases may be provided in an external device.

Also, in the example of FIG. 1, the server apparatus 200 implements the form recognition processing unit 220, but the present invention is not limited to this. The form recognition processing unit 220 may be realized by a plurality of information processing devices.

In the example of FIG. 1, one image forming device 300 and one terminal device 400 are included in the form recognition system 100. However, the number of image forming devices 300 and terminal devices 400 included in the form recognition system 100 is can be any number.

Next, with reference to FIG. 2, the hardware configuration of the server device 200 of this embodiment will be described. FIG. 2 is a diagram showing an example of the hardware configuration of the server device 200. As shown in FIG.

The server device 200 of this embodiment is constructed by a computer, and as shown in FIG. /F (Interface) 208 , network I/F 209 , data bus B, keyboard 211 , pointing device 212 , optical drive 214 and media I/F 216 .

Among these, the CPU 201 controls the operation of the server apparatus 200 as a whole. The ROM 202 stores programs used to drive the CPU 201, such as IPL. A RAM 203 is used as a work area for the CPU 201 . The HD 204 stores various data such as programs. The HDD controller 205 controls reading or writing of various data to/from the HD 204 under the control of the CPU 201 . A display 206 is a display unit that displays various types of information such as a cursor, menus, windows, characters, and images.

The external device connection I/F 208 is an interface for connecting various external devices. The external device in this case is, for example, a USB (Universal Serial Bus) memory, a printer, or the like. A network I/F 209 is an interface for data communication using the communication network N. FIG. The data bus B is an address bus, a data bus, etc. for electrically connecting each component such as the CPU 201 shown in FIG.

Also, the keyboard 211 is a kind of input means having a plurality of keys for inputting characters, numerical values, various instructions, and the like. The pointing device 212 is a kind of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, and the like. The optical drive 214 controls reading and writing of various data to and from an optical storage medium 213, which is an example of a removable recording medium. Note that the optical storage medium 213 may be a CD, DVD, Blu-ray (registered trademark), or the like. A media I/F 216 controls reading or writing (storage) of data to a recording medium 215 such as a flash memory.

The server device 200 of this embodiment is, for example, a smartphone, a tablet terminal, a PDA
(Personal Digital Assistant), a wearable PC, or the like.

The terminal device 400 of this embodiment is a computer having an arithmetic processing device and a storage device, and may be, for example, a PC (Personal Computer), a tablet terminal, a smartphone, or the like. FIG. 3 is a diagram showing an example of the hardware configuration of the terminal device 400. As shown in FIG.

The terminal device 400 of this embodiment includes a CPU 401 , a ROM 402 , a RAM 403 , an EEPROM 404 , a CMOS sensor 405 , an image sensor I/F 406 , an acceleration/direction sensor 407 , a media I/F 409 and a GPS receiver 411 .

Among these, the CPU 401 is an arithmetic processing unit that controls the operation of the terminal device 400 as a whole. The ROM 402 stores programs used to drive the CPU 401 such as the CPU 401 and IPL. A RAM 403 is used as a work area for the CPU 401 . The EEPROM 404 reads or writes various data such as smartphone programs under the control of the CPU 401 . The ROM 402 , RAM 403 and EEPROM 404 are examples of storage devices of the terminal device 400 .

A CMOS (Complementary Metal Oxide Semiconductor) sensor 405 is a type of built-in imaging means for capturing an object (mainly a self-portrait) and obtaining image data under the control of the CPU 401 . The CMOS sensor may be imaging means such as a CCD (Charge Coupled Device) sensor.

An imaging device I/F 406 is a circuit that controls driving of the CMOS sensor 305 . The acceleration/direction sensor 407 is various sensors such as an electronic magnetic compass, a gyro compass, and an acceleration sensor for detecting geomagnetism. A media I/F 409 controls reading or writing (storage) of data to a recording medium 408 such as a flash memory. The GPS receiver 411 receives GPS signals from GPS satellites.

The terminal device 400 also includes a long-distance communication circuit 412, an antenna 412a of the long-distance communication circuit 412, a microphone 415, a speaker 416, a sound input/output I/F 417, a display 418, an external device connection I/F (Interface) 419, a near A distance communication circuit 420, an antenna 420a of the short distance communication circuit 420, and a touch panel 421 are provided.

Among these, the telecommunications circuit 412 is a circuit that communicates with other devices over a communication network. Microphone 415 is a built-in circuit that converts sound into an electrical signal. The speaker 416 is a built-in circuit that converts electrical signals into physical vibrations to produce sounds such as music and voice. A sound input/output I/F 417 is a circuit for processing input/output of sound signals between the microphone 415 and the speaker 416 under the control of the CPU 401 .

A display 418 is a kind of display means such as liquid crystal or organic EL (Electro Luminescence) that displays an image of a subject, various icons, and the like. The external device connection I/F 419 is an interface for connecting various external devices. The short-range communication circuit 420 is a communication circuit such as NFC (Near Field Communication) or Bluetooth (registered trademark). The touch panel 421 is a kind of input means for operating the terminal device 400 by the user pressing the display 418 . A display 418 is an example of a display unit included in the terminal device 400 .

<Database configuration example>
Next, the business category determination database 240 and the specification definition database 250 of the server device 200 of this embodiment will be described.

FIG. 4 is a diagram showing an example of a business category determination database. The business type determination database 240 of this embodiment is provided for each type of business type.

In the business type determination database 240, the business type is associated with the issuer of the form. In the example of FIG. 3, the business type "retail" is associated with A company, B company, and C company. Therefore, it can be seen that Company A, Company B, and Company C are organizations engaged in retail business.

Next, the specification definition database 250 of this embodiment will be described with reference to FIGS. 5 to 8. FIG. FIG. 5 is a diagram showing types of specification definition information stored in the specification definition database.

In the specification definition database 250 of this embodiment, individual company specification definition information 251, industry specification definition information 252, and general specification definition information 253 are stored. The individual company detail definition information 251 is the detail item name and description position included in the form originally designed by the issuer (or used in a management system etc. independently managed by the issuer) among the forms. , and is the issuer-specific specification definition information. The individual company specification definition information 251 is the definition that should be respected the most among the three specification definition information, and is the specification definition information with the highest priority.

The industry specification definition information 252 of the present embodiment is specification definition information that defines the specification item names and description positions included in the forms generally used in each industry among the forms. The priority of the industry specification definition information 252 is intermediate among the three specification definition information.

The general-purpose specification definition information 253 of this embodiment is specification definition information that defines specification item names and description positions included in a highly versatile form that is used by a plurality of users. The general-purpose specification definition information 253 is specification definition information with the lowest priority among the three specification definition information.

The form recognition processing unit 220 of this embodiment refers to these three items of specification definition information to recognize form image data. Therefore, in this embodiment, for example, even if the form image data is for a form that includes a detail item name that is not defined in the individual company detail definition information 251, the industry detail definition information 252 and the general detail definition information 253 , the form image data can be recognized.

In the following description, a bill is used as an example of a form. Note that the form may be any form as long as it includes a detail part. Specifically, for example, the form may be a statement of delivery, an estimate, a purchase slip, or the like.

If the form is a statement of delivery, the issuer is the delivery source that issued the statement of delivery, and if the form is a purchase slip, the issuer is the purchaser that issued the purchase slip.

FIG. 6 is a diagram showing an example of general-purpose specification definition information. The general-purpose specification definition information 253 defines the specification item names adopted in the specification of a general bill, and the general-purpose specification definition information 253 has specification item names and keywords as information items. and both are associated.

The value of the item "description item name" indicates the name of the detail item used in the description part of the invoice. The value of the item "keyword" indicates a keyword (word) that is considered synonymous with the description item name of the description part.

That is, the general-purpose specification definition information 253 is information that defines specification items used in general forms and keywords extracted as specification items from the forms.

The example of FIG. 6 indicates that the item name "product name" is treated as synonymous with keywords such as "product", "product", and "product name". In the example of FIG. 6, the detail item name "amount" is treated as being synonymous with keywords such as "amount", "total amount", and "total".

In the example of FIG. 6, the item name "order number" is treated as being synonymous with keywords such as "order number" and "slip number". In addition, the example of FIG. 6 indicates that the item name "product code" is treated synonymously with keywords such as "product No." and "product number".

FIG. 7 is a diagram showing an example of industry specification definition information. The industry specification definition information 252 is information provided for each industry, and defines item names adopted in general invoice details in the industry indicated by the industry.

The business type specification definition information 252 has information specifying the business type, an item "specification item name", and an item "keyword", and both are associated with each other. The example of FIG. 7 indicates that the business type is "retail business".

The value of the item "description item name" indicates the name of the detail item used in the description part of a general invoice. The value of the item "keyword" indicates a keyword (word) corresponding to a general detail item name in an invoice for which the billing source is a retail business.

That is, the industry specification definition information 252 is information that defines the industry, specification items used in the form, and keywords extracted as the specification items from the form.

In the example of FIG. 7, the item name "product name" of general forms may be replaced with keywords such as "product name" and "product name" in the details of invoices issued by retailers. It is shown that.

FIG. 8 is a diagram showing an example of individual company specification definition information. The individual company statement definition information 251 of this embodiment is registered by the user for each issuer. Even if the issuer is the same, the individual company statement definition information 251 may differ depending on the user.

In the individual company statement definition information 251, the information specifying the invoicing party, the item "description item name", and the item "extraction information" are associated with each other.

The information identifying the invoicing party includes the invoicing party's name, telephone number, account number, and the like. In the example of FIG. 8, the information identifying the invoicing party includes the name, telephone number, and account number of the invoicing party. can be one.

The value of the item "description item name" indicates the name of the detail item used in the description part of a general invoice. In other words, the detail item name in the individual company detail definition information 251 indicates the detail item used in the form.

The value of the item "extracted information" is a keyword (word) extracted as a keyword (word) corresponding to a general detailed item in the invoice issued by the invoice source, and this keyword is described in the form image showing the invoice. and information indicating the location of the

That is, the extraction information in the individual company specification definition information 251 includes keywords extracted as specification items from the form and information indicating the position of the keyword in the form. Therefore, the individual company specification definition information 251 is information that defines specification items used in the form, keywords to be extracted as the specification items, and information indicating the position of the keyword in the form.

The example in FIG. 8 shows that the item name "product name" of a general form may be replaced with a keyword such as "product name" in the details of an invoice from Company A. ing.

Further, in the example of FIG. 8, as information indicating the position of the keyword "product name" in the form image showing the invoice for which the invoice is made by Company A, a cell in which the keyword coordinates indicating the coordinates of the keyword and the keyword "product name" are entered. and column coordinates indicating the coordinates of the column in which the keyword "product name" is entered are extracted.

Note that the information indicating the position where the keyword is written in the form image indicating the invoice may be keyword coordinates. The keyword coordinates are coordinates of two diagonal points in a rectangle circumscribing the keyword. From these coordinates, the width and height of the circumscribing rectangle of the keyword can be found. Similarly, the cell coordinates are coordinates of two diagonal points in a rectangle circumscribing the character string in the cell immediately below the keyword. Assume that cells in the same column have the same width and height. The column coordinates are the coordinates of two diagonal points in a rectangle that circumscribes the character string in the cell immediately below the keyword and the character string in the lowest cell.

<Server device functions>
Next, with reference to FIG. 9, functions of the server device 200 of this embodiment will be described. FIG. 9 is a diagram for explaining the functions of the server device 200. As shown in FIG.

In the server apparatus 200 of this embodiment, for example, the CPU 201 reads out and executes an information processing program stored in the HD 204, the RAM 203, etc., thereby realizing the functions of each unit described later.

The server device 200 of this embodiment has a form recognition processing unit 220 , a definition registration unit 290 and a communication unit 298 . The form recognition processing section 220 has an input reception section 221 , an authentication processing section 222 and a form recognition section 270 .

The input reception unit 221 receives various inputs to the server device 200 . Specifically, the input reception unit 221 receives input of authentication information for logging into the form recognition system 100 . Authentication information is, for example, a user ID and a password. The input reception unit 221 also receives input of form image data transmitted from the image forming apparatus 300, for example. The input reception unit 221 may receive an input to the server device 200 via the communication unit 298 or may receive an input made directly to the server device 200 .

The authentication processing unit 222 performs authentication based on the authentication information received by the input receiving unit 221 . Note that the authentication processing unit 222 may, for example, transmit the input authentication information to an authentication server provided outside the server device 200 and acquire the result of authentication by the authentication server. The authentication processing unit 222 may pass the form image data received by the input receiving unit 221 to the form recognition unit 270 when the user is authenticated.

The form recognition unit 270 of this embodiment includes an image data storage unit 271, a job list generation unit 272, an image data acquisition unit 273, a preprocessing unit 274, a character recognition unit 275, a definition identification unit 276, an item extraction unit 277, and a display screen. It has a generation unit 278 and an output unit 279 .

When the input reception unit 221 receives form image data from the image forming apparatus 300 via the communication unit 298 , the image data storage unit 271 stores the data in the form image database 230 .

When input receiving portion 221 receives input of form image data from image forming apparatus 300 via communication portion 298, job list generating portion 272 registers a job in the job list and manages the job list. Details of the job list will be described later.

The image data acquisition unit 273 acquires form image data stored in the form image database 230 . The preprocessing unit 274 performs preprocessing for form recognition on the form image data acquired by the image data acquiring unit 273 .

The character recognition unit 275 extracts a character string and information indicating the position where the character string is arranged from the form image data, and holds it as reading result data. Details of the read result data will be described later.

The definition identification unit 276 is an acquisition unit that identifies the specification definition information to be referred to when recognizing the form based on the character string included in the read result data, and acquires the specified specification definition information from the specification definition database 250 .

The item extraction unit 277 extracts the items included in the form image data and the item values based on the read result data, generates recognition result data in association with each other, and stores the recognition result data in the recognition result database 260 .

At this time, the item extraction unit 277 refers to the specification definition information specified by the definition specification unit 276, and extracts the specification item names described in the specification part of the form in association with the item values of the specification items.

In the following description, the information that associates the detail item name written in the detail part of the form with the item value of the detail item may be referred to as detail information. The detailed information of this embodiment is information included in the recognition result data, and is a part of the recognition result data.

Further, the recognition result data of the present embodiment includes information (issuer information) indicating the issuer of the form described in the form. That is, the recognition result data may include all character strings written on the form.

The display screen generator 278 uses the recognition result data stored in the recognition result database 260 to generate a recognition result confirmation screen. The display screen generation unit 278 instructs the terminal device 400 to display a screen via the communication unit 298 . Specifically, the display screen generation unit 278 causes the terminal device 400 to display a screen for generating individual company specification definition information. The display screen generation unit 278 also generates an XML file for displaying the screen, and displays the input screen on the web browser of the terminal device 400 via the communication unit 298 .

In this embodiment, the display screen generation unit 278 causes the terminal device 400 to display a screen for generating the individual company specification definition information 251, but the present invention is not limited to this. A device on which the display screen generation unit 278 displays a screen may be the server device 200 or a device other than the terminal device 400, as long as it has a display function.

The output unit 279 outputs the recognition result confirmation screen generated by the display screen generation unit 278 via the communication unit 298 . In other words, the output unit 279 causes the recognition result confirmation screen generated by the display screen generation unit 278 to be displayed on the web browser of the terminal device 400 via the communication unit 298 .

The definition registration unit 290 of this embodiment has an operation content acquisition unit 291 , a definition generation unit 293 and a definition update unit 294 .

The operation content acquisition unit 291 receives an operation input to the screen displayed on the web browser of the terminal device 400 via the communication unit 298 . More specifically, in this embodiment, the communication unit 298 receives information indicating an operation from the terminal device 400, and the operation content acquisition unit 291 receives input of information received by the communication unit 298. FIG.

The definition generation unit 293 generates individual company statement definition information based on the information input from the terminal device 400 to the input screen and stores it in the statement definition database 250 .

The definition update unit 294 updates individual company specification definition information by reflecting corrections to the individual company specification definition information already registered in the specification definition database 250 .

The communication unit 298 of this embodiment is, for example, a function realized by the network I/F 209 or the like, and transmits and receives data between the server device 200 and an external device. Specifically, for example, the communication unit 298 transmits a notification for instructing display of a screen to the terminal device 400 according to the control of the display screen generation unit 278 .

<Terminal device functions>
Next, with reference to FIG. 10, functions of the terminal device 400 of this embodiment will be described. FIG. 10 is a diagram explaining the functions of the terminal device 400. As shown in FIG.

The functions of each unit described later are realized by the CPU 401 of the terminal device 400 reading and executing programs stored in the EEPROM 404, the ROM 402, and the like.

The terminal device 400 of this embodiment has an operation reception unit 431 , a display control unit 432 and a communication unit 433 . The operation reception unit 431 receives an input to the screen displayed on the web browser of the terminal device 400 by the display control unit 432 .

The display control unit 432 controls display on the display 418 of the terminal device 400 . Specifically, display control unit 432 causes display 418 to display a screen corresponding to an instruction from server device 200 .

The communication unit 433 is a function realized by the long-distance communication circuit 412, the antenna 412a of the long-distance communication circuit 412, etc., and transmits and receives data between the terminal device 400 and another device.

<Operation or processing>
Next, the operation of the form recognition system 100 of this embodiment will be described with reference to FIGS. 11A and 11B. 11A and 11B are sequence diagrams illustrating the operation of the form recognition system.

In the form recognition system 100, when the image forming apparatus 300 receives a request to start an application for performing form recognition from a user (step S1101), this application is started (step S1102).

Subsequently, when the image forming apparatus 300 accepts input of authentication information from the user (step S1103), the image forming apparatus 300 transmits this authentication information to the server apparatus 200 and requests authentication (step S1104).

The communication unit 298 of the server apparatus 200 receives the authentication request, performs authentication by the authentication processing unit 222, and notifies the result to the image forming apparatus 300 (step S1105). Note that here, the operation when the user authentication is successful is shown.

Subsequently, the image forming apparatus 300 receives an instruction to scan a form (step S1106), scans the form, and acquires form image data (step S1107). Subsequently, the image forming apparatus 300 transmits the form image data to the server apparatus 200 (step S1108).

When the input reception unit 221 of the server device 200 receives the input of the form image data via the communication unit 298, the authentication processing unit 222 passes the form image data to the form recognition unit 270 (step S1109). In the example of FIG. 11A, the form image data is passed to the form recognition section 270 via the authentication processing section 222, but the present invention is not limited to this. The form image data may be passed to the form recognition section 270 without going through the authentication processing section 222 .

When the form recognition unit 270 of the server device 200 receives the form image data, the job list generation unit 272 registers the job in the job list (step S1110). Subsequently, the form recognition unit 270 causes the image data storage unit 271 to store the form image data in the form image database 230 (step S1111).

Subsequently, the form recognition unit 270 acquires form image data to be recognized from the form image database 230 by the image data acquisition unit 273 (step S1112), and executes processing for recognizing the form image (step S1113). . Details of the processing in step S1113 will be described later.

Subsequently, the form recognition unit 270 stores the recognition result data generated by the item extraction unit 277 in the recognition result database 260 (step S1114).

In the form recognition system 100, when the terminal device 400 receives an input of a browser display instruction from the user through the operation reception unit 431 (step S1115), the terminal device 400 transmits recognition result data to the server device 200 via the communication unit 433. is requested to display the browser (step S1116).

Further, the terminal device 400 receives input of authentication information from the user through the operation reception unit 431 (step S1117), and transmits this authentication information to the server device 200 through the communication unit 433 (step S1118).

The communication unit 298 of the server device 200 receives the authentication request, and the authentication processing unit 222 authenticates the user (step S1119). The communication unit 298 transmits the authentication result and the job list to the terminal device 400 (step S1120). The communication unit 433 of the terminal device 400 receives the recognition result and the job list, and the display control unit 432 displays them on the browser. Here, processing when user authentication is successful is shown.

Subsequently, when the terminal device 400 accepts selection of a job from the list of job lists by the operation accepting unit 431 (step S1121), the communication unit 433 transmits the selection to the server as an instruction to view the recognition result of the form image. A browsing request is issued to the device 200 (step S1122).

The communication unit 298 of the server device 200 receives the browsing request, the display screen generation unit 278 acquires recognition result data stored in the recognition result database 260 (step S1123), and generates a recognition result confirmation screen. The communication unit 298 transmits screen information of the recognition result confirmation screen to the terminal device 400 . The communication unit 433 of the terminal device 400 receives the screen information of the recognition result confirmation screen, and the display control unit 432 displays it on the browser (step S1124). The user can confirm the recognition result on the displayed recognition result confirmation screen.

Next, description will be made with reference to FIG. 11B. In general, recognition results using the industry specification definition information 252 or the general specification definition information 253 do not correctly reflect the delimiters of rows and columns in a form. This is the same when the individual company detail definition information 251 is used, and the height of the line changes depending on the contents of the form (long product name, etc.), and the recognition result correctly reflects the line separation in the form. There may be cases where there is no For this reason, the user operates the terminal device 400 to adjust the positions of rows and columns with respect to the recognition result (S1125). The operation reception unit 431 of the terminal device 400 receives the user's operation, and the display control unit 432 displays the row and column adjustment results according to the user's operation (S1126). The screen for row adjustment will be described with reference to FIG. In this embodiment, the adjustment result is a new line break, but the user can similarly adjust the column position. Therefore, the adjustment result includes the row and column position information of the form.

In the present embodiment, lines are connected by a line connection process, which will be described later. Therefore, the user only has to make an adjustment to the extent that the form image data is divided into lines at equal intervals. Also, if the form recognition unit 270 divides the form image data into lines at equal intervals, the user's adjustment may not be necessary.

After finishing the adjustment, the user instructs the terminal device 400 to recognize the form including line connection (S1127). Since form recognition has already ended in step S1113, form recognition is performed again. The operation reception unit 431 receives the instruction, and the communication unit 433 transmits a form recognition request including the user's adjustment result (row and column position information) to the server apparatus 200 (S1128).

The communication unit 298 of the server apparatus 200 receives the form recognition request, and the form recognition unit 270 performs form recognition including line concatenation (S1129). Details of the processing in step S1129 will be described with reference to FIG.

In the communication unit 298 of the server device 200, the form recognition unit transmits the recognition result data including the line connection result to the terminal device 400 (S1130).

The communication unit 433 of the terminal device 400 receives the recognition result data, and the display control unit 432 displays the recognition result including the line connection result (S1131).

As a result, lines that have been recognized as different lines depending on the height of the line are displayed as recognized details that are connected.

The user instructs the terminal device 400 to input an instruction to register individual company statement definition information. When the terminal device 400 receives the input of the registration instruction of the individual company specification definition information by the operation reception unit 431 (step S1132), the terminal device 400 transmits the registration request of the individual company specification definition information to the server device 200 via the communication unit 433. (step S1133).

When the communication unit 298 of the server device 200 receives this registration request, the display screen generation unit 278 generates a registration screen for individual company specification definition information. The communication unit 298 transmits the screen information of the registration screen of the individual company specification definition information to the terminal device 400 . The communication unit 433 of the terminal device 400 receives the screen information of the individual company specification definition information registration screen, and the display control unit 432 displays it on the browser (step S1134).

Subsequently, when the operation reception unit 431 of the terminal device 400 receives input of individual company specification definition information (step S1135), the communication unit 433 transmits the input information to the server device 200 (step S1136). The communication unit 298 of the server device 200 receives the input information, and the definition generation unit 293 generates individual company statement definition information based on the information transmitted from the terminal device 400 (step S1137). Store (step S1138) and terminate the process.

<Job List>
Next, a job list generated by the job list generation unit 272 of this embodiment will be described with reference to FIG. FIG. 12 is a diagram showing an example of a job list.

When the form image data is input from the image forming apparatus 300, the server apparatus 200 of the present embodiment holds the form image data and the job ID as a job list in which the form image data and the job ID are associated with each other.

The job list 101 of this embodiment has, as information items, a job ID, an invoicing party, a form image file path, a recognition result file path, a status, and a user ID.

The value of the item "job ID" is identification information that identifies a job. In other words, the value of the item “job ID” is identification information for specifying form image data received from the image forming apparatus 300 .

The item "claimer" is the company name of the issuer of the form. Since the invoicing party is unknown at the time of job registration, it may be set after form recognition.

The value of the item "image file path" is information indicating the location where the form image data is stored. The value of the item "recognition result file path" is information indicating the location where the recognition result data obtained by recognizing the form image is stored.

The value of the item "status" indicates the progress of recognition of the form image. In other words, the value of the item "status" indicates the status of the job. In this embodiment, the four values of the item "status" are exemplified as the unprocessed state, the draft save state, the finalized state, and the externally output state, but the present invention is not limited to these. Each state is as follows.

The unprocessed state (state 1) is a state immediately after the recognition result data is acquired by form recognition, and confirmation of the recognition result is not performed.

The draft storage state (state 2) is a state in which work such as recognition result data confirmation is in progress, and the recognition result data is in a state before finalization.

The finalized state (state 3) is a state in which work such as confirmation of recognition result data is completed and stored in the recognition result database 260 as finalized information.

The external output completed state (state 4) is a state in which output of recognition result data to an accounting system or the like that cooperates with the form recognition system 100 has been completed.

The value of the item “user ID” is identification information for specifying a user who has logged in to the form recognition system 100 .

When the server apparatus 200 of the present embodiment receives the form image data from the image forming apparatus 300, the job list generation unit 272 assigns a job ID, associates the user ID obtained as the authentication information, and creates the job list 101. Add records. Then, when the form image data is stored in the form image database 230, the job list generation unit 272 adds information indicating the storage destination as the value of the item "image file path".

Further, when the recognition result data is stored in the recognition result database 260, the job list generation unit 272 adds the issuer to the item "billing party", and adds the information indicating the storage destination to the item "recognition result file path". Add it as a value and set the value of the item "Status" to the finalized state.

In this embodiment, the terminal device 400 may display the job list generated by the job list generation unit 272, for example. In this case, the user can check the progress of recognition of the form image through the job list screen displayed based on the job list.

<Recognition of forms>
Next, referring to FIG. 13, processing of the form recognition unit 270 of this embodiment will be described. FIG. 13 is a first flowchart for explaining the processing of the form recognition unit 270. As shown in FIG. FIG. 13 shows details of the processing in step S1113 of FIG. 11A. 13 shows processing when a plurality of form image data are received from the image forming apparatus 300. FIG.

The image data acquiring unit 273 acquires the form image data for which the form recognition result data is not generated from the form image database 230 (S1300).

The preprocessing unit 274 performs processing for recognizing rows and columns for the form image data acquired by the image data acquiring unit 273 (S1301). Recognition here may be rough based on horizontal and vertical lines, regions with different colors, and the like.

The preprocessing unit 274 performs preprocessing for form recognition on the form image data acquired by the image data acquiring unit 273 (step S1302). Specifically, the pre-processing is, for example, correction processing such as binarization, noise removal, and top-bottom alignment of the form image.

Subsequently, the form recognition unit 270 uses the character recognition unit 275 to cut out characters in an area in which characters are formed (hereinafter referred to as a character field) from the form image and recognize the characters (step S1303). At this time, the character recognition unit 275 also acquires coordinates indicating the position of the extracted character.

Subsequently, the form recognition unit 270 generates and stores reading result data based on the positional relationship of each character string recognized by the character recognition unit 275 (step S1304). Specifically, based on the coordinates of each recognized character, the form recognition unit 270 recognizes characters whose distance from each other is within a preset threshold value as one character string, and forms character strings. The reading result data is generated in association with information such as coordinates indicating the position of the marked area.

Subsequently, the form recognition unit 270 causes the definition identification unit 276 to refer to the information for identifying the billing party (hereinafter referred to as billing party information) included in the read result data and the industry determination database 240, and identify the business type of the billing party. Identify (step S1305).

Subsequently, the form recognition unit 270 determines whether or not individual company statement definition information 251 corresponding to the claimant information exists in the statement definition database 250 by means of the definition identification unit 276 (step S1306).

In step S1306, if the individual company specification definition information 251 exists, the form recognition unit 270 causes the item extraction unit 277 to extract the specification information based on the individual company specification definition information 251 (step S1307). move on. Details of the processing in step S1307 will be described later.

In step S1306, if the individual company specification definition information 251 does not exist, the definition identification unit 276 determines whether or not the specification definition database 250 contains the industry specification definition information 252 corresponding to the industry identified in step S1305. (step S1308).

In step S1307, if the industry specification definition information 252 exists, the form recognition unit 270 causes the item extraction unit 277 to extract the specification information based on the industry specification definition information 252 (step S1309), and proceeds to step S1311, which will be described later. Details of the processing in step S1309 will be described later.

In step S1308, if the industry specification definition information 252 does not exist, the form recognition unit 270 causes the item extraction unit 277 to extract the specification information based on the general-purpose specification definition information 253 (step S1310), and proceeds to step S1311 to be described later. Details of the processing in step S1310 will be described later.

The form recognition unit 270 determines whether or not the detailed information has been successfully extracted (step S1311). Specifically, the form recognition unit 270 determines whether or not there is detailed information extracted in any one of steps S1307, S1309, and S1310.

In step S1311, if the extraction of the detailed information fails, the form recognition unit 270 notifies the terminal device 400 or the image forming apparatus 300 of an error (step S1312), and terminates the process.

In step S1311, if the detailed information is successfully extracted, the form recognition unit 270 causes the item extraction unit 277 to generate recognition result data by merging the extracted detailed information, and stores the data in the recognition result database 260 (step S1313). ).

Subsequently, the form recognition unit 270 registers the storage destination of the recognition result data in the job list 101 by the job list generation unit 272 (step S1314).

Subsequently, the form recognition unit 270 causes the display screen generation unit 278 to generate a recognition result confirmation screen generated using the recognition result data. The communication unit 433 of the terminal device 400 receives the screen information of the recognition result confirmation screen, the display control unit 432 displays the recognition result confirmation screen (step S1315), and the process ends.

In addition, in the example of FIG. 13, the process up to the process of displaying the recognition result data has been described as a series of operations, but the present invention is not limited to this. The form recognition unit 270 may display the recognition result data after accepting a request for viewing the recognition result data from the terminal device 400 .

<Extracting detail information>
Next, extraction of detailed information by the item extraction unit 277 of the present embodiment will be described with reference to FIGS. 14 and 15. FIG. FIG. 14 is a second flowchart for explaining the processing of the form recognition unit 270. As shown in FIG. FIG. 15 shows details of the processing in step S1307 of FIG.

In the form recognition unit 270 of this embodiment, the item extraction unit 277 acquires the coordinate values of each character when the reading result data is held (step S1401). Subsequently, the item extraction unit 277 acquires the extracted character string (extracted in step S1304) and its coordinate values (step S1402).

Next, the item extracting unit 277 searches for the item names included in the individual company item definition information 251 using the acquired character strings (step S1403), and determines whether or not there is a search result (step S1404). ). Specifically, the item extraction unit 277 determines whether or not the individual company specification definition information 251 has a specification item name or keyword that matches the character string.

In step S1404, if the corresponding item name or keyword does not exist, the form recognition unit 270 terminates the process. Also, in step S1404, if there is a corresponding detail item name or keyword, the item value is extracted from the column aligned vertically under the detail item name or keyword matching the search (step S1405), and the process ends. do.

Note that the processing in step S1309 in FIG. 13 is the same as the processing in FIG. 14 except that the detailed definition information to be referred to in step S1403 in FIG.

FIG. 15 is a third flowchart for explaining the processing of the form recognition unit 270. As shown in FIG. FIG. 15 shows details of the processing in step S1310 of FIG. In other words, FIG. 15 shows the process of referring to the general-purpose specification definition information 253 and extracting the specification information when neither the individual company specification definition information 251 nor the industry specification definition information 252 corresponding to the invoicing party exists.

The processing in steps S1501 and S1502 in FIG. 15 is the same as the processing in steps S1401 and S1402 in FIG. 14, so description thereof will be omitted.

After obtaining the character string and the coordinate values, the item extraction unit 277 determines whether or not there is a vertical ruled line (vertical line) in the form image data (step S1503). In step S1503, if there is no vertical ruled line, item extraction unit 277 proceeds to step S1510, which will be described later. A well-known method such as Hough transform is available as a method for detecting straight lines (vertical ruled lines).

In step S1503, if there is a vertical ruled line, the item extraction unit 277 extracts the coordinate values of each cell of the vertical ruled line (step S1504). At this time, the item extraction unit 277 may assign identification information (ID) for identifying each cell to each cell. Also, the identification information may be given as, for example, a column ID that identifies the columns of the table and a row ID that identifies the rows of the table. By assigning a column ID and a row ID in this way, a cell is specified by a combination of the column ID and the row ID.

Subsequently, the item extraction unit 277 identifies the character string included in each cell of the table, and stores it in association with the coordinate value of the cell (step S1505).

Subsequently, the item extraction unit 277 identifies the character strings (column headings) included in each cell of the top row of the table (step S1506).

Subsequently, the item extraction unit 277 searches the general-purpose specification definition information 253 with the character strings (column headings) included in each cell of the first row (step S1507).

Subsequently, the item extracting unit 277 determines whether or not the general-purpose specification definition information 253 includes a specification item name or keyword that matches the character string (column heading) contained in each cell of the first row (step S1508). ).

If the determination in step S1508 is No, the item extractor 277 terminates the process.

If the determination in step S1508 is Yes, the item extraction unit 277 extracts the item value of the detail item from the column containing the character string contained in each cell of the top row (step S1509), and ends the process.

In step S1503, if there is no vertical ruled line, the form recognition unit 270 identifies the detail area of the form image (step S1510). Specifically, the form recognition unit 270 may, for example, identify an area in which character strings are arranged in the vertical direction as the detail area.

Subsequently, the form recognition unit 270 uses the item extraction unit 277 to identify the character string in the first line in the area identified as the detail part, and stores it in association with the coordinate values of the identified character string (step S1511). The character string on the first line in the specified area is the character string positioned at the top of the specified area.

Subsequently, the item extraction unit 277 searches the general-purpose specification definition information 253 with the character string specified in step S1511 (step S1512). Subsequently, the item extraction unit 277 determines whether or not the general-purpose specification definition information 253 has a specification item name or keyword that matches the specified character string (step S1513).

If the determination in step S1513 is No, the form recognition unit 270 terminates the process.

If the determination in step S1513 is YES, the item extraction unit 277 obtains the width of the region where the character string circumscribes from the coordinate values of the character string identified in step S1511, and extracts the character strings arranged vertically within this width. is extracted as the item value of the detail item (step S1514), and the process ends.

In the example of FIG. 15, when the form image has vertical lines, the coordinate values of the cells are extracted, but horizontal lines may be used instead of the vertical lines. In this case, the form recognition unit 270 determines whether or not a horizontal line exists in the form image. Good luck.

If the general-purpose specification definition information 253 contains a description item name or keyword that matches the leftmost character string among the character strings between the horizontal lines, the form recognition unit 270 recognizes the horizontal line and the horizontal line. The second and subsequent character strings between are set as the item value of the detail item.

<Extracting detailed information using a form>
Next, the processing of the form recognition unit 270 will be further described with reference to FIGS. 16 to 18. FIG. FIG. 16 is a first diagram for explaining extraction of detailed information.

FIG. 16 is a first diagram for explaining form image data. A bill 161 shown in FIG. 16 includes billing party information 161a that identifies the billing party, and a detailed portion 161b in which data of the same format is repeatedly printed in the vertical direction for each detailed item. In the invoice 161 shown in FIG. 16, the billing party information 161a is the name of the billing party, and the detail part 161b is described in a tabular form.

In the present embodiment, the image forming apparatus 300 reads the invoice 161 to generate form image data, and the form recognition unit 270 processes the form image data to generate recognition result data.

FIG. 17 is a second diagram for explaining extraction of detailed information. Information 171 shown in FIG. 17 is the result of character recognition performed by the character recognition unit 275 on the form image data representing the invoice 161 .

Information 171 shows an example of extracting a character field from the specification part 161b of the invoice 161, recognizing the character in the character field, and extracting the coordinates indicating the position of the character.

From this information 171, the character recognition unit 275 of this embodiment recognizes a character string (character string) by connecting characters at adjacent positions. Specifically, the character recognition section 275 may recognize a combination of characters whose distance between characters is within a predetermined number of pixels as one word.

Information 172 shown in FIG. 17 is information indicating character strings recognized by the character recognition unit 275 and areas of the character strings, and is read result data of the form image by the character recognition unit 275 .

Further, the character recognition unit 275 recognizes a region delimited by vertical lines and horizontal lines from the form image data as one cell, and assigns identification information (cell ID) for each cell to identify the cell.

The information 173 shown in FIG. 17 is information indicating the correspondence between the area for each cell recognized by the specification section 161b and the cell ID.

In FIG. 17, for example, the region in which the column heading “product name” is recognized is specified in the information 172 from the height and width based on the position indicated by the X and Y coordinates.

Also, in the information 173, it can be seen that the area where the column heading "product name" is recognized is included in the cell ID "2". Therefore, the column heading "product name" is associated with the cell ID "2" as a column heading included in the detail portion 161b.

In the information 173, the cell ID is associated with the region in which the column heading "product name" is recognized, but the present invention is not limited to this. A cell ID may be identified by a row ID identifying a column in which row headings are arranged vertically and a column ID in which column headings are arranged horizontally.

The form recognition unit 270 of the present embodiment thus associates the character strings with the cells from the form image data.

The detailed information 174 shown in FIG. 18 is a third diagram showing the detailed information of the detailed part 161b generated based on the reading result data corresponding to the invoice 161. FIG. Therefore, the detailed information 174 shown in FIG. 18 is part of the recognition result data of the entire form image data.

The detail information 174 shown in FIG. 18 is generated by the item extraction unit 277, and has column IDs, column headings, column coordinates, row IDs, item values, and detail item names as information items.

The value of the item "column ID" is identification information for specifying the columns included in the detail portion 161b (table), and the value of the item "column heading" is the column heading in each cell of the top row of the table. is. The value of the item "column coordinates" indicates column coordinates. In this embodiment, the column coordinates are the coordinate values of the upper left point and the lower right point of the detail column in the form image.

The value of the item "line ID" is identification information that identifies the line, and the value of the item "item value" indicates the item value corresponding to the detail item.

The value of the item “description item name” indicates which item in the individual company description definition information 251 is associated with the corresponding description item name and the coordinates indicating the position of the item name. The coordinates indicating the position of the item name are indicated, for example, by a combination of column coordinates and row IDs.

In the detail information 174, the column header "slip date" is associated with the value "date" of the item "description item name". In addition, column coordinates and row IDs are associated with the column header "slip date".

When the user decides to update the individual company statement definition information 251 and performs this operation, based on the column coordinates and row ID of the "document date", the keyword: date", keyword coordinates (one record of the product name in the column header of the information 172), cell coordinates (one record of the cell corresponding to the product name in the information 173), and column coordinates (cell coordinates x coordinates for the number of rows) are registered. .

In the example of the detailed information 174 in FIG. 18, the information items include column coordinates, column IDs, and row IDs, but the information items are not limited to these. The detailed information 154 may contain detailed item names and item values as information items, and other items may not be included in the detailed information 174 .

Further, in the present embodiment, when displaying the detailed information 174 on the recognition result confirmation screen, the display screen generation unit 278 may display only the detailed item names and item values.

<Display example of recognition result>
Next, a display example of recognition results of form image data will be described with reference to FIG. FIG. 19 is a diagram showing a recognition result confirmation screen 501. As shown in FIG. The recognition result confirmation screen 501 is displayed on the terminal device 400 in step S1125 of FIG. 11B, for example.

The recognition result confirmation screen 501 includes a display column 502 , a display column 503 , and display components 508 and 509 . A display field 502 displays a form image based on the form image data. In the example of FIG. 19, the form is a bill, and the form image includes an area 504 indicating the details part.

A display field 503 displays recognition result data obtained as a result of the form recognition performed by the form recognition unit 270 on the form image data representing the form image displayed in the display field 502 .

Display field 503 includes display fields 505 , 506 , and 507 . A display column 505 displays the value of the item “status” of the job list 101 . In a display column 506, billing party information is displayed. The billing party information is part of the recognition result data.

A display column 507 displays detailed information. Specifically, the display column 507 displays a list of the values of the detail items included in the detail information. A display column 507 repeatedly displays the extracted detail items of each line. Although two lines are displayed in FIG. 19, detailed items of other lines can be displayed by scrolling or the like. Display column 507 includes display columns 511 , 513 , 514 , 515 , and 516 .

A display column 511 displays the value of the detail item with the detail item name “product name” displayed in the area 504 indicating the detail part in the form image. A display column 513 displays the value of the detail item with the detail item name “amount” displayed in the area 504 .

A display column 514 displays the value of the detail item with the detail item name “unit price” displayed in the area 504 . A display column 515 displays the value of the detail item with the detail item name “quantity” displayed in the area 504 . A display column 516 displays the value of the detail item with the detail item name “slip No.” displayed in the area 504 .

In this embodiment, as described above, the terminal device 400 extracts the item values from the items corresponding to the item names in the item description portion of the form image, and displays them in the display columns 511, 513, 514, 515, and 516. do. In other words, items corresponding to these detailed item names are items to be extracted (extraction target items). Also, the items to be extracted may be fixed items, or may be changed by the user.

The detail extraction button 510 is a button for the user to request form recognition in order to correctly recognize line breaks in a form including lines with different heights in step S1127 of FIG. 11B.

A display component 508 is a display component for changing the status value of the job list 101 from an unprocessed state to a draft save state. When the display component 508 is operated, the display field 505 changes from "unprocessed" to "save draft".

A display component 509 is a display component for setting the status value of the job list 101 to a confirmed state. In the example of FIG. 19, the extraction of the recognition result data from the form image data is successful, so the recognition result data is not corrected. Therefore, in the example of FIG. 19, when the display component 509 is operated, the status value of the job list 101 is changed to the finalized state, and the individual company specification definition information 251 is not generated.

<Adjustment of recognition result>
As described above, even when recognizing a form using the individual company detail definition information 251, line breaks may not be recognized correctly. In such a case, the user adjusts the line (adjustment by the user is not required) to indicate form recognition.

FIG. 20 is an example of a recognition result in which erroneous recognition of a line has occurred. For convenience of explanation, only the form image data displayed in the area 504 is shown in FIG. Also, for explanation, the line numbers are shown on the right bridge. In FIG. 20(a), as an example, every two lines are recognized. As for the recognition result, a translucent part 540 that distinguishes between rows and columns by color or the like is superimposed on the form image data so that the separation between rows and columns can be recognized. In FIG. 20, a dotted line indicates a delimiter between rows and columns. By dragging the translucent component 540 with a mouse or the like, the user can align each row at the same height, add rows, or delete rows.

Here, the user wants to separate the 1st line and the 4th to 8th lines, and wants to connect the 2nd and 3rd lines. Therefore, the user operated the translucent component 540 to divide the form image data into eight lines at regular intervals, as shown in FIG. 20(b). In this state, the server device 200 can connect the second and third rows. This process corresponds to the process of step S1129 in FIG. 11B.

In this embodiment, the adjustment of column delimiters is not particularly described, but the user can also adjust the columns. The translucent component 540 displays the keywords 521 to 526 registered in the individual company specification definition information 251, industry specification definition information 252, and general specification definition information 253 in association with each column of the form image data. In FIG. 20(a), keywords 521 to 526 are displayed above the column headings. The display position is only an example. If the correspondence between the keywords 521-526 and the columns is incorrect, the user can move the translucent part 540 for each column and adjust the correspondence between the column headings and the keywords 521-526. For example, the keyword 524 "quantity" corresponds to "quantity" and "unit" of the form image data, but "unit" can be excluded.

FIG. 21 is an example of a flowchart for explaining the procedure for the form recognition unit 270 to concatenate a plurality of lines.

First, the form recognition unit 270 extracts rows in which item values are entered in the item of amount or the item of quantity and unit price, and sets label A (S1129-1). The label is line type information that is set for convenience because it is necessary to distinguish lines in subsequent processing. Note that the row with label A is an example of the first row.

The reason why attention is paid to the amount item is that the amount is the object of processing in the form. The reason why the quantity and unit price are noted is that there are cases where the quantity and unit price are written on the form even if the amount is not written.

Note that if items such as the amount are not specified, the item of the amount is not focused. The amount is an item set by the user, but if the user sets another item, the form recognition unit 270 focuses on that item. Also, if neither amount, quantity nor unit price is set, label A is set for all rows.

The form recognition unit 270 connects a line labeled D to a line labeled A, which will be described later. Therefore, the form recognition unit 270 sets a label for each line as follows.
・Label B is a row in which one of the items contains the words "consumption tax" or "subtotal" or "total" or "slip total" (corresponding to exclusion terms). "Consumption tax", "subtotal", "total", or "slip total" are terms that are not used as item names. Note that the row with label B is an example of the third row. In addition, the user can appropriately set exclusion terms, and the exclusion terms in this embodiment are merely examples.
• Label C is a row in which no item has a value. Note that the row with label C is an example of a fourth row.
• Label D is the other row. Note that the row with label D is an example of the second row.

Therefore, the form recognition unit 270 changes the label of the line containing any of the characters "consumption tax", "subtotal", "total", and "slip total" from A to B (S1129-2).

The form recognition unit 270 sets a label C to a row in which no item has a value (S1129-3).

The form recognition unit 270 sets the label D to other rows (S1129-4). Note that the following processing will be supplemented with reference to FIG. 22, so please refer to it.

Next, the form recognition unit 270 determines whether or not the label A is set in the rows above and below (both of) the row with the label D (S1129-5). If there are further rows with label D above and below the row with label D, the row with label D at the farthest end is determined.

If the determination in step S1129-5 is No, the form recognition unit 270 determines whether or not the label A is set in one of the rows above and below the row labeled D (S1129-6).

If the determination in step S1129-6 is Yes, the form recognition unit 270 connects the row labeled D to the row labeled A (S1129-7). If the determination in step S1129-6 is No, the form recognition unit 270 does not connect label D to any line.

If the determination in step S1129-5 is YES, the form recognition unit 270 determines that there is no line above the line of label A that appears first in the form (details) (the line itself does not exist. The range of form image data outside.), label B, or label C (S1129-8).

If the determination in step S1129-8 is Yes, the form recognition unit 270 connects the row labeled A above the row labeled D (S1129-9).

If the determination in step S1129-8 is No, the line above the line with label A is labeled D, but the form recognition unit 270 recognizes the line with label D in the line with label A below the line with label D. Connect (S1129-10).

The form recognition unit 270 determines whether or not "quantity" and "unit price" are present in multiple lines as a result of linking label A to label D (S1129-11).

The form recognition unit 270 adopts the "quantity" and "unit price" closer to the row of label A (S1129-12).

FIG. 22 is a diagram schematically illustrating the connection of rows labeled A and D. As shown in FIG. First, FIG. 22(a) shows a case where a label A row exists in the rows above and below (both of) the label D row. In this case, as described in the flowchart of FIG. 21, it is determined whether the row above the row labeled A in the row adjacent to the upper side of label D is none, labeled B, or labeled C ( S1129-8).

FIG. 22(b) shows the case of none. In this case, the form recognition unit 270 connects the lines of the labels A and D in the line adjacent to the upper side of the label D. FIG.

FIG. 22(c) shows the case where the row above the row of label D is label D. FIG. In this case, the form recognition unit 270 connects the row of the label D, which has the label A above and below, and the row of the label A of the row adjacent to the lower side of the label D. If a row with label B is below label A, then label B is also concatenated.

Note that, as shown in FIG. 23, label D may be adjacent to the upper side of label D, and label A lines may exist above and below it. The number of rows labeled D is not limited to two. In this case as well, it is determined that the label A is set in the rows above and below (both of) the row with the label D. FIG. In this case, it is determined whether the line above the outermost label A is none, label B, or label C (S1129-8).

FIG. 24 shows labels A to D set for each line of the form image data by the linking process of FIG. As shown in FIG. 24, label D is set in the third row. Since only the row above the row to which label D is set has label A, the form recognition unit 270 connects the second and third rows. Therefore, the form recognition unit 270 can extract "xyz data migration fee" and "[your company's order No. 123456789]" in two lines as the product name in one line.

Note that the server device 200 does not store the result of linking the row labeled D with the row labeled A as described above. That is, the server device 200 does not store the coordinates of the cells and columns in the individual company specification definition information 251 in a state where the row of the label A and the label D are linked. This is because the location of the row labeled D is determined not by the form but by the content of the description, so there is little merit in saving it. Therefore, the individual company specification definition information 251 registered in step S1138 of FIG. 11B is in a state in which rows and columns have been adjusted in step S1126.

FIG. 25 is a diagram for explaining product names linked by the linking process of FIG. 21 . 25 shows the display column 507 of the recognition result confirmation screen 501. As shown in FIG. FIG. 25(a) shows the display field 507 before the connection process, and FIG. 25(b) shows the display field 507 after the connection process. Before the consolidation process, the product name in the display field 511 is divided into "xyz data transfer fee" and "[your company's order number: 123456789]". .:123456789]”.

<Main effects>
As described above, the server device 200 of the present embodiment can connect and extract item values that include rows with different heights or that are described in a plurality of rows.

In this example, virtual ruled lines and logical ruled lines are used to correctly reflect the line breaks in the form even if there are rows with different heights or item values are written across multiple lines. The server device 200 that extracts rows will be described. In this embodiment, since the line breaks in the form are correctly recognized, the same result as in the case of concatenation in the first embodiment can be obtained.

Further, the line break recognition processing of the present embodiment may be performed completely independently of the first embodiment, or may be performed before or after the connection processing of the first embodiment.

In this embodiment, it is assumed that the hardware configuration diagrams of FIGS. 2 and 3 and the functional block diagrams of FIGS. 9 and 10 described in the above embodiment can be used.

<Recognition of line breaks using virtual ruled lines and logical ruled lines>
First, with reference to FIGS. 26 and 27, the outline of the method for recognizing line breaks in this embodiment will be described. FIG. 26 shows a method for recognizing line breaks to be compared, and FIG. 27 shows an outline of a line break recognition method in this embodiment.

Assume that the server device 200 saves the individual company specification definition information 251 based on the arrangement of rows and columns (with or without user adjustment) for the form A in FIG. 26(a). When the server device 200 extracts lines from the form B in FIG. A line that is different from the correct line delimiter in is extracted.

Therefore, as shown in FIG. 27, the server apparatus 200 uses virtual ruled lines and logical ruled lines to extract lines at correct line breaks in the form. Note that the form image data in FIG. 27 is displayed on the recognition result confirmation screen 501 .

First, as shown in form A in FIG. 27(a), the user designates which column to use as a row separator. The user, for example, drags the width of the column headings with the mouse. This renders an arrow 530 revealing the column specified by the user. Here, it is assumed that the "money amount" column has been selected as a row delimiter. The server device 200 extracts the upper end of the item value as the upper end of the row in the money column.

When the form B in FIG. 27(b) is also scanned, the server device 200 uses the information that the "amount" column has been selected to set the upper end of the item value to the upper end of the row in the "amount" column. Extract. By doing so, it is possible to correctly determine the line break whether the product name has one line or multiple lines. The ruled lines extracted in this manner are referred to as "logical ruled lines 531".

However, if line delimiters are determined only by the method of FIGS. 27A and 27B, there is a problem with the following forms.

In the form C of FIG. 27(c), the product name corresponding to the first amount "1000" is only "△△△", and "○○○" is not included in the product name.

Therefore, as shown in FIG. 27(d), the user designates "virtual ruled lines 532" that divide the detail area at equal intervals. Evenly spaced lines may be recognized without user specification. In FIG. 27D, three imaginary ruled lines 532 are displayed, and the logical ruled lines 531 overlap the second and third imaginary ruled lines 532 from the top. When the logical ruled line 531 and the virtual ruled line 532 overlap, the server apparatus 200 determines that the ruled line is a correct line break.

As described above, in the row recognition method of this embodiment, information indicating which column the user uses as a row delimiter, the logical ruled line 531 (an example of the second ruled line), and the virtual ruled line 532 (the first An example of a ruled line), it is possible to extract lines by correctly reflecting the line breaks in the form.

<Operation or processing of form recognition system>
Next, the operation or processing of the form recognition system 100 will be described with reference to FIG. The process of FIG. 28 is executed instead of steps S1125 to S1131 of FIG. 11B or after S1131.

FIG. 28 is a sequence diagram for explaining the operation of the form recognition system.

The user moves or transforms the translucent part 540 in the area 504 of the recognition result confirmation screen 501 displayed on the terminal device 400 to specify the detail area (S2101). The user overlays the translucent component 540 on the form image data so as to cover the entire form image data just enough. An operation accepting unit 431 accepts an operation.

Next, the user designates by how many lines the form image is to be separated (S2102). User simply enters a number. This number of lines can be the number of lines including blank lines. An operation accepting unit 431 accepts an operation. Alternatively, the user does not have to specify the number of rows. Since the rows and columns have been recognized in step S1301 of FIG. 13, the form recognition unit 270 can draw virtual ruled lines 532 using them.

Next, the communication unit 433 of the terminal device 400 specifies the number of lines and transmits a request to display the virtual ruled lines 532 to the server device 200 (S2103). The virtual ruled lines 532 may be displayed at regular intervals by the terminal device 400 dividing the height of the form image by the number of lines.

The communication unit 298 of the server device 200 receives the request to display the virtual ruled line 532, and the form recognition unit 270 draws the virtual ruled line 532 (S2104). Drawing is to determine the position of the virtual ruled line 532 in the form image data. The form recognition unit 270 divides the height of the form image data by the number of lines to draw virtual ruled lines 532 at regular intervals. If the user does not specify the number of lines, the form recognition unit 270 draws equally spaced imaginary ruled lines 532 with the height of the lowest line among the lines recognized in step S1301 of FIG.

298 of the server device 200 transmits the position information of the virtual ruled line 532 to the terminal device 400 (S2105). The communication unit 433 of the terminal device 400 receives the position information of the virtual ruled line 532, and the display control unit 432 displays the virtual ruled line 532 superimposed on the form image data of the area 504 (S2106). If there is an error in the virtual ruled line 532, the user corrects it as appropriate.

Next, the user designates a column to be used as a line break with a mouse or the like (S2107). For example, the column used to separate rows is the amount column. This is because, in many cases, product names, etc. in rows are separated by amounts. An operation accepting unit 431 accepts an operation. Note that at least one of the column ID, column name, and column coordinates is specified by dragging the width of the column header.

The communication unit 433 of the terminal device 400 specifies the column ID, column name, or column coordinates, and transmits a display request for the logical ruled line 531 to the server device 200 (S2108).

The communication unit 298 of the server apparatus 200 receives the display request for the logical ruled lines 531, and the form recognition unit 270 draws the logical ruled lines 531 (S2109). That is, the form recognition unit 270 draws the logical ruled line 531 at the upper end of the item value in the designated column.

298 of the server device 200 transmits the position information of the logical ruled line 531 to the terminal device 400 (S2110). The communication unit 433 of the terminal device 400 receives the position information of the logical ruled line 531, and the display control unit 432 superimposes it on the form image data of the area 504 and displays it (S2111). Since the display control unit 432 displays the virtual ruled lines 532 and the logical ruled lines 531 in different colors, the user can distinguish between the logical ruled lines 531 and the virtual ruled lines 532 . If there is an error in the logical ruled line 531, the user corrects it as appropriate.

Next, the user instructs the terminal device 400 to recognize the form (extract the item values) based on the virtual ruled lines 532 and the logical ruled lines 531 (S2112). Note that the user may indicate which item values are to be concatenated. Unless otherwise instructed by the user, product names are concatenated. The user presses the detail extraction button 510 . An operation accepting unit 431 accepts an operation.

The communication unit 433 of the terminal device 400 specifies the specified item, the virtual ruled line 532, and the logical ruled line 531, and transmits an item value extraction request to the server device 200 (S2113).

The communication unit 298 of the server apparatus 200 receives the item value extraction request, and the form recognition unit 270 determines the character string between the line breaks where the virtual ruled line 532 and the logical ruled line 531 overlap as the item value (S2114). ).

The communication unit 298 of the server device 200 transmits the recognition result in which the item and the item value are associated to the terminal device 400 (S2115). The communication unit 433 of the terminal device 400 receives the recognition result, and the display control unit 432 displays it in the detailed information in the display column 507 (S2116).

FIG. 29 is a diagram for explaining product names linked by the linking process of FIG. Note that FIG. 29 mainly describes differences from FIG. 25 . A display column 507 of the recognition result confirmation screen 501 is shown. FIG. 29(a) shows a display column 507 by line breaks determined only by logical ruled lines 531, and FIG. 29(b) shows a display column 507 by line breaks determined by logical ruled lines 531 and virtual ruled lines 532. With only the logical ruled line 531, the product name in the display field 511 is divided into "○○○" and "△△△", but the product name is connected to "○○○△△△" by the logical ruled line 531 and the virtual ruled line 532. was done.

In this way, for example, even if the product name is spread over two or more lines for one line of money, the character strings of the two or more product name columns are concatenated to form the display column 511 (product name) of the recognition result confirmation screen 501 is displayed).

<Multi-column form with another character string in the specified item "Amount"column>
With reference to FIG. 30, a multi-column form in which there is another character string in the specified item "amount" column will be described. FIG. 30 shows part of a form in which consumption tax is also displayed in the amount column. As in the form in FIG. 30, in the case of a multi-column form in which other information (for example, consumption tax) is also written in the "amount" column, the logical ruled line 531 is drawn at the upper end of the characters in the "amount" column. When rendered, the amount and tax lines are extracted as separate lines.

Therefore, for multi-column forms, the number of lines to be skipped in drawing the logical ruled lines 531 is set (for each skip number) when the user designates the columns used for line breaks with a mouse or the like. For example, when the user right-clicks with the mouse on the arrow 530 specifying the above columns, a dialog for setting how many lines to skip is displayed. In the example of FIG. 30, the user inputs one line because one line may be skipped.

FIG. 31 shows a rendering example of the logical ruled lines 531 of a multi-column form when the user designates the column of "amount" and skips one line as the line break of the logical ruled lines 531. FIG. In FIG. 31, the amount (and consumption tax) and the product name are correctly associated and extracted.

<Main effects>
According to this embodiment, it is possible to correctly reflect the line breaks in the form and extract the lines by using the information indicating which column the user uses as the line break, the logical ruled line 531 and the virtual ruled line 532 .

<Other application examples>
Although the best mode for carrying out the present invention has been described above using examples, the present invention is by no means limited to such examples, and various modifications can be made without departing from the scope of the present invention. and substitutions can be added.

For example, in the present embodiment, the image forming apparatus 300 and the server apparatus 200 communicate with each other, and the server apparatus 200 recognizes the form. However, the image forming apparatus 300 may perform the form recognition alone. In the present embodiment, the terminal device 400 displays the recognition result confirmation screen 501 after the server device 200 recognizes the form, but the image forming device 300 may display the recognition result confirmation screen 501 .

Alternatively, instead of the image forming apparatus 300 , a scanner device connected to a PC may scan, and the PC may transmit the form image data to the server device 200 . In this case, the PC replaces the terminal device 400 .
Also, in the present embodiment, the item values of the product name are linked, but the user may be able to set which item values are linked. Also, item values of a plurality of items may be concatenated.

Also, the configuration examples in FIG. 9 and the like are divided according to the main functions in order to facilitate understanding of the processing by the server device 200 . The present invention is not limited by the division method or name of the unit of processing. The processing of the server device 200 can also be divided into more processing units according to the content of the processing. Also, one processing unit can be divided to include more processing.

Moreover, the devices described in the examples are only representative of one of several computing environments for implementing the embodiments disclosed herein. In some embodiments, server apparatus 200 includes multiple computing devices, such as a server cluster. Multiple computing devices are configured to communicate with each other over any type of communication link, including a network, shared memory, etc., to perform the processes disclosed herein.

Further, the server device 200 can be configured to share the disclosed processing steps, such as FIG. 11, in various combinations. For example, a process executed by a predetermined unit can be executed by multiple information processing devices included in server device 200 . Also, the server device 200 may be integrated into one server device, or may be divided into a plurality of devices.

Each function of the embodiments described above may be implemented by one or more processing circuits. Here, "processing circuitry" as used herein refers to processors programmed by software to perform the functions, such as processors implemented by electronic circuitry, or processors designed to perform the functions described above. It includes devices such as ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), FPGAs (Field Programmable Gate Arrays), and conventional circuit modules.

100 form recognition system 200 server device 400 terminal device

Japanese Patent Application Laid-Open No. 2021-140788

Claims (22)

An information processing system for providing a terminal device with a recognition result obtained by recognizing a character string from tabular image data having rows and columns,
From the recognition result, concatenate the item value of the first row that has a value in the preset column heading column and the item value of the second row that does not contain the preset exclusion term and is not blank. a form recognition unit;
a display screen generation unit that provides the terminal device with the item values of the first row and the second row that are linked by the form recognition unit;
An information processing system characterized by having: The item values of the first row and the second row that are concatenated are the item values in the column whose column heading is the item name,
2. The information processing system according to claim 1, wherein said exclusion term is a term not used as a product name. 3. The information processing system according to claim 2, wherein said exclusion term is consumption tax, subtotal, total, or slip total. 4. The information processing system according to claim 3, wherein the preset column headings are price or quantity and unit price. When either the upper row or the lower row adjacent to the second row is the first row, the form recognition unit selects the first row adjacent to the second row 5. The information processing system according to claim 4, wherein the item value of and the item value of the second row are concatenated. The row containing the excluded term is the third row, and the row without values in all columns is the fourth row,
When both the upper row and the lower row adjacent to the second row are the first row,
If the row above the first row that first appears in the tabular image data is the third row, the fourth row, or does not exist,
5. The information processing system according to claim 4, wherein the item value of said first row and the item value of said second row adjacent above said second row are concatenated. The row containing the excluded term is the third row, and the row without values in all columns is the fourth row,
When both the upper row and the lower row adjacent to the second row are the first row,
When the row above the first row that first appears in the tabular image data is the second row, the form recognition unit
5. The information processing system according to claim 4, wherein the item values of said first row and the item values of said second row adjacent below said second row are concatenated. An information processing system for providing a terminal device with a recognition result obtained by recognizing a character string from tabular image data having rows and columns,
set a first ruled line at the row delimiter, set a second ruled line at the top of the column heading column item value set in advance, and set the second ruled line based on the first ruled line and the second ruled line , a form recognition unit that determines line breaks;
a display screen generation unit that provides the terminal device with item values extracted based on the line breaks determined by the form recognition unit;
An information processing system characterized by having: 9. The information processing system according to claim 8, wherein the form recognition unit determines the second ruled line overlapping the first ruled line as the line break. the first ruled lines are determined to divide the image data at equal intervals;
10. The information processing system according to claim 8, wherein the form recognition unit draws the first ruled line for each specified number of skips. A form recognition method performed by an information processing system for providing a terminal device with recognition results obtained by recognizing a character string from image data in a tabular format having rows and columns,
The form recognition unit determines, from the recognition result, the item values of the first row that have values in the columns of the preset column headings, and the second rows that do not contain preset exclusion terms and are not blank rows. concatenating field values;
a step in which the display screen generation unit provides the terminal device with the item values of the first row and the second row linked by the form recognition unit;
A form recognition method characterized by comprising: The item values of the first row and the second row that are concatenated are the item values in the column whose column heading is the item name,
12. The form recognition method according to claim 11, wherein said excluded term is a term that is not used as a product name. 13. The form recognition method according to claim 12, wherein the exclusion terms are consumption tax, subtotal, total, or slip total. 14. The form recognition method according to claim 13, wherein the preset column headings are amounts or quantities and unit prices. When either the upper row or the lower row adjacent to the second row is the first row, the form recognition unit selects the first row adjacent to the second row 15. The form recognition method according to claim 14, further comprising concatenating the item value of the second row with the item value of the second row. The row containing the excluded term is the third row, and the row without values in all columns is the fourth row,
When both the upper row and the lower row adjacent to the second row are the first row,
If the row above the first row that first appears in the tabular image data is the third row, the fourth row, or does not exist,
15. The form recognition method according to claim 14, wherein the item value of said first line and the item value of said second line adjacent above said second line are concatenated. The row containing the excluded term is the third row, and the row without values in all columns is the fourth row,
When both the upper row and the lower row adjacent to the second row are the first row,
When the row above the first row that first appears in the tabular image data is the second row, the form recognition unit
15. The form recognition method according to claim 14, wherein the item values of said first line and the item values of said second line adjacent below said second line are concatenated. A form recognition method performed by an information processing system for providing a terminal device with recognition results obtained by recognizing a character string from image data in a tabular format having rows and columns,
set a first ruled line at the row delimiter, set a second ruled line at the top of the column heading column item value set in advance, and set the second ruled line based on the first ruled line and the second ruled line , a form recognition unit that determines line breaks;
a display screen generation unit that provides the terminal device with item values extracted based on the line breaks determined by the form recognition unit;
A form recognition method characterized by comprising: 19. The form recognition method according to claim 18, wherein said form recognition unit determines said second ruled line overlapping said first ruled line as said line break. the first ruled lines are determined to divide the image data at equal intervals;
20. The form recognition method according to claim 18, wherein the form recognition unit draws the first ruled line for each specified number of skips. An information processing system that provides terminal devices with recognition results obtained by recognizing character strings from tabular image data having rows and columns,
From the recognition result, concatenate the item value of the first row that has a value in the preset column heading column and the item value of the second row that does not contain the preset exclusion term and is not blank. a form recognition unit;
a display screen generation unit that provides the terminal device with the item values of the first row and the second row that are linked by the form recognition unit;
A program to function as A form recognition system comprising: an information processing system for providing a terminal device with recognition results obtained by recognizing character strings from tabular image data having rows and columns; and a terminal device for displaying the recognition results,
The information processing system is
From the recognition result, concatenate the item value of the first row that has a value in the preset column heading column and the item value of the second row that does not contain the preset exclusion term and is not blank. a form recognition unit;
a display screen generation unit that provides the terminal device with item values of the first row and the second row linked by the form recognition unit;
The terminal device
A form recognition system comprising a display control section for displaying the recognition result.

Images