JP7110723B2 - Data conversion device, image processing device and program - Google Patents

Description

The present invention is capable of extracting characters and calculation formulas from image data including a plurality of characters to be calculated and characters indicating calculation results when these characters are calculated by a predetermined calculation formula. The present invention relates to a data conversion device such as an image processing device, an image processing device having the data conversion device, and a program.

In recent years, by performing image analysis processing on image data, for example, OCR (optical character reader), ICR (intelligent character recognition), or combining these with object separation technology, printed characters In addition, it has become possible to convert handwritten characters into text data or the like with high accuracy. Along with this, convenience has been improved by making converted data into a format that is easier to edit.

One such attempt is to convert printed or handwritten tables into data in spreadsheet form.

To give a specific example, as shown in FIG. 11, if the vertical direction of the table is the column and the horizontal direction is the row, the squares (cells) of the 2nd to 5th rows of column A are "1Q", "2Q", and "3Q". ” and “4Q” are written respectively, the character string “total” is written in the cell of column A, row 6, the individual’s name is written in each cell of columns B to D, row 1, and E It is assumed that there is image data for a table in which a character string "total" is written in a cell in the first row of the column. Also, in each cell of the 2nd to 5th rows of columns B to D, a numerical value for each individual is written.

Such image data is converted into spreadsheet format data. Specifically, the calculation formula is embedded so that the total value of 1Q to 4Q for each individual is automatically calculated and entered in each cell of columns B to D, row 6, and each of columns E, rows 2 to 5 Embed the calculation formula so that the total value for each 1Q to 4Q is automatically calculated and filled in the cell, and embed the formula so that the total value of all is automatically calculated and filled in the cell of column E, row 6. . Therefore, for example, the data in cell B6, row 6, column B, indicated by the dashed frame, must be converted to the embedded formula "=sum(B2:B5)" instead of the number "18". .

However, at present, the accuracy of image analysis processing (especially offline handwritten character recognition processing) is still insufficient, and the following erroneous conversion may occur.

For example, when reading a handwritten table as shown in the diagram on the left side of FIG. 3”, the total value calculated by the formula is 13. In other words, the total value is 18 in the original table, but when converted to spreadsheet format data, if the numerical values to be calculated are erroneously converted, the calculation result of the formula will also be erroneously converted.

Patent Document 1 proposes a table data conversion method that derives a calculation formula used for aggregation from table data in which items and numerical values are described and outputs table data in which the calculation formula is embedded. , when a specific keyword such as "total" or "average" is used as an item name in table data with cells arranged in a grid, the cell (keyword Cells that contain numerical values in the same "row" (horizontal direction) or "column" (vertical direction) as the corresponding cell) are changed to cells that store the aggregation results corresponding to the "total" keyword (aggregation results storage cell) and replace the value of the aggregate result storage cell with a formula that references other cells, or if the result of the formula does not match the value originally registered in that cell, It is described that the user is warned by notifying that the calculation results are inconsistent.

JP 2017-187931 A

However, in the method described in Patent Document 1, when the calculation result by the calculation formula and the numerical value originally registered in the cell do not match, the user is warned that the calculation result does not match. However, erroneous conversions are corrected manually by users, and erroneous conversions cannot be corrected automatically.

The present invention has been made in view of such a technical background, and includes a plurality of characters to be operated on, and a character indicating the operation result when the plurality of characters are operated by a predetermined formula. To provide a data conversion device, an image processing device, and a program capable of automatically correcting erroneous conversions when extracting characters and calculation formulas from image data containing data and converting them into data in spreadsheet format, etc. aim.

The above object is achieved by the following means.
(1) receiving means for receiving, as input data, image data including a plurality of characters to be operated on and characters indicating a result of calculating the plurality of characters by a predetermined calculation formula; extracting means for extracting the plurality of characters to be operated, the calculation formula, and characters indicating the calculation result from the input data obtained; and extracting the input data from the plurality of characters to be operated and the calculation formula and a conversion means for converting into editable data using characters indicating the calculation result; a calculation result when a plurality of characters to be calculated extracted from the extraction means are calculated by the extracted calculation formula; determining means for determining whether or not at least one of the extracted characters to be operated on, the calculation formula, and the characters indicating the operation result have been erroneously converted by comparing them with the characters indicating the operation result; correction means for correcting the characters or formulas to correct characters or calculation formulas when the determination means determines that at least one of them has been erroneously converted, and the extraction of the characters by the extraction means is performed by character recognition processing. , the extracting means extracts a character with the highest character recognition rate, and if the determining means determines that the extracted character to be operated has been erroneously converted, the correcting means The character is corrected to a character with the next highest character recognition rate, and the determination means recalculates according to the formula after the correction of the character, thereby determining whether or not the corrected character is erroneously converted again. A data conversion device characterized by determining:
(2) receiving means for receiving, as input data, image data including a plurality of characters to be operated on and characters indicating a result of calculating the plurality of characters by a predetermined formula; extracting means for extracting the plurality of characters to be operated, the calculation formula, and characters indicating the calculation result from the input data obtained; and extracting the input data from the plurality of characters to be operated and the calculation formula and a conversion means for converting into editable data using characters indicating the calculation result; a calculation result when a plurality of characters to be calculated extracted from the extraction means are calculated by the extracted calculation formula; determining means for determining whether or not at least one of the extracted characters to be operated on, the calculation formula, and the characters indicating the operation result have been erroneously converted by comparing them with the characters indicating the operation result; correction means for correcting to a proper character or calculation formula when at least one of them is determined to be erroneously converted by the determination means, wherein the calculation formula extracted by the extraction means includes rounding. , the determining means determines whether rounding is to be performed by rounding down or rounding up, and the correcting means determines rounding of the formula based on the determination result as to whether rounding is to be performed by rounding down or rounding up. A data conversion device characterized by correcting.
(3) receiving means for receiving, as input data, image data including a plurality of characters to be computed and characters indicating a computation result when the plurality of characters are computed by a predetermined formula; extracting means for extracting the plurality of characters to be operated, the calculation formula, and characters indicating the calculation result from the input data obtained; and extracting the input data from the plurality of characters to be operated and the calculation formula and a conversion means for converting into editable data using characters indicating the calculation result; a calculation result when a plurality of characters to be calculated extracted from the extraction means are calculated by the extracted calculation formula; determining means for determining whether or not at least one of the extracted characters to be operated on, the calculation formula, and the characters indicating the operation result have been erroneously converted by comparing them with the characters indicating the operation result; correcting means for correcting to a proper character or calculation formula when at least one of the characters is determined to be erroneously converted by the determining means; If the extraction is impossible, the correction means corrects the part of the characters to proper characters by calculating backward from the other characters to be operated, the calculation formula, and the characters indicating the calculation result extracted by the extraction means. A data conversion device characterized by:
(4) The input data is image data in a spreadsheet format having a character string indicating a calculation formula, and the extracting means extracts the calculation formula based on the character string in the input data, and extracts the calculation target and the conversion means converts into spreadsheet data using the extracted plurality of characters to be operated, the calculation formula, and the characters indicating the calculation result. 4. The data conversion device according to any one of 3.
(5) The input data is image data having operation symbols, and the extracting means extracts the calculation formula based on the operation symbols, and based on the area to be operated and the operation result area indicated by the operation symbols, 4. The data conversion device according to any one of the preceding items 1 to 3, which extracts the plurality of characters to be operated and the characters indicating the operation result.
(6) The data conversion device according to the preceding item 1, wherein the result of recalculation is recorded on a separate sheet.
(7) The data conversion device according to the preceding item 6, wherein the result of recalculation is recorded on a separate sheet in a manner in which the characters after correction can be identified.
(8) An image processing apparatus comprising the data conversion apparatus according to any one of items 1 to 7 above.
(9) a receiving step of receiving, as input data, image data including a plurality of characters to be computed and characters representing a computation result when the plurality of characters are computed by a predetermined formula; an extracting step of extracting the plurality of characters to be operated on, the calculation formula, and characters representing the calculation result from the input data obtained from the input data; and a conversion step of converting into editable data using characters indicating the calculation result, a calculation result when the plurality of characters to be calculated extracted from the extraction step are calculated with the extracted calculation formula, and the extraction a determination step of determining whether or not at least one of the extracted characters to be operated on, the calculation formula, and the character indicating the operation result has been erroneously converted by comparing with the character indicating the operation result; a correction step of correcting to a proper character or calculation formula when at least one of them is determined to be erroneously converted by the determination step, and causing the computer of the data conversion device to execute extraction of the character by the extraction step. is performed by character recognition processing, the extracting step extracts a character with the highest character recognition rate, and the determining step determines that the extracted character to be operated has been erroneously converted, the correcting In the step, the erroneously converted character is corrected to a character with the next highest character recognition rate, and in the determination step, after the character is corrected, recalculation is performed according to the calculation formula, thereby making the corrected character erroneous again. A program for causing the computer to execute processing for determining whether or not conversion has been performed.
(10) a receiving step of receiving, as input data, image data including a plurality of characters to be computed and characters indicating a computation result when the plurality of characters are computed by a predetermined formula; an extracting step of extracting the plurality of characters to be operated on, the calculation formula, and characters representing the calculation result from the input data obtained from the input data; and a conversion step of converting into editable data using characters indicating the calculation result, a calculation result when the plurality of characters to be calculated extracted from the extraction step are calculated with the extracted calculation formula, and the extraction a determination step of determining whether or not at least one of the extracted characters to be operated on, the calculation formula, and the character indicating the operation result has been erroneously converted by comparing with the character indicating the operation result; and a correction step of correcting to a proper character or calculation formula if at least one of them is determined to be erroneously converted by the determination step, and causing the computer of the data conversion device to execute the calculation extracted by the extraction step. When the formula includes rounding, the determining step determines whether the rounding is rounding down or rounding up, and the modifying step determines rounding of the formula based on the determination result of rounding down or rounding up. A program for causing the computer to execute a process of correcting the calculation formula by
(11) a receiving step of receiving, as input data, image data including a plurality of characters to be computed and characters indicating a computation result when the plurality of characters are computed by a predetermined formula; an extracting step of extracting the plurality of characters to be operated on, the calculation formula, and characters representing the calculation result from the input data obtained from the input data; and a conversion step of converting into editable data using characters indicating the calculation result, a calculation result when the plurality of characters to be calculated extracted from the extraction step are calculated with the extracted calculation formula, and the extraction a determination step of determining whether or not at least one of the extracted characters to be operated on, the calculation formula, and the character indicating the operation result has been erroneously converted by comparing with the character indicating the operation result; If at least one of the characters is determined to be erroneously converted by the determination step, the computer of the data conversion device is caused to execute a correction step of correcting it to a proper character or a calculation formula, and one of the plurality of characters to be calculated. If the part cannot be extracted by the extracting step, the correcting step back-calculates from other characters of the operation target extracted by the extracting step, the calculation formula, and the character indicating the calculation result to obtain the part A program for causing the computer to execute a process of correcting characters to proper characters.

According to the invention described in the preceding item (1), image data including a plurality of characters to be operated and characters indicating the operation result when the plurality of characters are operated by a predetermined formula is accepted as input data. , a plurality of characters to be calculated, a calculation formula, and characters indicating the calculation result are extracted from the received input data and converted into editable data. On the other hand, each of the extracted operation target is compared by comparing the operation result when the extracted operation target characters are operated by the extracted calculation expression and the extracted operation result character. It is determined whether or not at least one of characters, calculation formulas, and characters indicating calculation results has been erroneously converted, and if it is determined that at least one of them has been erroneously converted, it is corrected to a proper character or calculation formula. be.

Therefore, even if an erroneous conversion occurs when extracting these characters and calculation formulas from the original image data and converting them into editable data, the erroneously converted characters and calculation formulas is automatically corrected, saving the user the trouble of manually correcting misconverted characters and formulas. Character extraction is performed by character recognition processing, and the character with the highest character recognition rate is extracted. character is corrected to the character with the next highest character recognition rate. Then, after the correction, recalculation is performed using the calculation formula, and it is determined whether or not the corrected character has been erroneously converted again.
According to the invention described in the preceding item (2), when the calculation formula includes rounding, rounding down or rounding up is determined and then the calculation formula is corrected by determining the rounding. It can be modified to have a formula.
According to the invention described in the preceding item (3), when extraction of a part of a plurality of characters to be operated is impossible, backward calculation is performed from other extracted characters to be operated, calculation formulas, and characters indicating operation results. Therefore, since some characters that cannot be extracted are corrected to proper characters, erroneous conversion can be reliably prevented.

According to the invention described in the preceding item ( 4 ), the calculation formula is extracted based on the character string in the image data in the spreadsheet format having the character string indicating the calculation formula, and the plurality of characters to be calculated and the calculation result are extracted. is extracted, and converted into spreadsheet data using the extracted plurality of characters to be operated, the calculation formula, and the characters indicating the operation result.

According to the invention described in the preceding item ( 5 ), the calculation formula is extracted based on the calculation symbol of the image data having the calculation symbol, and based on the calculated area and the calculation result area indicated by the calculation symbol, the plurality of calculation targets and the character indicating the operation result are extracted.

According to the invention described in the preceding item ( 6 ), the results of recalculation are recorded on a separate sheet, so the results before and after recalculation can be easily confirmed.

According to the invention described in the preceding item ( 7 ), the result of recalculation is recorded on a separate sheet in a manner in which the corrected characters can be identified, so the corrected characters can be easily recognized.

According to the invention described in the preceding paragraph ( 8 ), erroneous conversion occurs when extracting characters and calculation formulas from the original image data and converting them into editable data. can also be an image processing device capable of automatically correcting erroneously converted characters and calculation formulas.

According to the invention described in the preceding paragraph ( 9 ), when extracting these characters and calculation formulas from the characters and calculation formulas contained in the original image data and converting them into editable data, there is an error in the characters to be operated. If a conversion occurs, the computer of the data processing apparatus can be caused to automatically correct the erroneously converted characters .

1 is a block diagram of an information processing device such as a personal computer as an example of a data processing device; FIG. FIG. 3 is a block diagram of an image forming apparatus as another example of a data processing apparatus; 2 is a block diagram showing a configuration of a function of extracting characters and calculation formulas from image data and converting them into editable data, realized by a CPU of an information processing apparatus and a CPU of an image forming apparatus; FIG. FIG. 4 is a diagram for explaining a method of correcting erroneous conversion according to one embodiment of the present invention; It is a table|surface which shows the conversion precision of each numerical value used for said correction method. FIG. 10 is a diagram showing a state after incorrect conversion correction; FIG. 5 is a diagram for explaining a method of correcting erroneous conversion according to another embodiment of the present invention; FIG. 11 is a diagram for explaining a method of correcting erroneous conversion according to still another embodiment of the present invention; FIG. 11 is a diagram for explaining a method of correcting erroneous conversion according to still another embodiment of the present invention; 4 is a flow chart showing an erroneous conversion correcting operation of the data processing device in the first embodiment. It is a figure for demonstrating the conventional problem.

Embodiments of the present invention will be described below with reference to the drawings.

A data processing apparatus according to an embodiment of the present invention may be configured by an information processing apparatus 1 such as a personal computer (hereinafter referred to as PC) as shown in FIG. image forming apparatus 2.

FIG. 1 is a block diagram of an information processing device 1 constituting a data processing device.

The information processing apparatus 1 includes a control unit 11, a storage device 12, an input interface (input I/F) 13, an output interface (output I/F) 14, and the like.

The control unit 11 has a CPU 111 and a RAM 112 .

The CPU 111 comprehensively controls the entire information processing apparatus 1 by operating according to an operation program stored in the storage device 12 or the like. In particular, in this embodiment, image data including a plurality of characters to be operated on and characters indicating the operation result when these plurality of characters are operated by a predetermined formula is accepted as input data, and the accepted input data By analyzing the image and extracting characters and calculation formulas, the input data is converted into editable data using the extracted characters and calculation formulas, etc. Details will be described later. do.

The RAM 112 is a memory that serves as a work area when the CPU 111 operates according to the operation program.

The storage device 12 is, for example, a hard disk device (HDD) or the like, and stores an operating program for the CPU 111, various applications, and other data.

The input interface 13 functions as receiving means for receiving various data transmitted from the external device 3 , and the output interface 14 functions as transmitting means for transmitting various data from the data processing device 1 to the external device 3 .

FIG. 2 is a block diagram of the image forming apparatus 2 that constitutes the data processing apparatus.

The image forming apparatus 2 includes a control unit 21, a storage device 22, an operation panel 23, an image input unit 24, an image printing unit 25, an input interface (input I/F) 26, and an output interface (output I/F). F) 27 and the like.

The control unit 21 has a CPU 211 , a RAM 212 and a ROM 213 .

The CPU 211 comprehensively controls the entire image forming apparatus 2 by operating according to the operation programs stored in the ROM 213, the storage device 22, etc., and executes the printing function and the like. Furthermore, in this embodiment, the CPU 211 accepts as input data image data including a plurality of characters to be operated on and characters indicating the operation result when the plurality of characters are operated by a predetermined formula. By analyzing the image of the input data and extracting characters and calculation formulas, the input data is converted into editable data using the extracted characters and calculation formulas. will be described later.

A RAM 212 is a memory that serves as a work area when the CPU 211 operates according to an operation program, and a ROM 213 is a memory that stores an operation program of the CPU 211 and the like.

The storage device 22 is, for example, a hard disk device (HDD) or the like, and stores various applications and other data.

The operation panel 23 is used by the user to operate the image forming apparatus 2 and to notify the user of messages, and has a display section with a touch panel, a start key, a stop key, and the like.

The image input unit 24 has a scanner device, and reads an image of a document and converts it into image data. It prints print data on paper.

The input interface 26 functions as receiving means for receiving print data and other data transmitted from the external device 3 , and the output interface 27 functions as transmitting means for transmitting various data from the data processing device 1 to the external device 3 . .

Note that the external device 3 is an information processing device other than the information processing device 1 and the image forming device 3, another image forming device, or the like.

FIG. 3 is a block diagram showing a configuration of functions implemented by the CPU 111 of the information processing apparatus 1 and the CPU 211 of the image forming apparatus 2 to extract characters and calculation formulas from image data and convert them into editable data.

The CPU 111 and CPU 211 functionally include an image processing unit 41 , an item part/numerical part separation unit 42 , a calculation candidate cell identification unit 43 , a calculation target group extraction unit 44 , and a conversion result processing unit 45 . .

The image processing unit 41 performs image processing on image data D1 in tabular format or the like received as input data, separates objects such as ruled lines and characters, and extracts characters by OCR processing or the like to generate tabular data D2. The item part/numerical part separation part 42 separates table item parts such as "total", "average", "Tanaka", "1Q", and "2Q" and numerical parts such as "4" and "2" based on the extracted characters. separate.

Image data as input data may be input from the external device 3 to the information processing device 1 or the image forming device 2, or may be acquired by reading an image with the image input unit 24 of the image forming device 2. good.

The calculation candidate cell specifying unit 43 extracts and specifies a calculation formula from character strings such as “total” and “average” written in the item part, and also identifies the area to be operated indicated by the character string, that is, the cell (mass cell) where the numerical value is calculated. ) and the calculation result area, that is, the cell in which the calculation result is written. The calculation target group extracting unit 44 extracts cells constituting a calculation target group from among a plurality of cells in each row (horizontal direction) and each column (vertical direction).

The conversion result processing unit 45 writes the extracted items and numerical values in each cell, embeds the extracted calculation formula in the cell indicating the calculation result, converts the data into editable spreadsheet format data, and converts the data into editable spreadsheet format data. Calculates the numerical value written in the cell according to the formula. Then, by comparing the operation result with the character (numerical value) indicating the operation result extracted from the original image data, at least one of the extracted characters to be operated, the calculation formula, and the character indicating the operation result. is erroneously converted, and if it is determined that at least one of them is erroneously converted, it is corrected to an appropriate character or calculation formula, and table data D3 which can be easily re-edited is obtained. Specific examples are described below.
[First Embodiment]
This example shows the case where the numbers "8" and "3" are erroneously converted.

The figure on the left side of FIG. 4 is an image of the original handwritten spreadsheet format, and for Mr. Tanaka, the numerical values corresponding to 1Q, 2Q, 3Q, and 4Q are 4, 2, 4, and 8. The text in the cell is "Total". Therefore, "18", which is the sum of the numerical values of the cells in the operation area, is written in the cell in the operation result area.

The image shown on the left side of FIG. 4 is input as image data, characters and calculation formulas are extracted by image analysis processing including character recognition processing, and the state converted into spreadsheet data is shown on the right side of the same drawing. . The extracted characters 1Q, 2Q, 3Q, and 4Q are described in cells A2 to A5 of columns 2 to 5, respectively, and the extracted characters of "total" are described in cell A6 of column A, row 6. The extracted character "Tanaka" is written in cell B1 of column B, row 1, and the extracted characters 4, 2, 4, and 3 are written in cells B2 to B5 of column B, rows 2 to 5. Have been described. From the description of the character string "total" and each numerical value, etc., the area to be operated is cells B2 to B5 in columns 2 to 5 of B, and an area indicating the operation result in which the extracted formula is embedded (operation result area) becomes cell B6 in column B, row 6.

Also, the calculation formula is "=sum(B2:B5)", which adds the numerical values of cells B2 to B5, which are areas to be operated on.

Further, as can be understood from a comparison of the left and right diagrams in FIG. 4, the numerical value "8" of 4Q in the original image data is erroneously converted to "3" in cell B5. As a result, the total value obtained by the calculation formula is 4+2+4+3=13, which is different from the total value 18 of the original image data extracted by the image analysis processing.

Therefore, the conversion result processing unit 45 shown in FIG. 3 determines that there is an erroneous conversion, reconfirms the conversion accuracy of each numerical value, which is the character recognition rate of the application that executes the character recognition processing, and then determines the possibility. Recalculate using a higher number of

An example of the conversion accuracy of each numerical value is shown in the table of FIG. As shown in the figure, each numerical value of 4, 2, 4, 3, and 18 indicates high conversion accuracy as the first candidate, and these numerical values are extracted first, but there are conversion errors as described above. If so, the one with the highest possibility, in other words, the numerical value with the highest character recognition rate is selected from the second candidates. In the example of FIG. 5, the hatched 89% "8" corresponds, and the number "3" first extracted and written in cell B5 is corrected to "8" and embedded in cell B6 in the calculation result area. Recalculate using the formula.

As a result of the recalculation, as shown in FIG. 6, the value converted by the formula "=sum(B2:B5)" is 18, which matches the numerical value 18 converted from the character string of the original image. Therefore, the numerical value of cell B5 is determined as the modified value "8".

In this way, even if an erroneous conversion occurs when extracting these characters and calculation formulas from the original image data and converting them into editable data, the erroneously converted characters and calculation formulas Since calculation formulas are automatically corrected, the user does not need to manually correct erroneously converted characters or calculation formulas, saving time and effort.
[Second embodiment]
This embodiment shows a case where the calculation formula includes rounding by division because the calculation formula is a formula for obtaining an "average" rather than a "sum".

The figure on the left side of FIG. 7 is an image of the original handwritten spreadsheet format, and the numerical values corresponding to Mr. Tanaka's 1Q, 2Q, 3Q, and 4Q are 4, 2, 4, and 8. The text in the cell is "Average". Also, "4" obtained by averaging the numerical values of the cells in the calculation area and rounding off the fractions is written in the cell in the calculation result area.

The image shown on the left side of FIG. 7 is input as image data, and the image analysis processing including character recognition processing is performed to extract characters and calculation formulas. show. The extracted characters 1Q, 2Q, 3Q, and 4Q are described in each cell A2 to A5 of columns 2 to 5 of column A, and the extracted characters of "average" are described in cell A6 of column A, row 6. The extracted character "Tanaka" is written in cell B1 of column B, row 1, and the extracted characters 4, 2, 4, and 8 are written in cells B2 to B5 of column B, rows 2 to 5. Have been described. From the description of the character string "average" and each numerical value, etc., the area to be operated is cells B2 to B5 in columns 2 to 5 of B, and the calculation result area in which the extracted formula is embedded is the cell in row 6 of column B. becomes B6.

The calculation formula is "=AVERAGE(Bsum(B2:B5)" for obtaining the average value of the numerical values of the cells B2 to B5, which are the areas to be calculated, and the result of calculation according to this formula is 4.5. is rounded to determine whether it is rounding down or rounding up, it becomes 5 as shown in the right-hand diagram of FIG.

On the other hand, the average value of the original image data extracted and converted by the image analysis process is 4, and both are different.

Therefore, the conversion result processing unit 45 shown in FIG. 3 determines that there is an erroneous conversion in the rounding of the calculation formula, determines that the rounding in the calculation formula is rounding down rather than rounding up or rounding up, and reflects it in the calculation formula.

In this way, when the calculation formula includes rounding, rounding down or rounding up is confirmed, and then the calculation formula is corrected by determining the rounding.
[Third Embodiment]
This embodiment shows a case in which even if some of the numeric values to be converted cannot be converted, they are converted by calculating backward from the other numeric values and the content of the table.

The figure on the left side of FIG. 8 is an image of the original handwritten spreadsheet format, and for Mr. Tanaka, the numerical values corresponding to 1Q, 3Q, and 4Q are 4, 4, and 8, and the characters in the cell below 4Q are Although the column is "Total", the numerical value corresponding to 2Q is unknown.

In addition, "18", which is the sum of the numerical values of the cells in the calculation area, is written in the cell in the calculation result area.

The image shown on the left side of FIG. 8 is input as image data, and the image analysis processing including character recognition processing is performed to extract characters and calculation formulas. show. The extracted characters 1Q, 2Q, 3Q, and 4Q are described in cells A2 to A5 of columns 2 to 5, respectively, and the extracted characters of "total" are described in cell A6 of column A, row 6. , and the extracted character "Tanaka" is written in cell B1 of column B, row 1, and the extracted characters 4, 4, and 8 are written in cells B2, B4, and B5 of columns B, 2, 4, and 5, respectively. characters are written. However, cell B3 in column B, row 3 cannot be read and cannot be converted. From the description of the character string "total" and each numerical value, the operation area is cells B2 to B5, and the operation result area in which the extracted formula is embedded is cell B6 in column B, row 6.

The formula is "=sum(B2:B5)", which adds the values of cells B2 to B5, which are the areas to be operated on. cannot perform operations with

Therefore, the conversion result processing unit 45 shown in FIG. 3 determines that there is an erroneous conversion, and uses numerical characters 4, 4, and 8 to be calculated extracted from the original image data and a numerical value 18 indicating the calculation result. to calculate backwards. That is, another numerical value is subtracted from the numerical value 18 indicating the calculation result (18-4-4-8=2) to obtain the numerical value 2.

Then, the conversion result processing unit 45 applies 2 as the numerical value of cell B3 in column B, row 3, and recalculates using the formula. As a result of the recalculation, as shown in FIG. 8, the value converted by the formula "=sum(B2:B5)" is 18, which matches the numerical value 18 of the original image.

In this way, when it is impossible to extract some of the multiple characters to be operated on, the non-extractable part is calculated backward from the other characters of the extracted operation target, the calculation formula, and the characters indicating the operation result. is corrected to a proper character, erroneous conversion can be reliably prevented.
[Fourth embodiment]
This embodiment shows the case where the original data is not in tabular form but the calculation formula is written as it is.

FIG. 9 is an original handwritten image, in which "2+4+8=14" is written. This image is input as image data, image analysis processing including character recognition processing is performed, and characters and calculation formulas are extracted. Specifically, based on the operation symbols “+” and “=” on the left side, the left side is determined as the region to be operated, and the right side is determined as the operation result region. Then, a calculation formula for adding the numerical value on the left side is extracted from the operation symbol "+".

Here, if "8" on the left side is erroneously converted to "3", when the extracted numerical values on the left side are added by the extracted calculation formula, the total value becomes 9, which is different from the numerical value 14 of the extracted operation result. .

Therefore, the conversion result processing unit 45 shown in FIG. 3 determines that there is an erroneous conversion. (see FIG. 5), and then recalculate using the next most likely value.

Specifically, the one with the highest possibility, in other words, the numerical value with the highest character recognition rate is selected from the second candidates. In the example of FIG. 5, "8" corresponds, and the numerical value "3" corresponding to the first extracted "3" is corrected to "8" and recalculated by the extracted formula.

As a result of the recalculation, the value converted by the formula becomes 14, which matches the numerical value 14 converted from the character string of the original image. Therefore, the correction value "8" is determined.

As described above, according to this embodiment, a calculation formula can be extracted based on the operation symbol of the image data, and a plurality of characters to be operated and the operation result can be extracted based on the operation target area and the operation result area indicated by the operation symbol. You can extract characters that indicate .

FIG. 10 is a flow chart showing the operation of correcting erroneous conversion of the data processors 1 and 2 in the first embodiment described with reference to FIGS. This operation is executed by the CPUs 111 and 211 of the data processors 1 and 2 operating according to operation programs stored in the storage devices 12 and 22 or the like.

After receiving input of image data in step S01, image analysis processing is executed in step S02, and it is determined whether or not characters (keywords) such as "total", "average", "+" and "=" are extracted in step S03. do. If not extracted (NO in step S03), conversion is not possible and the process ends. If extracted (YES in step S03), the process proceeds to step S04.

In step S04, after extracting a region to be operated as a target of the keyword and a calculation result region indicating the calculation result, in step S05, the calculation result region is converted into a spreadsheet. That is, the calculation formula is extracted and embedded in the calculation result area.

Next, in step S06, the variable n is set to 0, and T is set to the number of operands+1. For example, if the number of operands is 4, set T=5.

Next, in step S07, the calculation result of the calculation formula is compared with the conversion result of the character string described in the calculation result area of the original image data. If they are the same (“=” in step S07), then the process is finished since all the values have been properly converted. If not the same (“≠” in step S07), it is determined whether n=T in step S08.

If n=T (NO in step S08), in step S09, the second candidate with the n-th highest conversion accuracy is obtained from the list of conversion accuracies shown in FIG. , in step S10, recalculation is performed by the formula using the replaced numerical value, and the process proceeds to step S11. The recalculation results are preferably recorded on a sheet separate from the sheet showing the conversion results so that the user can easily check them. It is also desirable that the results of the recalculation be recorded in a manner that allows identification of the revised figures.

In step S11, after setting n to n+1, the process returns to step S07.

In step S07, the calculation result of the calculation formula and the conversion result of the character string are compared again. Such processing is repeated until it is determined that n=T in step S08. If n=T is determined in step S08 (YES in step S08), proper conversion cannot be performed, so in step S12 the user is notified by displaying a warning screen.

1 Information processing device (data processing device)
2 Image forming device (data processing device)
3 external device 11, 21 control unit 111, 211 CPU
41 image processing unit 42 item part/numerical part separation unit 43 calculation candidate cell identification unit 44 calculation target group extraction unit 45 conversion result processing unit

Claims (11)

receiving means for receiving, as input data, image data including a plurality of characters to be calculated and characters indicating a calculation result when the plurality of characters are calculated by a predetermined formula;
extracting means for extracting the plurality of characters to be calculated, the calculation formula, and characters indicating the calculation result from the input data received by the receiving means;
conversion means for converting the input data into editable data using characters representing a plurality of characters to be operated, calculation formulas, and calculation results extracted by the extraction means;
A plurality of characters to be operated extracted by the extracting means are compared with the operation result when the extracted calculation formula is operated, and the character indicating the extracted operation result is compared to obtain the extracted operation target. Determination means for determining whether or not at least one of the characters, the calculation formula, and the characters indicating the calculation result has been erroneously converted;
Correction means for correcting to a proper character or calculation formula when the determination means determines that at least one of them has been erroneously converted;
with
The extraction of the characters by the extraction means is performed by character recognition processing, and the extraction means extracts characters with the highest character recognition rate,
When the determination means determines that the extracted character to be operated has been erroneously converted, the correction means corrects the erroneously converted character to a character with the next highest character recognition rate,
The data conversion device , wherein the determination means determines whether or not the corrected character is erroneously converted again by performing recalculation according to the calculation formula after the correction of the character . receiving means for receiving, as input data, image data including a plurality of characters to be calculated and characters indicating a calculation result when the plurality of characters are calculated by a predetermined formula;
extracting means for extracting the plurality of characters to be calculated, the calculation formula, and characters indicating the calculation result from the input data received by the receiving means;
conversion means for converting the input data into editable data using characters representing a plurality of characters to be operated, calculation formulas, and calculation results extracted by the extraction means;
A plurality of characters to be operated extracted by the extracting means are compared with the operation result when the extracted calculation formula is operated, and the character indicating the extracted operation result is compared to obtain the extracted operation target. Determination means for determining whether or not at least one of the characters, the calculation formula, and the characters indicating the calculation result has been erroneously converted;
Correction means for correcting to a proper character or calculation formula when the determination means determines that at least one of them has been erroneously converted;
with
When the calculation formula extracted by the extraction means includes rounding, the determination means determines whether the rounding is rounding down or rounding up,
The data conversion device, wherein the correction means corrects the calculation formula by determining rounding of the calculation formula based on a determination result as to whether the rounding is rounding down or rounding up. receiving means for receiving, as input data, image data including a plurality of characters to be calculated and characters indicating a calculation result when the plurality of characters are calculated by a predetermined formula;
extracting means for extracting the plurality of characters to be calculated, the calculation formula, and characters indicating the calculation result from the input data received by the receiving means;
conversion means for converting the input data into editable data using characters representing a plurality of characters to be operated, calculation formulas, and calculation results extracted by the extraction means;
A plurality of characters to be operated extracted by the extracting means are compared with the operation result when the extracted calculation formula is operated, and the character indicating the extracted operation result is compared to obtain the extracted operation target. Determination means for determining whether or not at least one of the characters, the calculation formula, and the characters indicating the calculation result has been erroneously converted;
Correction means for correcting to a proper character or calculation formula when the determination means determines that at least one of them has been erroneously converted;
with
When extraction by the extracting means is impossible for some of the plurality of characters to be operated, the correction means extracts from the other characters to be operated, calculation formulas, and characters indicating the operation result extracted by the extraction means 1. A data conversion device for correcting said part of characters to proper characters by calculating backward. The input data is image data in a spreadsheet format having a character string indicating a calculation formula,
The extraction means extracts the calculation formula based on the character string in the input data, and extracts the plurality of characters to be operated and characters indicating the operation result,
4. The data conversion device according to any one of claims 1 to 3, wherein said conversion means converts data into spreadsheet data using a plurality of extracted characters to be operated, calculation formulas, and characters indicating calculation results. The input data is image data having operation symbols,
The extracting means extracts the calculation formula based on the operation symbol, and extracts the plurality of characters to be operated and characters indicating the operation result based on the operation target area and the operation result area indicated by the operation symbol. The data conversion device according to any one of claims 1 to 3 . 2. The data conversion device according to claim 1 , wherein the recalculation result is recorded on a separate sheet. 7. The data conversion apparatus according to claim 6 , wherein the recalculation result is recorded on a separate sheet in a manner that allows identification of the characters after correction. An image processing device comprising the data conversion device according to any one of claims 1 to 7 . a receiving step of receiving, as input data, image data including a plurality of characters to be calculated and characters indicating a calculation result when the plurality of characters are calculated by a predetermined formula;
an extracting step of extracting the plurality of characters to be calculated, the calculation formula, and characters indicating the calculation result from the input data received by the receiving step;
a conversion step of converting the input data into editable data using a plurality of characters extracted by the extraction step, a calculation formula, and characters indicating a calculation result;
By comparing the operation result when a plurality of characters of the operation object extracted in the extraction step are operated by the extracted calculation formula and the character indicating the extracted operation result, the extracted operation object a determination step of determining whether or not at least one of characters indicating each character, calculation formula, and calculation result has been erroneously converted;
a correction step of correcting to a proper character or calculation formula when it is determined that at least one of them is erroneously converted by the determination step;
is executed by the computer of the data conversion device,
The extraction of the characters by the extraction step is performed by character recognition processing, and in the extraction step, the character with the highest character recognition rate is extracted,
If the determination step determines that the extracted character to be operated on has been erroneously converted, the correction step corrects the erroneously converted character to a character with the next highest character recognition rate,
A program for causing the computer to execute, in the determination step, a process of determining whether or not the corrected character is erroneously converted again by performing recalculation according to the calculation formula after the correction of the character. a receiving step of receiving, as input data, image data including a plurality of characters to be calculated and characters indicating a calculation result when the plurality of characters are calculated by a predetermined formula;
an extracting step of extracting the plurality of characters to be calculated, the calculation formula, and characters indicating the calculation result from the input data received by the receiving step;
a conversion step of converting the input data into editable data using a plurality of characters extracted by the extraction step, a calculation formula, and characters indicating a calculation result;
By comparing the operation result when a plurality of characters of the operation object extracted in the extraction step are operated by the extracted calculation formula and the character indicating the extracted operation result, the extracted operation object a determination step of determining whether or not at least one of characters indicating each character, calculation formula, and calculation result has been erroneously converted;
a correction step of correcting to a proper character or calculation formula when it is determined that at least one of them is erroneously converted by the determination step;
is executed by the computer of the data conversion device,
If the calculation formula extracted by the extraction step includes rounding, the determining step determines whether the rounding is rounding down or rounding up,
A program for causing the computer to execute a process of correcting a calculation formula in the correction step, by determining rounding of the calculation formula based on a determination result as to whether the rounding is rounding down or rounding up. a receiving step of receiving, as input data, image data including a plurality of characters to be calculated and characters indicating a calculation result when the plurality of characters are calculated by a predetermined formula;
an extracting step of extracting the plurality of characters to be calculated, the calculation formula, and characters indicating the calculation result from the input data received by the receiving step;
a conversion step of converting the input data into editable data using a plurality of characters extracted by the extraction step, a calculation formula, and characters indicating a calculation result;
By comparing the operation result when a plurality of characters of the operation object extracted in the extraction step are operated by the extracted calculation formula and the character indicating the extracted operation result, the extracted operation object a determination step of determining whether or not at least one of characters indicating each character, calculation formula, and calculation result has been erroneously converted;
a correction step of correcting to a proper character or calculation formula when it is determined that at least one of them is erroneously converted by the determination step;
is executed by the computer of the data conversion device,
When extraction by the extraction step is impossible for some of the plurality of characters to be operated, in the correction step, from the other characters to be operated extracted by the extraction step, the calculation formula, and the characters indicating the operation result A program for causing the computer to execute a process of calculating back and correcting the part of the characters into proper characters.

Images