US8428356B2 - Image processing device and image processing method for generating electronic document with a table line determination portion - Google Patents

Image processing device and image processing method for generating electronic document with a table line determination portion Download PDF

Info

Publication number
US8428356B2
US8428356B2 US12/824,860 US82486010A US8428356B2 US 8428356 B2 US8428356 B2 US 8428356B2 US 82486010 A US82486010 A US 82486010A US 8428356 B2 US8428356 B2 US 8428356B2
Authority
US
United States
Prior art keywords
line
information
cell
region
generated
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active, expires
Application number
US12/824,860
Other languages
English (en)
Other versions
US20110002547A1 (en
Inventor
Makoto Enomoto
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Canon Inc
Original Assignee
Canon Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Canon Inc filed Critical Canon Inc
Assigned to CANON KABUSHIKI KAISHA reassignment CANON KABUSHIKI KAISHA ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: ENOMOTO, MAKOTO
Publication of US20110002547A1 publication Critical patent/US20110002547A1/en
Application granted granted Critical
Publication of US8428356B2 publication Critical patent/US8428356B2/en
Active legal-status Critical Current
Adjusted expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
    • G06V30/40—Document-oriented image-based pattern recognition
    • G06V30/41—Analysis of document content
    • G06V30/412—Layout analysis of documents structured with printed lines or input boxes, e.g. business forms or tables

Definitions

  • the present invention relates to a method of storing results of document image processing.
  • a data format which a user desires to reuse is different depending on the circumstances, but it is desirable for objects of the document image to be so arranged that the user easily utilizes the individual objects.
  • objects of the document image For example, for a user who desires to edit and use, as a table, a table in a document image, it is desirable for the table to be so arranged on an electronic document that the table can be edited as a table object.
  • the table For a user who desires to reprint a table and use the table as a paper document, it is desirable for the table to be arranged such that visual information such as table lines is reproduced according to the document image as accurately as possible.
  • the table lines of the document image may however fail to be completely reproduced, depending on the specifications of the format of an electronic document, when a table on an image is converted into a table object.
  • the method disclosed in Japanese Patent Laid-Open No. 2005-346137 or the like can be used to draw the table as a vector object; however, in this way, it is impossible for the user to edit the table as a table, for example, it is impossible to add or remove a row.
  • the specification of an optimal format is different depending on purposes in cases such as where the user desires to edit the table as a table and when the user desires to print or display a table to use it. Hence, it is difficult to generate an electronic document in a format specification that is optimal for many purposes of the electronic document.
  • an image processing device that generates, from an input image, an electronic document having a predetermined format
  • the image processing device comprising: a region-discrimination unit configured to analyze the image and discriminate a table region; a table-structure-analysis unit configured to analyze a table structure and a ruled line in the discriminated-table region and create table-structure information including information on a cell of a table; a determination unit configured to determine whether or not a side of the cell included in the created table-structure information can be represented as a ruled line of the table according to the format; a unit configured to create information of a ruled line to be represented according to the format from the side of the cell which is determined to be representable as the ruled line; a unit configured to create a vector-line object by performing vector-conversion processing on the side of the cell which is determined to be unrepresentable as the ruled line; and an electronic-document-generation unit configured to generate the electronic document from the created table-structure information, the created ruled-line information, and the
  • an image processing device and an image processing method that generate an electronic document in a format specification that is optimal for many purposes of the electronic document.
  • FIG. 1 is a diagram showing an example of the configuration of a system according to an embodiment of the present invention
  • FIG. 2 is a diagram illustrating the operation of an image processing device in the embodiment
  • FIG. 3 is a diagram illustrating table structure information in the embodiment
  • FIG. 4 is a flowchart showing table-line-determination processing in the embodiment
  • FIG. 5 is a flowchart showing table-line-generation processing in the embodiment
  • FIG. 6 is a diagram showing an example of an input image in the embodiment
  • FIG. 7 is a diagram showing an example of the result of a table-structure-information analysis in the embodiment.
  • FIG. 8 is a diagram showing an example of the result of character recognition in a character region in the embodiment.
  • FIG. 9 is a diagram showing an example of the specification of a workbook document to be converted in the embodiment.
  • FIG. 10 is a diagram showing an example of the converted workbook document in the embodiment.
  • FIG. 11 is a diagram showing an example where the workbook document is displayed in a display/editing program in the embodiment.
  • FIG. 12 is a diagram showing an example where the workbook document is edited with the display/editing program in another embodiment
  • FIG. 13 is a diagram showing an example of an input image in the another embodiment
  • FIG. 14 is a diagram showing an example of the specification of a workbook document to be converted in the another embodiment
  • FIG. 15 is a flowchart showing table-line-determination processing in the another embodiment.
  • FIG. 16 is a diagram showing an example where the workbook document is displayed in a display/editing program in the another embodiment.
  • FIG. 1 is a diagram showing an example of the configuration of a system used in embodiment 1.
  • Reference numeral 100 represents an example of the configuration of an image processing device embodying the present invention.
  • the image processing device 100 includes a scanner 101 , a CPU 102 , a memory 103 , a hard disk 104 and a network I/F 105 .
  • the scanner 101 is a reading device that coverts, into image data, information on the paper of a document which has been read.
  • the CPU 102 is a processor that executes an electronic-document-generation program for performing processing of the present invention on the image data.
  • the memory 103 is utilized to, for example, temporarily store work memory or data when a program read from the hard disk 104 or the like is executed.
  • the hard disk 109 is a storage device that stores a program such as the electronic-document-generation program and data.
  • the network I/F 105 is an interface for exchanging data with an external device.
  • a personal computer (PC) 120 is a terminal device that is connected through a network such as a LAN 110 to the image processing device 100 , and that receives data transmitted from the image processing device.
  • the PC 120 can perform a display/editing program 121 that allows the received electronic document to be displayed and edited on a screen.
  • FIG. 2 is a diagram showing the operation of the image processing device 100 in embodiment 1.
  • Each type of processing shown in FIG. 2 is realized through the execution of the electronic-document-generation program by the CPU 102 , but part or the whole of the processing may be realized by an electrical circuit.
  • Reference numeral 200 represents an input image input from the scanner 101 .
  • Reference numeral 210 represents an electronic-document-generation portion for generating an electronic document from the input image 200 .
  • a workbook document 220 is an electronic-document-generated by the electronic-document-generation portion 210 .
  • a predetermined format specification is defined for the workbook document 220 . An example of the format specification will be described later. Note that a format of a workbook document in the embodiment is one of a spreadsheet available in a spreadsheet software (e.g., Microsoft Office Excel), but not limited to this.
  • Blocks 211 to 215 schematically represent the individual types of processing performed by the electronic-document-generation portion 210 .
  • Reference numeral 211 represents a region-discrimination portion that analyzes a document image which has been input and that extracts regions such as for characters, tables and natural images.
  • Reference numeral 212 represents a table-structure-analysis portion that analyzes a table structure and a ruled line on the table region extracted from the region-discrimination portion 211 , and that creates table-structure information.
  • FIG. 3 is a diagram showing, in the form of a tree, the table-structure information obtained from the table-structure-analysis portion 212 .
  • a tree structure is formed with rows and the minimum constituent units of the table, namely, cells that are obtained through the division of the rows into columns.
  • the cells have, as subelements, the information of data, rectangles and ruled lines.
  • the data stores results obtained by recognizing characters within the cells.
  • the rectangles are formed with positions x and y and sizes w and h, and store coordinates within the document of the cell.
  • the ruled lines are the information of line borders enclosing the cells, and are formed with upper, lower, left and right sides; the sides have line-type information, which is line thickness and pattern information, and line-color information. Corners are each formed with upper left, upper right, lower right and lower left, and store the information of the curvature of the corners.
  • the information-holding method shown in FIG. 3 is one example; the method of dividing the rows into the columns may be reversed, and the information of the table structure may be formed in any structure other than the tree structure.
  • Reference numeral 213 shown in FIG. 2 represents a character-recognition portion that performs character recognition on the character regions extracted from the table-structure-analysis portion 212 . The results obtained by the recognition are stored in data within the cell element of the table-structure information.
  • Reference numeral 214 represents a table-line-determination portion that checks the table lines against the format specification of the workbook document to be converted and that determines whether or not the table lines can be represented as the border lines of the cells.
  • Reference numeral 215 represents a table-line-generation portion that creates the ruled-line information of the table, and creates a vector-line object for the cell which is determined by the table-line-determination portion 214 such that it cannot be drawn as the border lines of the cells, and arranges the vector-line object in an appropriate position.
  • Reference numeral 216 represents a workbook-document-conversion portion (electronic-document-generation portion) that converts the character-recognition data, the vector-line object, and the border line information of the cells into the format of the workbook document 220 .
  • the workbook document 220 generated by the workbook-document-generation portion 210 has a sheet 221 , and can be displayed and edited by the display/editing program 121 held by the PC 120 .
  • the sheet 221 is the data of a sheet that is the unit of pages within the workbook document, and includes table information 222 and ruled-line vector information 223 .
  • step S 401 a cell region that has not undergone the table-line-determination processing is selected from the cell elements of the table-structure tree extracted by the table-structure-analysis portion 212 .
  • step S 402 a vectorization flag for determining whether or not to vectorize the side elements of the cell is prepared, and is set to false.
  • the information set in the vectorization flag is stored in a storage portion such as the memory 103 and the hard disk 104 . The same is true for the information of results obtained by performing various types of processing described below.
  • step S 403 repetition ends where repetition processing is performed on all ⁇ sides, adjacent sides ⁇ of the selected cell region, that is, ⁇ upper, right ⁇ , ⁇ right, lower ⁇ , ⁇ lower, left ⁇ and ⁇ left, upper ⁇ are shown.
  • the adjacent side refers to a side that joins a side of interest in a clockwise direction.
  • step S 404 a determination is made as to whether or not the curvature of the corner where the side and the adjacent side are joined together falls within the range of the format specification of the workbook document to be converted. If the curvature falls within the range, the process proceeds to step S 405 whereas, if it falls outside the range, the process proceeds to step S 408 .
  • step S 405 a determination is made as to whether or not the line color of the side falls within the range of the format specification of the workbook document to be converted. If the line color falls within the range, the process proceeds to step S 406 whereas, if it falls outside the range, the process proceeds to step S 409 .
  • step S 406 a determination is made as to whether or not the line type of the side falls within the range of the format specification of the workbook document to be converted. If the line type falls within the range, the process proceeds to step S 407 whereas, if it falls outside the range, the process proceeds to step S 409 .
  • Step S 407 is the end of the repetition starting from step S 403 .
  • step S 410 a determination is made as to whether or not an unprocessed cell is left. If the unprocessed cell is not present, the process is completed whereas, if the unprocessed cell is present, the process proceeds to step S 401 .
  • the table-line-determination portion 214 determines whether or not the curvature of the corner where the side and the adjacent side are joined together, the line color of the side and the line type of the side fall within the range of the format specification of the workbook document to be converted. If they are determined not to fall within the range of the format specification, that is, if the side is determined to be unrepresentable (not able to be represented) according to the format specification, the vectorization flag of the side is set to true.
  • the method of determining whether or not the side can be represented is not limited to this method; the determination is made based on the attribute of the format specification.
  • step S 501 a cell region that has not undergone the table-line-generation processing is selected from the cell information of the table-structure information extracted by the table-structure-analysis portion 212 .
  • step S 502 repetition ends where processing is repeatedly performed on the upper side, the right side, the lower side and the left side of one cell region.
  • step S 503 a determination is made as to whether or not the ruled-line-generation processing has already been performed on the side of the selected cell region. If the ruled-line-generation processing has not been performed, the process proceeds to step S 509 whereas, if the line-generation processing has been performed, the process proceeds to step S 508 .
  • step S 504 a determination is made as to whether the vectorization flag added by the table-line-determination portion 214 is true or false. If the flag is true, the process proceeds to step S 505 whereas, if the flag is false, the process proceeds to step S 509 .
  • step S 505 processing for coupling the vectorization sides of the adjacent sides is performed. If the vectorization flags of the adjacent sides are true, the vectorization sides are coupled, and if the vectorization flags of the remaining parallel sides are true, further coupling is performed. If both the adjacent sides are false, the sides are not coupled together.
  • step S 506 vectorization processing is performed on the vectorization sides coupled in step S 505 .
  • step S 507 the vector-line object created in step S 506 is arranged on the cell.
  • the cell is arranged in such a position that circumscribed-rectangle, upper-left-point coordinates and circumscribed-rectangle, lower-right-point coordinates of the vector-line object coincide with the relative position of the upper-left coordinates of the cell corresponding to the vector-line object and the relative position of the lower-right coordinates of the cell.
  • the table-line information in which the vector-line object is arranged in the relative coordinates of the table region is created.
  • Step S 508 shows the end of the repetition starting from step S 502 .
  • step S 510 the border-line information of the cell is set as the table-line information from the information of the curvature, the line type and the line color in the table-structure information, using the specification of the ruled lines of the cell of the document format to be converted.
  • step S 510 a determination is made as to whether or not a cell that has not undergone the table-line-generation processing is present. If the unprocessed cell is present, the process proceeds to step S 501 whereas, if the unprocessed cell is not present, the processing is completed.
  • the adjacent sides whose vectorization flags are true are coupled and thus the vectorization processing is performed, and the table-line information in which the created vector-line object is arranged on the cell (that is, in the relative position of the table region) is created.
  • the table-line information, in which the border-line information of the cell is set is created using the specification of the ruled lines of the cell of the document format (the ruled lines representable in the format of the table in the document).
  • the region-discrimination portion 211 first uses known image-analysis processing to extract the character region and the table region in the input image.
  • One example of the region-discrimination processing is proposed in, for example, Japanese Patent Laid-Open No. 2004-086436.
  • the specification of U.S. Pat. No. 5,680,478 discloses that a set of black pixels and white pixels in the document image is extracted, and that, based on the shape, the size, the state of the set and the like, characteristic regions such as for characters, pictures and drawings, tables, frames and lines are extracted.
  • the table-structure-analysis portion 212 uses a known table-structure-analysis technology to extract information on rows within the table, the structure of columns, the line color and line type of ruled lines, the curvature of corner portions and the like.
  • table structure analysis technology is proposed in, for example, Japanese Patent Laid-Open No. 2004-086436.
  • the Japanese Patent Laid-Open No. 2004-086436 discloses that, based on histograms determined from each of edges in two directions, ruled-line information is acquired from the vicinity of the border of a mass of white pixels within a table region.
  • FIG. 7 is a diagram showing, in a table, an example of the information extracted by the table-structure-analysis portion 212 .
  • a cell of 1 - 1 represents a cell in the first row and the first column from the upper left corner. It is shown that, in a circumscribed rectangle of this cell, the upper left coordinates are (100, 200) (upper left origin, unit pixel), the size is (200, 70) (width, height, unit pixel). It is shown that, in the sides of this cell, the line color of all the sides is #000 (black) in 12-bit RGB and the line type of all the sides is 1. It is shown that, in the curvature of this cell, the curvature of the upper left corner portion is 1/30 (1/pixel), and that the curvature of the other corner portions is cc (the radius of curvature is zero.)
  • the information of the line type is represented by the repetition of the display region of a ruled-line pattern and a pattern of the length with respect to the thickness of a non-display region.
  • the line type when the line type is 1, it shows a solid line in the display region alone; when the line type is 1 - 1 , it shows a broken line in which the display region and the non-display region of the same width as the thickness are repeated.
  • the method of storing the pattern information of the line type is one example; the information may be stored by another method.
  • a cell in 1 - 2 represents a cell in the first row and the second column from the upper left corner. It represents the extraction of cell information in which the upper left coordinates of a circumscribed rectangle of the cell are (300, 200), the size is (900, 70), the line color of all the sides is #000 (black), the line type of all the sides is a solid line, and the curvature of the upper right corner portion is 1/30 and the curvature of the other corner portions is ⁇ (the radius of curvature is zero).
  • a cell in 2 - 1 represents a cell in the second row and the first column from the upper left corner. It represents the extraction of cell information in which the upper left coordinates of a circumscribed rectangle of the cell are (100, 270), the size is (200, 70), the line color of all the sides is #000 (black), the line type of all the sides is a solid line, and the curvature of the lower left corner portion is 1/30 and the curvature of the other corner portions is ⁇ (the radius of curvature is zero).
  • a cell in 2 - 2 represents a cell in the second row and the second column from the upper left corner. It represents the extraction of cell information in which the upper left coordinates of a circumscribed rectangle of the cell are (300, 270), the size is (400, 70), the line color of all the sides is #000 (black), the line type of all the sides is a solid line, and the curvature of the lower right corner portion is 1/30 and the curvature of the other corner portions is ⁇ (the radius of curvature is zero).
  • the character recognition portion 213 uses a known character-recognition technology to recognize characters within each character region, and generates character-code data and OCR display data along with the sizes of characters.
  • An example of the character-recognition processing will be briefly described. This description is simply one example; another method may be used.
  • One way to do this is to binarize an image, acquire projections in vertical and horizontal directions and determine which one of the projections is low in projection diffusion is in the row direction.
  • the image is divided into individual character images.
  • the image is preferably divided into the character images by performing the following operations: the line space to be cut is found by using the projection of the binary image in the row direction; thus the image is divided into row images; furthermore the character space of the row image to be cut is found by using the row and the projection in the vertical direction.
  • the feature of each of the character images is obtained, feature information closest to the feature is searched from a dictionary that previously stores features of all types of characters and a character code indicated by the dictionary is obtained as the result of the recognition.
  • the size of the character can be obtained as the average value of the sizes of the characters acquired when the individual characters are recognized; another method may be used.
  • FIG. 8 is a diagram showing, in a table, an example of the results of the processing performed by the region-discrimination portion 211 and the character-recognition portion 213 in the example of FIG. 6 .
  • character strings “ApplicationForm”, “ID”, “1234567”, “name” and “Maruko Taro” are extracted from regions 602 , 603 , 604 , 605 and 606 , respectively, as the recognition results of the character-recognition portion.
  • the region 603 is stored as the data of the cell of 1 - 1 . Since the region 604 is within the cell of 1 - 2 , the region 604 is stored as the data of the cell of 1 - 2 . Since the region 605 is within the cell of 2 - 1 , the region 605 is stored as the data of the cell of 2 - 1 . Since the region 606 is within the cell of 2 - 2 , the region 606 is stored as the data of the cell of 2 - 2 .
  • step S 401 the cell of 1 - 1 is selected as a cell that has not undergone the ruled-line determination.
  • a vectorization flag is defined as the information of the upper side, the lower side, the left side and the right side of the 1 - 1 cell, and false information is set for each of them.
  • step S 403 a value of the upper side is substituted into the variable of the side, a value of the right side is substituted into the variable of the adjacent side, and thus the repetition processing is started.
  • step S 404 a determination is made as to whether or not the curvature of the upper right corner portion, which is the corner between the upper side and the adjacent right side, falls within the range of the specification.
  • the specification of the workbook document to be converted will be described using FIG. 9 . It is assumed that the workbook document to be converted in this case is workbook document A which is the workbook document of its own XML format, that the specification of the line color is a monochromatic 32-bit color, that the specification of the line type is a solid line and a three pattern broken line and that the curvature of the corner portion is ⁇ . Since, in the upper right corner portion of the 1 - 1 cell, its curvature is ⁇ , that is, it falls within the range of the specification, the process proceeds to step S 405 .
  • step S 405 a determination is made as to whether or not the line color of the side falls with the range of the specification of the line color. Since the line color is an RGB 12-bit monochrome and can be converted into an RGB 32-bit monochrome, that is, it falls within the range of the specification, the process proceeds to step S 406 .
  • step S 406 a determination is made as to whether or not the line type of the side falls within the range of the specification of the line type. Since the line type is 1 (solid line), that is, it falls within the specification, the process proceeds to step S 407 .
  • step S 407 since the repetition processing is not completed, the process proceeds to step S 403 .
  • step S 403 the right side is substituted into the side, and the lower side is substituted into the adjacent side, and the process proceeds to step S 404 .
  • ⁇ side, adjacent side ⁇ falls within the range of the specification with respect to ⁇ right, lower ⁇ and ⁇ lower, left ⁇ , and the vectorization flag remains false.
  • ⁇ left, upper ⁇ is substituted into ⁇ side, adjacent side ⁇ , and the process proceeds to step S 404 .
  • step S 404 since the curvature of the upper left corner portion, which is the corner between the upper side and the adjacent left side, is 1/30, that is, it falls outside the range of the specification, the process proceeds to step S 408 .
  • step S 408 the vectorization flag of the left side and the upper side, which are the side and the adjacent side currently considered, is set true, and the process proceeds to step S 407 .
  • step S 407 it is determined that the repetition processing is completed, and the process proceeds to step S 410 .
  • step S 410 the cells of 1 - 2 , 2 - 1 and 2 - 2 are unprocessed, and thus the process proceeds to step S 401 .
  • step S 410 the vectorization-determination processing is completed on all the cells, and thus the processing is completed.
  • step S 501 the cell of 1 - 1 is selected as an unprocessed cell.
  • step S 502 the upper side is substituted into the variable of the side, and the repetition processing is started.
  • step S 503 the upper side is the side of the unprocessed vector, and thus the process proceeds to step S 504 .
  • step S 504 the upper side of the cell of 1 - 1 is determined to be true from the vectorization flag stored by the table-line-determination portion 214 , and thus the process proceeds to step S 505 .
  • step S 505 since the vectorization flag of the left side is true, the left side is coupled, and the vectorization is determined to be performed.
  • step S 506 the vector-line object is created, on the target side coupled in step S 505 , from the input image, by using a known vectorization technology.
  • a known vectorization technology there are methods disclosed in Japanese Patent No. 3026592 and Japanese Patent Laid-Open No. 2005-346137.
  • Japanese Patent No. 3026592 there is disclosed a technology that generates, by performing the followings, information in which the rounds of coupled pixel data called an outline vector are described by a set of vectors between pixels.
  • step S 507 in the vector line object created in step S 506 , the vector-line-object, circumscribed-rectangle, upper-left-end coordinates are determined as the relative coordinates of a cell at the upper left end.
  • Vector-line-object, circumscribed-rectangle, lower-left-end coordinates are determined as the relative coordinates of a cell at the lower right end (or the upper left end of a lower right cell).
  • An arrangement coordinate value of the vector-line object is determined.
  • step S 508 conditions for completing the repetition processing are not satisfied, and thus the process proceeds to step S 502 .
  • step S 502 the value of the side is substituted into the right side.
  • step S 503 since the right side is not vectorized yet, the process proceeds to step S 504 .
  • step S 504 the vectorization flag of the side is false, and thus the process proceeds to step S 509 .
  • step S 509 conversion from information on the corner portion curvature of ⁇ , the solid line and #000 of the table structure information into the line-border format of the cell of the workbook document to be converted is performed.
  • step S 508 the conditions for completing the repetition processing are not satisfied, and thus the process proceeds to step S 502 .
  • step S 502 the lower side is substituted into the side, and the processing is repeated. As with the right side, the lower side is converted into the border-line format of the cell. Finally, the value of the side is substituted into the left side, and the process proceeds to step S 503 .
  • step S 503 since the left side is coupled with the upper side, and the vectorization is completed when the upper side is processed, the process proceeds to step S 508 .
  • step S 508 the repetition processing is completed, and thus the process proceeds to step S 510 .
  • step S 510 the cells of 1 - 2 , 2 - 1 and 2 - 2 are unprocessed, and thus the process proceeds to step S 501 .
  • step S 501 the processing is performed, from step S 501 , on 1 - 2 , 2 - 1 and 2 - 2 , and thus, on the cell of 1 - 2 , the upper side and the right side are coupled with each other and are converted into the vector-line object, and the lower side and the left side are converted into the cell-line-border format.
  • step S 510 the right side and the lower side are coupled with each other and are converted into the vector-line object, and the upper side and the left side are converted into the cell-line-border format.
  • the workbook-document-conversion portion 216 Based on the table structure information created by the table-structure-analysis portion 212 , the characters recognized by the character-recognition portion 213 and the ruled-line information of the table created by the table-line-generation portion 215 , the workbook-document-conversion portion 216 performs conversion into an electronic document in the workbook-document format.
  • An electronic document 1000 of FIG. 10 is an example of the workbook document 220 generated according to the specification of an imaginary XML format made for describing the present example.
  • a code 1001 is a portion that stores table information, and corresponds to an area covered by an element name “Table”.
  • a “Column” element portion 1003 describing the information of the columns of the table and a “Row” element portion 1004 describing the information of the rows are stored.
  • the sheet of the workbook document is generated from one large piece of table information.
  • the number of rows and columns within the image is 2 ⁇ 2 but one row and one column are added to express margins in the upper part and left part of the table, thus forming 3 ⁇ 3.
  • three “Column” elements 1003 indicating the columns and three “Row” element portions 1004 indicating the rows are present.
  • the “Column” element 1003 stores a column number “c” attribute and each column width “width” attribute.
  • the “Row” element a row number “r” attribute and each row height “height” attribute are stored; the “Row” element is composed of cell information “Cell” elements corresponding in number to the columns.
  • the “Cell” element stores a “c” attribute indicating that the “Cell” element itself is the c-th column, a “type” attribute indicating the type of data described in the cell and a “value” attribute indicating the value of data in the cell; the “Cell” element is formed with border-line information “Line” of the cell.
  • a “Line” element stores a “position” attribute indicating which one of the four sides the element corresponds to, a “type” attribute indicating the line type and a “color” attribute indicating the line color.
  • a code 1002 is freely arranged on the table, and is a portion in which graphic information is stored and corresponds to an area enclosed by an element name “Shape”.
  • the code 1002 is composed of a plurality of “Shape” elements describing the graphic information.
  • the “Shape” element has a graphic object as a subelement, and stores, as an attribute, information on a position where the graphic object is drawn.
  • the “Shape” element corresponds to one vector-line object.
  • the code 1002 four “Shape” elements 1005 , 1006 , 1007 and 1008 formed with subelements “Path” representing Bezier curve are included.
  • the vector-line object defines the upper left end and the lower right end as the relative coordinates of the cell.
  • the “Shape” element is composed of information of fromCell representing the matrix of the target cell relative to the upper left end, information of fromPos representing the relative coordinates, information of toCell representing the matrix of the target cell relative to the lower right end and information of toPos representing the relative coordinates.
  • the “Shape” element 1005 which is the vector-line object on the upper left side of the cell 1 - 1 , will be specifically described.
  • the “Path” element has information on the shape of an object, and describes, on a plane whose size is set in a “canvasSize” attribute, the shape of a curve in the “data” attribute with the cubic Bezier curve.
  • a window 1101 is a window displayed by the display/editing program 121 , and includes a sheet editing window 1102 .
  • a display corresponding to the contents of the sheet information 221 in the workbook document 220 is generated.
  • the contents of table information 1001 and ruled-line data 1002 are drawn.
  • the table is generated as its input image is.
  • the user can add a row of the table, change the contents of the cell and change the size of the cell. Moreover, the user can store the workbook document that has been edited, and print the appearance of the displayed sheet editing window 1102 on paper.
  • FIG. 12 shows an example of the result obtained through the addition of a row and the change of a cell size by the user. Since the vector-line object is described as the coordinates relative to the cell, even if an operation such as for changing the size of the cell and inserting a row is performed, the vector-line object follows like the border line of the cell.
  • the workbook is generated that represents the ruled lines which cannot be reproduced by the specification of the format and that can be edited as a table.
  • the workbook thus generated can be easily selected through a menu or the like and used by the user with the display/editing program 121 .
  • embodiment 1 deals with the case where the ruled lines which cannot be represented simply by the specification are switched, ruled lines which cannot be represented due to a complicated structure itself may additionally be switched.
  • regions 1301 and 1302 are assumed to be extracted as table regions.
  • step S 1501 since the number of criteria for the specification of ruled lines is increased, processing for determining the specification of a table structure itself in step S 1501 is added to the flowchart of the table-line determination in embodiment 1 shown in FIG. 4 .
  • step S 1501 a determination is made as to whether or not the range of the specification on the position of the table itself is satisfied. In this embodiment, if the target table of the determination exists inside of another table, the determination is made as the specification on the position of the table itself being unsatisfied. On the other hand, if the target table of the determination does not exist inside of another table, the determination is made as the specification on the position of the table itself being satisfied. With respect to the region 1301 , the specification on the position of the table is satisfied, and thus the process proceeds to step S 1502 .
  • step S 1512 the vectorization flags of all the sides of the table structure are set true, and an internal character string is also represented as a text within a graphic object.
  • the table-line-generation portion 215 Based on the determination result of the table-line-determination portion 214 , the table-line-generation portion 215 generates table lines. With respect to the region 1301 , the upper left side of the cell in the first row and the first column is converted into a vector-line object, the upper right side of the cell in the first row and the second column is converted into a vector-line object and the right bottom and left bottom sides of the cell in the third row and the first column are converted into a vector-line objects. With respect to the region 1302 , all the sides are converted into vector-line objects.
  • the workbook-document-conversion portion 216 Based on the results of the character-recognition portion 213 and the table-line-generation portion 215 , the workbook-document-conversion portion 216 performs processing for converting the character-recognition data, the vector-line objects and the border-line information of the cells into the workbook document.
  • FIG. 16 is an example where the workbook document obtained by converting the image 1300 is opened by the display/editing program 121 . As shown in FIG. 16 , it is possible to reproduce the region 1302 that cannot be represented by the conventional specification.
  • aspects of the present invention can also be realized by a computer of a system or apparatus (or devices such as a CPU or MPU) that reads out and executes a program recorded on a memory device to perform the functions of the above-described embodiment (s), and by a method, the steps of which are performed by a computer of a system or apparatus by, for example, reading out and executing a program recorded on a memory device to perform the functions of the above-described embodiment (s).
  • the program is provided to the computer for example via a network or from a recording medium of various types serving as the memory device (e.g., computer-readable medium).

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Artificial Intelligence (AREA)
  • Character Input (AREA)
  • Processing Or Creating Images (AREA)
  • Document Processing Apparatus (AREA)
  • Editing Of Facsimile Originals (AREA)
  • Image Processing (AREA)
  • Image Analysis (AREA)
US12/824,860 2009-07-01 2010-06-28 Image processing device and image processing method for generating electronic document with a table line determination portion Active 2031-09-27 US8428356B2 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2009156971A JP5361574B2 (ja) 2009-07-01 2009-07-01 画像処理装置、画像処理方法、及びプログラム
JP2009-156971 2009-07-01

Publications (2)

Publication Number Publication Date
US20110002547A1 US20110002547A1 (en) 2011-01-06
US8428356B2 true US8428356B2 (en) 2013-04-23

Family

ID=42932674

Family Applications (1)

Application Number Title Priority Date Filing Date
US12/824,860 Active 2031-09-27 US8428356B2 (en) 2009-07-01 2010-06-28 Image processing device and image processing method for generating electronic document with a table line determination portion

Country Status (5)

Country Link
US (1) US8428356B2 (fr)
EP (1) EP2270714B1 (fr)
JP (1) JP5361574B2 (fr)
KR (1) KR101447932B1 (fr)
CN (1) CN101944179B (fr)

Families Citing this family (19)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP5146190B2 (ja) * 2008-08-11 2013-02-20 オムロン株式会社 文字認識装置、文字認識プログラム、および文字認識方法
US9042653B2 (en) * 2011-01-24 2015-05-26 Microsoft Technology Licensing, Llc Associating captured image data with a spreadsheet
JP6399872B2 (ja) * 2014-07-07 2018-10-03 キヤノン株式会社 情報処理装置、情報処理方法、プログラム
US20160026613A1 (en) * 2014-07-28 2016-01-28 Microsoft Corporation Processing image to identify object for insertion into document
US20160026858A1 (en) * 2014-07-28 2016-01-28 Microsoft Corporation Image based search to identify objects in documents
JP2016167156A (ja) * 2015-03-09 2016-09-15 キヤノン株式会社 システム、プログラム及び制御方法
US20170093896A1 (en) * 2015-09-25 2017-03-30 Intel Corporation Techniques for radio frequency identification (rfid) input/output (i/o) port management
US11995428B2 (en) 2017-01-26 2024-05-28 Nice Inc. Method and system for providing image-based interoperability with an application
US10540167B2 (en) 2017-01-26 2020-01-21 Nice Ltd. Image based method and system for building object model and application states comparison and graphic-based interoperability with an application
US10740123B2 (en) * 2017-01-26 2020-08-11 Nice Ltd. Method and system for accessing table content in a digital image of the table
US11036356B2 (en) * 2017-07-31 2021-06-15 Microsoft Technology Licensing, Llc Service backed digital ruled paper templates
CN107679024B (zh) * 2017-09-11 2023-04-18 畅捷通信息技术股份有限公司 识别表格的方法、系统、计算机设备、可读存储介质
CN109753641B (zh) * 2017-11-01 2023-11-28 珠海金山办公软件有限公司 一种更改对象位置的方法、装置、电子设备及存储介质
CN109522816B (zh) * 2018-10-26 2021-07-02 北京慧流科技有限公司 表格识别方法及装置、计算机存储介质
CN111859874B (zh) * 2019-04-17 2023-06-13 百度在线网络技术（北京）有限公司 表格生成方法及其系统、视频播放设备和计算机可读介质
US11010543B1 (en) * 2020-08-11 2021-05-18 Fmr Llc Systems and methods for table extraction in documents
US11663842B2 (en) * 2020-11-05 2023-05-30 Jpmorgan Chase Bank, N.A. Method and system for tabular information extraction
JP2022092119A (ja) 2020-12-10 2022-06-22 キヤノン株式会社 画像処理装置、画像処理方法およびプログラム
US11790110B2 (en) 2021-02-09 2023-10-17 Nice Ltd. System and method for preventing sensitive information from being recorded

Citations (19)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5048107A (en) * 1989-06-06 1991-09-10 Ricoh Company, Ltd. Table region identification method
US5680478A (en) 1992-04-24 1997-10-21 Canon Kabushiki Kaisha Method and apparatus for character recognition
US5881381A (en) * 1994-09-16 1999-03-09 International Business Machines Corporation Method and system for mapping non-uniform table-structure input data to a uniform cellular data structure
US5907631A (en) * 1993-05-12 1999-05-25 Ricoh Company, Ltd. Document image processing method and system having function of determining body text region reading order
JPH11167532A (ja) 1997-12-02 1999-06-22 Canon Inc データ加工システムおよび装置、データ加工方法、記録媒体
JP3026592B2 (ja) 1990-10-22 2000-03-27 キヤノン株式会社 輪郭抽出方法及びその装置
KR20000047655A (ko) 1998-12-16 2000-07-25 아끼구사 나오유끼 표 화상 처리 장치 및 그 프로그램 기억 매체
US20020021840A1 (en) * 2000-08-15 2002-02-21 Fujitsu Limited Apparatus for extracting ruled line from multiple-valued image
JP2004086436A (ja) 2002-08-26 2004-03-18 Canon Inc 画像処理装置および画像処理方法、プログラムおよび記憶媒体
US20040223197A1 (en) * 2003-02-13 2004-11-11 Canon Kabushiki Kaisha Image processing method
JP2005028678A (ja) 2003-07-10 2005-02-03 Dainippon Printing Co Ltd 罫線用帳票、属性用帳票、設計システム及びプログラム
KR20050071843A (ko) 2004-01-03 2005-07-08 노키아 인크 제한된 영역의 디스플레이 상에 렌더링하기 위한 표 정보변환
US20050238244A1 (en) 2004-04-26 2005-10-27 Canon Kabushiki Kaisha Function approximation processing method and image processing method
JP2005346137A (ja) 2004-05-31 2005-12-15 Canon Inc 関数化処理方法、関数近似処理装置及び画像処理装置
US20060215233A1 (en) * 2005-03-23 2006-09-28 Tatsuhiko Hirai Image processing apparatus and its method
US20090234820A1 (en) * 2008-03-14 2009-09-17 Canon Kabushiki Kaisha Image processing apparatus, image processing method, and computer-readable storage medium
US7623710B2 (en) * 2006-02-14 2009-11-24 Microsoft Corporation Document content and structure conversion
US20090324080A1 (en) 2008-06-30 2009-12-31 Canon Kabushiki Kaisha Image processing device, image processing method and storage medium
US7860266B2 (en) * 2004-07-07 2010-12-28 Canon Kabushiki Kaisha Image processing system and image processing method

Family Cites Families (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR930009639B1 (ko) * 1989-07-09 1993-10-08 가부시끼가이샤 히다찌세이사꾸쇼 화상데이타를 이용하는 문서데이타 처리방법 및 장치
JPH06131497A (ja) * 1992-10-16 1994-05-13 Fuji Xerox Co Ltd 表認識方式
JP2002007953A (ja) * 2000-06-22 2002-01-11 Hitachi Ltd 電子帳票処理装置
JP4012140B2 (ja) * 2003-11-20 2007-11-21 キヤノン株式会社 画像処理装置、情報処理装置及びそれらの制御方法、プログラム
JP3796500B2 (ja) * 2003-11-21 2006-07-12 キヤノン株式会社 画像処理装置及びその制御方法、プログラム
JP4378208B2 (ja) * 2004-04-26 2009-12-02 キヤノン株式会社 情報処理装置及び情報処理方法
JP2006106971A (ja) * 2004-10-01 2006-04-20 Canon Inc 表ベクトルデータ生成方法及び文書処理装置
JP4895340B2 (ja) * 2005-03-31 2012-03-14 キヤノン株式会社 情報処理装置およびその方法
US7583841B2 (en) * 2005-12-21 2009-09-01 Microsoft Corporation Table detection in ink notes
JP4973063B2 (ja) * 2006-08-14 2012-07-11 富士通株式会社 表データ処理方法及び装置
JP4921335B2 (ja) * 2007-12-10 2012-04-25 キヤノン株式会社 ドキュメント処理装置及び検索方法

Patent Citations (24)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5048107A (en) * 1989-06-06 1991-09-10 Ricoh Company, Ltd. Table region identification method
US6404921B1 (en) 1990-10-22 2002-06-11 Canon Kabushiki Kaisha Contour extracting method and apparatus
JP3026592B2 (ja) 1990-10-22 2000-03-27 キヤノン株式会社 輪郭抽出方法及びその装置
US5680478A (en) 1992-04-24 1997-10-21 Canon Kabushiki Kaisha Method and apparatus for character recognition
US5907631A (en) * 1993-05-12 1999-05-25 Ricoh Company, Ltd. Document image processing method and system having function of determining body text region reading order
US5881381A (en) * 1994-09-16 1999-03-09 International Business Machines Corporation Method and system for mapping non-uniform table-structure input data to a uniform cellular data structure
JPH11167532A (ja) 1997-12-02 1999-06-22 Canon Inc データ加工システムおよび装置、データ加工方法、記録媒体
KR20000047655A (ko) 1998-12-16 2000-07-25 아끼구사 나오유끼 표 화상 처리 장치 및 그 프로그램 기억 매체
US20050031208A1 (en) * 2000-08-15 2005-02-10 Fujitsu Limited Apparatus for extracting ruled line from multiple-valued image
US7440618B2 (en) * 2000-08-15 2008-10-21 Fujitsu Limited Apparatus for extracting rules line from multiple-valued image
US7164795B2 (en) * 2000-08-15 2007-01-16 Fujitsu Limited Apparatus for extracting ruled line from multiple-valued image
US20020021840A1 (en) * 2000-08-15 2002-02-21 Fujitsu Limited Apparatus for extracting ruled line from multiple-valued image
JP2004086436A (ja) 2002-08-26 2004-03-18 Canon Inc 画像処理装置および画像処理方法、プログラムおよび記憶媒体
US20040223197A1 (en) * 2003-02-13 2004-11-11 Canon Kabushiki Kaisha Image processing method
JP2005028678A (ja) 2003-07-10 2005-02-03 Dainippon Printing Co Ltd 罫線用帳票、属性用帳票、設計システム及びプログラム
KR20050071843A (ko) 2004-01-03 2005-07-08 노키아 인크 제한된 영역의 디스플레이 상에 렌더링하기 위한 표 정보변환
US20050238244A1 (en) 2004-04-26 2005-10-27 Canon Kabushiki Kaisha Function approximation processing method and image processing method
US7873218B2 (en) * 2004-04-26 2011-01-18 Canon Kabushiki Kaisha Function approximation processing method and image processing method
JP2005346137A (ja) 2004-05-31 2005-12-15 Canon Inc 関数化処理方法、関数近似処理装置及び画像処理装置
US7860266B2 (en) * 2004-07-07 2010-12-28 Canon Kabushiki Kaisha Image processing system and image processing method
US20060215233A1 (en) * 2005-03-23 2006-09-28 Tatsuhiko Hirai Image processing apparatus and its method
US7623710B2 (en) * 2006-02-14 2009-11-24 Microsoft Corporation Document content and structure conversion
US20090234820A1 (en) * 2008-03-14 2009-09-17 Canon Kabushiki Kaisha Image processing apparatus, image processing method, and computer-readable storage medium
US20090324080A1 (en) 2008-06-30 2009-12-31 Canon Kabushiki Kaisha Image processing device, image processing method and storage medium

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
Aug. 7, 20012 Korean Office Action in Korean Application No. 10-2010-0063286.

Also Published As

Publication number Publication date
EP2270714A2 (fr) 2011-01-05
JP2011013897A (ja) 2011-01-20
CN101944179A (zh) 2011-01-12
KR20110002439A (ko) 2011-01-07
KR101447932B1 (ko) 2014-10-07
EP2270714A3 (fr) 2017-03-01
CN101944179B (zh) 2013-07-17
EP2270714B1 (fr) 2019-01-09
US20110002547A1 (en) 2011-01-06
JP5361574B2 (ja) 2013-12-04

Similar Documents

Publication Publication Date Title
US8428356B2 (en) Image processing device and image processing method for generating electronic document with a table line determination portion
KR101334483B1 (ko) 문서를 디지털화하는 장치 및 방법과, 컴퓨터 판독가능 기록 매체
RU2437152C2 (ru) Устройство обработки изображений, способ и компьютерная программа обработки изображений
US8320019B2 (en) Image processing apparatus, image processing method, and computer program thereof
US8050499B2 (en) Image-processing apparatus, image-processing method, and computer program used therewith
JP5121599B2 (ja) 画像処理装置、画像処理方法およびそのプログラムならびに記憶媒体
US9558433B2 (en) Image processing apparatus generating partially erased image data and supplementary data supplementing partially erased image data
JP5049922B2 (ja) 画像処理装置及び画像処理方法
JP2012205181A (ja) 画像処理装置およびプログラム
JP5020698B2 (ja) 画像処理装置、画像処理方法、画像処理プログラム
US8295602B2 (en) Image processing apparatus and image processing method
US8340434B2 (en) Image processing apparatus, image processing system and computer readable medium
JP4651407B2 (ja) 画像処理装置およびコンピュータプログラムおよび記憶媒体
JPH0728934A (ja) 文書画像処理装置
JP4587167B2 (ja) 画像処理装置及び画像処理方法
JPH08185475A (ja) 画像認識装置
JP2006330875A (ja) 文書処理装置、方法およびプログラム

Legal Events

Date Code Title Description
AS Assignment

Owner name: CANON KABUSHIKI KAISHA, JAPAN

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:ENOMOTO, MAKOTO;REEL/FRAME:025156/0031

Effective date: 20100618

STCF Information on status: patent grant

Free format text: PATENTED CASE

FPAY Fee payment

Year of fee payment: 4

MAFP Maintenance fee payment

Free format text: PAYMENT OF MAINTENANCE FEE, 8TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1552); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

Year of fee payment: 8

MAFP Maintenance fee payment

Free format text: PAYMENT OF MAINTENANCE FEE, 12TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1553); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

Year of fee payment: 12