JP2008033960A - Apparatus and method for document image processing - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To reduce user load in check/correction operations for recognition/editing processing results of document images, and to realize efficient operations. <P>SOLUTION: A document image which is inputted from an input part 21 is area identified by an identification part 22, and recognized by a recognition part 23. Then, a type code of each area and an individual code of a recognition result are stored in a storage part 24, and displayed on a display part 25. A user inputs a correction instruction through a correction part 26, and then corrects the results of area identification and recognition processing by one operation. In addition, an original image is displayed near the recognition results on the display part 25, so that movement of a user's line of sight is reduced. If there is not a correct answer in recognition candidates, a code is added to the original image to consider the image as a recognition result, so that the image is corrected with keeping the possibility of editing. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a document image processing apparatus that uses a document image as an input, determines an image of a document image constituent element such as a character, a table, a figure, or a frame, and performs a coding process by recognizing the document image constituent element.

  With the spread of personal computers and the development of communication networks in recent years, many electronic documents have been distributed. However, the main medium for information distribution is still paper documents, and there are many existing paper documents. Therefore, the demand for a document image recognition / editing apparatus for converting a paper document into an electronic document and editing the conversion result is increasing.

  The document image recognition / editing apparatus is an apparatus that determines an image of a document image constituent element such as a character, a table, a figure, a frame, and the like, and performs a coding process by recognizing the document image constituent element, using the document image as an input. In the encoding process, in particular, a character image is converted into a character code.

  However, in the recognition process in the document image processing apparatus, since the correct answer rate does not reach 100%, the handling of recognition results that are not correct is a problem, and in particular, a mechanism that can perform correction work efficiently is desired.

  FIG. 38 is a block diagram of a conventional document image recognition / editing apparatus. The document image input unit 1 inputs a document image to be processed, and the region identification unit 2 identifies an individual region in the image and stores the result in the region identification result storage unit 3. Here, the display unit 8 displays the region identification result on the screen, and the user corrects it as necessary. At this time, the data in the region identification result storage unit 3 is corrected by the first correction unit 6.

  Next, the individual area recognition unit 4 recognizes characters in the individual area and stores the result in the recognition result storage unit 5. And the display part 8 displays a recognition result on a screen, and a user corrects it as needed. At this time, the data in the recognition result storage unit 5 is corrected by the second correction unit 7.

In such a document image recognition / editing apparatus, handling and correction work for recognition results whose correct answer rate does not reach 100% are processed as follows.
(1) As an area identification process by the area identification unit 2, the individual area recognition unit 4 determines the attributes such as text, table, figure, frame, etc. of the document image component of the individual area by correcting if necessary. , Recognition of individual document image constituent elements according to attributes is performed. If it is a text area, an individual character image is determined and character recognition is performed. If it is a table area, ruled line extraction is performed, a character area in each cell is determined, and characters are recognized. The recognition result is corrected as necessary.

  (2) The result of the character recognition processing includes a sequence of candidate character codes arranged in a probable order as shown in FIG. The first candidate character code is the initial value of the recognition result. The 2nd correction part 7 displays the candidate character code below the 2nd rank, and the user can select one of them. When the character recognition result is corrected, the corresponding character image is displayed at the original position P1 in the input image.

However, the conventional document image recognition / editing apparatus has a problem that a great deal of labor is required to correct the recognition result as described below.
(1) Conventional document image processing includes two stages of area identification and intra-area recognition, and includes a user correction process at each stage. As described above, the user needs two correction operations, which is troublesome. Even if there is no identification error at the region identification stage, it is necessary to confirm the presence or absence of the identification error. If this confirmation is omitted, it is not possible to correct the location where the identification error occurred after the intra-region recognition. . In this case, in order to obtain a correct processing result, it is necessary to repeat the processing from the beginning and correct the identification error at the region identification stage.

  (2) The information included in the recognition result display of the document image component is only code information as shown in FIG. Therefore, in order to confirm whether the character recognition result is correct, when a target character is designated in the recognition result display, the position P1 of the corresponding document image component in the input image is surrounded by a frame. It was displayed. However, when comparing and collating the code information of the recognition result display with the character image in the input image, the movement of the user's viewpoint is large, and collation work becomes a burden on the user.

  In addition, in the candidate character code correction selection, there are cases where there is no correct character among the candidate characters. In this case, it is necessary to input a correct character code from the beginning, and the input work becomes a burden on the user.

  SUMMARY OF THE INVENTION An object of the present invention is to provide a document image processing apparatus and method for reducing the burden on the user and realizing an efficient operation in the confirmation / correction of processing results by the document image recognition / editing apparatus.

  FIG. 1 is a configuration diagram of a document image processing apparatus according to the present invention. The document image processing apparatus of FIG. 1 includes an identification unit 11, a recognition unit 12, an output unit 13, a correction unit 14, an extraction unit 15, a code addition unit 16, and an editing unit 17, and performs a recognition process on an input image. .

According to the first principle of the present invention, the identification unit 11, the recognition unit 12, the output unit 13, and the correction unit 14 operate as follows.
The identification unit 11 identifies the pattern area of the input image and determines the type of the pattern area. The recognition unit 12 performs recognition processing for patterns included in the pattern area. The output unit 13 outputs the type information indicating the type of the pattern area and the individual information indicating the pattern as recognition result candidates of the image constituent elements constituting the input image. The correcting unit 14 corrects the recognition result candidate.

  The pattern area refers to an image area such as a sentence, a table, a figure, a frame, or a character included in the input image, and a certain pattern area may include another pattern area. For example, the pattern area of a sentence is usually composed of a pattern area of a plurality of characters. An image component refers to a partial image of an input image and corresponds to a pattern area or a pattern in the pattern area.

  The identification unit 11 determines which type of the pattern area to be recognized corresponds to a sentence, a table, a figure, a box, a character, and the like. Recognition processing such as character recognition and ruled line recognition is performed on a pattern area having a structure.

  Then, the output means 13 collectively outputs the pattern area type information and individual information such as a character font representing the recognized pattern as recognition result candidates. The user can collectively correct the type information and the individual information using the correcting unit 14 while looking at the output result.

  As described above, according to the first principle, it is possible to collectively perform region identification and intra-region recognition of document images and collectively correct the result. For this reason, it is not necessary to perform the conventional two-stage correction work, and the burden on the user in the correction work is reduced.

Further, according to the second principle of the present invention, the output means 13, the extraction means 15, the code addition means 16, and the editing means 17 operate as follows.
The extraction means 15 extracts the image component which comprises it from an input image. The code adding means 16 adds new code information to the image component. The output unit 13 outputs document information in which image data corresponding to image components and character patterns corresponding to existing code information are mixed. The editing means 17 edits document information using new code information and existing code information.

  By adding code information to an image component extracted from an input image, it can be handled in the same manner as a character pattern corresponding to existing code information. Therefore, it is possible to display and edit a document in which a partial image of the input image and a character pattern given as code information are mixed.

  According to the second principle, it is possible to display the original image near the candidate of the character recognition result by using the code information added to the image component, and to confirm the comparison between the recognition result and the input image. Viewpoint movement is reduced.

According to the third principle of the present invention, the recognition unit 12, the output unit 13, and the extraction unit 15 operate as follows.
The extraction means 15 extracts the image component which comprises it from an input image. The recognition unit 12 performs image component recognition processing. The output unit 13 separates the image data corresponding to the image component from the input image and outputs the image data together with one or more candidates in the recognition result of the image component.

  According to the third principle, the image of the image component extracted from the input image can be displayed on the screen near the recognition result candidate, and the viewpoint movement for comparing and confirming the recognition result and the input image is reduced. Is done. Further, when there is no correct answer among the recognition result candidates, the original image can be selected and corrected, so that it is not necessary to input a correction character code again.

  For example, the identification unit 11, the recognition unit 12, and the correction unit 14 in FIG. 1 correspond to an area identification unit 22, an individual region recognition unit 23, and a correction unit 26 in FIG. The code adding unit 16 and the editing unit 17 correspond to a document image constituent element extracting unit 42, a code adding unit 43, and an editing unit 49 in FIG. Further, for example, the output unit 13 in FIG. 1 corresponds to the display unit 25 in FIG. 2 and the display unit 46 in FIG.

  According to the present invention, area identification and in-area recognition of document images can be performed at once, and the results can be corrected collectively, thereby reducing the burden on the user in correction work. In the conventional two-stage correction work, the user is restrained during the first and second stage correction work, but this restraint time is eliminated.

  In addition, since the original image is displayed as one of the recognition result candidates in the intra-region recognition, the viewpoint movement for comparing and confirming the recognition result and the input image is reduced, and the burden on the user is reduced. Further, when there is no correct answer among the recognition result candidates, the original image can be selected and corrected, so that it is not necessary to input a correction character code again.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
In the present invention, the following measures are taken in response to the above-mentioned conventional problems (1) and (2).

  (1) An identification result candidate for area identification is retained even in the intra-area recognition stage so that the identification result candidate and the intra-area recognition result candidate can be selected and corrected simultaneously. Thereby, area identification and intra-area recognition can be performed collectively, and the processing result can be corrected collectively.

  The document image processing apparatus receives a document image, identifies a region such as a sentence, table, figure, and box, and has an internal structure such as a sentence, table, and frame for areas such as character recognition and ruled line recognition. Perform internal recognition processing. At this time, as a recognition result candidate code of the document image constituent element composed of the area and the area inside the area, the area type code such as “text”, “table”, “figure”, “frame”, “character code”, etc. By including the individual codes together, the results of area identification and intra-area recognition can be corrected collectively.

In order to correct the recognition code, means for indicating the document image component is required. The document image constituent element is a partial image of the document image, and means an element constituting the document image such as a character. Normally, the document image component is indicated by the coordinate position of the document image region. For this purpose, the region of the document image component is defined by one of the following methods (a) and (b).
(A) The region of the document image component is defined as a pixel region corresponding to the character / graphic of the document image component.
(B) The region of the document image constituent element is defined as a circumscribed rectangle of the character / graphic of the document image constituent element.

The following (a), (b), and (c) are conceivable as means for first instructing the document image constituent element based on the document image constituent element area defined above and the designated coordinate position of the document image area.
(A) The innermost document image constituent element including the instruction coordinates of the document image is set as an instruction target.
(B) The largest document image constituent element including the designated coordinates of the document image is set as the designation target.
(C) The document image constituent element whose circumscribing frame is closest to the designated coordinates of the document image is set as the designation target.

In the above methods (a) and (b), there may occur document image components that cannot be specified. These document image components include document images that have already been specified, including the case of (c) above. From the relationship with the constituent elements, the following instructions (a) and (b) can be given.
(A) The innermost document image constituent element including the already instructed document image constituent element is set as an instruction target.
(B) The largest document image constituent element including the designated coordinates of the document image among the already designated document image constituent elements is set as the instruction target.

The method of presenting the recognition result candidate code of the document image component to the user is, for example, as shown in (a) and (b) below.
(A) When the first recognition result candidate is “character”, character recognition is performed, the candidate character code of the character recognition result is set as a higher recognition result candidate code, and “table”, “figure”, The type code of the document image constituent element such as “frame” is set as a lower recognition result candidate code.
(B) When the recognition result candidate number 1 is a document image component type such as “text”, “table”, “figure”, “frame” other than “character”, the type code of the document image component is The first recognition result candidate code is used, and the other types of document image component type codes are used as lower recognition result candidate codes.

The operation at the time of a correction instruction is as follows (a) and (b).
(A) When an instruction for correcting the type code of “character” is given as the recognition result candidate code, the character recognition of the corresponding input character image is performed, and “character” of the recognition result candidate code is used with one or more character recognition result candidate character codes. Is replaced.
(B) When a correction instruction is given for a type code of a document image constituent element such as “table”, “figure”, “frame” other than “character” as a recognition result candidate code, it is instructed for the corresponding input character image. Recognition as a type of document image component.

  (2) A character image is displayed in the candidate character display area of the character recognition result to reduce viewpoint movement for comparison confirmation between the recognition result and the input image. The document image processing apparatus receives a document image, determines an image of a document image constituent element such as a character, a table, a figure, or a frame, and performs a coding process by recognizing the document image constituent element. At this time, as the recognition result candidate display, the image of the document image constituent element is displayed together with the recognition result candidate code. Thereby, viewpoint movement for comparing and confirming the recognition result and the input image is reduced.

  In addition, it is possible to instruct correction of the document image constituent element image in the recognition result candidate display of the document image constituent element, and when the correction instruction is given, the new code is made to correspond to the document image constituent element image and can be edited. As a result, even when there is no correct character in the recognition result candidate characters, the character image can be corrected and edited without fail by selecting and correcting the character image.

  In order to modify the document image component image and edit the document image component image in association with the new code, the display information of the new code is displayed on the display unit, so that the existing code and the new code are mixed. Make the document editable. More generally, when a document image is input and a document image component image is determined, a new code is associated with the document image component image and the display information of the new code is displayed on the display unit. It is possible to edit a document that contains both the new code and the new code.

The display information of the new code corresponding to the document image constituent element image includes the following (a) and (b).
(A) A reduced or enlarged image including a normal image of the document image component image is used.
(B) Data obtained by vectorizing the contour of the document image constituent element image is used.

  When a new code is made to correspond to the document image component image that has been instructed to be edited and editing is possible, the document image component elements such as “character”, “table”, “figure”, “frame” are added to the document image component image. This type attribute can add value to the subsequent editing process. As this type attribute, a method of automatically assigning the attribute of the first code of the recognition candidate can be considered. If an interface unit for instructing correction of the type attribute of the document image constituent element image is provided, more flexible correction can be performed.

  When editing a document in which an existing code and a new code are mixed, a certainty factor (reliability) may be given to a recognition result candidate code. In this case, in order to treat the document image constituent element image as a recognition result candidate, the document image constituent element image is given a certain level of certainty, and the recognition result candidate code or document image constituent element image having the highest certainty is ranked in order. Attach. When the document image component image is handled as a recognition result candidate, there is an effect that the correction operation becomes smoother.

  In addition, having an interface unit that can input a certainty value given to a document image component image from the outside has an effect of adapting the recognition result to the user's application. That is, if the certainty given to the document image component image is high, the ratio that the document image component image is first in the recognition result is high, and the rate that the result is encoded is low. The recognition rate is also lowered. If the certainty given to the document image constituent element image is low, the ratio that the document image constituent element image is first in the recognition result is low, and the ratio of existing coding is high, but the misrecognition rate is also high. Become.

  Furthermore, if a means for interactively inputting the certainty factor of the document image component image, having a window for displaying only the first recognition result candidate, and sequentially changing the recognition result candidate first display, a recognition target document is provided. Can be adjusted interactively.

Next, the configuration of the document image processing apparatus and a specific example of the processing as described above will be described in order.
FIG. 2 is a first block diagram of the document image processing apparatus of the present invention. The document image processing apparatus of FIG. 2 includes a document image input unit 21, an area identification unit 22, an individual area recognition unit 23, a recognition result storage unit 24, a display unit 25, and a correction unit 26, and performs processing as shown in FIG. I do.

  First, the document image input unit 21 inputs a digital image obtained by digitizing a document (step S1). As the document image input unit 21, for example, a scanner device for digitizing a paper document is used. Next, the region identification unit 22 receives a document image as input and identifies individual regions such as sentences, tables, diagrams, and surrounding frames (step S2). Next, the individual area recognition unit 23 performs recognition processing inside the area, such as character recognition and ruled line recognition, for an area having an internal structure such as a sentence, a table, or a frame (step S3).

  The processing results by the area identification unit 22 and the individual area recognition unit 23 are stored in the recognition result storage unit 24. That is, as a recognition result candidate code of a document image component composed of a region and a region inside the region, a region type code such as “text”, “table”, “figure”, “enclosed frame”, “character code” Are stored together with the individual codes.

  FIG. 4 shows examples of area type codes and individual codes. In FIG. 4, “sentence”, “table”, “figure”, “enclosed frame”, and “character” represent type codes, and “character code” represents an individual code corresponding to “character”. Here, the individual codes corresponding to “text”, “table”, “figure”, and “box” are not defined.

  The data stored in the recognition result storage unit 24 is displayed on the screen of the display unit 25 (step S4), and at the same time, is corrected according to the user's correction instruction input from the correction unit 26 (step S5). Specifically, the data of the recognition result candidate code of the document image constituent element is corrected.

  In the correction of the recognition result candidate code data by the correction unit 26, an operation for selecting a document image constituent element to be corrected is required. The selection of a document image component is generally performed by a coordinate instruction using a pointing device or the like on the display screen. For this reason, it is necessary to determine the image range (region) of the document image constituent element in advance.

  As the image range of the document image constituent element, the black pixel connection area of the document image constituent element can be used. For example, in the case of a document image constituent element as shown in FIG. 5, the black pixel portion constituting the image “M” is the image range.

  A circumscribed rectangle of the document image constituent element can also be used as the image range of the document image constituent element. For example, in the case of a document image constituent element as shown in FIG. 5, as shown in FIG. 6, the circumscribed rectangle of the black pixel connection area of the image “M” is the image range. If such an image range is used, even when the white pixel portion around the black pixel portion is designated, the corresponding document image constituent element can be designated, and there is an advantage that it is easier to designate than the black pixel portion.

  Since document image components include those that are conceptually hierarchically related, such as a text region and a character region included in the document image component, the corresponding document image configuration is based on one designated coordinate on the image. An element may not be uniquely determined. In general, when two document image components are in a conceptual hierarchical hierarchical relationship, their regions are in an inclusive relationship. For example, in the case of the document image component shown in FIG. 7, the text area includes a character area, and the table area includes a ruled line area and a character area.

  In such a case, in order to uniquely determine the document image constituent element by one designated coordinate, it is considered that the innermost document image constituent element is designated among the plurality of document image constituent elements including the designated coordinate. To.

  For example, in the image shown in FIG. 8, the document “media” that is the document image component further includes four document image components “me”, “de”, “i”, and “a”. Has been. Here, the image range of each document image constituent element is defined by its circumscribed rectangle. In this example, when the user designates the coordinates of a point in the circumscribed rectangle 31 of the document image component “M”, the document image component “M” is detected as an instruction target.

  Further, it can be considered that the outermost document image constituent element including the designated coordinates is designated. In the example of FIG. 8, regardless of the point in the circumscribed rectangle 32 of the sentence “media”, “media” is detected as an instruction target. Therefore, even when a point in the circumscribed rectangle 31 of “Me” is designated, “Media” is designated.

  It can also be considered that the document image constituent element whose circumscribing frame is closest to the designated coordinates is designated. In the example of FIG. 8, a perpendicular line is drawn from the designated point on each circumscribed rectangle side of five document image components such as “media” and “me”, and the length is between the designated point and each circumscribed rectangle. It is calculated as a distance. Then, the document image constituent element corresponding to the circumscribed rectangle with the shortest distance to the indication point is detected as the indication target. According to this method, any document image constituent element may be an instruction target without being restricted by the inclusion relationship.

  By the way, in the instruction method as described above, one document image constituent element can be selected from a plurality of document image constituent elements whose areas are in an inclusive relationship, but it is not possible to directly select other document image constituent elements. Can not. Therefore, it is necessary to provide a process for indirectly selecting another document image constituent element from already selected document image constituent elements.

  Therefore, a selection operation for designating the innermost one among the other document image components including the already specified document image component is provided. In the example of FIG. 8, when the user performs this selection operation in a state in which “M” has already been instructed, “Media” on the outside thereof is instructed.

  In addition, a selection operation may be provided that designates the largest document image constituent element including the already designated coordinates among other document image constituent elements in the already designated document image constituent element. In the example of FIG. 8, it is assumed that the document image component “media” is instructed by instructing the coordinates in the “me” area. In this state, when the user performs this selection operation, “me”, which is the largest document image constituent element including the designated coordinates in the “me” region, is designated.

  Next, as a method of displaying the type code and individual code of the region of the document image constituent element as a recognition result candidate, a binary display method in which the type code and the individual code are individually displayed as shown in FIG. 9 is considered. It is done. In FIG. 9, since the first place of the recognition result candidate is “character”, character recognition is performed and an individual code of the character recognition result is displayed. However, in this display method, the relationship between the type code and the individual code is often difficult to understand intuitively.

  Therefore, in the recognition of the type of document image component by the individual area recognition unit 23, when the first recognition result candidate is “character”, character recognition is performed, and the candidate character code obtained as a result is shown in FIG. As shown, it is displayed as a higher recognition result candidate code. Then, the type codes of the document image constituent elements such as “table”, “figure”, “frame” other than “character” are displayed as lower recognition result candidate codes. In this way, by displaying the type code and the individual code as a single list, the recognition result candidates can be recognized at a glance.

  In such a one-way display method, when the first recognition result candidate is a type such as “table”, “figure”, or “frame” other than “character”, it is obtained by character recognition as shown in FIG. The obtained candidate character code is displayed as a lower recognition result candidate code. In this case, it is unlikely that the recognition result is “character”, and a process for obtaining a character recognition result with a low degree of correctness is executed.

  Therefore, when the first recognition result candidate is a type other than “character”, the type of the document image constituent element is set as the first recognition result candidate code, and only the type codes of other document image constituent elements are subordinate. May be a recognition result candidate code. In this method, as shown in FIG. 12, only the types of document image constituent elements are displayed as candidates, and candidate characters for character recognition are not displayed.

  The reason why such a display is used is to make the display easier to view without displaying the second and lower detailed information. In general, since there is a high probability that the first place of the recognition candidate is correct, detailed information below the second place is often unnecessary as a result. Furthermore, this method eliminates the need for recognition processing corresponding to the second or lower recognition result candidate such as “character”, and speeds up the processing.

  The user can see the recognition result candidates displayed in this way and correct them. For example, as shown in FIG. 13, when “table” which is the first type code is instructed to be corrected to “character”, the rank of the type codes other than “character” is shifted downward one by one. Then, if necessary, character recognition of the corresponding input character image is performed, and “characters” in the recognition result candidate code are replaced with one or a plurality of candidate character codes obtained as a result.

  In addition, when a type code such as “table”, “figure”, or “frame” other than “character” is instructed as a recognition result candidate code, it is instructed to the corresponding input character image as necessary. Recognition processing as a document image constituent element is performed.

  For example, as illustrated in FIG. 14, when the “type” that is the first type code is instructed to be corrected to a “table” or “enclosure” having an internal structure, the indicated type code “table” And the internal structure of the “box” is recognized. In the process of recognizing the internal structure of the “table”, ruled lines are extracted and vectorized, cells surrounded by the ruled lines are extracted, and characters in the cells are recognized. In the process of recognizing the internal structure of the “enclosed frame”, extraction and vectorization of the frame, area identification within the frame, and the like are performed.

  FIG. 15 is a second block diagram of the document image processing apparatus of the present invention. 15 includes a document image input unit 41, a document image component extraction unit 42, a code addition unit 43, an edit data storage unit 44, a document data storage unit 45, a display unit 46, a code document input unit 47, An editing operation input unit 48 and an editing unit 49 are provided, and processing as shown in FIGS. 16 and 17 is performed.

  First, the document image input unit 41 inputs a digital image obtained by digitizing a document (step S11). The document image input unit 41 is, for example, a digital scanner device. Next, the document image constituent element extraction unit 42 extracts the document image constituent elements constituting the document image (step S12).

  The code adding unit 43 adds a new code to the extracted document image constituent element (step S13), and stores the document image constituent element to which the new code is added in the editing data storage unit 44 (step S14). If necessary, the code document input unit 47 inputs an existing electronic document (electronic code document) to the edit data storage unit 44 (step S15). The data of an existing electronic document is a set of existing codes and corresponds to an existing character pattern. Therefore, the edit data stored in the edit data storage unit 44 includes two types of data as shown in FIG.

  In the editing data storage unit 44, the document image constituent element is represented by image data such as a bitmap, and an external character code or the like is used as the new code. Since the new code is automatically added to the bitmap data of the document image component, the user does not need to design the character shape or the like as in the case of normal external character registration. An existing character pattern is represented by font data or the like.

Furthermore, if necessary, the document data that has already been edited and stored in the document data storage unit 45 can be read out and used in the editing data storage unit 44 (step S16).
Next, the display unit 46 displays the document to be edited on the screen using the data in the editing data storage unit 44 (step S17). Image data is used to display document image components, and font data is used to display existing character patterns.

  When a document editing operation by the user is input to the editing unit 49 from the editing operation input unit 48 (step S18), the editing unit 49 edits the data in the editing data storage unit 44 (step S19). At this time, a set of the new code and the existing code becomes a direct object of the editing process, and image data and font data are used for the display process.

  When a document image component is copied or moved, the user gives an instruction on the displayed image using a pointing device or the like. In response to this, the editing unit 49 performs the instructed editing process on the new code corresponding to the instructed document image constituent element. In this way, the new code corresponding to the document image component is automatically processed by the system, so the user does not have to deal with it directly.

  The document data for which the editing operation has been completed is stored in the document data storage unit 45 from the editing data storage unit 44 (step S20), and the process ends. The document data stored in the document data storage unit 45 can be read into the edit data storage unit 44 and re-edited.

  For example, it is assumed that a code document as shown in FIG. 19 is printed on a paper medium and is input as an image from the document image input unit 41 after being copied or faxed several times. Since the print quality deteriorates during this process, the input image becomes a document image as shown in FIG.

  The document image component extraction unit 42 extracts document image components as shown in FIG. 21 from the document image. Here, each extracted document image component is displayed on the screen using a circumscribed rectangle. Each document image component is given a new code by the code adding unit 43 so that it can be edited in units of document image components.

  The user can create an image as shown in FIG. 22 by changing the order of the document image constituent elements included in the displayed image. Here, the image of “multimedia system” is edited to “media multimedia system”.

  Further, the mixed editing of the code document input from the code document input unit 47 and the document image constituent element can be performed. For example, when editing is performed by replacing the document image component “multi” with the code document “integrated”, a document as shown in FIG. 23 is generated. After all editing is completed, as shown in FIG. 24, the display of the document image component by the circumscribed rectangle is canceled. The edited document is printed or recognized according to a user instruction.

  In this way, by associating a new code with the document image component image and displaying the display information of the code on the display unit 46, it is possible to edit a document in which the existing code and the new code are mixed. Become.

  The display unit 46 uses a reduced or enlarged image including a normal image of the document image component image as display information of a new code corresponding to the document image component image. Since the document image constituent element images have various sizes, it is more convenient to display the document image by changing the size in order to edit it together with existing document data.

  The most common character image as the document image component image is displayed with the same size as the existing character pattern. For this purpose, the character image is reduced or enlarged and displayed. In FIG. 24, the size of “media” and “system” as character images and the size of “integrated” as existing character patterns are unified and displayed by this method.

Further, when the character image is reduced and displayed while maintaining the size and consistency of the existing character pattern, the result is as shown in FIG. 25, and further reduced as shown in FIG.
However, when the document image component image is enlarged or reduced with the image data as it is, the shape may be collapsed and displayed. Therefore, when outline data (outline font) obtained by vectorizing the outline of the document image constituent element image is used for displaying the document image constituent element, it is possible to prevent the shape from collapsing even when enlarged or reduced.

  For example, when the outline data obtained by vectorizing the character image “M” is enlarged or reduced, a display as shown in FIG. 27 is obtained. In FIG. 27, it can be seen that the shapes of all the characters are similar.

  By the way, when the recognition processing of the edited document image is performed, the document data is input to the document processing apparatus of FIG. Then, after performing the region identification and recognition processing as described above, the recognition result candidates are displayed and corrected.

  In the conventional recognition result display method, as shown in FIG. 39, the character pattern of the recognition result candidate code and the corresponding character image are displayed separately. For this reason, there is a problem that the line of sight movement for confirmation is large, which is a burden on the user.

  Therefore, in the present embodiment, as shown in FIG. 28, the corresponding document image component image is displayed in the recognition result candidate display area (display window) 51 together with the recognition result candidate character pattern. In this way, if the document image component is separated from the input document image and displayed near the recognition result candidate, the line-of-sight movement is greatly reduced, and the user can easily confirm the validity of the recognition result. .

  Similarly, when the document image component is other than a character, the original image can be displayed in the display area of the recognition result candidate. For example, in the case of the recognition result 52 as shown in FIG. 29, the document image constituent element image 53 is displayed in the display area 51 together with “table”, “enclosed frame”, and the like representing the type code of the recognition result candidate. Thereby, the user can easily confirm the validity of the recognition result 52.

  Also, the document image component image displayed in this way can be used for correction. In the recognition result display of normal character recognition, there are cases where the correct answer is not included in the candidates. In such a case, when the user selects a document image constituent element image, the candidate character code in the recognition result is replaced with the code of the document image constituent element. The document is displayed.

  For example, in the recognition result as shown on the left side of FIG. 30, when the correct candidate “information” is not included in the displayed candidate characters, the user selects the document image component image 54 and issues a correction instruction. . Thereby, the recognition result is corrected and displayed using the image 54 as shown on the right side.

  In addition, in the recognition result of the table as shown on the left side of FIG. 31, when the displayed recognition result 52 is not correct, the user selects the document image component image 55 and issues a correction instruction. Thereby, the recognition result is corrected and displayed using the image 55 as shown on the right side.

  As described above, when the document image component image in the recognition result candidate display can be instructed to be corrected and the user selects it, editing processing is performed using a code corresponding to the document image component image. Therefore, at least on the display, a valid recognition result candidate is always included.

  Further, by holding the document image constituent element type attributes such as “character”, “table”, “figure”, “frame” in the document image constituent element image, handling according to the type attribute becomes possible. The type attribute is stored in the recognition result storage unit 24 of FIG. 2 together with the code of the document image constituent element image. In the recognition result, the first recognition candidate usually has the highest accuracy. Therefore, an attribute corresponding to the type of the code is automatically added to the document image constituent element image as a type attribute.

For example, in the recognition result as shown on the left side of FIG. 32, since the type attribute of the first “惰” is “character”, the attribute of the document image component image 54 is also “character”.
In such recognition result display, the user can select a document image constituent element image and perform re-recognition processing according to the type attribute. In the example of FIG. 32, when the user selects and corrects the document image component image 54, as shown on the right side, the corresponding character pattern is replaced with the document image component image 54, and character recognition is performed again. Is called.

  By the way, even if the type attribute corresponding to the first recognition candidate is given to the document image constituent element image, it may be wrong. Therefore, an external interface unit for instructing correction of the type attribute of the document image constituent element image is provided in the correction unit 26 of FIG. The external interface unit changes the type attribute of the document image constituent element image in accordance with a user instruction. For example, in FIG. 33, the attribute of “information” which is a document image component image is changed from “table” to “character”.

  In the recognition process, a distance value between each candidate code of the recognition result and the recognition dictionary is often calculated to give a certainty degree to each candidate code. Therefore, in the present embodiment, a certain degree of certainty is given to the document image constituent element images, and ranking is performed in order from candidate codes or document image constituent elements having a high certainty degree. Even if the certainty of the candidate code that is the original recognition result candidate is low, if such ranking is performed, the document image constituent element image becomes the first recognition result candidate, so that the recognition result document is appropriate. Is displayed.

  34 and 35 show an example in which recognition result candidates are displayed together with certainty factors. In FIG. 34, the certainty factor of “information”, which is a document image constituent element image, is 60, and this is displayed as the second candidate. On the other hand, in FIG. 35, the certainty factor of “symbol” is 70, which is displayed as the first candidate.

  As described above, the ranking is different depending on the relative certainty difference between the document image component image and the other candidate codes. Therefore, the certainty factor of the document image constituent element image can be used as a threshold value for replacing the original recognition result candidate with the document image constituent element image. Therefore, an external interface unit for inputting the certainty factor of the document image component image is provided in the correction unit 26 in FIG. 2 so that the user can adjust the threshold value.

  The correction unit 26 compares the certainty factor of the document image constituent element image with the certainty factors of other candidates, and updates the first display of the recognition result candidate if necessary. Thus, the user can interactively adjust the certainty threshold while confirming the result.

  At this time, the display unit 25 of FIG. 2 may display a predetermined number of recognition result candidates as shown in FIGS. 34 and 35, but sets a window for displaying only the first candidate. Also good. By displaying only the first candidate, the movement of the user's line of sight is reduced, so that the threshold adjustment operation is made efficient.

  The document image processing apparatus of the present embodiment can be configured using an information processing apparatus (computer) as shown in FIG. 36, for example. 36 includes a CPU (central processing unit) 61, a memory 62, an input device 63, an output device 64, an external storage device 65, a medium driving device 66, a network connection device 67, and a photoelectric conversion device 68. They are connected to each other by a bus 69.

  The memory 62 stores programs and data used for document image processing as described above. As the memory 62, for example, a read only memory (ROM), a random access memory (RAM), or the like is used. The CPU 61 performs necessary processing by executing a program using the memory 62.

  The input device 63 is, for example, a keyboard, a pointing device, a touch panel, and the like, and is used for inputting instructions from the user and necessary information. The output device 64 is, for example, a display or a printer, and is used for outputting processing results.

  The external storage device 65 is, for example, a magnetic disk device, an optical disk device, a magneto-optical disk device, or the like. The above-described program and data can be stored in the external storage device 65, and loaded into the memory 62 for use as necessary.

  The medium driving device 66 drives the portable recording medium 70 and accesses the recorded contents. As the portable recording medium 70, any computer-readable recording medium such as a memory card, a flexible disk, a CD-ROM (compact disk read only memory), an optical disk, or a magneto-optical disk is used. The above-described program and data can be stored in the portable recording medium 70, and can be loaded into the memory 62 and used as necessary.

  The network connection device 67 communicates with an external device via an arbitrary network (line) such as a LAN (local area network) and performs data conversion accompanying the communication. Thereby, the document image processing apparatus can receive the above-described program and data from an external apparatus and use them by loading them into the memory 62 as necessary.

The photoelectric conversion device 68 is a digital scanner, for example, and inputs an image of a document described on a paper medium.
FIG. 37 shows a computer-readable recording medium that can supply a program and data to the information processing apparatus of FIG. Programs and data stored in the portable recording medium 70 or the external database 71 are loaded into the memory 62. Then, the CPU 61 executes the program using the data and performs necessary processing.

1 is a principle diagram of a document image processing apparatus of the present invention. It is a 1st block diagram of a document image processing apparatus. It is a flowchart of a 1st document image process. It is a figure which shows a kind code and an individual code. It is a figure which shows the 1st image range. It is a figure which shows the 2nd image range. It is a figure which shows the hierarchy of a document image component. It is a figure which shows the document image component which is in an inclusive relationship. It is a figure which shows the 1st display of a recognition result. It is a figure which shows the 2nd display of a recognition result. It is a figure which shows the 3rd display of a recognition result. It is a figure which shows the 4th display of a recognition result. It is a figure which shows the 1st correction instruction | indication. It is a figure which shows the 2nd correction instruction | indication. It is a 2nd block diagram of a document image processing apparatus. It is a flowchart (the 1) of the 2nd document image processing. It is a flowchart (the 2) of the 2nd document image processing. It is a figure which shows edit data. It is a figure which shows a code document. It is a figure which shows a document image. It is a figure which shows the extracted document image component. It is a figure which shows the 1st display of an edit result. It is a figure which shows the 2nd display of an edit result. It is a figure which shows the 3rd display of an edit result. It is a figure which shows the 1st reduction display. It is a figure which shows the 2nd reduction display. It is a figure which shows an outline display. It is a figure which shows the 5th display of a recognition result. It is a figure which shows the 6th display of a recognition result. It is a figure which shows the 3rd correction instruction | indication. It is a figure which shows the 4th correction instruction | indication. It is a figure which shows the 5th correction instruction | indication. It is a figure which shows the 6th correction instruction | indication. It is a figure which shows the 7th display of a recognition result. It is a figure which shows the 8th display of a recognition result. It is a block diagram of information processing apparatus. It is a figure which shows a recording medium. It is a block diagram of the conventional document image recognition and editing apparatus. It is a figure which shows the conventional recognition result display.

Explanation of symbols

1, 21, 41 Document image input unit 2, 22 Region identification unit 3 Region identification result storage unit 4, 23 Individual region recognition unit 5, 24 Recognition result storage unit 6 First modification unit 7 Second modification unit 8, 25 46 Display unit 11 Identification unit 12 Recognition unit 13 Output unit 14 Correction unit 15 Extraction unit 16 Code addition unit 17 Editing unit 26 Correction unit 31 and 32 circumscribed rectangle 42 Document image component extraction unit 43 Code addition unit 44 Edit data storage unit 45 Document Data Storage Unit 47 Code Document Input Unit 48 Editing Operation Input Unit 49 Editing Unit 51 Display Area 52 Recognition Results 53, 54, 55 Document Image Component Image 61 CPU
62 memory 63 input device 64 output device 65 external storage device 66 medium drive device 67 network connection device 68 photoelectric conversion device 69 bus 70 portable recording medium 71 database

Claims (10)

Extraction means for extracting image components constituting the input image from the input image;
Recognizing means for recognizing the image components;
An image processing apparatus comprising: output means for separating image data corresponding to the image component from the input image and outputting the image data together with one or more candidates in the recognition result of the image component.   A selection means for selecting one of the one or more candidates and image data; and when the image data is selected, new code information is added to an image component corresponding to the image data, 2. The image processing apparatus according to claim 1, further comprising editing means for editing a document in which image data and character patterns corresponding to existing code information are mixed.   The image processing apparatus according to claim 1, wherein the recognition unit adds type information to the image constituent element.   The image processing apparatus according to claim 3, wherein the recognition unit adds type information corresponding to a first candidate in the recognition result to the image constituent element.   The image processing apparatus according to claim 3, further comprising a correcting unit that corrects the type information of the image component.   The recognizing unit obtains the certainty factor of the one or more candidates, and the output unit gives a predetermined certainty factor to the image data corresponding to the image constituent element, and the one or more candidates and the image data are obtained. The image processing apparatus according to claim 1, wherein the images are ranked and output in descending order of certainty.   The image processing apparatus according to claim 6, further comprising an input unit configured to input a certainty factor of image data corresponding to the image constituent element.   The output means outputs the first rank information among the one or more candidates and image data, and changes the first rank information according to the certainty factor inputted by the input means. The image processing apparatus according to claim 7. A recording medium recording a program for a computer,
A function of extracting an image component constituting the input image from the input image;
A function of performing recognition processing of the image component;
A computer having recorded thereon a program for causing the computer to realize a function of separating image data corresponding to the image component from the input image and outputting together with one or more candidates in the recognition result of the image component A readable recording medium. Enter the image into the computer,
Extracting image components constituting the input image from the input image,
Recognizing the image component,
An image processing method comprising: separating image data corresponding to the image component from the input image and displaying the image data together with one or more candidates in the recognition result of the image component.

Images