JP2017199288A - Image processing device, image processing method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To solve a problem that in modification processing of character recognition from a document, there is a big burden of a modification work operation for a modification worker, and it is impossible to easily instruct a part to be corrected and the manner to correct the part.SOLUTION: An image processing device images a paper document, performs character recognition of an area on which characters are arranged, in an imaged image, extracts a character string in question from the character string of the recognized result, displays data of the extracted character string and attribute data of the character string, acquires an indication position to at least either of the displayed data and the paper document, and performs different modification processing based on the acquired indication position when character string data displayed by display means is modified.SELECTED DRAWING: Figure 3

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program that support correction of character recognition.

  Information extraction processing that recognizes a paper document using OCR (optical character recognition) and obtains electronic character data is a very important technology that enables electronic processing of a paper document. For example, it is possible to automate accounting processing and tax refund processing by linking paper documents such as invoices and delivery notes with corporate systems. However, recognition techniques such as OCR are difficult to guarantee the recognition contents, and it has been necessary to manually confirm and correct the recognition results. Therefore, a method for reducing the burden of correction has been proposed. For example, Patent Document 1 proposes a method of reducing the burden on the user by reflecting the correction on the same character when the user corrects the misrecognized character.

  On the other hand, in the past, it was common to read a document using a scanner and operate using a pointing device such as a mouse or a keyboard. However, in recent years, a camera or the like is used instead of a contact-type reading device such as a scanner. However, an apparatus for projecting with a projector has been proposed. Patent Document 2 proposes an apparatus configuration in which a document is placed upward on a platen, the document is read in a non-contact manner using an imaging device mounted on the top of the platen, and the read electronic data is projected by a projection device. Yes. By using such an apparatus configuration, it is possible to perform a document reading operation and a confirmation and correction operation as a series of operations, and the application range of the OCR technique can be widened. For example, in a face-to-face business negotiation, the result of reading and digitizing a document placed on a desk can be immediately projected and confirmed, so that the time required for business negotiation can be reduced.

JP-A-5-20492 JP2015-41233

  However, in the method of Patent Document 1, it is necessary to instruct the character of the correction content using an instruction device such as a keyboard. When the hardware keyboard cannot be prepared due to limitations on the work space and hardware configuration, and there is a burden on the input of characters on the software keyboard itself, the burden on the operation of the correction operator is large. In addition, it is not possible to easily instruct which part the correction operator wants to correct.

  The present invention has been made in view of the above problems, and an object thereof is to provide an image processing apparatus capable of easily correcting a character recognition result. It is also intended to provide the method and program

  An image processing apparatus according to the present invention has the following configuration. That is, an imaging unit that captures a paper document, a recognition unit that recognizes a region where a character is arranged in an image captured by the imaging unit, and a character string that is a target from a character string that is a recognition result of the recognition unit Extraction means for extracting the character string data, display means for displaying the character string data extracted by the extraction means and attribute data for the character string, data displayed by the display means, and at least one of the paper document Acquisition means for acquiring an indicated position for the character, and correction means for performing different correction processing based on the indicated position acquired by the acquisition means when correcting the character string data displayed by the display means.

  According to the present invention, the configuration including the photographing device and the projection device can assist the correction operator to easily perform the correction work when correcting the character recognition result.

Example of appearance when using the image processing apparatus according to the embodiment Example of hardware configuration in an embodiment The flowchart which shows the flow of the whole process in embodiment Flowchart showing confirmation / correction processing in the embodiment Example of document image and extraction result in the embodiment Example of UI display in the embodiment Example showing angle of view during re-shooting process in the embodiment Example showing the effect of re-shooting processing in the embodiment Example of UI display in the embodiment Example explaining touch determination of UI operation in the embodiment

  Embodiments of the present invention will be described below with reference to the drawings.

<First Embodiment>
First, an outline of processing of an image processing apparatus (hereinafter also referred to as “document processing apparatus” or “photographing / projection apparatus”) in the present embodiment will be described with reference to FIG.

The work table 101 is a work table such as a table or a counter for a user to perform work.
The photographing / projecting apparatus 102 is an image processing apparatus having a function of photographing a work table 101 and a document 103 placed on the work table, and projecting and displaying the photographed contents and UI on the work table 101. A document 103 on the work table 101 is a paper document such as an identity verification document or an application form, and is a document entity that is a target of information extraction processing using the image processing apparatus.

  FIG. 2 is an example of a hardware configuration of the photographing projection apparatus 102 according to the embodiment of the present invention.

  A CPU (Central Processing Unit) 201 performs calculations and logic determinations for various processes, and controls various elements connected to the bus 202.

  The memory 203 holds a program memory and a data memory. The program memory stores a program for control by the CPU including various processing procedures described later with reference to flowcharts. This memory may be a ROM (Read-Only Memory) or a RAM (Random Access Memory) in which a program is loaded from an external storage device or the like. Or you may implement | achieve with these combination.

  The storage device 204 is a device such as a hard disk for storing data and programs according to the present embodiment.

  The photographing device 205 is a device having a lens and an image sensor, photographing light incident from the lens and developing it into image data. In the present embodiment, the work table 101 is mainly photographed and used to acquire image data for extracting information of the document 103.

  The input device 206 is a device for inputting an instruction from the user. The user gives an instruction to the photographing / projecting apparatus via the input device 206. The input device 206 can be realized by, for example, a distance sensor that can detect a touch instruction on a desk by a user based on a distance between the desk and a finger, or a visible light sensor that can detect a user's gesture instruction. Other sensors, touch panels, hardware such as a pen and a mouse may be used.

  The projection device 207 is a device that displays a UI. The projection device 207 can be realized by a projector device using a liquid crystal display or a laser display, for example.

  The configuration described in FIGS. 1 and 2 is an example, and may be realized in other forms. For example, the imaging device 205 and the projection device 207 may exist as independent hardware, or each hardware may operate in cooperation with a USB or a network cable.

  FIG. 3 is a flowchart for explaining the processing flow of the photographing projection apparatus in the present embodiment. Hereinafter, the flowchart is realized by the CPU executing the control program.

In step S301, the imaging device 205 images the work table 101 including the document 103, generates a captured image, and proceeds to step S302.
In step S302, the CPU 201 detects the area of the document 103 from the captured image generated in step S301, and proceeds to step S303. The document includes an area where characters are arranged.

  For the detection of the document area, for example, a method of extracting the sides of the paper by straight line extraction and estimating the four sides may be used. The straight line extraction may be performed by converting a captured image into a gray scale, applying a known edge enhancement such as a Sobel or Laplacian filter, and extracting a line segment by a known method such as a Hough transform or a minimum approximation method. In addition, if shooting conditions such as an angle of view and illumination are fixed, the work table 101 in a state where the document 103 does not exist may be shot in advance, and a pixel difference from the shot image may be taken. Alternatively, it may be detected by a dedicated sensor device and mapped to coordinates on the captured image.

  In step S303, the CPU 201 corrects the area of the document 103 obtained in step S302 into a rectangular shape as a document image, and proceeds to step S304. The region of the document 103 obtained in step S302 has an unequal side rectangle due to a tilt due to the positional relationship between the photographing apparatus 205 and the work table 101. What is necessary is just to use what is called reverse perspective transformation calculation for the calculation which correct | amends an unequal side rectangle to a rectangular shape. The parameters of the transformation matrix are extracted by solving the simultaneous equations obtained by giving the coordinates of the four vertices to the inverse perspective transformation formula, as disclosed in Japanese Patent Laid-Open No. 2003-288588 (Patent Document 3). it can. In addition, if the imaging device 205 is always at a certain angle with respect to the table, the parameters for inverse perspective transformation may be given in advance.

  In step S304, the CPU 201 performs character recognition processing on the document image obtained in step S303 to obtain a recognized character string, and proceeds to step S305.

  In step S305, the CPU 201 extracts character string information having a desired attribute from the character recognition result obtained in step S304 according to a predetermined rule, and proceeds to step S306. The character string information of a desired attribute is information indicating a specific meaning such as individual contents for attributes such as a driver's license, a name of a resident card, and an address, and contents described in an application form.

  As a method of obtaining the character string information corresponding to the attribute, a character string existing at a specific coordinate in the document may be acquired if the document has a fixed layout. If the layout cannot be specified, candidate character strings may be narrowed down by regular expressions or dictionaries from the document recognition results, or a character string with a name indicating an attribute is searched and a character string is specified from the vicinity. May be. When there are a plurality of candidates for the character recognition result or when a plurality of searches are performed by neighborhood search for attribute names, each is extracted as a candidate character string. For a plurality of candidate character strings, the order of candidates and the reliability may be determined by a rule, or the order may be determined by the probability of character recognition. This method is an example, and other methods may be used as long as the target character string and its coordinates can be acquired. In the example of the document 103, it can be seen that the attribute “name” has character string information “Taro Yamada”.

  In step S306, the CPU 101 presents the correction support UI to the user with respect to the character string information obtained in step S305, corrects the character string information based on the user input, and determines the character string information for a desired attribute. As will be described later with reference to FIG. 6, the result of character recognition and extraction is corrected and confirmed. Details of the processing in step S306 will be described later with reference to the flowchart of FIG.

A document image 501 in FIG. 5A is an example of a document image obtained as a result of processing from step S301 to step S303 by placing a resident card on the work table 101 as the document 103.
The purpose of this embodiment is to extract a householder's name (hereinafter referred to as name attribute) and address (hereinafter referred to as address attribute) from the resident card. In the example of the document image 501, a character string “Taro Yamada” of the name attribute 502 and a character string “XXX Shimomaruko XXX, Tokyo Ota-ku” are expected as correct values.

  As an example of the information extraction method in step S305, the character string “household owner” is extracted from the document, and the character string existing in the right-hand vicinity region is extracted as the name attribute. Then, a character string “address” is extracted from the document, and a character string existing in the right-hand vicinity region is extracted as an address attribute.

The information extraction result 504 in FIG. 5B is a diagram showing the extraction result obtained as a result of the processing in step S304 and step S305 in a tree structure for the document image 501. X and Y coordinates with the extracted result as the root, name attribute and address attribute below it, multiple candidates below it, character string and its value below it, rectangular coordinate and its value as the origin at the top left of the document , W (width), and H (height) size.
In the extraction result 504, as the first candidate 505 for the name attribute, the character string “Taro Yamada” and the rectangular coordinates X, Y, W, and H are extracted. Similarly, three candidates 506, 507, and 508 are extracted as address attributes.

  Hereinafter, the document image 501 and the information extraction result 504 will be described as examples of input to step S306, and the processing will be described using the flowchart of FIG.

  In step S401, the CPU 201 obtains relative coordinates of the document 103 with respect to the projected video area in order to perform UI display on the photographing projection apparatus 102, and the process proceeds to step S402. In the present embodiment, for the sake of explanation, it is assumed that the surface of the work table 101, the projection area of the projection device 207, and the document photographing area in step S301 coincide for convenience. That is, the two-dimensional coordinates of the surface of the work table 101 coincide with the coordinates on the captured image acquired in step S301 and the coordinates on the projection screen projected by the projection device 207.

  Therefore, if the position of the document 103 on the work table 101 has not changed from the time of step S301, the document area coordinates on the captured image obtained by document detection in step S302 can be used as they are. If the position of the document 103 on the work table 101 has moved from the time of step S301, the position of the document is specified by performing the same processing as in steps S301 and S302 again.

  Actually, since the document photographing area of the photographing apparatus 205 is different from the projection area of the projecting apparatus 207, coordinate conversion is required. As a method of coordinate conversion, if the positional relationship between the image capturing device 205 and the projection device 207 of the image capturing / projecting device 102 is known in advance, mapping may be appropriately performed with a geometric conversion parameter. In addition, the coordinates may be adjusted by projecting a marker for alignment with the projection device 207 and photographing and recognizing with the photographing device 205. This method is an example, and other methods may be used.

  In step S402, the CPU 201 displays / updates a UI for confirmation / correction support, and proceeds to step S403.

  The basic configuration of the UI uses the document 103 as a UI component for display as it is, and the extraction result, the correction field, and a completion button for instructing confirmation / correction completion exist in other spaces of the work table 101. This is an example, and there may be other UI parts, or the UI parts may be omitted and input by a user operation such as a gesture.

  FIG. 6A shows a display example of the UI here. A document 601 is an entity of the document 103. The character string of the first candidate 505 having the name attribute of the information extraction result 504 is projected by the projection device 207 as the UI display 602 in the right space. A character string of the first candidate 506 having an address attribute is similarly projected as a UI display 603. The UI displays 602 and 603 also serve as input fields for correction input. The completion button 604 is projected as a UI part that notifies the completion of the confirmation / correction work. A broken-line rectangle 605 is a highlight display showing a current selection state for confirming / correcting the name attribute. Here, a state in which the name attribute value is selected as a correction target in advance as an item that needs to be confirmed from the reliability of the information extraction result 504 is shown, but may be initially unselected. A broken-line rectangle 606 is a highlighted display showing the rectangular coordinates on the paper surface of the currently selected name attribute, that is, the rectangular coordinates of the first candidate 505, and is projected on the paper of the document 601 in a superimposed manner.

  In step S403, the CPU 201 waits for input from the user. If there is an operation from the user, the CPU 201 acquires the operation content, and proceeds to step S404. In the description of the present embodiment, the operation from the user is an operation in which any part on the workbench is touched with a finger as a user operation, and the touch coordinates (instructed position) are acquired. This is an example, and may be a mouse operation, a pressing operation using a separate input sensor, or other operations using a gesture.

  In step S404, the CPU 201 branches the UI process based on the operation content obtained in step S403. If an input field other than the currently selected attribute, that is, the UI display 603 of the address attribute is touched and selected, the process proceeds to step S405. In step S405, the correction target attribute is changed to the address attribute, and the process proceeds to step S402. If a specific character string of the document 601 is touched and selected as the correction character string of the currently selected attribute, the process proceeds to step S412 to acquire the character string at the touched position. In step S413, the correction target character string The attribute character string information is corrected to the character string acquired in step S412, and the process proceeds to step S402. Details of the processes in steps S412 and S413 will be described later.

  If the input field of the currently selected attribute, that is, the UI display 602 of the name attribute is touched, the process proceeds to step S406.

  In this example, it is desired to obtain “Taro Yamada” as the correct value of the name attribute, but the first candidate for name attribute 505 is correct in the extracted rectangular coordinates, but the character string is “Taro Yamada” due to misrecognition of character recognition. It has become. In order to correct this, it is assumed that the input field of the name attribute extraction result 602 is touched, and the following description will be continued.

  In step S406, the CPU 201 changes to the character input mode and proceeds to step S407. Shifting to the character input mode while the input field is touched is a general operation in a PC or the like, and a portable terminal without a hardware keyboard is in a state where characters can be input by displaying a software keyboard. . The description of this embodiment also uses a software keyboard. This is an example, and characters may be input by other methods such as voice.

In steps S407 to S410, it is estimated that the user has an intention to correct the character input. That is, the character recognition result is misrecognized, and it is interpreted that a correction operation is required, and the character string is photographed again for recognition.
In step S407, the CPU 201 acquires the rectangular coordinates of the first candidate 505 for the name attribute to be corrected, and the process proceeds to step S408. The region indicated by the highlight display 606 in FIG. 6A corresponds to this.

  In step S408, the CPU 201 sets the shooting conditions of the shooting device 205, and proceeds to step S409. The character area acquired in step S407 is photographed under conditions more favorable than in step S301. In this embodiment, the focal length is increased using optical zoom, that is, the resolution of the target character string is improved by performing telephoto zoom.

  FIG. 7 is an image of a shooting angle of view. In the present embodiment, since the shooting angle of view in step S301 coincides with the work table 101 as described above, the same surface 701 is the shooting angle of view. An angle of view 702 indicated by a broken line is an angle of view of the focal length set in step S408. If the lens extension distance and the like are ignored, the resolution per character string is quadrupled when the focal length is doubled.

In step S409, the imaging device 205 performs imaging, acquires a captured image, and proceeds to step S410. Shooting is performed so that the angle of view 702 includes the region 606.
In step S410, the CPU 201 extracts a character area from the captured image, and proceeds to step S411.
The character area is extracted by combining the character area acquired in step S407 with the coordinates of the captured image captured in step S409. If there is distortion due to the shooting angle, it is also corrected here.

  FIG. 8 shows an example of the extracted character area image. The character area image 801 is an example of the character area image obtained by extracting the target character string from the captured image acquired in step S301, and the character area image 802 is an example of the character area image acquired in step S410. In the character area image 801, some pixels that are dots of “thick” characters are missing due to lack of resolution, but the character area image 802 can be acquired properly.

  In step S411, the CPU 201 performs character recognition on the character area image 802 acquired in step S410, and proceeds to step S402. Since the character area image 802 has sufficient resolution for character recognition, the character recognition succeeded and the character string “Taro Yamada” was obtained.

In step S402, the CPU 201 updates the correction UI based on the character recognition result acquired in step S411, and proceeds to step S403.
In the present embodiment, for the sake of explanation, it is expressed as a sequential process according to a flowchart, but an input by a user may be accepted in parallel during steps S407 to S411.

FIG. 6B shows an example of display of the modified UI here.
A software keyboard 607 is displayed in step S406. The correction candidate presentation 608 is the new character recognition result acquired in step S411. If this candidate is correct, the user can complete the correction operation by touching the candidate. If the recognition result acquired in step S411 is not correct, the user inputs correct characters using the software keyboard 607 and completes the correction operation. When a character input operation from the user using the software keyboard 607 continues after the support candidate is presented, the shooting setting may be changed again, and the processing from step S407 to step S411 may be performed to additionally present a correction candidate.
When all the correction operations are completed, the user touches the completion button 604 to complete the confirmation / correction process.

  In addition, when the photographing apparatus 205 further includes a mechanism such as pan / tilt and the photographing direction can be controlled from the photographing and projecting apparatus, the focal length can be further extended by changing the rectangular coordinate to be corrected to the center of the angle of view. It is possible to increase the resolution around the target character string. In addition, a setting other than the focal length may be changed to set a condition more suitable for character recognition and re-shoot. For example, it is conceivable to adjust the exposure so that the focus is set to the target rectangular coordinates or the contrast in the rectangular coordinates is maximized. Further, the setting may be changed in consideration of information such as lens distortion of the photographing apparatus 205 and optical performance with respect to the aperture.

  As described above, when information is extracted from a form using this photographing and projection apparatus, document correction and character recognition processing can be easily performed again when correcting the erroneous recognition result. The possibility of presenting the UI as a candidate is increased, and the user can complete the correction operation with fewer operations.

Step S412, step S413 will be described in detail.
In this photographing and projection apparatus, when the location of the candidate character string of the attribute is incorrect, the touched location can be changed as the attribute character string by touching the correct character string. It is.

  FIG. 9 is an example of a modified UI in which the user touches the UI display 603 of the address attribute with respect to the modified UI of FIG. 6 and the correction target is changed to the address attribute by the process of the flowchart S405. A highlight display 901 indicating that the current correction target attribute is an address attribute is displayed by the projection device 207. In addition, the character string “Kanagawa case Nakahara-ku YYY” in the first candidate 506 is projected on the UI display 603 as the address attribute character string by the projection device 207 as the highlight display 902.

  In the process of step S305 described above, as the address attribute, the character string near the right direction of the character string “address” is acquired as the address attribute.

  In the address attribute candidates 506, 507, and 508 of the extraction result 504, as character strings satisfying the conditions, “Yaray, Nakahara-ku, Kawasaki-shi, Kanagawa-ken”, “Shimomaruko XXX, Ota-ku, Tokyo”, “ZZZ, Tama-ku, Kawasaki-shi, Kanagawa-ken, respectively. "Is acquired as satisfying this condition. Here, since the priority is not set in particular when a plurality of conditions are matched, the candidate order is indefinite, and the address attribute 503 originally intended to be acquired is the second candidate 507.

  Therefore, the user can perform a correction operation by touching the character string area 903 on the document 601 and can change the value to a correct value by the processing in steps S412 and S413. The correct value is a character string that the user wants to acquire from the resident card 601 as an address attribute.

  However, when the character of the document 601 is small or the line spacing is narrow, it may be determined that the adjacent area is touched instead of the character area 903 intended by the user. In the present embodiment, the correction operation is supported by expanding the touch determination area with respect to the rectangular area that is a candidate for the target attribute.

  For example, FIG. 10A is a diagram showing the touch determination area of each character string on the document 601 when the present embodiment is not used by a broken line. When the user performs a touch operation on the character area 903 and the touch coordinates are 1001, it is determined that the touch determination area 1003 is pressed instead of the touch determination area 1002 of the character area 903.

  FIG. 10B is a diagram showing the touch determination area of each character string in the present embodiment by a broken line. The touch determination areas 1006, 1005, and 1007 show the expanded states of the touch determination areas of the character areas corresponding to the address attribute candidates 506, 507, and 508, respectively. According to this, even if the same position 1001 as before is touched, it is determined that the touch determination area 1004 of the target character area 903 is touched, and the correct character string is corrected as the character string of the target attribute.

  In this description, the touch determination area is similarly expanded with respect to the candidate character string of the target attribute. However, the candidate order may be enlarged with weighting. In addition, the touch determination area may be enlarged using separate dictionary information and analysis results. For example, even if it is a value that is not acquired as a candidate character string in the extraction result, if it is an address, the value of the address is calculated using the general place name dictionary, and the touch determination area may be expanded when the value of the address is high Good.

  Further, it is not limited to expanding the hit area for the touch operation, and the click area for the mouse operation and the threshold for selection determination at the time of the gesture operation may be expanded.

  As described above, when information extraction from a form is performed using this photographing and projection apparatus, it is possible to easily input a character string that seems to be correct by weighting candidate areas when correcting an extraction position error. It becomes possible to reduce mistakes in input.

<Other embodiments>
The preferred embodiments of the present invention have been described in detail above, but the present invention is not limited to such specific embodiments, and various modifications can be made within the scope of the gist of the present invention described in the claims.・ Change is possible.

  The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

201 CPU
202 Bus 203 Memory 204 Storage Device 205 Imaging Device 206 Input Device 207 Projector

Claims (8)

An imaging means for imaging a paper document;
A recognizing unit for recognizing a region where a character is arranged in an image captured by the imaging unit;
Extraction means for extracting a target character string from a character string of a recognition result by the recognition means;
Display means for displaying the character string data extracted by the extraction means and the attribute data of the character string;
An acquisition means for acquiring an indicated position for at least one of the data displayed by the display means and the paper document;
Correction means for performing different correction processing based on the indicated position acquired by the acquisition means when correcting the data of the character string displayed by the display means;
An image processing apparatus comprising: When the indication position acquired by the acquisition unit is a position where the display unit displays character string data, in the correction unit,
The imaging unit changes the setting to increase the resolution of the image to be captured and re-images the paper document,
The recognizing unit re-recognizes at least the recognition source area of the character string in the re-captured image,
The image processing apparatus according to claim 1, wherein the display unit displays a character string obtained as a result of character recognition again in place of the character string displayed at the designated position.   The image processing apparatus according to claim 2, wherein the imaging unit changes a focal length setting.   The image processing apparatus according to claim 2, wherein the imaging unit changes a setting of a shooting direction. When the indicated position acquired by the acquisition means is the position of the character area of the paper document,
The display unit displays the character string recognized by the recognition unit in the designated position area of the paper document with the character string displayed side by side with the attribute data corresponding to the character string. The image processing apparatus according to claim 1.   The image processing apparatus according to claim 1, wherein the display unit performs projection. Causing the imaging means to image the paper document;
A recognition step for recognizing a region where a character is arranged in an image captured by the imaging unit;
An extraction step of extracting a target character string from the character string of the result recognized in the recognition step;
Displaying the extracted character string data and attribute data of the character string on a display means;
An acquisition step of acquiring an indication position for at least one of the data displayed by the display means and the paper document;
When correcting the data of the character string displayed by the display means, a correction step for performing different correction processing based on the indicated position acquired in the acquisition step;
An image processing method comprising: An imaging process for imaging a paper document;
A recognition step for recognizing a region where characters are arranged in the image captured in the imaging step;
An extraction step of extracting a target character string from the character string of the result recognized in the recognition step;
Displaying the extracted character string data and attribute data of the character string on a display means;
An acquisition step of acquiring an indication position for at least one of the data displayed by the display means and the paper document;
When correcting the data of the character string displayed by the display means, a correction step for performing different correction processing based on the indicated position acquired in the acquisition step;
A program that causes a computer to execute.