JP2020135022A - Image processor, image processing method, and program - Google Patents

Abstract

To accurately detect a normal direction of a manuscript even when the manuscript has a small number of characters.SOLUTION: An image processor includes: a first dictionary holding a feature amount of each single character; a second dictionary holding weighting magnification of each prescribed character; extraction means extracting a feature amount of character image data included in manuscript image data; reliability acquisition means comparing a feature amount extracted by the extraction means with a feature amount of the first dictionary with respect to a plurality of prescribed directions of the manuscript image data to acquire reliability about a character associated with the feature amount of the compared first dictionary; weighting means weighting the reliability of the matched character by the weighting magnification when the character acquiring the reliability is in the second dictionary; and normal direction determining means determining a normal direction of the manuscript image data on the basis of the weighted reliability from the plurality of prescribed directions.SELECTED DRAWING: Figure 12

Description

The present invention relates to a technique for reading a document placed on a platen of a scanner.

Conventionally, multiple sheets of forms, receipts of atypical area, business cards, cards, etc. are read side by side on the platen of a scanner, and the image area corresponding to each document is detected from the generated scanned image data. The multi-crop process for cutting out is known.

When using the multi-crop process, the user manually arranges multiple originals on the platen, so the orientations of the originals may not be aligned in the same direction (upward, downward, leftward, or rightward). is there. Therefore, in the multi-crop process, first, the boundary line between the image area corresponding to the original and the background in the scanned image data is detected, the four vertices of the rectangular area composed of the detected boundary line are specified, and the rectangular area is cut out. .. At this time, if the rectangular area, which is the image area corresponding to the original, is tilted with respect to the scanned image data, the original image data cut out so as to correct the tilt is rotated. After cutting out the image area corresponding to each original in this way to generate the original image data, if necessary so that the orientation of each original image data is in the positive direction (for example, the characters are in the correct orientation). And further rotation processing is performed. The image processing for correcting the original image data so as to be in the positive direction is referred to as "direction detection (or direction correction) processing".

As the direction detection process, there is a method in which OCR (Optical Character Recognition) processing or the like is performed on the original image data and the result is used for the direction detection process. In Patent Document 1, a method of performing OCR processing on manuscript image data generated by reading one manuscript and performing direction detection processing based on the character direction recognized as a result is studied. Further, Patent Document 2 proposes a method of improving the direction detection accuracy of the original image data by omitting characters that are not appropriate as characters used in the direction detection process.

Japanese Unexamined Patent Publication No. 10-181137 Japanese Unexamined Patent Publication No. 2011-338263

However, when targeting manuscript image data such as receipts with a relatively small number of characters such as "receipts" received by customers as proof of payment in the retail and service industries, the direction detection process as in Patent Document 1 is sufficient. There is a problem that it is difficult to realize the detection accuracy. Although a method as in Patent Document 2 can be used to improve the direction detection accuracy, there is a problem that a sufficient improvement effect cannot be obtained when the number of characters is small as in a receipt.

Therefore, an object of the present invention is to accurately detect the forward direction of a document even if the document has a small number of characters.

In order to solve the above problems, the present invention is an image processing apparatus having an image acquisition means for scanning a document and generating document image data, and a first dictionary that holds a feature amount for each single character and The extraction with respect to a second dictionary that holds a weighting magnification for each predetermined character, an extraction means for extracting the feature amount of the character image data included in the manuscript image data, and a plurality of predetermined orientations of the manuscript image data. The reliability acquisition means for comparing the feature amount extracted by the means with the feature amount of the first dictionary and acquiring the reliability of the character associated with the feature amount of the compared first dictionary, and the above-mentioned When the character whose reliability has been acquired exists in the second dictionary, the weighting means for weighting the reliability of the matching character by the weighting factor and the reliability weighted from the plurality of predetermined directions. It is characterized by providing a forward direction determining means for determining the positive direction of the original image data based on the degree.

According to the present invention, it is possible to accurately detect the forward direction of a document even if the document has a small number of characters.

It is a figure which shows the whole structure of the applicable system which concerns on one Embodiment of this invention. It is a figure which shows the hardware configuration example of the image reader which concerns on one Embodiment of this invention. It is a figure which shows the sequence of the whole image processing function which concerns on one Embodiment of this invention. It is a figure which shows the UI display which concerns on one Embodiment of this invention. It is a figure for demonstrating such as a sample image which concerns on one Embodiment of this invention. It is a figure which shows the orientation of the manuscript and the definition of a character area which concerns on one Embodiment of this invention. It is a figure explaining the direction of the character area which concerns on one Embodiment of this invention. It is a figure which shows the orientation of the character area and the example of the OCR result which concerns on one Embodiment of this invention. It is a figure which shows the outline example of the contents of the OCR dictionary which concerns on one Embodiment of this invention. This is a summary example of the contents of the image data type character string dictionary according to the embodiment of the present invention. It is a figure which shows the example of the cumulative reliability by the OCR result which concerns on one Embodiment of this invention. It is a flowchart which shows the detail of the direction detection processing which concerns on one Embodiment of this invention.

Hereinafter, embodiments according to the present invention will be described in detail with reference to the drawings. However, the components described in this embodiment are merely examples, and the scope of the present invention is not intended to be limited thereto.

<System configuration>
FIG. 1 is a diagram showing an overall configuration of a system example to which the image processing apparatus according to the embodiment of the present invention can be applied.

In a typical system to which the present invention can be applied, as shown in FIG. 1, the image processing device 100 and the PC / server terminal 101 are connected to a LAN 104 composed of Ethernet (registered trademark), wireless LAN, etc., and the LAN 104 is further connected to the Internet. It is a configuration connected to 105. Further, the mobile terminal 103 is connected to the Internet 105 from a public wireless communication network 102 or the like. The image processing device 100, the PC / server terminal 101, and the mobile terminal 103 are connected to the Internet 105 from the LAN 104 or the public wireless communication network 102, and can communicate with each other. The PC / server terminal 101 and the mobile terminal 103 may be configured with only one of them, or only the image processing device 100 may perform the processing performed by the PC / server terminal 101, the mobile terminal 103, or the like.

The image processing device 100 is a copying multifunction device having an operation unit, a scanner unit, and a printer unit. In the system of this embodiment, the image processing device 100 is used as a scanning terminal for reading a document such as one or more business cards, a driver's license, and a postcard. Further, the image processing device 100 performs a multi-crop process for extracting the original image data for each original from the scanned image data obtained by reading a plurality of originals arranged on the original table. Further, the image processing device 100 has a display unit and operation units such as a touch panel and hard buttons, and the operation unit displays error notifications and instruction notifications, and the user performs operations such as scanning operations and setting operations. Can be done.

The PC / server terminal 101 can display the original image generated by the image processing device 100. Further, the PC / server terminal 101 saves the original image generated by the image processing device 100, performs OCR processing, and the like to generate reusable content data. The multi-crop process performed by the image processing device 100 may be performed by the PC / server terminal 101. Further, the PC / server terminal 101 can also communicate with an external storage such as a cloud or a server, and can transmit the saved manuscript image data and metadata to the external storage. In this embodiment, the flow of storing the original image data, generating the metadata, and transmitting the original image data to the external storage will be described in the image processing device 100, but the same function may be realized in the PC / server terminal 101. ..

Further, the mobile terminal 103 is a smartphone or tablet terminal having an operation unit, a wireless communication unit, and an application unit for operating a web browser. In the system of this embodiment, the mobile terminal 103 is used as a display terminal, an operation terminal, and a content data generation / storage terminal in the same manner as the PC / server terminal 101. The PC / server terminal 101 and the mobile terminal 103 may have either configuration such as display, operation, and metadata generation / content data generation / storage functions.

The above components are merely examples, and not all components are required.

<Hardware configuration of image processing device 100>
FIG. 2 is a block diagram showing the configuration of the image processing device 100. The control unit 110 includes a CPU 111, a storage device 112, a network I / F unit 113, a scanner I / F unit 114, and a display / operation unit I / F unit 115, which are communicably connected to each other via the system bus 116. Has been done. The control unit 110 controls the operation of the entire image processing device 100.

The CPU 111 reads a control program stored in the storage device 112 and performs various controls such as read control and transmission control.

The storage device 112 stores and holds the above program, image data, metadata, setting data, processing result data, and the like. The storage device 112 includes a ROM 117 which is a non-volatile memory, a RAM 118 which is a volatile memory, an HDD 119 which is a large capacity storage area, and the like.

The ROM 117 holds a control program and the like. The CPU 111 reads the control program stored in the ROM 117 and controls the image processing device 100.

The RAM 118 is used as a temporary storage area such as a main memory and a work area of the CPU 111.

The HDD 119 is an HDD that is a large-capacity storage area, and is used as a storage area for storing image data, metadata, and the like.

The network I / F unit 113 is an interface for connecting the control unit 110 (image processing device 100) to the LAN 104. The network I / F unit 113 transmits image data to an external device on the LAN 104 such as a PC / server terminal 101 or a mobile terminal 103, and receives various information from the external device on the LAN 104.

The scanner I / F unit 114 is an interface that connects the scanner unit 120 and the control unit 110. The scanner unit 120 is an image acquisition means that reads a document on a platen, generates scanned image data, and inputs the scanned image data to the control unit 110 via the scanner I / F unit 114.

The display / operation unit I / F unit 115 is an interface for connecting the display / operation unit 121 and the control unit 110. The display / operation unit 121 is provided with a liquid crystal display unit having a touch panel function and hard keys such as a numeric keypad, a start button, and a cancel button. The start button is a button for starting the copy or scan process. The cancel button is a button for suspending or canceling the processing being executed by the image processing device 100.

In addition, some image processing devices 100 also have a printer unit and the like, but they are omitted because they are not used in this embodiment.

As described above, the image processing apparatus 100 according to the present embodiment can provide the image processing function by the above hardware configuration.

<Execution flow of "scan and send"function>
A processing sequence for the user to execute the multi-crop processing by using the "scan and send" function will be described with reference to FIG. The process described in this embodiment is realized by the CPU 111 included in the image processing device 100 reading the control program stored in the storage device 112 and executing the control program.

The "scan and transmit" function is a function of scanning a document with an image processing device 100 connected to a network such as a LAN and transmitting the obtained scanned image data to an external device. Specifically, it is a function of executing image processing and format conversion on the scanned image data read by a scanner and transmitting the data to a folder on a server specified by the user, an e-mail, or HDD 119 in a copying machine.

In the function use instruction S400, the user can instruct the image forming apparatus 100 to use the scan function by operating the display / operation unit 121 and selecting the “scan and send” function button. The image processing device 100 accepts the selection of the "scan and send" function button via the display / operation unit 121. FIG. 4A is a main menu UI500 displayed on the display / operation unit 121. The main menu UI 500 is a screen in which functions that can be performed by the image processing device 100 are displayed as buttons. For example, the "copy" function button 501, the "scan and send" function button 502, the "scan and save" function button 503, the "use saved file" function button 504, the "print" function button 505, and the like are displayed. .. The image processing device 100 accepts a user's selection of a function to be executed via the main menu UI 500. In this embodiment, it is assumed that the user taps and selects the "scan and send" function button 502.

In the setting UI display S401, the image processing device 100 displays the setting screen of the "scan and transmit" function on the display / operation unit 121. FIG. 4B is an example of the “scan and send” setting UI 510, which is a setting screen displayed on the display / operation unit 121. The "scan and send" setting UI 510 shows the states of various settings of the "scan and send" function. For example, the "destination" block 511 displays the address of the destination to which the scanned image data generated by scanning is transmitted. When the user taps the "destination" block 511, a destination setting screen (not shown) is displayed, and the user can input the destination of the scanned image data. In this embodiment, the image processing device 100 transmits the original image data generated by performing multi-crop processing on the scanned image data to the PC / server terminal 101. Therefore, the URL (Uniform Resource Locator), IP address, and the like of the PC / server terminal 101 are set in the "destination" block 511. The "scan / transmit setting" button 512 displays the color setting of the scan image data to be generated, the format of the image file to be generated, and the state of the type of the original. Further, the "other function" button 513 is a button for setting an applied function that is not displayed in the "scan and send" setting UI 510.

In the basic setting instruction S402, the user instructs the image forming apparatus 100 to set the basic function by operating the display / operation unit 121 and selecting each button on the “scan and send” setting UI 510. be able to. The image processing device 100 receives a setting instruction of a setting item that can be set by the "scan and send" setting UI 510 from the user. The settings accepted by the basic setting instruction S402 include, for example, color setting of the scanned image data to be generated, format selection of the image file to be generated, and the like. The image processing device 100 accepts a tap operation of any one of the "destination" block 511 and the "scan / transmission setting" block 512, and accepts the input of the corresponding setting items.

In the basic setting S403, the image forming apparatus 100 stores the set value of the basic setting of the “scan and transmit” function instructed by the user in S402 in the RAM 118 of the image processing apparatus 100.

Next, in the application setting instruction S404, the user can instruct the image forming apparatus 100 to set the application function by tapping and selecting the "other function" button 513.

In the detailed setting UI display S405, when the image processing device 100 receives the user's selection of the "other function" button 513, the image processing device 100 displays a screen for setting the applied function on the display / operation unit 121. FIG. 4C is an example of the “other function” setting UI 520 for setting the applied function. The "other functions" setting UI 520 displays buttons for setting various application functions of the "scan and send" function that can be executed by the image processing device 100. The "other functions" setting UI 520 includes, for example, "page aggregation" button 521, "color type" button 522, "original type" button 523, "color adjustment" button 524, "file name" button 525, and "multi". The "Crop" button 526 and the like are displayed. The "multi-crop" button 526 is a button for instructing execution of a process of extracting the original image data corresponding to each original from the scanned image data generated by scanning the original. In this embodiment, the "other functions" setting UI 520 displays setting items that overlap with the setting items that can be set from the "scan and send" setting UI 510. The "other function" setting UI 520 may display setting items excluding the setting items that can be set from the "scan and send" setting UI 510.

In the multi-crop setting instruction S406, the user can instruct the image forming apparatus 100 to set the multi-crop process by tapping and selecting the "multi-crop" button 526 of the "other function" setting UI 520. it can.

When the user taps and selects the "multi-crop" button 526, the image processing device 100 sets the multi-crop processing flag ON indicating that the multi-crop processing is to be executed in S407. The multi-crop processing flag is stored in the RAM 118. Further, in S407, the image processing device 100 displays the display / operation unit 121 indicating that the "multi-crop" button 526 has been selected in the "other function" setting UI 520 as shown in FIG. 4 (d). Let me. For example, when the user selects the "multi-crop" button 526, the color of the "multi-crop" button 526 of the "other functions" setting UI 520 is inverted, and a screen indicating that the multi-crop process is set to on is displayed. Will be done.

Further, when the user taps the "multi-crop" button 526, the screen 530 for setting the orientation of the character string of the document is displayed to the user as shown in FIG. 4 (e). Here, the "horizontal writing" button 531 corresponding to the receipt, the "vertical writing" button 532 corresponding to the business card, etc., and the "mixed" button to be set when the vertical writing and horizontal writing documents are mixed and placed on the platen. 533 is arranged. Note that the button type name such as "receipt" or "business card" may be used instead of the orientation of the character string to display the button as an option.

In the character string orientation setting instruction S408 of the original, the user taps and selects any one of the "horizontal writing" button 531 and the "vertical writing" button 532 and the "mixed" button 533 to select the image forming apparatus 100. It is possible to instruct the direction of the character string of the document. Since the target manuscript in this embodiment is a receipt, it is assumed that the "horizontal writing" button 531 is selected by the user by tapping.

According to this user operation, in S409, the image processing device 100 sets the character string orientation setting indicating the character string orientation of the document to "horizontal writing". The character string orientation setting is stored in the RAM 118. Further, in S409, the image processing device 100 indicates that the horizontal writing document setting is set to ON by inverting the color of the "horizontal writing" button 531 shown in FIG. 4 (e) on the display / operation unit 121. .. Subsequently, when the user taps and selects the "close" button 534, the display / operation unit 121 displays the "other function" setting UI 520. Further, when the user taps and selects the "close" button 527, the display / operation unit 121 displays the "scan and send" setting UI 510.

In the scan instruction S410, the user can instruct the image processing device 100 to start scanning the original by tapping the start key 506. When the start key 506 is tapped, the image processing device 100 writes various setting information for scanning the document to the RAM 118, and starts preparing for the document reading process.

When the scanning process of the document is started, first, in scan S411, the image processing device 100 drives the scanner unit 120 to read the document placed on the platen of the scanner unit 120.

In the image forming S412, the image processing apparatus 100 converts the signal value obtained from the scanner unit 120 into image data (scanned image data) in a bitmap format that can be handled by the image processing by scanning the document with the scan S411. For example, in S412, the image processing device 100 converts the luminance signal value input from the scanner unit 120 into an 8-bit digital signal and stores it in the HDD 119 as scanned image data.

FIG. 5 is a schematic view showing a state in which a plurality of receipt documents are placed on a platen. Here, it is assumed that seven documents are arranged in the scanner unit 120 capable of reading an A3 size area. Receipts are relatively small documents, and there are various sizes and aspect ratio types. In addition, the appearance design is also different, and considering that it is placed inside out toward the user, it is generally placed in a messy manner as shown in the figure. Of course, the number and arrangement of documents placed on the platen are not limited to this.

In the multi-crop process S413, the image processing device 100 acquires the scanned image data stored in the HDD 119 in S412. The image processing device 100 identifies the boundary (edge) between each document area and the background from the acquired scanned image data by the CPU 111, and detects the coordinates of the four vertices constituting the rectangular document area corresponding to each document. The four-vertex coordinate values of each detected document area are stored in the RAM 118. The image processing device 100 acquires the four-vertical coordinate values of each document area detected by the multi-crop coordinate detection process S413, and obtains image data (original image data) corresponding to each document area from the scanned image data generated in S412. break the ice. When each side of the document area is not parallel or perpendicular to each side of the scanned image, the image processing device 100 uses projective transformation, trapezoidal transformation, affine transformation, or the like based on the four vertex coordinates of each document region. The original image data is cut out so that the tilt correction is also performed at the same time. The image processing device 100 stores each original image data after cutting out in the HDD 119. At this time, the image processing device 100 may compress the original image data and store the original image data after data compression in the HDD 119.

In the character recognition preprocessing S414, the image processing device 100 acquires the original image data from the HDD 119. The image processing device 100 performs image preprocessing (for example, binarization, line segment removal, noise removal, layout analysis, etc.) necessary for OCR processing on the acquired original image data by the CPU 111, and the processed original. The image data is stored in the RAM 118. If the original image data is compressed, the original image data is decoded before the image preprocessing.

In the direction detection process S415, the image processing device 100 performs a document-oriented narrowing process on the document image data stored in the RAM 118 by the CPU 111. After that, the image processing device 100 executes OCR processing and determines the direction of the original image data from the feature vector for each character obtained from the character image. The details of this process will be described later.

In the direction correction image processing S416, the image processing apparatus 100 performs rotation processing based on the rotation angle detected in S415 so as to be in the positive direction with respect to the original image data cut out in S413. Generally, the positive direction of the original image data is one of 0 °, 90 °, 180 °, and 270 °, which is the direction in which each side of the original area is a horizontal upper side when the original area is rectangular. Since it is a direction, the rotation process here is the direction correction process in the above four directions. Needless to say, when the positive direction is determined to be 0 ° in S415, the rotation process of S416 may be skipped.

In the multi-crop processing result UI display S417, the image processing device 100 displays each document image data after the multi-crop processing and the document direction correction processing on the display / operation unit 121. FIG. 4F is an example of a screen showing the processing result of the multi-crop and the document direction correction displayed on the display / operation unit 121. As shown in FIG. 4 (f), the "manuscript detection result display" UI540 displays each manuscript image obtained by the multi-crop and the manuscript direction correction processing side by side.

In the transmission instruction S418, the user confirms the result of cutting out / direction correction processing of the original image data from the original image displayed on the display / operation unit 121. Then, the user gives an instruction to save and transmit the cut-out and direction-corrected original image data. By pressing the start key 506, the user instructs the image processing device 100 to save and transmit each original image data after the cutting / direction correction processing. In S418, the image processing device 100 receives a transmission instruction from the user for transmitting the original image data to the PC / server terminal 101.

In the image transmission S419, the image processing device 100 transmits the original image data after the crop processing to the PC / server terminal 101. The original image data transmitted to the PC / server terminal 101 in S419 is image data that reflects the setting values set via the "scan and transmit" setting UI 510 and the "other function" setting UI 520.

In the storage S420, the PC / server terminal 101 stores each original image data transmitted from the image processing device 100. Here, the PC / server terminal 101 may only save the original image data, but for example, the original image data is subjected to character recognition processing (OCR processing), and the character recognition processing result is added to the original image data as metadata. You may try to memorize it. By doing so, the searchability of the manuscript image data can be improved, and the information extracted from the manuscript image data can be registered in the system.

In the stored data viewing instruction S421, the user can instruct the display of the original image data by operating the PC / server terminal 101.

Display of stored data In the provision S422, the PC / server terminal 101 displays the manuscript image data instructed to be displayed by the user on the display unit of the PC / server terminal 101. When the original image data stored in the PC / server terminal 101 is processed by OCR processing or the like, the PC / server terminal 101 can also display the result of the OCR processing or the like in S420. In this embodiment, it is assumed that the saved manuscript image data is displayed on the display unit of the PC / server terminal 101. The PC / server terminal 101 may transmit the manuscript image data to the client terminal such as the user's PC or tablet terminal according to the instruction from the user.

<Calculation of reliability of OCR used for direction detection>
In the direction detection process S415 of this embodiment, the image processing device 100 executes character recognition by the CPU 111 by the existing technology "OCR using the feature vector". What will be described here is a process corresponding to steps S603 and S604 described later.

Here, the present invention pays attention to the fact that characters most accurately represent the direction of the manuscript, and from the directions of 0 °, 90 °, 180 °, and 270 ° with respect to several types of character areas in the manuscript. Perform character recognition. That is, the original image data is rotated to 90 °, 180 °, and 270 ° by image rotation processing, and OCR is performed in each direction including 0 ° before rotation. The positive direction is determined so that the direction in which the reliability of character recognition in each of the resulting directions is the highest is the positive direction of the document. The reliability can be read as the reliability of the character recognition result or the similarity between the character recognition result and the feature amount for each character included in the character recognition dictionary. The details of the character recognition dictionary will be described later.

In this embodiment, since the candidates are narrowed down to 0 ° and 180 ° or 90 ° and 270 ° as the orientation of the document in the process of step S601 described later, the two narrowed directions will be described with reference to the drawings. To do.

Generally, as a preprocessing of OCR, rectangular information of a character area is extracted from original image data. Here, the character area is a sentence part, a title part, a character part in a table, and the like. For example, in the case of the manuscripts shown in FIGS. 6 (a) and 6 (c), the rectangular information of the character area as shown in FIGS. 6 (b) and 6 (d) is extracted. The extracted character area is further divided into character blocks within the character area (blocks for each character unit), and character recognition by OCR is performed for each character block in all the character areas. The character block in the character area refers to rectangular information in units of characters as shown in FIG. 6 (f) with respect to the character area shown in FIG. 6 (e).

7 (a) and 7 (b) are examples of extracting a character block string in a character area consisting of "go" and "total". FIG. 7A shows a case where the character string is in the positive direction, and FIG. 7B shows a state where the character string is rotated by 180 °. Here, pay attention to the first "go" of this character block string. When the character direction is determined by "go", as shown in FIG. 8A, character recognition is performed for one character image "go" from two directions of 0 ° and 180 °. As a method of performing character recognition in two directions, for example, the cut-out original image data (or each character image data) is rotated in two directions of 0 ° and 180 ° to extract a feature vector of each character image data. However, character recognition may be performed by comparing with a character recognition dictionary. Further, a feature vector may be extracted from each character image data, and the extracted feature vector may be rotated in two directions of 0 ° and 180 ° for comparison with a character recognition dictionary.

The character recognition results at each rotation angle are different from each other as shown in FIG. 8 (b). Further, FIG. 8C shows the reliability of the character recognition processing result at each rotation angle shown in FIG. 8B. The character recognition processing results and reliability in FIGS. 8 (b) and 8 (c) are examples, and depend on the OCR algorithm and the scanning environment (noise, etc.), so that the actual results are not always the same.

As shown in FIG. 8B, when character recognition is performed from the positive direction (0 °), it is correctly recognized as “go” and the reliability is as high as 0.90. When character recognition is performed from the direction rotated by 180 °, it is erroneously recognized as "No." and the reliability is lowered to 0.30. The reason why the erroneous recognition occurs and the reliability is lowered in this way is that the character recognition is performed based on the feature vector when viewed from the rotation direction. It should be noted that the more complicated the characters are, the more remarkable the difference in the reliability of the direction determination of the character recognition becomes.

<Structure of OCR dictionary (character recognition dictionary)>
FIG. 9 is an example of the OCR dictionary used in this embodiment. The OCR dictionary contains a reference feature vector (feature amount) associated with each single character to be recognized in OCR using a feature vector, which is an existing technology, and is stored in ROM 117 or the like. .. This OCR dictionary is used in step S604 described later. There is no limit to the number of characters that can be stored in the OCR dictionary. However, as shown in FIG. 9, characters that frequently appear on the receipt, numbers (0 to 9) that always appear on the receipt, and other characters stored in the character string dictionary of the manuscript type described later are always stored. .. Characters that frequently appear on receipts include characters such as "go", "total", "gold", "amount", "dot", "number", "article", and "eye". It should be noted that the characters shown in the OCR dictionary example of FIG. 9 contain only a part of the above. In addition, there are only a few examples of frequently-used characters, which will be described later, and it goes without saying that additional adjustments should be made based on actual usage examples.

In the dictionary, these characters are converted into n-dimensional feature vectors and stored. This feature vector is taken out during the OCR process and compared with the character image data in each character block extracted from the original image data. The value of the feature vector shown in FIG. 6 is an example, and it is not necessary that the feature vector to be implemented is actually the same.

<Calculation of cumulative reliability considering OCR reliability and weighting>
11 (a) to 11 (d) show the original by rotating one receipt document to 0 °, 90 °, 180 ° and 270 ° as described above, and FIGS. 11 (a) and 11 (c) show the original. ) Indicates an example of cumulative reliability when OCR is performed in each direction. In this embodiment, since the candidates are narrowed down to 0 ° and 180 ° for the document orientation in the process of step S601 described later, the cumulative reliability is shown as Graph 1100 and Graph 1105 only in the narrowed down two directions. There is.

Here, using the OCR dictionary described above, the reliability for each orientation of the document when each character is detected by OCR is represented as a black bar graph with the cumulative value of the total of the entire document as the “cumulative reliability”. That is, the cumulative reliability of the character "go" in the document orientation shown in FIG. 11 (a) is the black bar graph 1101, and the cumulative reliability of the character "go" in the document orientation shown in FIG. 11 (c) is the black bar graph 1103. It becomes. Generally, in OCR, when one character is recognized and processed, it is possible to extract a plurality of candidates and calculate the reliability for those candidates. The cumulative reliability can be thought of as a histogram of the reliability values for each candidate in the manuscript.

Up to this point, it is an example of a general method for obtaining the cumulative reliability, but from here, the weighting for the reliability of a specific character, which is a feature of the present invention, will be described. This description corresponds to the processing corresponding to step S605, step S606, and step S607 described later.

First, a manuscript type dictionary or a character string dictionary created separately for each type of manuscript corresponding to the manuscript image data of FIG. 10 will be described. Here, a single character is also included in the character string, and hereinafter, the manuscript type character string dictionary will be used. The manuscript type character string dictionary in this embodiment is a character string that frequently appears in the type of manuscript (receipt in this embodiment) read when generating manuscript image data, and the reliability of direction detection corresponding to each character string. It is a dictionary in which a predetermined weighting ratio for a degree is described. The character strings that frequently appear on the receipt are, for example, character strings such as "total", "amount", "points", and "item". When these are detected by OCR processing, the reliability of each character included in the character string (for example, "total" for "total" and "total") is multiplied by a weighting factor of 2.0 times. .. This weighting ratio can be changed for each character string, and it is also possible to change the weighting ratio of each character to 3.0 times for a character string consisting of three characters such as "receipt". Is. On the contrary, if "go", which is a character that frequently appears even if it does not appear as a character string, is detected, it can be adjusted by 1.5 times. It should be noted that the characters shown in the dictionary example of FIG. 10 describe only a part of the character strings that frequently appear on the receipt. Needless to say, the frequently-used character examples given here are only a part and should be added and adjusted based on the usage examples.

A plurality of character string dictionaries for image data types can be created for each manuscript image data type such as "receipt" and "business card", and can be stored in ROM 117 or the like. These can be switched and used according to the document to be scanned.

Subsequently, the cumulative reliability is calculated by adding up the reliability including the weighting of all the characters in the original for each original orientation. The white bar graph 1102 in the cumulative reliability graph 1100 of the document orientation FIG. 11 (a) and the white bar graph 1103 in the cumulative reliability graph 1105 of the document orientation FIG. 11 (c) are increments of reliability in consideration of weighting. Is. Therefore, the values of the black bar graph 1101 + white bar graph 1102 and the black bar graph 1103 + white bar graph 1104 are the cumulative reliability of "go" in each document orientation in this embodiment. Further, FIG. 11 (e) shows the white bar graphs 1102 and 1104 of FIGS. 11 (a) and 11 (c) numerically. The cumulative reliability values in the figure are examples, and the feature vectors to be implemented do not have to be exactly the same.

<Flowchart of "direction detection process S415">
FIG. 12 is a flowchart showing the processing of the image processing apparatus 100 that performs direction detection by integrating the above-mentioned configuration and the calculation method. The contents of S415 of FIG. 3 are described in detail. The following processing is realized by the CPU 111 executing a control program stored in the ROM 117, HDD 119, or the like.

In step S601, the image processing device 100 performs a process of narrowing down the orientation of the document by the CPU 111. Specifically, first, the original image data is read from the HDD 119, and the character string orientation setting "horizontal writing" of the original stored in the RAM 118 is read. Subsequently, the original image data is rotated to 0 °, 90 °, 180 °, 270 ° (FIGS. 11A to 11D), and black pixels are obtained in the vertical and horizontal directions of the original image data in each direction. By taking a histogram of the numbers, it is determined in which direction the character strings are arranged. As a specific example of this method, the method of JP2012-83500 can be considered. Then, out of the above four directions, "two directions in which character strings are lined up in the horizontal direction" corresponding to horizontal writing are narrowed down. In this embodiment, FIGS. 11A and 11C correspond to these two directions, and these two directions are extracted as candidates for the forward direction of the document.

In step S602, the image processing apparatus 100 reads the contents of the dictionary from the ROM 117 or the like by the CPU 111 and stores them in the RAM 118. The dictionaries developed here are the OCR dictionary shown in FIG. 9 and the character string dictionary of the image data type illustrated in FIG.

In step S603, the image processing apparatus 100 selects one character area from the original image data and extracts the feature vector of each character block in the character area.

In step S604, the image processing apparatus 100 compares the feature vector in the OCR dictionary of S602 with the feature vector of each character block in the same character area extracted in S603, and trusts the OCR processing of each recognized character. Calculate the degree and acquire the reliability.

In step S605, the image processing apparatus 100 selects one of the character blocks, and the character recognized by the character block is a character that is recognized by other character blocks before and after the continuation in the same character area. Determines whether or not to constitute the character string registered in the column dictionary. If the character recognized in the selected character block constitutes a character string that matches the character string registered in the character string dictionary, the process proceeds to step S606. If there is no matching character string, the process proceeds to step S607.

In step S606, the image processing device 100 weights the reliability of the characters recognized by the character block selected in S605 according to the weighting ratio set in the character string dictionary.

In step S607, the image processing apparatus 100 adds the reliability calculated in the character block selected in S605 to the cumulative reliability of the characters recognized in the character block. The details of the processes from steps S603 to S607 are as described above.

In step S608, the image processing apparatus 100 determines whether or not the reliability calculated for each character block is added to the cumulative reliability for each character for all the character blocks in the current character area selected in S603. .. If a character block for which addition to the cumulative reliability has not been completed exists in the same character area, the process returns to step S605. If the addition to the cumulative reliability is completed in all the character blocks in the same character area, the process proceeds to step S609.

In step S609, the image processing device 100 determines whether or not the extraction of the feature vector has been completed for the entire character area in the original image data. If there is a character area for which the extraction of the feature vector has not been completed, the process proceeds to step S611, the next character area is selected, and the process returns to step S603.

In step S610, the image processing apparatus 100 determines whether or not the previous processing has been performed on the document orientation 0 ° shown in FIG. 11A and the document orientation 180 ° shown in FIG. 11C. .. If there is an unfinished document orientation, the process proceeds to step S612, the document image data is rotated to an angle to be processed next, and the process returns to step S603. If the processing has been completed for all the document orientations, the process proceeds to step S613.

In step S613, the image processing apparatus 100 determines the forward direction of the document based on the method described later. Here, this process will be described with reference to FIG. FIG. 11 is a calculation example in which the direction of the document is detected from the result of the OCR reliability shown in FIG. 10 according to the present embodiment. In FIG. 11 (e), the cumulative reliability of each character is totaled for each orientation of the documents in FIGS. 11 (a) and 11 (c). Here, the total value of 0 ° for the original (FIG. 11 (a)) is 95.68, which is higher than the total value of 180 ° for the original (FIG. 11 (c)) of 52.84. Therefore, the image processing apparatus 100 determines that the positive direction of the original image data is the direction of 0 ° shown in FIG. 11A.

As described above, in the present invention, even if the receipt is a form having a relatively small number of characters, the difference in cumulative reliability for each orientation of the document is increased by weighting the reliability of the characters constituting a specific character string. It is possible.

<"Mixed" in the character string orientation setting instruction S408 of the document>
In the above embodiment, the case where the original is limited to the horizontal writing receipt and "horizontal writing" is specified for the character string orientation setting at the time of multi-crop in step S408 of FIG. 3 has been described. Here, the case where "mixed" is specified at the time of setting the character string orientation will be described.

The case where "mixed" is specified by the user is, for example, the case of scanning a plurality of business cards in which both vertical writing and horizontal writing are present. In this case, there is a method of adding a process of detecting the character string orientation before the process of narrowing down the document orientation in step S601 in FIG. As a method for detecting the character string orientation, there is a method shown in Japanese Patent Application Laid-Open No. 2012-83500.

If the character string orientation detection process is not added, the document orientation narrowing process in step S601 is skipped, and the cumulative reliability calculation process is performed for all of the document orientations 0 °, 90 °, 180 °, and 270 °. It may be done.

When "vertical writing" is specified in the original character string orientation setting, the original orientation in which the character strings are arranged in the vertical direction is adopted when the original orientation is narrowed down in step S601.

Further, in this embodiment, the description is made on the premise that the character string orientation and the document type are always set, but when the character string orientation and the document type are not set, it is specified as "mixed". The same processing as in the case may be performed.

(Other Examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

Image processing device 100
Control unit 110
Scanner unit 120

Claims (13)

An image processing device having an image acquisition means for scanning a document and generating image data of the document.
A first dictionary that holds features for each single character,
A second dictionary that holds a weighting factor for each given character,
An extraction means for extracting the feature amount of the character image data included in the manuscript image data, and
With respect to a plurality of predetermined orientations of the manuscript image data, the feature amount extracted by the extraction means and the feature amount of the first dictionary were compared and associated with the feature amount of the first dictionary compared. A reliability acquisition method for acquiring the reliability of characters, and
When the character for which the reliability has been acquired exists in the second dictionary, the weighting means for weighting the reliability of the matching character by the weighting factor and
An image processing apparatus comprising: a forward direction determining means for determining a positive direction of the original image data based on the weighted reliability from the plurality of predetermined orientations. The first aspect of claim 1, wherein the positive direction determining means acquires the cumulative values of all the reliabilitys for each of the predetermined directions, and sets the predetermined direction having the largest cumulative value as the positive direction. Image processing equipment. The second dictionary further holds a weighting factor for each predetermined character string composed of a plurality of characters.
When the character for which the reliability has been acquired constitutes a predetermined character string of the second dictionary, the weighting means corresponds to the reliability of each character constituting the predetermined character string of the second dictionary. The image processing apparatus according to claim 1 or 2, wherein the weighting is performed by a weighting factor of the predetermined character string. The image processing apparatus according to claim 3, wherein the predetermined character string held by the second dictionary is a character string that frequently appears for each type of the manuscript. The second dictionary includes a plurality of manuscript type dictionaries separately created for each type of manuscript.
The image processing apparatus according to any one of claims 1 to 4, wherein the weighting means is weighted based on the manuscript type dictionary corresponding to the manuscript type corresponding to the manuscript image data. When the orientation of the original image data does not match the positive direction determined by the positive direction determining means, a rotation processing means for rotating the original image data so that the orientation of the original image data is the positive direction is further added. The image processing apparatus according to any one of claims 1 to 5, wherein the image processing apparatus is provided. The image acquisition means generates scanned image data including an image area corresponding to a plurality of documents placed on a platen.
The invention according to any one of claims 1 to 6, further comprising an image processing means for cutting out the image area from the scanned image data and generating a plurality of original image data corresponding to each of the plurality of originals. The image processing apparatus described. The image processing apparatus according to any one of claims 1 to 7, further comprising a character string orientation acquisition means for acquiring the character string orientation of the original image data. The image processing according to claim 8, wherein the character string orientation acquisition means acquires the character string orientation associated with the type of the original document based on the type of the original document corresponding to the original image data. apparatus. Further provided with a character string orientation detecting means for detecting a character string orientation candidate in the original image data,
The image processing according to any one of claims 1 to 9, wherein the reliability acquisition means and the forward direction determining means set the detected character string orientation candidate in the plurality of predetermined orientations. apparatus. Any one of claims 1 to 9, wherein the reliability acquisition means and the forward direction determining means have a plurality of predetermined directions in which each side of the original image data is a horizontal upper side. The image processing apparatus according to the section. An extraction means for extracting the feature amount of the character image data included in the original image data generated by scanning the original with the image acquisition means, and an extraction means.
It has a first dictionary that holds a feature amount for each single character, and compares the feature amount extracted by the extraction means with the feature amount of the first dictionary with respect to a plurality of predetermined orientations of the manuscript image data. Then, the step of acquiring the reliability of the characters associated with the feature quantities of the first dictionary compared with each other, and
When a second dictionary having a weighting factor for each predetermined character is held and a character having acquired the reliability is present in the second dictionary, the reliability of the matching character is weighted by the weighting factor. Steps and
An image processing method comprising a step of determining a positive direction of the original image data based on the weighted reliability from the plurality of predetermined orientations. A program for causing a computer to execute the image processing method according to claim 12.

Images