JP2019159374A - Information processing apparatus and program - Google Patents

Abstract

To determine a threshold value reflecting information of an accumulated past determination result as a threshold dividing sections of a determination accuracy of the determination result for selecting post-stage processing to a system for processing the determination result of determination means for input to selected one of the plurality of post-stage processing.SOLUTION: N-number of post-stage processing are associated with sections having different recognition accuracy (corresponding to a determination accuracy) of a character recognition unit (example of determination means) in a pre-stage. A number of pairs of the recognition accuracy obtained by a past character recognition and correctness information (information indicating whether the recognition is correct) are received as learning data. In order from the section with the highest recognition accuracy, the recognition accuracy rate obtained from the correctness information of the learning data belonging to the section determines the threshold defining the section so that the target correction rate of the determination means corresponding to the section is satisfied.SELECTED DRAWING: Figure 5

Description

  The present invention relates to an information processing apparatus and a program.

  The method disclosed in Patent Document 1 performs character recognition on an input image on a form, obtains a similarity as a result of the character recognition, and obtains the similarity and the character recognition registered in advance. Compared with the certainty required for the comparison, the character recognition result is output based on the comparison result without requiring manual verification processing, or the character recognition candidate for the character recognition result based on the comparison result. An option for prompting a manual verification process is presented, or an output for prompting a manual input process by presenting a new manual input and confirmation for the character recognition result based on the comparison result.

  A character recognition device disclosed in Patent Document 2 recognizes a coordinate point sequence of a character input by handwriting and outputs a recognition candidate character group, and a determination target recognition candidate character group output from the character recognition unit As feature quantities for calculating the reliability, a feature extraction means for calculating an average writing speed of a coordinate point sequence of handwritten characters, a feature quantity from the feature extraction means, and a statistical tendency of sample data And a reliability calculation means for calculating the reliability of the determination target recognition candidate character group, and a post-processing control means for controlling the post-processing of the determination target recognition candidate character group based on the reliability from the reliability calculation means. Have.

  The method disclosed in Patent Document 3 extracts a logical element from an input document image, identifies whether the extracted logical element is a character string area, recognizes the identified character string area, and recognizes the character string area. When the certainty of the result is greater than or equal to the threshold, it is displayed as text, and when it is less than the threshold, it is displayed as a partial image.

  The classification unit of the information processing apparatus disclosed in Patent Document 4 classifies the character recognition target into one of three types, and the extraction unit recognizes the character recognition when the classification unit classifies the first type. The character recognition result of the target is extracted, and the first control means extracts the character recognition result of the character recognition target when it is classified into the second type by the classification means, and manually selects the character recognition target. The second control unit controls the character recognition target to be input manually by a plurality of persons when the classification unit classifies the third type.

  Patent Documents 5 to 10 show various calculation methods for the recognition accuracy of character recognition.

JP 2003-346080 A JP 2003-296661 A JP 2000-259847 A JP 2016-211281 A JP-A-5-40853 JP-A-5-20500 JP-A-5-290169 JP-A-8-101880 JP-A-9-134410 JP-A-9-259226

  In a system that performs a determination on an input and causes a subsequent process corresponding to a section belonging to the determination accuracy of the determination result to process the determination result among a plurality of subsequent processes, it is necessary to set a threshold value that divides the sections. It is desirable that these threshold values reflect information on accumulated past determination results. However, no device or method for determining such a threshold value has been proposed.

  The invention according to claim 1 includes: a determination unit that performs determination on an input; a calculation unit that calculates a determination accuracy of the determination unit with respect to the input; and a subsequent process on the determination result of the determination unit. A plurality of post-processing units capable of generating an output in response to the input and having different degrees of dependence on the determination result of the determination unit in generating the output, wherein the possible range of the determination accuracy is divided by one or more threshold values; A plurality of post-processing units associated with each section, and the post-processing unit corresponding to the section to which the determination accuracy calculated by the calculation unit belongs is controlled to generate an output for the input. An information processing apparatus for determining a threshold for dividing the section with respect to the determination accuracy, for a determination system including: Acquisition means for acquiring a set of the determination accuracy for the input and correct / incorrect information indicating whether the determination result of the determination means for the input is correct or incorrect for each past input; and the acquisition means The accuracy rate of the determination means obtained from the set of sets belonging to the section in order from the section with the highest determination accuracy using the set acquired by the above condition satisfies the target accuracy rate of the determination means corresponding to the section Determining means for determining the threshold value defining the section.

  In the invention according to claim 2, the target recognition rate of the determination unit corresponding to each section is a higher value as the determination accuracy is higher, and the determination unit sequentially starts from the section with the higher target recognition rate. The information processing apparatus according to claim 1, wherein the threshold value defining each section is determined.

  The invention according to claim 3 uses a method with a lower cost as a method for generating the output using the determination result as the post-processing unit corresponding to the section having the higher determination accuracy. The information processing apparatus according to claim 1, wherein the means determines the threshold value defining each section in order from the section with the lowest cost.

  According to a fourth aspect of the present invention, the subsequent processing means corresponding to the section with the highest determination accuracy uses the determination result of the determination means as it is as the output, and corresponds to the section with the highest determination accuracy. The information processing apparatus according to any one of claims 1 to 3, wherein a target correct answer rate set for the determination system is used as the target correct answer rate for the post-processing unit.

  In the invention according to claim 5, the plurality of post-processing units include a second type post-processing unit that generates an output for the input without using a determination result of the determination unit, and the second type 5. The information processing apparatus according to claim 1, wherein the post-processing unit is associated with a section having the lowest determination accuracy among the sections.

  The invention according to claim 6 is the information processing apparatus according to claim 5, wherein the target correct answer rate corresponding to the second-stage post-processing means is zero.

  According to a seventh aspect of the present invention, the plurality of post-processing units include a first post-processing unit that uses the determination result of the determination unit as it is as the output, and a person for the input without using the determination result of the determination unit. Second post-processing means for generating the output based on the determination result by the third post-processing means for generating the output by matching the determination result of the determination means and the determination result by the person with respect to the input, And using the target accuracy rate set for the determination system as the target accuracy rate for the first post-processing means, and using 0 as the target accuracy rate for the second post-processing means, The information processing device according to any one of claims 1 to 3, wherein the target accuracy rate for the third post-processing unit is obtained from the accuracy rate of the person and the accuracy rate of the determination unit. It is.

  The invention according to claim 8 is characterized in that the acquisition means corresponds to each determination accuracy within a range from the maximum value of the determination accuracy to the determination accuracy, instead of a set of determination accuracy and correctness information. A set of correct and incorrect information cumulative result information is acquired, and the determination means uses the cumulative result information corresponding to each of the determination accuracy to obtain a correct answer rate of a section in which the threshold is to be determined. It is an information processing apparatus of any one of Claims 1-7.

  The invention according to claim 9 includes: a determination unit that performs determination on an input; a calculation unit that calculates a determination accuracy of the determination unit with respect to the input; and a subsequent process on the determination result of the determination unit. A plurality of post-processing units capable of generating an output in response to the input and having different degrees of dependence on the determination result of the determination unit in generating the output, wherein the possible range of the determination accuracy is divided by one or more threshold values; A plurality of post-processing units associated with each section, and the post-processing unit corresponding to the section to which the determination accuracy calculated by the calculation unit belongs is controlled to generate an output for the input. A computer as an information processing apparatus for determining a threshold for dividing the section with respect to the determination accuracy. Correct information indicating whether the determination accuracy for the input and the determination result of the determination unit for the input are correct or incorrect for each of the past inputs to the determination unit Using the set acquired by the acquisition means, the accuracy rate of the determination means obtained from the set of sets belonging to the section in order from the section with the highest determination accuracy, It is a program for functioning as a determination unit that determines the threshold value that defines the section so as to satisfy the target accuracy rate of the determination unit corresponding to the section.

  According to a tenth aspect of the present invention, there is provided: a determination unit that performs determination on an input; a calculation unit that calculates a determination accuracy of the determination unit with respect to the input; and a subsequent process on the determination result of the determination unit. A plurality of post-processing units capable of generating an output in response to the input and having different degrees of dependence on the determination result of the determination unit in generating the output, wherein the possible range of the determination accuracy is divided by one or more threshold values; A plurality of post-processing units associated with each section, and the post-processing unit corresponding to the section to which the determination accuracy calculated by the calculation unit belongs is controlled to generate an output for the input. Control means, and for each past input to the determination means, the determination accuracy for the input and the determination means determination for the input An acquisition unit that acquires a set of correct / incorrect information indicating whether the result is correct or incorrect, and the set acquired by the acquisition unit, in order from the interval with the highest determination accuracy, Determining means for determining the threshold value defining the section so that a correct answer rate of the determining means obtained from the set satisfies a target correct answer rate of the determining means corresponding to the section;

  According to an eleventh aspect of the present invention, the computer performs determination processing for the input, determination means for calculating the determination accuracy of the determination means for the input, and post-processing for the determination result of the determination means. Can generate an output for the input, and a plurality of subsequent processing units having different degrees of dependency on the determination result of the determination unit in the generation of the output, and the range that the determination accuracy can take is a threshold value of 1 or more Control so that a plurality of subsequent processing means associated with each divided section, and the subsequent processing means corresponding to the section to which the determination accuracy calculated by the calculation means belongs, generate output for the input. For each of past inputs to the control means and the determination means, the determination accuracy for the input and the determination means for the input Acquisition means for acquiring a set of correct and incorrect information indicating whether the determination result is correct or incorrect, and using the set acquired by the acquisition means, the sets belonging to the section in descending order of determination accuracy This is a program for functioning as a determination unit that determines the threshold value that defines the section so that the accuracy rate of the determination unit obtained from the set satisfies the target accuracy rate of the determination unit corresponding to the section.

  According to the first, second, fourth, fifth, sixth, ninth, tenth, or eleventh aspects of the present invention, the accumulated past values are used as threshold values that delimit a plurality of sections for recognition accuracy corresponding to a plurality of subsequent processes. A threshold value reflecting the information of the determination result can be determined.

  According to the invention which concerns on Claim 3, the cost for the determination as the whole determination system can be reduced compared with the case where this invention is not used.

  According to the invention which concerns on Claim 7, the target correct answer rate of each area can be determined automatically.

  According to the eighth aspect of the present invention, it is not necessary to calculate information on the accumulation result of correct / incorrect information when determining the threshold value.

It is a figure which shows the example of the determination system with which the threshold value setting processing apparatus of embodiment is applied. It is a figure for demonstrating the data for learning input into a threshold value setting processing apparatus. It is a figure which illustrates the function structure of a threshold value setting processing apparatus. It is a figure which illustrates the process sequence of a threshold value calculation part. It is a figure for demonstrating the process which a threshold value calculation part performs. It is a figure for demonstrating how to advance the process of a threshold value calculation part. It is a figure which illustrates the detailed procedure of the threshold value determination process in a threshold value calculation part. It is a figure which illustrates the principal part of the specific example of a determination system.

  FIG. 1 shows an example of a threshold setting processing apparatus 20 that is an embodiment of an information processing apparatus according to the present invention and a determination system that uses the threshold setting processing apparatus 20.

  This determination system converts a character string included in input image data (“input image data”) into OCR 10 and N post-processing units 18-1, 18-2,. An integer greater than or equal to 2. Hereinafter, when there is no need to distinguish between each other, it is determined by the post-processing unit 18).

  The OCR 10 recognizes a character string included in the input image data by performing a known OCR (optical character recognition) process on the input image data. The OCR 10 outputs a combination of a text code indicating a character string recognized from input image data and a recognition accuracy. The recognition accuracy is a degree indicating the probability that the recognition result text code correctly represents a character string (may be handwritten) included in the input image data. The higher the recognition accuracy, the higher the possibility that the text code of the recognition result is correct (that is, correctly represents the character string in the input image data). The possibility that the recognition result is correct is hereinafter referred to as a recognition rate or a correct answer rate. The OCR 10 may output a plurality of different recognition results for the input image data in association with recognition accuracy in descending order of recognition accuracy. The unit in which the OCR 10 performs character recognition (that is, a unit for outputting a recognition result) is not particularly limited, and may be any of character unit, row or column (horizontal writing or vertical writing) unit, page unit, document unit, and the like. There may be.

  The character recognition method used by the OCR 10 and the calculation method of the recognition accuracy are not particularly limited, and any of conventional methods such as those exemplified in Patent Documents 5 to 10 and methods developed in the future may be used. .

  Each of the N post-processing units 18 receives the text code of the recognition result by the OCR 10 in principle, and the text code and the recognition result of the character string in the input image data by zero or more other means. Is used to determine the final character recognition result of the post-processing unit 18. For example, one recognition result is selected and output as a final character recognition result from a recognition result of the OCR 10 and a recognition result of “other means” according to a predetermined (ie, predetermined) rule. Rules for selecting what “other means” to use (some other means may not be used) and selecting a recognition result to be output are determined for each post-processing unit 18. “Other means” used by the post-processing unit 18 for character recognition includes a person, a character recognition service outside the system, and the like. As this external character recognition service, for example, an average recognition rate (accuracy rate) is expected to be higher than that of OCR 10, but it is expensive to use (when the usage cost of OCR 10 can be regarded as 0), or Use cost higher than OCR10. Note that the N post-processing units 18 may include those that do not use the recognition result of the OCR 10 at all.

  The order is defined in the order of 1, 2, 3,..., N in the N post-processing units 18, and the greater the number of the order number, the higher the dependency on the OCR 10. More precisely, the dependency on the OCR 10 monotonously increases as the sequence number increases. In addition, the later processing unit 18 having a larger sequence number has a lower cost (final cost converted into an amount of money) required for the processing of the subsequent processing unit 18. More precisely, the processing cost decreases monotonically as the sequence number increases.

  For example, a later-described post-processing unit 18-N (hereinafter, the post-processing unit 18-K is also referred to as “post-processing K” in order to avoid complexity, where K is an integer from 1 to N) is described later. As in “Post-processing 3” in FIG. 8, a text code that is a character recognition result of the OCR 10 may be output as it is as a final character recognition result. Since the post-process N in this example uses only the recognition result of the OCR 10 to determine the final character recognition result and does not use “other means”, the degree of dependence on the OCR 10 is 100%. Further, since the post-process N in this example does not use a means for character recognition other than the OCR 10, the additional cost for character recognition other than the OCR 10 is zero.

  Further, the first-stage post-processing unit 18-1 ("post-stage process 1") having the first order determines the final recognition result from only the character recognition result of one or more "other means" without using the recognition result of the OCR 10. You may do. In other words, the dependency of the post-processing N in this example on the OCR 10 is 0%. As the post-stage process N, as shown in “post-stage process 1” in FIG. 8 described later, when two persons see and recognize the input image data and the input character strings match, the character string is finalized. A recognition result may be used in which the result of character recognition by another person is used as a final recognition result when they do not match. In this case, since a minimum of two people and a maximum of three people are required, the required cost for processing is high.

  Also, for example, when the recognition result of the OCR 10 matches the character string of the recognition result input by the first person looking at the same input image data, as in “post-processing 2” illustrated in FIG. Is used as a final recognition result, and if they do not match, the second processing unit 18 adopts a recognition result input by the second person looking at the same input image data as the final recognition result. (Referred to as post-processing A) may be used. The degree of dependency and cost of the post-processing A on the OCR 10 are as follows: the post-processing N that uses the recognition result of the OCR 10 exemplified above as it is as the final recognition result, and the post-processing unit 1 that does not use the OCR 10 described above. Between.

  Further, as one of the post-processing units 18, for example, the above-described post-processing A uses an external higher-level (and higher-cost) character recognition system instead of the first person (referred to as post-processing B). ) May be used. The post-processing unit 18 adopts the recognition result as a final recognition result when the recognition result of the OCR 10 and the recognition result of the external character recognition system match, and when they do not match, The recognition result input by looking at the data is adopted as the final recognition result. Since the post-processing B uses the recognition result of the OCR 10 in the same manner as the post-processing A, the dependency of the post-processing B on the OCR 10 may be regarded as being equivalent to that of the post-processing A. Generally, since the cost for a person is higher than the cost of a character recognition system using a computer, the post-process B is lower than the post-process A. For this reason, the order of the post-stage process B is the order after the post-stage process A (the number is larger).

  In addition, for example, when the input image data and the recognition result of the OCR 10 are presented to a person and the person determines that the recognition result of the OCR 10 is correct, a simple input indicating that fact (for example, pressing the correct button) is accepted. If it is determined that there is an error, a post-processing unit 18 (referred to as post-processing C) that accepts the input of the character string of the person's recognition result as the final recognition result may be used. This post-stage process C is more expensive than the example of the post-stage process N described above that does not use “other means” at all, but is lower in cost than the post-stage processes A and B described above (the “other means” is 1). Only humans). Further, it can be said that the degree of dependence on the OCR 10 is higher than that of the post-stage processes A and B in the sense that there are fewer means related to the determination of the final recognition result than the post-stage processes A and B. Accordingly, the order of the post-stage process C is between the above-described post-process N and the post-stage process A that do not use “other means” at all.

  Further, as a variation of the post-processing C, a plurality of recognition result candidates recognized by the OCR 10 from the same input image data are presented to several people in descending order of recognition accuracy. A simple operation of selecting the correct answer may be performed, and if there is no correct answer, a post-process D of inputting a character string recognized by the person may be used. Since the post-processing D is less time-consuming for human input, the number of processes that can be processed per hour increases, so the cost per hour is expected to be lower than the post-processing C. For this reason, the post-process D is later than the post-process C.

  In the present embodiment, the recognition accuracy required by the OCR 10 is divided into N sections, and the sections are associated one by one in the order of rank with the N post-processing units 18. That is, a higher recognition accuracy section is associated with a higher-order post-processing unit 18. Then, in order to verify the final character recognition result for the input image data, the determination system recognizes the recognition accuracy for the input image data output by the OCR 10 among these N ranked post-processing units 18. The post-processing unit 18 associated with the section to which the user belongs is selected and operated. The post-processing unit 18 not selected is not operated.

  The threshold value DB 14 shown in FIG. 1 holds (N−1) threshold values that delimit these N intervals. The threshold comparison processing unit 12 compares the recognition accuracy obtained by the OCR 10 for the text code of the recognition result to be output with those (N−1) threshold values, and which of the N intervals the recognition result belongs to. Determine. The result of this determination is any number from 1 to N indicating the section, and this number functions as information for specifying the post-processing unit 18 corresponding to the section. The separation processing unit 16 receives the section number output by the threshold comparison processing unit 12 and selectively validates the subsequent processing unit 18 corresponding to the section number among the N subsequent processing units 18. The validated post-processing unit 18 determines and outputs the final character recognition result for the input image data using the input information (the recognition result of the OCR 10, the input image data, etc.). Other post-processing units 18 do not operate. The integration processing unit 19 uses the output of the subsequent processing unit 18 corresponding to the section number obtained from the threshold comparison processing unit 12 among the N subsequent processing units 18 as the character recognition result of this determination system for the input image data. Output. The integrated processing unit 19 discards the output of the other subsequent processing unit 18 (if it exists).

  The threshold value group held in the threshold value DB 14 is set by the threshold value setting processing device 20. The threshold value setting processor 20 determines (N−1) threshold values by processing a large number of learning data. As the data for learning, as illustrated in FIG. 2, the recognition accuracy value and the character recognition result for each of M (very many) character recognitions performed by the OCR 10 in the past are correct, A pair with correct / incorrect information indicating which of the errors is used is used. As the recognition accuracy value, the value output by the OCR 10 may be recorded. The correct / incorrect information is a binary value indicating whether the character recognition result is correct or incorrect. In the following description, the correct / incorrect information is information that is “1” when the answer is correct and “0” when there is an error. For example, it is only necessary that a person confirms whether the recognition result of the OCR 10 is correct and inputs correct / incorrect information.

The points of threshold setting processing performed by the threshold setting processing device 20 are as follows.
(1) A considerable number of pairs of recognition accuracy and correct / incorrect information are input as learning data.
(2) The target recognition rate required for the text code of the recognition result used by the subsequent processing unit 18 is set in each of the subsequent processing 1 to N. The target recognition rate is set higher as the number of N in the subsequent process is larger.
(3) From the subsequent process N to the subsequent process 1 in order, a threshold that can achieve the target recognition rate of each subsequent process K (1 ≦ K ≦ N) is calculated.

  The threshold setting processing device 20 includes a learning data input unit 22, a cumulative data calculation unit 24, a target recognition rate setting unit 26, and a threshold calculation unit 28.

  The learning data input unit 22 inputs M learning data (a pair of recognition accuracy and correct / incorrect information), and sorts the M learning data in the order of recognition accuracy.

  The accumulated data calculation unit 24 uses the sorted learning data to calculate the cumulative number of correct answers from the first learning data to the learning data in each order. Details will be described later.

  The target recognition rate setting unit 26 sets a target recognition rate for each of the subsequent processes 1 to N. The target recognition rate of the subsequent process K is a recognition rate that the subsequent process K should satisfy. In one example, this setting is made by the user. In the example of FIG. 8 described later, the target recognition rate can be automatically set.

  Based on the cumulative number of correct answers calculated by the cumulative data calculation unit 24 and the target recognition rate of each subsequent process set by the target recognition rate setting unit, the threshold calculation unit 28 divides a recognition accuracy interval corresponding to each subsequent process. (N-1) threshold values are calculated.

  A threshold value calculation method performed by the threshold value calculation unit 28 will be described with reference to FIGS.

In order to divide the possible range of the recognition accuracy X into N sections, it is necessary to determine (N−1) threshold values. The N threshold values set here are T ₁ , T ₂ ,..., T _N−1 . Even if the recognition accuracy X can be a real number from 0 to 1, the generality is not lost, so in the following example, it is defined as such. In addition, the number of the section (the subsequent process) is K.

The threshold calculation unit 28 executes the threshold calculation procedure illustrated in FIG. In this procedure, first, both ends of the threshold are set to T ₀ = 0 and T _N = 1, and the initial value J ₀ of the threshold index J _K is set to 0 (S10). In this example, the index j when T _K = X _j is expressed as j = J _NK . That is, the procedure described below, to determine the threshold value T _K, it can be regarded as an algorithm to calculate a threshold index J _NK. X _j is the recognition accuracy of the j-th learning data. However, 1 ≦ j ≦ M. As will be described later, it is assumed that if i> j, sorting is performed so that X _i ≦ X _j .

Next, a target recognition rate Y _K for each section K is set (S12). The target recognition rate Y _K is a recognition rate (that is, a correct recognition rate of character recognition) that is a target that the OCR 10 should achieve for the subsequent processing K corresponding to the section K. Here, the target recognition rate Y _K is set to increase as the number of the section K increases. The reason is as follows.

That is, the determination system illustrated in FIG. 1 selects any one of the post-processing 1 to N for the input image data, and the selected post-processing K regarding the character string included in the input image data The final recognition result as a judgment system is output. Since it is necessary to satisfy a certain recognition rate required for the entire determination system (that is, to obtain a higher recognition rate as an average), the selected post-processing K needs to satisfy the certain recognition rate. The post-stage process K obtains the recognition result as the post-stage process K by combining the recognition result of the OCR 10 with the recognition result by other means. Here, as described above, as the number K increases, the dependency of the post-processing K on the recognition result of the OCR 10 increases. Therefore, in order for the recognition rate as the post-processing K to satisfy the recognition rate required for the determination system, it is necessary to increase the recognition rate of the OCR 10 as K increases. For this reason, the target recognition rate Y _K of the OCR 10 is set so as to increase as K increases.

The user sets the target recognition rate Y _{K in} S12, for example. An example of automatically calculating the target recognition rate Y _K will be described later.

Next, the threshold value calculation unit 28 sorts the learning data in descending order of recognition accuracy (S14). As described above, each learning data is a pair of recognition accuracy X and correct / incorrect information F. In the learning data group sorted in descending order of the recognition accuracy X, the relationship X _i ≦ X _j is established if i> j. That is, as the value of the index j increases, the recognition accuracy X _j in the learning data j decreases monotonously. By sorting the learning data in advance according to the recognition accuracy, the calculation of the cumulative correct number S (i), which will be described later, becomes faster. In addition, by calculating the cumulative correct answer number S (i) in advance, the threshold value calculation process becomes faster (it is not necessary to add the number of learning data items that enter the desired subsequent process every time).

FIG. 5 schematically shows the relationship between the recognition accuracy X _j , the threshold index J _K and the threshold T _K in each learning data j, and each section (post-stage processing) K after this sorting. As shown in FIG. 5, the section K to which the post-stage processing K is applied is a section in which the recognition accuracy X is T _K−1 or more and less than T _K. The value of the target recognition rate set in this section K is Y _K. Recognition accuracy X _j of each learning data j is the value as the index j is increased is reduced. When the number of learning data is M, X _M is the minimum value of the recognition accuracy in the learning data set. From the definition, the recognition accuracy X _j when j = J _NK is the threshold value T _K.

In the procedure of FIG. 4, as shown in FIG. 5, the threshold value T _K is determined in the direction of decreasing K from K = N. In other words, the threshold index J _m is increased from 0 to m. It is also determined to become the direction.

  Returning to the description of the procedure in FIG. After S14, the threshold value calculation unit 28 calculates the cumulative number of correct answers S (i) for each index i using the equation (1) shown (S16). That is, the cumulative correct answer number S (i) is the sum of correct / incorrect information Fj (1 for correct answer, 0 for incorrect answer) from 1 to i in each learning data j.

Next the threshold calculation unit 28 initializes the index K of the threshold T _K to N (= total number of post-processing) (S18).

Next, the threshold value calculation unit 28 executes a process for determining the threshold value T _K-1 (S20). In the section N processed in the first loop, the upper limit threshold T _N is set to 1, and the lower limit threshold T _N-1 of the section N is determined in S20. A detailed example of the process of S20 will be described later with reference to FIG.

When the determination of the threshold value T _K-1 is finished, the threshold value calculation unit 28 decreases the index K by 1 (S22), and determines whether or not the result K has reached 1 (S24). When K is not 1 (that is, when K is 2 or more), determination of all threshold values has not been completed yet, so the process returns to S20 to determine threshold value T _K-1 . When K reaches 1, it means that the determination of all threshold values T _{1 to} T _N-1 to be obtained has been completed, and the processing of FIG. 4 ends.

Next, with reference to FIG. 7, a detailed procedure of the process (S20) for determining the threshold value T _K-1 is illustrated.

At the start of this procedure, the thresholds T _N , T _N−1 , T _N−2 ,..., T _K have been determined.

  In this procedure, the threshold value calculation unit 28 first initializes the index j of the cumulative number of correct answers S (j) to M (that is, the total number of learning data) (S202).

Next, the threshold value calculation unit 28 determines whether or not the illustrated equation (2) is satisfied (S204). Referring to FIG. 5, S (J _NK ) is a threshold index J _NK (=) of a threshold value T _K that has already been determined from correct / incorrect information F ₁ in the learning data including the maximum value X ₁ of recognition accuracy. j (K)) is the sum of up to correct / incorrect information F _{j (K)} in the learning data including the recognition accuracy X _{j (K)} corresponding to _{( j (K))} . On the other hand, S (j) is the sum from the correct / incorrect information F ₁ to the correct / incorrect information F _j corresponding to the index j larger than the threshold index J _NK . These differences (S (j) −S (J _NK )) are the total number of correct answers in the section from J _NK to j, and when this is divided by (j−J _NK ), the correct answer of OCR 10 in that section Rate is obtained.

If this accuracy rate is equal to or higher than the target recognition rate Y _K of the section K that is the target of the current threshold determination process, the section from J _NK to j satisfies the target recognition rate condition for that section K ( The determination result of S204 is Yes). In this case, the threshold value calculation unit 28 employs the recognition accuracy X _j corresponding to j at this time as the threshold value T _K-1 that defines the lower limit of the recognition accuracy of the section K (S206). At this time, the j is stored as a threshold index J _{N-K + 1} corresponding to the threshold T _K−1 .

If S204 the determination result is No, the threshold value calculation unit 28, the index j 1 Herashi (S208), whether a new index j as a result of reduced has reached the threshold index J _NK corresponding to the upper limit of the interval K Determine (S210). When the result of this determination is No, the threshold value calculation unit 28 returns to S204 and evaluates the expression (2) for the new index j.

The evaluation of Expression (2) is repeatedly performed while decreasing the index j by 1 in order from the maximum value M (S208). When j reaches J _NK (the determination result in S210 is Yes), the recognition accuracy that can enter the section K. This means that X does not exist. In this case, the threshold calculation unit 28 invalidates the subsequent process K corresponding to the section K (S212). That is, in the determination process for actual input image data executed by the determination system using the threshold value group set in the threshold value setting process, the subsequent process K is not used.

In this way, in the procedure of FIG. 7, the evaluation proceeds in the direction in which the index j decreases in order from the maximum value M, so the section K determined by this procedure is the section of the maximum width that satisfies the target recognition rate Y _K. . The procedure of FIG. 4 determines the section K (lower threshold T _K-1 ) in order from the section K in which the corresponding recognition accuracy X (corresponding target recognition rate Y _{K in} another viewpoint) is high. The sections with the maximum width satisfying the target recognition rate Y _K are secured in order from the section K with the highest recognition accuracy X. For example, in the example of FIG. 6, first, the lower limit threshold value T _N−1 of the section N is set so that the section N of the subsequent processing N with the highest recognition accuracy has a maximum width in a range satisfying the target recognition rate Y _N. Next, the threshold value T _N−2 is determined so that the section (N−1) has the maximum width in a range satisfying the target recognition rate Y _N−1 . Such determination process, the corresponding recognition accuracy is repeated until it is determined (lower threshold i.e. the section 2) T ₁ upper threshold of lowest section 1.

The later processing K corresponding to the section K having a higher recognition accuracy (or target recognition rate Y _K ) has a higher dependency on the OCR 10, that is, a lower dependency on one or more “other means” that costs more than the OCR 10. Therefore, in the procedure of FIG. 4, since the maximum width section that satisfies the target recognition rate Y _K is secured in order from the low-cost post-processing K, the low-cost post-processing K is selected when processing the input image data. As a result, the processing cost of the entire determination system is reduced (theoretically, the cost is minimized under a given learning data group).

  Heretofore, the configuration and operation of the threshold setting processing device 20 of the present embodiment have been described. Next, referring to FIG. 8, in accordance with a specific example of a determination system including three specific post-processing units 18 (post-processing 1 to 3), target recognition of a section corresponding to each post-processing unit 18 An example of automatically determining the rate is shown.

  FIG. 8 shows the post-processing units 18-1, 18-2, and 18-3, the OCR 10 that supplies recognition results to the processing units 18-1, 18-2, and the separation processing unit 16 in the determination system of this specific example. FIG. 8 further shows manual input devices 30-1, 30-2, and 30-3 that provide input image data by a human operator to the determination system (manual input devices when there is no need to distinguish them from each other). 30).

  The manual input device 30 displays input image data to be character-recognized on a screen, accepts input of a recognition result of a character string included in the input image data from an operator who is a person, and receives characters of the received recognition result The column is transmitted to the subsequent processing units 18-1, 18-2, and 18-3. The manual input device 30 is, for example, application software on each operator's personal computer connected to the determination system via the Internet.

  The post-processing unit 18-3 (post-processing 3) corresponds to a section having the highest recognition accuracy (target recognition rate in another viewpoint) among the three post-processing units 18, and in this example, the recognition result of the OCR 10 And outputs the recognition result as it is as its own recognition result.

  The post-stage processing unit 18-2 (post-stage process 2) corresponds to the section in which the recognition accuracy is the middle of the three post-stage processing units 18. In addition to the recognition result of the OCR 10, a character string of the recognition result of each operator is input from the manual input devices 30-1 and 30-2 to the post-processing unit 18-2. When the post-processing unit 18-2 is selected from the separation processing unit 16 in accordance with the recognition accuracy obtained by the OCR 10, the post-processing unit 18-2 supplies input image data to the manual input device 30-1, and A character string (text code) input by the operator as a recognition result of the input image data is acquired. Then, the recognition result obtained from the OCR 10 and the recognition result obtained from the manual input device 30-1 are matched, and when both match, the matching recognition result is used as the recognition result of the subsequent processing unit 18-2. Output. On the other hand, if they do not match, the post-processing unit 18-2 supplies the input image data to another manual input device 30-2, and the operator of the manual input device 30-2 receives the input image. The character string input as the data recognition result is acquired, and the character string is output as the recognition result of the post-processing unit 18-2. In this case, the operator of the manual input device 30-2 is assumed to have a higher accuracy in recognizing the character string in the input image data than the operator of the manual input device 30-1 (for example, past results are Good person).

  The post-stage processing unit 18-3 (post-stage process 3) corresponds to a section having the lowest recognition accuracy among the three post-stage processing units 18. The post-processing unit 18-3 does not use the recognition result of the OCR 10, but uses the recognition result input by each operator to the manual input devices 30-1, 30-2, and 30-3, and uses the recognition result of the post-processing unit 18-3. -2 is performed. That is, when the post-processing unit 18-3 is selected from the separation processing unit 16 according to the recognition accuracy obtained by the OCR 10, it supplies the input image data to the manual input devices 30-1 and 30-3, A character string input as a recognition result of the input image data by each operator of the input devices 30-1 and 30-3 is acquired. If the recognition results obtained from the two manual input devices 30-1 and 30-3 are matched, and the two match, the matching recognition results are used as the recognition results of the subsequent processing unit 18-3. Output. On the other hand, if they do not match, the post-processing unit 18-2 supplies the input image data to another manual input device 30-2, and the operator of the manual input device 30-2 receives the input image data. The character string input as the recognition result is acquired, and the character string is output as the recognition result of the post-processing unit 18-2. In this case, the operator of the manual input device 30-2 is assumed to be more accurate in recognizing the character string in the input image data than the operators of the manual input devices 30-1 and 30-3. It is good.

The three sections 1, 2, and 3 of the recognition accuracy associated with the three subsequent processing units 18-1, 18-2, and 18-3 have two threshold values T ₁ and T ₂ (T ₁ <T ₂ ) separated by. Separation processing unit 16 selects the post-processing unit 18-1 if recognition accuracy X is less than T ₁ to OCR10 outputs, select the post-processing unit 18-2 is less than above T ₁ T _2, T ₂ If so, the post-processing unit 18-3 is selected.

The threshold setting processing apparatus 20 includes a section 1 (T ₁ > X), a section 2 (T ₂ > X ≧ T ₁ ), a section 3 (corresponding to the three subsequent processing units 18-1, 18-2, and 18-3. The target recognition rates Y ₁ , Y ₂ , Y ₃ of the OCR 10 corresponding to each of X ≧ T ₂ ) are calculated as follows.

First, let R be the target recognition rate of the determination system (that is, the target value of the correct answer rate of the final output of the determination system). Since the post-processing unit 18-1 does not use the recognition result of the OCR 10 at all, the recognition rate of the OCR 10 may be zero. Therefore, the threshold setting processing device 20 sets the target recognition rate Y ₁ = 0. The post-processing unit 18-1 itself satisfies the target recognition rate R of the determination system by appropriately selecting each operator of the manual input devices 30-1, 30-2, and 30-3.

On the other hand, since the post-processing unit 18-3 uses the recognition result of the OCR 10 as its output as it is, the target recognition rate Y ₃ = R.

For the remaining post-processing unit 18-2, the target recognition rate Y2 of the OCR ₁₀ is calculated as follows.

First, let λ be an error rate when a person performs data entry (that is, processing for recognizing a character string included in input image data and inputting it to the manual input device 30). In other words, the correct answer rate (recognition rate) of the human data entry is (1−λ). On the other hand, the error rate of the recognition result of the OCR 10 is ω. That is, the accuracy rate (recognition rate) of the OCR 10 is (1-ω). The approximate value of the error rate of the matching process between the OCR 10 and the human recognition result by the post-processing unit 18-2 is λω. Considering that the post-processing unit 18-2 needs to satisfy the target recognition rate R of the determination system,
λω = (1-R)
Holds. Therefore, since the target recognition rate Y ₂ of the OCR 10 when the post-processing unit 18-2 is selected may be considered to be equal to (1-ω), finally Y ₂ is the error rate λ of a known person. It is calculated from the target recognition rate R of the determination system as follows.
Y ₂ = 1−ω = 1− (1-R) / λ

The threshold setting processor 20 may calculate the threshold Y ₂ according to this equation.

  The embodiment described above is merely an example of realization of the present invention.

In the above example, the cumulative number of correct answers S (j) is used in determining the threshold value, but the cumulative number of errors may be used instead. The number of errors is the number of learning data with F _j = 0. Even if the cumulative correct answer rate (recognition rate) obtained by dividing the cumulative correct answer number by the number of samples is used instead of the cumulative correct answer number S (j) (or error number), the same processing as in the case of using the cumulative correct answer number is performed. Is possible. Whether each section K includes the upper limit or lower limit threshold value T _K or T _K-1 is not limited to the above example, and may be determined as appropriate.

In the above-described embodiment, learning data that is a pair of recognition accuracy X and correct / incorrect information F is input to the threshold setting processing device 20, but this is merely an example. Instead, information on the accumulation result obtained by accumulating the correctness / incorrectness information F to the recognition accuracy X _j to which attention is paid in ascending order of the corresponding recognition accuracy X is calculated in advance, and the information and its recognition accuracy are calculated. A pair with X _j may be input to the threshold setting processing device 20 as learning data. Here, as the information of the accumulation result, any of the above-mentioned accumulated correct answer number, accumulated error number, accumulated correct answer rate, accumulated error rate, etc. may be used.

  In the above embodiment, the determination system recognizes a character string in input image data. However, in addition to character recognition, the determination system determines the content of input data and outputs the determination result. The method of the above embodiment can be applied to the entire system. That is, the determination system to which the present invention is applied includes a primary determination means (corresponding to OCR10) for determining the content of input data, a determination result of the primary determination means, and zero or more other determination means (for example, humans). And a plurality of subsequent processing units that obtain the determination result of the content of the data by combining the determination results of the determination means of higher accuracy but higher cost than the primary determination means. This determination system has means for obtaining determination accuracy (corresponding to recognition accuracy in the case of character recognition) for the determination result of the primary determination means, and which of the plurality of subsequent processing units is used according to this determination accuracy. decide. That is, each post-processing unit is associated with a section with different determination accuracy, and the post-processing unit corresponding to the section to which the determination accuracy of the determination by the primary determination unit belongs selectively operates.

  The determination system and the threshold setting processing device 20 exemplified above can be configured as a hardware logic circuit in one example. As another example, the determination system and the threshold setting processing device 20 may be realized, for example, by causing a built-in computer to execute a program representing the function of each functional module in the system or device. Here, the computer includes, for example, a processor such as a CPU, a memory (primary storage) such as a random access memory (RAM) and a read only memory (ROM), an HDD controller that controls an HDD (hard disk drive), and the like as hardware. A network interface that performs control for connection to a network such as an I / O (input / output) interface or a local area network has a circuit configuration connected via a bus, for example. Also, portable non-volatile recording of various standards such as a disk drive and a flash memory for reading and / or writing to a portable disk recording medium such as a CD or a DVD via the I / O interface, for example. A memory reader / writer for reading from and / or writing to a medium may be connected. A program in which the processing contents of each functional module exemplified above are described is stored in a fixed storage device such as a hard disk drive via a recording medium such as a CD or DVD, or via a communication means such as a network, and stored in a computer. Installed. The program stored in the fixed storage device is read into the RAM and executed by a processor such as a CPU, whereby the functional module group exemplified above is realized. Further, the determination system and the threshold setting processing device 20 may be configured by a combination of software and hardware.

10 OCR, 12 Threshold comparison processing unit, 14 Threshold DB, 16 Separation processing unit, 18 Subsequent processing unit, 19 Integration processing unit, 20 Threshold setting processing device, 22 Learning data input unit, 24 Cumulative data calculation unit, 26 Target recognition Rate setting unit, 28 threshold calculation unit, 30 manual input device.

Claims (11)

A determination means for determining the input;
Calculation means for calculating the determination accuracy of the determination means for the input;
A plurality of post-processing units that can generate an output for the input by performing post-processing on the determination result of the determination unit, and have different degrees of dependence on the determination result of the determination unit in the generation of the output. A plurality of subsequent processing means respectively associated with each section obtained by dividing the range that the determination accuracy can take with one or more threshold values;
Control means for controlling the subsequent processing means corresponding to the section to which the determination accuracy calculated by the calculation means belongs to generate an output for the input;
An information processing apparatus that determines a threshold value that divides the section for the determination accuracy, for a determination system including:
For each past input to the determination means, an acquisition means for acquiring a set of the determination accuracy for the input and correct / incorrect information indicating whether the determination result of the determination means for the input is correct or incorrect;
Using the set acquired by the acquisition means, the correct answer rate of the determination means obtained from the set of sets belonging to the section in order from the section with the highest determination accuracy is the target correct answer of the determination means corresponding to the section. Determining means for determining the threshold value defining the section so as to satisfy the rate;
An information processing apparatus including: The target recognition rate of the determination means corresponding to each section is a higher value as the section has a higher determination accuracy.
The information processing apparatus according to claim 1, wherein the determination unit determines the threshold value that defines each section in order from the section having the highest target recognition rate. The later stage processing means corresponding to the section with higher determination accuracy uses a method with a lower cost as a method for generating the output using the determination result,
The information processing apparatus according to claim 1, wherein the determination unit determines the threshold value that defines each section in order from the section having the lowest cost. The post-processing unit corresponding to the section with the highest determination accuracy is the output of the determination result of the determination unit as it is,
The target accuracy rate set for the determination system is used as the target accuracy rate for the post-processing unit corresponding to the section with the highest determination accuracy. Information processing device.   The plurality of post-stage processing means includes second type post-stage processing means for generating an output for the input without using the determination result of the determination means, and the second type of post-stage processing means includes the section of the section. The information processing apparatus according to claim 1, wherein the information processing apparatus is associated with a section having the lowest determination accuracy.   The information processing apparatus according to claim 5, wherein the target accuracy rate corresponding to the second-stage post-processing unit is zero. The plurality of post-processing units output the output based on a first post-processing unit that uses the determination result of the determination unit as it is as the output, and a determination result by a person with respect to the input without using the determination result of the determination unit. A second post-processing unit for generating, and a third post-processing unit for generating the output by matching the determination result of the determination unit and the determination result by the person with respect to the input,
The target accuracy rate set for the determination system is used as the target accuracy rate for the first post-processing means, and 0 is used as the target accuracy rate for the second post-processing means. The information processing apparatus according to any one of claims 1 to 3, wherein the target correct answer rate for three subsequent processing means is obtained from the correct answer rate of the person and the correct answer rate of the determination means. The acquisition means, instead of a set of determination accuracy and correct / incorrect information, includes determination accuracy, and information on the accumulation result of the correct / incorrect information corresponding to each determination accuracy within a range from the maximum value of the determination accuracy to the determination accuracy. Get a pair of,
8. The information processing according to claim 1, wherein the determining unit obtains a correct answer rate of a section in which the threshold is to be determined using information on the cumulative result corresponding to each determination accuracy. apparatus. A determination means for determining the input;
Calculation means for calculating the determination accuracy of the determination means for the input;
A plurality of post-processing units that can generate an output for the input by performing post-processing on the determination result of the determination unit, and have different degrees of dependence on the determination result of the determination unit in the generation of the output. A plurality of subsequent processing means respectively associated with each section obtained by dividing the range that the determination accuracy can take with one or more threshold values;
Control means for controlling the subsequent processing means corresponding to the section to which the determination accuracy calculated by the calculation means belongs to generate an output for the input;
For a determination system including:
For each past input to the determination means, an acquisition means for acquiring a set of the determination accuracy for the input and correct / incorrect information indicating whether the determination result of the determination means for the input is correct or incorrect;
Using the set acquired by the acquisition means, the correct answer rate of the determination means obtained from the set of sets belonging to the section in order from the section with the highest determination accuracy is the target correct answer of the determination means corresponding to the section. Determining means for determining the threshold value defining the section so as to satisfy the rate;
Program to function as. A determination means for determining the input;
Calculation means for calculating the determination accuracy of the determination means for the input;
A plurality of post-processing units that can generate an output for the input by performing post-processing on the determination result of the determination unit, and have different degrees of dependence on the determination result of the determination unit in the generation of the output. A plurality of subsequent processing means respectively associated with each section obtained by dividing the range that the determination accuracy can take with one or more threshold values;
Control means for controlling the subsequent processing means corresponding to the section to which the determination accuracy calculated by the calculation means belongs to generate an output for the input;
For each past input to the determination means, an acquisition means for acquiring a set of the determination accuracy for the input and correct / incorrect information indicating whether the determination result of the determination means for the input is correct or incorrect;
Using the set acquired by the acquisition means, the correct answer rate of the determination means obtained from the set of sets belonging to the section in order from the section with the highest determination accuracy is the target correct answer of the determination means corresponding to the section. Determining means for determining the threshold value defining the section so as to satisfy the rate;
An information processing apparatus including: Computer
A determination means for determining the input;
Calculation means for calculating the determination accuracy of the determination means for the input;
A plurality of post-processing units that can generate an output for the input by performing post-processing on the determination result of the determination unit, and have different degrees of dependence on the determination result of the determination unit in the generation of the output. A plurality of subsequent processing means associated with each section obtained by dividing the range that the determination accuracy can take by one or more threshold values,
Control means for controlling the post-processing means corresponding to the section to which the determination accuracy calculated by the calculation means belongs to generate an output for the input;
For each past input to the determination means, an acquisition means for acquiring a set of the determination accuracy for the input and correct / incorrect information indicating whether the determination result of the determination means for the input is correct or incorrect;
Using the set acquired by the acquisition means, the correct answer rate of the determination means obtained from the set of sets belonging to the section in order from the section with the highest determination accuracy is the target correct answer of the determination means corresponding to the section. Determining means for determining the threshold value defining the section so as to satisfy the rate;
Program to function as.

Images