JP7404625B2 - Information processing device and program - Google Patents

Description

The present invention relates to an information processing device and a program.

Techniques are being considered to improve the accuracy of character recognition. Patent Document 1 discloses that a text area of an input image is divided into a text line area and a line spacing area, a line spacing string that is a character string in the line spacing area is extracted, and a character string in the body line area is extracted for each line spacing string. A ruby dictionary that determines a tentative parent character string from among them, recognizes characters in the text line area and interline area, and shows ruby candidates for the parent character using the recognition result of the temporary parent string as a reference key. It is determined whether the recognition result of the line spacing character string matches at least one of the ruby candidates obtained by referring to A document recognition device is described that determines the remaining character string after excluding the characters , as the final parent character string.

Patent Document 2 discloses that image data that is a first character string is character-recognized and converted into a character string code, and image data that is a second character string that has the same pronunciation as the first character string but a different character type is generated. Recognize the characters and convert them into character string codes, convert the first recognized character string into a character string of the same character type as the second character string, and combine the first character string with the first character string with the same character type as the second character string. A character recognition device is described that compares a second character string with a second character string and corrects the second character string based on the first character string if the first character string and the second character string are different. .

Patent Document 3 discloses that when selecting a corresponding kanji or Roman character based on shape characteristics for each character image information extracted from image information of a manuscript, when a plurality of kanji are selected for specific kanji image information, A character recognition method that searches for a kanji that corresponds to specific kanji image information from among a plurality of selected kanji based on a roman character selected for roman character image information that has a predetermined relationship with this kanji image information within image information. is listed.

Note that Patent Document 4 describes an information processing device that eliminates the need for information regarding boundaries between single characters as a teacher signal in machine learning for recognizing characters to be recognized.

Japanese Patent Application Publication No. 2012-212293 Japanese Patent Application Publication No. 9-138835 Japanese Patent Application Publication No. 2010-282272 Japanese Patent Application Publication No. 2016-212473

One of the objects of the present invention is to obtain an indication as to which of the character strings respectively recognized from two related images is to be trusted.

The information processing device according to claim 1 of the present invention includes a first recognition unit that recognizes a first character string from a first image, and a second recognition unit that recognizes a second character string from a second image related to the first image. a recognition unit; an extraction unit that extracts one or more third character strings related to the first character string by referring to a dictionary in which different character strings are associated in advance; a calculation unit that respectively calculates a first similarity indicating a degree of similarity to a second character string, a first reliability indicating a reliability of character recognition of the first character string , and a reliability of character recognition of the second character string; When the reliability of the third character string with respect to the second character string, which is evaluated using the second reliability indicating the degree of similarity and the first similarity, is less than a threshold, the second An information processing device comprising: an output unit that outputs a character string and outputs the third character string as the first information in place of the second character string when the reliability is equal to or higher than the threshold value. .

In the information processing apparatus according to claim 2 of the present invention, in the aspect according to claim 1 , the extraction unit is configured to extract the information from the third reliability when the first reliability and the second reliability satisfy a predetermined condition. This is an information processing device characterized by extracting character strings.

In the information processing device according to claim 3 of the present invention, in the aspect according to claim 1 or 2 , the extraction unit refers to a dictionary in which different character strings are associated in advance, and the calculation unit calculates, for each of the fourth character strings, a second degree of similarity indicating the degree of similarity to the first character string, and the output unit: Outputting second information based on at least one of the second similarity and the fourth character string according to at least one of the first reliability and the second reliability and the second similarity. This is an information processing device characterized by the following.

In the information processing device according to claim 4 of the present invention, in the aspect according to claim 3 , the output unit is evaluated using the first reliability, the second reliability, and the second similarity. outputting the first character string as the second information when the reliability of the fourth character string with respect to the first character string is less than a threshold; and when the reliability is greater than or equal to the threshold; The information processing apparatus is characterized in that the fourth character string is output as the second information in place of the first character string.

In the information processing apparatus according to claim 5 of the present invention, in the aspect according to claim 3 or 4 , the extracting unit extracts the information when the first reliability and the second reliability satisfy a predetermined condition. This is an information processing device characterized by extracting a fourth character string.

In the information processing device according to claim 6 of the present invention, in the aspect according to claim 1 , the second recognition unit calculates a weight determined for each of the one or more feature amounts calculated from the second image. When the second character string is recognized and the output unit outputs the third character string instead of the second character string, the third character string is recognized from the second image based on the amount added and totaled. The information processing apparatus is characterized in that the weights are modified so as to recognize columns.

In the information processing device according to claim 7 of the present invention, in the aspect according to claim 4 , the first recognition unit calculates a weight determined for each of the one or more feature amounts calculated from the first image. When the first character string is recognized and the output unit outputs the fourth character string instead of the first character string, the fourth character from the first image is recognized based on the amount added and totaled. The information processing apparatus is characterized in that the weights are modified so as to recognize columns.
The information processing device according to claim 8 of the present invention includes a first recognition unit that recognizes a first character string from a first image, and a second recognition unit that recognizes a second character string from a second image related to the first image. A recognition unit refers to a dictionary in which different character strings are associated in advance, and extracts one or more third character strings related to the first character string, and extracts one or more third character strings related to the second character string. an extraction unit that extracts a fourth character string; and an extraction unit that calculates, for each of the third character strings, a first degree of similarity indicating the degree of similarity to the second character string; a calculation unit that respectively calculates a second degree of similarity indicating a degree of similarity to a string; a first degree of reliability indicating a degree of reliability of character recognition of the first character string; and a degree of reliability indicating a degree of reliability of character recognition of the second character string; According to at least one of the second reliability and the first similarity, first information based on at least one of the first similarity and the third character string is output, and the first reliability, When the reliability of the fourth character string with respect to the first character string, which is evaluated using the second reliability and the second similarity, is less than a threshold, the first character string is used as second information. and an output unit that outputs the fourth character string instead of the first character string as the second information when the reliability is equal to or higher than the threshold value.

In the information processing device according to claim 9 of the present invention, in the aspect according to any one of claims 1 to 8 , the second image is a character string indicating the pronunciation of the character string included in the first image. The information processing apparatus is characterized in that the image is an image including the following information.

A program according to claim 10 of the present invention includes a first recognition unit that recognizes a first character string from a first image, and a second recognition unit that recognizes a second character string from a second image related to the first image. 2 recognition unit, an extraction unit that extracts one or more third character strings related to the first character string by referring to a dictionary in which different character strings are associated in advance, and for each third character string, a calculation unit that respectively calculates a first similarity degree indicating a degree of similarity to the second character string; a first reliability degree indicating a reliability degree of character recognition of the first character string; and a first degree of reliability indicating a degree of character recognition of the second character string; When the reliability of the third character string with respect to the second character string , which is evaluated using the second reliability indicating the reliability and the first similarity, is less than a threshold, the third character string is used as the first information. 2 character strings, and outputs the third character string as the first information in place of the second character string when the reliability is equal to or higher than the threshold value . be.
The program according to claim 11 of the present invention includes a computer that includes a first recognition unit that recognizes a first character string from a first image, and a second recognition unit that recognizes a second character string from a second image related to the first image. 2. A recognition unit refers to a dictionary in which different character strings are associated in advance, and extracts one or more third character strings related to the first character string, and extracts one or more third character strings related to the second character string. an extraction unit that extracts a fourth character string of a calculation unit that respectively calculates a second degree of similarity indicating a degree of similarity to a character string, a first degree of reliability indicating a degree of reliability of character recognition of the first character string, and a degree of reliability of character recognition of the second character string; outputting first information based on at least one of the third character string and the first similarity according to at least one of the second reliability shown and the first similarity; , when the reliability of the fourth character string with respect to the first character string, which is evaluated using the second reliability and the second similarity, is less than a threshold, the first character is used as second information. The program functions as an output unit that outputs the fourth character string as the second information in place of the first character string when the reliability is equal to or higher than the threshold value.

According to the inventions according to claims 1 and 10, an index can be obtained as to which of the character strings respectively recognized from two related images is to be trusted.
Further, according to the invention according to claims 1 and 10 , the reliability of the third character string related to the first character string, which is extracted with reference to the dictionary, is equal to or higher than a threshold value with respect to the second character string. Sometimes a third string of output is obtained.
According to the invention according to claim 2 , the conditions for performing the process of extracting the third character string related to the first character string with reference to the dictionary are set based on the reliability of each of the first character string and the second character string. Can be set based on.
According to the invention according to claim 3 , the first character string recognized from the first image, the second character string recognized from the second image related to the first image, and the second character string extracted with reference to a dictionary. , a third character string related to the first character string, and a fourth character string related to the second character string extracted with reference to a dictionary;
can be used to obtain an index as to whether to trust the first character string or the second character string.
According to the invention according to claim 4 , when the reliability of the fourth character string related to the second character string extracted with reference to the dictionary is equal to or higher than the threshold value with respect to the first character string, You will get 4 strings of output.
According to the invention according to claim 5 , the conditions for performing the process of extracting the fourth character string related to the second character string with reference to the dictionary are set based on the reliability of each of the first character string and the second character string. Can be set based on.
According to the invention according to claim 6 , the accuracy of recognizing the second character string is improved compared to the case where the weight given to the feature amount calculated from the second image is not corrected.
According to the inventions according to claims 7, 8, and 11, the accuracy of recognizing the first character string is improved compared to the case where the weight given to the feature amount calculated from the first image is not corrected.
According to the invention according to claim 9, the second character string including the character string indicating the pronunciation of the character string included in the first image is recognized from the second image.

FIG. 1 is a diagram showing the configuration of an information processing device 1. FIG. 3 is a diagram showing an example of an area correspondence table 121 stored in the storage unit 12. FIG. A diagram showing an example of a handwritten area. FIG. 3 is a diagram showing an example of a character recognition model 122 stored in the storage unit 12. FIG. 3 is a diagram showing an example of a dictionary DB 123 stored in the storage unit 12. FIG. 3 is a conceptual diagram for explaining a classification model 124 stored in the storage unit 12. FIG. FIG. 1 is a diagram showing a functional configuration of an information processing device 1. FIG. A diagram for explaining an example of character recognition. FIG. 2 is a flow diagram showing the flow of operations of the information processing device 1. FIG.

<Embodiment>
<Configuration of information processing device>
FIG. 1 is a diagram showing the configuration of an information processing device 1. As shown in FIG. As shown in FIG. 1, the information processing device 1 includes a control section 11, a storage section 12, a communication section 13, an operation section 14, a display section 15, and an image reading section 16.

The control unit 11 has a CPU (Central Processing Unit), a ROM (Read Only Memory), and a RAM (Random Access Memory), and the CPU is a computer program (hereinafter simply referred to as a program) stored in the ROM and the storage unit 12. Each part of the information processing device 1 is controlled by reading and executing the command.

The communication unit 13 is a communication circuit connected to a communication line (not shown) by wire or wirelessly. The information processing device 1 uses the communication unit 13 to exchange information with other devices (ie, external devices) connected to the communication line.

The operation unit 14 includes operators such as operation buttons, a keyboard, and a touch panel for issuing various instructions, and receives operations by the user and sends signals to the control unit 11 according to the contents of the operations.

The display section 15 has a display screen such as a liquid crystal display, and displays images under the control of the control section 11. A transparent touch panel of the operation unit 14 may be placed on top of the display screen.

The image reading unit 16 includes a platen glass, an irradiation device that irradiates light onto the medium, an optical system that collects reflected light, and an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor. Equipped with etc. The image reading unit 16 reads an image formed on a medium such as paper placed on a platen glass under the control of the control unit 11, generates image data representing the read image, and supplies the generated image data to the control unit 11. .

The storage unit 12 is a storage means such as a solid state drive or a hard disk drive, and stores various programs, data, etc. that are read by the CPU of the control unit 11. The storage unit 12 also stores an area correspondence table 121, a character recognition model 122, a dictionary DB 123, and a classification model 124.

<Structure of area correspondence table>
FIG. 2 is a diagram showing an example of the area correspondence table 121 stored in the storage unit 12. The area correspondence table 121 is a table showing the layout of areas such as entry columns included in a handwritten paper such as a form. The area correspondence table 121 stores area names for identifying areas and area information, which is information indicating the specific range and position of the area, in association with each other. For example, in the area correspondence table 121 shown in FIG. 2, the area information corresponding to "Name Furigana" is "A2". The area information is expressed, for example, as coordinate information in a corrected image after skew correction and enlargement/reduction correction are performed on an image read from handwritten paper. For example, if the area indicated by the area information is a rectangle, the area information is expressed by the coordinates of the upper left and lower right vertices of the rectangle.

FIG. 3 is a diagram showing an example of a handwritten area. For example, "A2" surrounded by a two-dot chain line frame shown in FIG. 3 is an area identified by the area name "Name Furigana" shown in FIG. This area is identified by the area name "Name Kanji" shown in the figure.

<Configuration of character recognition model>
FIG. 4 is a diagram showing an example of the character recognition model 122 stored in the storage unit 12. The character recognition model 122 stores, for each area identified by the area name included in the area correspondence table 121, model data used for recognition processing of characters handwritten in that area, in association with each other.

This model data is, for example, a trained model generated by loading image data showing images of handwritten characters that are associated with correct character codes in advance and performing machine learning on the correspondence between character codes and handwritten characters. It is. For example, the control unit 11 of the information processing device 1 decomposes image data generated by reading an image of handwritten characters written on paper into pixel by pixel, and inputs the gradation value of each pixel to a multilayer neural network. . The control unit 11 then applies the model data read from the character recognition model 122 to the multilayer neural network, and recognizes the character code corresponding to the handwritten character based on the calculated output.

<Configuration of dictionary DB>
FIG. 5 is a diagram showing an example of the dictionary DB 123 stored in the storage unit 12. The dictionary DB 123 is a database in which different character strings are associated in advance. The dictionary DB 123 shown in FIG. 5 includes a dictionary name list 1231 and dictionary data 1232. The dictionary name list 1231 is a list of dictionary names that are identification information for identifying the dictionary data 1232. The dictionary data 1232 is data associated with each dictionary name listed in the dictionary name list 1231, and is data that stores character strings respectively related to the target character string indicated by the dictionary name.

For example, the dictionary name "Name Dictionary" is entered in the dictionary name list 1231 of the dictionary DB 123 shown in FIG. One dictionary data 1232 is associated with the dictionary name "Name Dictionary." This dictionary data 1232 is data that associates a character string indicating a kanji used in a name with a character string indicating a katakana (namely, furigana) indicating the pronunciation of the kanji. Dictionary data 1232 associated with the name dictionary is used to identify the furigana of a kanji from a kanji. If one kanji has multiple pronunciations, multiple furigana may be identified from that kanji. For example, as shown in FIG. 5, a plurality of pronunciations are associated with the kanji ``Tomuki'', such as ``Tomoki'', ``Tomonori'', and ``Yuki''.

<Configuration of classification model>
The classification model 124 stored in the storage unit 12 is used to determine which of the character strings respectively recognized from two related images is to be trusted. The information processing device 1 uses the classification model 124 to classify the feature amount consisting of three numerical values, for example, a first reliability degree, a second reliability degree, and a first similarity degree, and makes the above-mentioned judgment. conduct.

Here, the first reliability is the reliability of the first character string recognized from the image included in the read image (hereinafter referred to as the first image). Moreover, the second reliability is the reliability of the second character string recognized from the image related to the first image (hereinafter referred to as the second image). The first similarity is the similarity of a character string related to the first character string (hereinafter referred to as a third character string) extracted from the dictionary DB 123 to the second character string. The degree of similarity is a numerical value indicating the degree to which two character strings are similar to each other, and is expressed as an edit distance such as Levenshtein distance or Jaro-Winkler distance.

FIG. 6 is a conceptual diagram for explaining the classification model 124 stored in the storage unit 12. In FIG. 6, the dimension of the feature amount is assumed to be 2 for the sake of explanation. The feature amount consisting of two numerical values x and y is plotted on the xy plane as shown in FIG. These feature amounts are associated with labels indicating the classes to which they belong, so-called "labeling" in advance. In other words, these feature amounts are recognition data with correct answers and teacher data.

In the example shown in FIG. 6, the points corresponding to each feature amount are either points represented by squares or points represented by circles. The classification model 124 is model data generated based on these feature amounts that have been labeled in advance. For example, the straight line L shown in FIG. 6 is a line that classifies the plurality of points described above by type, and the parameters indicating this straight line L are an example of model data.

The classification model 124 is generated by a machine learning classification method using training data.
This training data includes, for example, feature quantities consisting of three numerical values: first reliability, second reliability, and first similarity, and two or more classes respectively associated with these feature quantities. This is data associated with . The classes are labeled with either "Trust the second character string" or "Trust the third character string", for example. Examples of this machine learning classification method include support vector machine, linear regression, and ensemble learning. Furthermore, for example, Adaboost is applied to this machine learning algorithm. Note that the labels are not limited to the two types described above, and may include, for example, labels indicating the output content of information and the presence or absence of output.

<Functional configuration of information processing device>
FIG. 7 is a diagram showing the functional configuration of the information processing device 1. As shown in FIG. In FIG. 7, the communication unit 13 and operation unit 14 of the information processing device 1 are omitted.

The control unit 11 of the information processing device 1 functions as an analysis unit 111, a recognition unit 112, an extraction unit 113, a calculation unit 114, and an output unit 115 by reading and executing a program stored in the storage unit 12. .

The analysis unit 111 acquires image data representing the image read by the image reading unit 16, and analyzes the layout of entry fields that make up this image. When the analysis unit 111 acquires the image data from the image reading unit 16, the analysis unit 111 performs various corrections such as so-called skew correction, enlargement/reduction correction, offset correction, etc. based on the lines, marks, etc. drawn on the image indicated by this image data. I do. Then, the analysis unit 111 cuts out the first image and the second image from the corrected image by referring to the area correspondence table 121.

The first image is, for example, an image drawn in the area indicated by the area information "A1", and is an image showing kanji characters handwritten by the user. The second image is, for example, an image drawn in the area indicated by the area information "A2", and is an image showing furigana handwritten by the user. Since the furigana shown in the second image is the furigana of the kanji character shown in the first image, the first image and the second image are related. The kanji and furigana are, for example, the kanji and furigana that indicate the user's name.

The recognition unit 112 performs character recognition processing on each of the first image and second image extracted by the analysis unit 111 using the model data stored in the character recognition model 122, and recognizes the handwritten characters in each area. Recognize strings. The recognition unit 112 recognizes the first character string from the first image. At this time, the recognition unit 112 functions as a first recognition unit. Furthermore, the recognition unit 112 recognizes a second character string from a second image related to the first image. At this time, the recognition unit 112 functions as a second recognition unit.

The recognition unit 112 performs edge detection and the like on each of the first image and the second image based on the gradation value, and divides the image into characters. Then, the recognition unit 112 reads model data associated with each region from the character recognition model 122, and performs recognition processing one character at a time.

Here, when performing character recognition character by character, the recognition unit 112 evaluates how much the recognized handwritten character differs from the character image included in the teacher data used to generate the model data. do. This evaluation is performed based on, for example, the number of matching pixels, the arrangement and size of clusters of pixels showing a common gradation value, and the like. Then, the recognition unit 112 calculates the reliability of character recognition for each character (hereinafter referred to as character reliability) based on the results of this evaluation.

When the recognition unit 112 performs character recognition on all the divided images, it generates a character string by arranging the characters and calculates the reliability of this character string. The reliability of a character string is calculated using Equation 4 described in Patent Document 4, for example. For example, the recognition unit 112 calculates the sum of local energy functions calculated from the character reliabilities of adjacent characters, and thereby calculates the reliability of the character string.

If there are a plurality of character recognition results for each character, the recognition unit 112 calculates reliability for each character string obtained by combining those characters. Then, the character string with the highest calculated reliability is selected as the character string recognized from the image. For example, when selecting one character string from a plurality of character string candidates, the recognition unit 112 uses the Viterbi algorithm to select a character string that minimizes the energy function represented by the sum of the local energy functions. Explore using.

FIG. 8 is a diagram for explaining an example of character recognition. For example, with respect to the handwritten characters shown in FIG. 3, the recognition unit 112 recognizes the character string "Fuji Tomoki" as the first character string and the character string "Fujimauki" as the second character string. The recognition unit 112 also calculates the reliability of each of the first character string and the second character string. The reliability is indicated by a numerical value between 0 and 1. As shown in FIG. 8, the reliability of the first character string is 0.998, and the reliability of the second character string is 0.19. Since the reliability of the first character string is closer to 1 than 0, it is presumed that character recognition is successful. On the other hand, since the reliability of the second character string is closer to 0 than 1, it is presumed that character recognition has failed.

The extraction unit 113 refers to a dictionary in which different character strings are associated in advance, and extracts one or more third character strings related to the first character string. The extraction unit 113 extracts from the dictionary DB 123 a third character string related to the first character string "Fuji Yuki" described above. The extraction unit 113 refers to the dictionary DB 123 and extracts one type of furigana, "Fuji", from the "Fuji" part. Further, the extraction unit 113 refers to the dictionary DB 123 and extracts three types of furigana, "Tomoki", "Tomonori", and "Yuki" from the "Tomoki" part. Therefore, the extraction unit 113 extracts three third character strings: "Fujitomoki", "Fujitomonori", and "Fujiyuki".

The calculation unit 114 calculates, for each third character string, a first degree of similarity indicating the degree of similarity to the second character string. This degree of similarity is calculated based on the edit distance between the second character string and the third character string. Here, the edit distance is calculated based on the type and number of editing processes to change an initial character string (hereinafter referred to as initial character string) to a target character string (hereinafter referred to as target character string). This is the numerical value. Editing processing refers to processing such as "adding", "deleting", and "replacing", for example.

The calculation unit 114 sets the edit distances of the three editing processes of "add", "delete", and "replace" to "1". Then, the calculation unit 114 calculates the sum of the edit distances of the editing processes required from the initial character string to the target character string as the edit distance from the initial character string to the target character string. When this edit distance is the first similarity, the closer the first similarity is to 0, the more similar the second character string and the third character string are, and the larger the first similarity, the more dissimilar.

Note that one "add" and one "delete" correspond to one "swap," but the calculation unit 114 employs "swap" so that the total edit distance becomes small.

In the example shown in Figure 8, the third character string "Fujitomoki" is edited to the second character string "Fujitomoki" by replacing "to" and "mo" with "ma" and "u" respectively. . Therefore, this third character string changes to the second character string by performing the editing process of "swapping" twice, so the editing distance of the third character string with respect to the second character string, that is, the first similarity is " 2".

In addition, the third character string "Fujitomonori" is created by replacing "to," "mo," and "no" with "ma," "u," and "ki," respectively, and deleting "li" to create the second character string. It is edited into the column "Fujimauki". In other words, this third character string is changed into the second character string by performing the editing process of "replace" three times and "delete" once, that is, a total of four times. Therefore, the edit distance of the third character string to the second character string, that is, the first similarity is "4".

On the other hand, the third character string "Fuji Yuuki" is edited into the second character string "Fuji Mauuki" by replacing "yu" with "ma". Therefore, since this third character string changes to the second character string by performing the editing process of "swapping" only once, the editing distance of the third character string with respect to the second character string, that is, the first similarity is It is "1".

By the way, the furigana recognized from the second image is generally a string consisting of multiple katakana characters, and even if it is misrecognized, the error is often part of the entire string. . The first similarity represents the degree of difference between the furigana extracted from the dictionary DB 123 and the recognized furigana. Therefore, the more similar the first similarity is, generally the furigana extracted from the dictionary DB 123 is more reliable than the recognized furigana. That is, in this case, it can be said that the closer the first similarity is to 0 (indicating that they are similar), the more reliable the recognition accuracy of kanji is than the recognition accuracy of furigana.

However, the larger the first similarity (indicating that they are dissimilar), the more places there are differences between the furigana extracted from the dictionary DB 123 and the recognized furigana. It is no longer possible to say that you cannot trust the accuracy.

Therefore, when there are multiple third character strings, the information processing device 1 compares the first similarity calculated for each of these third character strings and selects the one that shows the most similarity (in this case, the one closest to 0). Select the third character string.

The output unit 115 outputs first information that is information based on at least one of the first similarity and the third character string, depending on the first similarity. For example, as shown in Figure 8, the character string "Fuji Yuuki" is recognized as the first character string, the character string "Fuji Mauuki" is recognized as the second character string, and the character string "Fuji Yuuki" is extracted as the third character string. Suppose that In this case, the first reliability that is the reliability of the first character string is "0.998", the second reliability that is the reliability of the second character string is "0.19", and the second reliability of the third character string is "0.998". The first similarity to the character string is "1".

At this time, the output unit 115 outputs the classification model 124 for the case where the feature amount indicated by (first reliability, second reliability, first similarity) is (0.998, 0.19, 1). and determine whether to trust the second character string or the third character string. Then, the output unit 115 outputs first information that is information based on at least one of the first similarity and the third character string, depending on the result of the determination.

In this case, in addition to the first similarity, the output unit 115 outputs at least one of a first reliability indicating the reliability of the first character string and a second reliability indicating the reliability of the second character string. The first information is output according to the first information. In particular, when using multi-dimensional features including the first similarity, such as the three-dimensional features (first reliability, second reliability, first similarity) mentioned above, for example, statistical By applying a classification method, the content of output of the first information and whether or not to output it are determined based on complex judgment criteria.

For example, the output unit 115 may output a control signal indicating a character string “Fuji mauki (maybe Fuji Yuuki?)” as the first information, and cause the display unit 15 to display this character string. In this case, since the third character string is shown following "maybe" in parentheses, the first information is information based on at least one of the first similarity and the third character string.

Further, the output unit 115 outputs, for example, a control signal indicating a character string “Fujimauki (there is another candidate with a similarity of 1)” as the first information, and causes the display unit 15 to display this character string. Good too. In this case, the similarity of other candidates is shown in parentheses, so the first information is information based on at least one of the first similarity and the third character string.

<Operation of information processing device>
FIG. 9 is a flow diagram showing the flow of operations of the information processing device 1. As shown in FIG. 9, the control unit 11 of the information processing device 1 controls the image reading unit 16 to read the image formed on the medium (step S101). The control unit 11 corrects the read image and cuts out the first image and the second image from this image based on the area correspondence table 121 (step S102). Note that the read image does not need to be corrected.

The control unit 11 recognizes the first character string from the first image (step S103), and calculates the first reliability of the first character string (step S104).

Further, the control unit 11 recognizes the second character string from the second image (step S105), and calculates the second reliability of the second character string (step S106). Step S105 may be performed before step S103.

The control unit 11 refers to the dictionary DB 123 to extract one or more third character strings related to the first character string (step S107), and calculates the first similarity to the second character string for each third character string. Calculate (step S108).

The control unit 11 evaluates the reliability of the third character string with respect to the second character string according to the first reliability, the second reliability, and the first similarity (step S109). In this evaluation, the control unit 11 refers to the classification model 124, classifies the feature amount indicated by (first reliability, second reliability, first similarity), and determines which label this feature is attached to. It is decided according to whether the class is classified into the specified class.

Then, if the evaluated reliability satisfies the conditions, the control unit 11 outputs the third character string instead of the second character string (step S110). For example, when the feature amount indicated by (first reliability, second reliability, first similarity) is classified into a class labeled "trust the third character string", the control unit 11 Then, the third character string is output in place of the second character string.

As described above, the information processing device 1 outputs the first information according to the first similarity of the third character string to the second character string.

For example, the first reliability mentioned above is an index indicating the recognition accuracy of the first character string, and the second reliability is an index indicating the recognition accuracy of the second character string. This is a numerical value calculated based on. Therefore, if the recognition accuracy is evaluated only based on the first reliability or the second reliability, there is a possibility that the judgment will be incorrect.

On the other hand, the first similarity is calculated based on a comparison result such as the edit distance between the third character string related to the first character string and the second character string extracted from the dictionary DB 123. That is, the first similarity is based on the dictionary DB 123 that stores the relationships between character strings in addition to character recognition processing of the first character string and second character string. Even if a plurality of third character strings are stored in relation to one first character string, the first similarity is calculated for each pair of the plurality of third character strings and second character strings. Since each is calculated, there is no possibility that the character string extracted from the dictionary cannot be determined uniquely.

In other words, by outputting the first information according to the first similarity, the information processing device 1 makes a decision based on both the character recognition process and other aspects as to which of the two character recognition processes to trust. Provide users with indicators of

<Modified example>
The above is the description of the embodiment, but the content of this embodiment can be modified as follows. Further, the following modifications may be combined.

<1>
In the embodiment described above, the first image and the second image are cut out from one read image, but the invention is not limited to this. For example, the control unit 11 of the information processing device 1 may cause the image reading unit 16 to separately read a first image from the front side of the business card and a second image from the back side. That is, the first image and the second image do not need to be included in a common image as long as they are related to each other.

<2>
In the embodiment described above, in addition to the first similarity, the information processing device 1 calculates at least the first reliability indicating the reliability of the first character string and the second reliability indicating the reliability of the second character string. Although the first information is output in accordance with either of the above, the present invention is not limited to this. For example, the information processing device 1 may output the first information according to the first similarity, regardless of the first reliability and the second reliability. In this case, the information processing device 1 does not need to calculate either or both of the first reliability and the second reliability.

<3>
In the embodiment described above, the control unit 11 refers to the classification model 124, classifies the feature amount indicated by (first reliability, second reliability, first similarity), and determines which class this feature amount belongs to. The reliability of the third character string with respect to the second character string was evaluated depending on whether the third character string was classified as such. However, the reliability may be evaluated using a numerical value calculated from the feature amount instead of the class to be classified.

For example, if a function is defined that has the feature amount shown as (first reliability, second reliability, first similarity) as an independent variable, the control unit 11 calculates the numerical value obtained by calculating this function. may be used as the reliability of the third character string with respect to the second character string. In this case, when the numerical value indicating reliability is equal to or greater than the threshold value, the information processing device 1 may output the third character string instead of the second character string. That is, in the information processing device 1 in this modification, the reliability of the third character string with respect to the second character string, which is evaluated using the first reliability, the second reliability, and the first similarity, is greater than or equal to the threshold. Sometimes, a third string is output in place of the second string.

<4>
In the embodiment described above, when the information processing device 1 recognizes the first character string and the second character string, it extracts the third character string related to the first character string from the dictionary DB 123. The third character string may be extracted when the second reliability satisfies a predetermined condition. For example, when the first reliability and the second reliability are each greater than or equal to a predetermined threshold, there is a low possibility that both the first character string and the second character string are incorrect. In this case, the information processing device 1 does not need to extract the third character string. In other words, if both character recognitions are reliable, the information processing device 1 in this modification does not extract the third character string, thereby reducing unnecessary processing load.

<5>
In the embodiment described above, the information processing device 1 refers to the dictionary DB 123 and extracts one or more third character strings related to the first character string, but extracts one or more third character strings related to the second character string. A plurality of fourth character strings may be extracted. For example, the information processing device 1 may refer to the dictionary DB 123 and extract, from the second character string Furigana, the kanji pronounced by the Furigana as the fourth character string. In this case, the dictionary DB 123 used to extract the fourth character string may or may not be the same as the dictionary DB 123 used to extract the third character string.

In this case, the information processing device 1 calculates, for each of the extracted fourth character strings, a second degree of similarity indicating the degree of similarity to the first character string, and calculates a second degree of similarity in accordance with the second degree of similarity. It is preferable to output the second information that is information based on at least one of the degree and the fourth character string.

For example, with respect to the handwritten characters shown in FIG. 3, the information processing device 1 recognizes the character string "Fuji Tanuki" as the first character string and the character string "Fuji Yuuki" as the second character string. At this time, the information processing device 1 calculates 0.1 as the reliability of the first character string and 0.9 as the reliability of the second character string. In this case, the reliability of the first character string is closer to 0 than 1, and it is presumed that character recognition has failed. On the other hand, the reliability of the second character string is closer to 1 than 0, indicating that character recognition is successful.

The extraction unit 113 realized by the control unit 11 of the information processing device 1 extracts a fourth character string related to the second character string “Fuji Yuuki” from the dictionary DB 123. The extraction unit 113 refers to the dictionary DB 123 and extracts, for example, one type of kanji character "Fuji" from the part "Fuji". Further, the extraction unit 113 refers to the dictionary DB 123 and extracts three types of kanji, ``Yuki'', ``Yuki'', and ``Yuki'' from the portion of ``Yuki'', for example. Therefore, the extraction unit 113 extracts the three fourth character strings: "Yuki Fuji," "Yuki Fuji," and "Yuki Fuji."

Then, the information processing device 1 calculates the second similarity with respect to the first character string for each of the three extracted fourth character strings. “Fuji Yuki” and “Fuji Yuki” both have an edit distance of “2” with respect to the first character string “Fuji Tanuki”, while “Fuji Yuki” has an edit distance of “1”. Therefore, the information processing device 1 selects "Fuji Yuki" from among the three fourth character strings.

The output unit 115 realized by the control unit 11 of the information processing device 1 has a feature amount indicated by (first reliability, second reliability, second similarity) of (0.1, 0.9, 1). In a certain case, the classification model 124 is referred to to determine which of the first character string and the fourth character string should be trusted. Then, based on the determination result, the information processing device 1 outputs second information based on at least one of the second similarity and the fourth character string.

The above-mentioned output unit 115 may output, for example, a control signal indicating a character string “Fuji Hanuki (maybe Fuji Tomoki?)” as the second information, and cause the display unit 15 to display this character string. In this case, since the fourth character string is shown following "maybe" in parentheses, the second information is information based on at least one of the second similarity and the fourth character string.

Further, the output unit 115 described above outputs, for example, a control signal indicating a character string “Fuji Hanuki (there is another candidate with a similarity of 1)” as second information, and displays this character string on the display unit 15. may be displayed. In this case, since the similarity of other candidates is shown in parentheses, the second information is information based on at least one of the second similarity and the fourth character string.

As described above, the information processing device 1 extracts the third character string using the first character string and the dictionary DB 123, thereby correcting the erroneously recognized second character string, or correcting the possibility of the erroneous recognition. Inform users of gender.

On the other hand, for example, if the reliability of the first character string is lower than that of the second character string, the information processing device 1 cannot correct the first character string by simply extracting the third character string. However, the information processing device 1 of this modification uses the first character string and the dictionary DB 123 to extract the third character string, and also uses the second character string and the dictionary DB 123 to extract the fourth character string. . Therefore, the information processing device 1 corrects the erroneously recognized first character string or notifies the user of the possibility of the erroneous recognition.

<6>
Further, when extracting the fourth character string from the dictionary DB 123, the information processing device 1 selects at least one of the first reliability level indicating the reliability level of the first character string and the second reliability level indicating the reliability level of the second character string. It is preferable to output the second information depending on either of the two.

In particular, when using a multi-dimensional feature including a second similarity, such as a three-dimensional feature (first reliability, second confidence, second similarity), statistical classification methods By applying the above, the content of output of the second information and whether or not to output it are determined based on complex judgment criteria.

<7>
Further, when extracting a fourth character string from the dictionary DB 123, the information processing device 1 extracts the fourth character with respect to the first character string, which is evaluated using the first reliability, the second reliability, and the second similarity. When the reliability of the string is equal to or higher than a threshold value, it is preferable to output the fourth character string instead of the first character string.

<8>
Furthermore, when extracting the fourth character string from the dictionary DB 123, the information processing device 1 may extract the fourth character string when the first reliability and the second reliability satisfy predetermined conditions. The information processing device 1 in this modification does not extract the fourth character string when the character recognition of both the first character string and the second character string is reliable, so unnecessary processing load is reduced.

<9>
In the embodiment described above, the control unit 11 applied the model data read from the character recognition model 122 to the multilayer neural network to recognize the character code corresponding to the handwritten character, but the method of character recognition is limited to this. I can't do it.

Further, the control unit 11 may not only read out the character recognition model 122 but also rewrite it according to the result of processing.

For example, the information processing device 1 inputs the gradation value of each pixel constituting the second image into a multilayer neural network, and performs character recognition by applying a weighting coefficient to each input obtained from the character recognition model 122. Sometimes. That is, the control unit 11 in this case recognizes the second character string based on the amount of one or more feature amounts calculated from the second image, each of which is given a predetermined weight and then totaled.

Then, the information processing device 1 calculates a first degree of similarity indicating the degree of similarity to the second character string for each third character string. When outputting the third character string instead of the second character string according to the first similarity, the information processing device 1 outputs the third character string (and the third character string used for extraction) rather than the second character string. This means that the first character string) is trusted. At this time, the information processing device 1 in this modification modifies the character recognition model 122 used to recognize the second character string according to the result of the process. Specifically, the information processing device 1 modifies the weighting coefficient described above so that the third character string is recognized from the second image. That is, when the information processing device 1 outputs the third character string instead of the second character string, it is preferable to modify the weight described above so that the third character string is recognized from the second image. According to this modification, the results of the processing of the information processing device 1 are fed back to the character recognition model 122, which is a trained model, so that the accuracy of character recognition increases.

<10>
Furthermore, when extracting the fourth character string from the dictionary DB 123, the information processing device 1 extracts the fourth character string from the first image based on the amount calculated by assigning predetermined weights to each of the one or more feature amounts calculated from the first image. It is best to recognize one character string. Then, when the information processing device 1 outputs the fourth character string instead of the first character string, it is preferable to modify the weight described above so that the fourth character string is recognized from the first image.

<11>
In the embodiment described above, the second image is an image showing the character string representing the pronunciation of the kanji handwritten by the user, that is, the furigana, but the second image is not limited to this. For example, the second image may be a translation of a sentence written by hand on the first image. In this case, the dictionary DB 123 may be an interlingual dictionary such as a Japanese-English dictionary or an English-Japanese dictionary.

For example, the user handwrites the character string "car" in the area of the first image, and handwrites the character string "car" in the area of the second image. Upon acquiring the first image and the second image, the information processing device 1 performs character recognition processing on each. As a result, the information processing device 1 recognizes the character string "automobile" from the first image, and recognizes the character string "dar" from the second image. In this case, character recognition of the first image is successful, but character recognition of the second image is unsuccessful.

The information processing device 1 extracts a third character string related to the first character string from the dictionary DB 123 based on the first character string "automobile" recognized from the first image. The extracted third character strings are "car", "automobile", "auto", "motorcar", etc., and the information processing device 1 extracts the second character string "dar" for each of these third character strings. A first degree of similarity with the first similarity is calculated. Then, the information processing device 1 selects the most similar "car" as the third character string, and calculates the first reliability of the first character string, the second reliability of the second character string, and the selected Based on the first similarity of the third character string, it is determined whether the third character string should be output in place of the second character string.

<12>
In the embodiments described above, image data representing an image is used for inputting image recognition, but the image data is not limited to a read image. The information processing device 1 may recognize characters from information indicating changes over time when writing characters, such as stroke order and strokes of handwritten characters, for example.

<13>
In the embodiment described above, the reliability of a character string is calculated using Equation 4 described in Patent Document 4, but the reliability is not limited to this. The information processing device 1 may calculate the reliability of a character string based on the character reliability of each character constituting the character string. The information processing device 1 may calculate the reliability of a character string, for example, based on the average value of the character reliability of characters included in the character string. For example, an arithmetic mean, a geometric mean, a harmonic mean, etc. may be used as the average value.

Further, the information processing device 1 may calculate, for example, the minimum value of the character reliability of each character included in the character string as the reliability of the character string. Further, the information processing device 1 may calculate the product of the character reliability of each character included in the character string as the reliability of the character string. In this case, the character reliability of each character is normalized to 0 or more and 1 or less.

<14>
In the embodiment described above, the edit distances for the three editing processes "add", "delete", and "replace" are all calculated as "1", but these editing processes have different distances for each type. Different weights may be given.

Further, the similarity may be calculated by dividing the edit distance by the length of the initial character string or the target character string. For example, if the second character string is "Fuji Mauki" and the third character string is "Fuji Yuuki", the length of the target character string is "5", and the edit distance of the third character string with respect to the second character string is is "1". In this case, the first similarity is "1/5", that is, "0.2".

Further, the degree of similarity may be expressed as a value obtained by subtracting the edit distance from the length of the initial character string or the target character string. For example, if the length of the target character string is "5" and the edit distance of the third character string to the second character string is "1", the first similarity is "5-1", that is, "4". . In short, the degree of similarity between the initial character string and the target character string is preferably calculated using the edit distance from the initial character string to the target character string, and further calculated using the length of the initial character string or the target character string. may be done.

<15>
In the embodiment described above, the information processing device 1 refers to the character recognition model 122 that is not associated with a user, but for example, refers to a trained model for character recognition that is associated with each writer. You may. That is, the information processing device 1 may use a different trained model for each user to recognize a character string from an image instructed by the user. According to this modification, for example, a trained model specialized in the handwriting, writing habits, etc. of each handwritten character is used for character recognition, so that the accuracy of character recognition is improved.

<16>
In the embodiment described above, the information processing device 1 is an image reading device having the image reading section 16, but it may not have the image reading section 16. The information processing device 1 may, for example, control an image reading device that reads an image from a medium via the communication unit 13 and a communication line, and acquire the image from the image reading device. Further, the information processing device 1 may recognize characters handwritten by the user by operating the touch panel of the operation unit 14. In this case, the information processing device 1 may acquire information including the stroke order, strokes, etc. based on the operation received by the touch panel as image data representing the image.

<17>
In the embodiment described above, the information processing device 1 recognizes one first character string and one second character string, but may recognize multiple first character strings and multiple second character strings, respectively. Good too. In this case, the information processing device 1 may perform the above-described process for each combination of each first character string and each second character string.

<18>
In the embodiment described above, the first similarity is calculated based on the edit distance of the third character string with respect to the second character string, but the first similarity is calculated based on the edit distance of the third character string with respect to the second character string. The information may include information about the edited location (hereinafter referred to as the edited location). In this case, the first similarity may be expressed not as a scalar value but as a vector.

For example, if the second character string is "Fuji Mauuki" and the third character string is "Fuji Yuuki", the edit distance is "1", and the difference between the third character string and the second character string, i.e. , the edited part is the third character. In this case, the information processing device 1 may calculate a vector composed of a plurality of elements such as (edited part, edited distance) = (3, 1) as the first similarity. Further, in this case, the information processing device 1 may determine the content to be output using the information on the editing location and the character reliability calculated for the character corresponding to that location in the second character string. . According to this configuration, the information included in the first similarity increases compared to the case where the first similarity is derived only from the edit distance, so for example, whether the second character string should be corrected to the third character string or not. The accuracy of judgment will be improved.

<19>
The program executed by the control unit 11 of the information processing device 1 is a record that can be read by a computer device, such as a magnetic recording medium such as a magnetic tape and a magnetic disk, an optical recording medium such as an optical disk, a magneto-optical recording medium, or a semiconductor memory. It may be provided stored on a medium. Further, this program may be downloaded via a communication line such as the Internet. Note that various devices other than the CPU may be applied as the control means exemplified by the control unit 11 described above, and for example, a dedicated processor or the like may be used.

DESCRIPTION OF SYMBOLS 1... Information processing device, 11... Control unit, 111... Analysis unit, 112... Recognition unit, 113... Extraction unit, 114... Calculation unit, 115... Output unit, 12... Storage unit, 121... Area correspondence table, 122... Character Recognition model, 123...Dictionary DB, 1231...Dictionary name list, 1232...Dictionary data, 124...Classification model, 13...Communication section, 14...Operation section, 15...Display section, 16...Image reading section.

Claims (11)

a first recognition unit that recognizes a first character string from a first image;
a second recognition unit that recognizes a second character string from a second image related to the first image;
an extraction unit that refers to a dictionary in which different character strings are associated in advance and extracts one or more third character strings related to the first character string;
a calculation unit that calculates, for each of the third character strings, a first degree of similarity indicating the degree of similarity to the second character string;
Evaluated using a first reliability indicating reliability of character recognition of the first character string , a second reliability indicating reliability of character recognition of the second character string , and the first similarity. When the reliability of the third character string with respect to the second character string is less than a threshold, the second character string is output as first information, and when the reliability is greater than or equal to the threshold, the first an output unit that outputs the third character string instead of the second character string as information ;
An information processing device having: The information processing apparatus according to claim 1 , wherein the extraction unit extracts the third character string when the first reliability and the second reliability satisfy a predetermined condition. The extraction unit refers to a dictionary in which different character strings are associated in advance, and extracts one or more fourth character strings related to the second character string,
The calculation unit calculates, for each of the fourth character strings, a second degree of similarity indicating a degree of similarity to the first character string,
The output unit is configured to generate a first reliability level based on at least one of the first reliability level and the second reliability level, and a third character string based on at least one of the second similarity level and the fourth character string, according to the second similarity level. The information processing device according to claim 1 or 2, wherein the information processing device outputs two pieces of information. When the reliability of the fourth character string with respect to the first character string is less than a threshold, the output unit evaluates using the first reliability, the second reliability, and the second similarity. outputting the first character string as the second information, and outputting the fourth character string as the second information in place of the first character string when the reliability is equal to or higher than the threshold value; The information processing device according to claim 3 , characterized in that: The information processing device according to claim 3 or 4 , wherein the extraction unit extracts the fourth character string when the first reliability and the second reliability satisfy a predetermined condition. The second recognition unit is
Recognizing the second character string based on an amount calculated by adding a predetermined weight to each of one or more feature amounts calculated from the second image and totaling the amount;
A claim characterized in that, when the output unit outputs the third character string instead of the second character string, the weight is corrected so that the third character string is recognized from the second image. The information processing device according to item 1 . The first recognition unit is
Recognizing the first character string based on an amount calculated by adding a predetermined weight to each of one or more feature amounts calculated from the first image and totaling the amount;
A claim characterized in that, when the output unit outputs the fourth character string instead of the first character string, the weight is corrected so that the fourth character string is recognized from the first image. The information processing device according to item 4 . a first recognition unit that recognizes a first character string from a first image;
a second recognition unit that recognizes a second character string from a second image related to the first image;
Referring to a dictionary in which different character strings are associated in advance, one or more third character strings related to the first character string are extracted, and one or more fourth character strings related to the second character string. an extraction unit that extracts
For each of the third character strings, calculate a first degree of similarity indicating the degree of similarity to the second character string, and for each fourth character string, calculate a second degree of similarity indicating the degree of similarity to the first character string. A calculation unit that calculates each,
According to at least one of a first reliability level indicating the reliability level of character recognition of the first character string, and a second reliability level indicating the reliability level of character recognition of the second character string, and the first similarity level. , outputs first information based on at least one of the first similarity and the third character string, and is evaluated using the first reliability, the second reliability, and the second similarity. , when the reliability of the fourth character string with respect to the first character string is less than a threshold, output the first character string as second information, and when the reliability is greater than or equal to the threshold, output the first character string. an output unit that outputs the fourth character string instead of the first character string as second information;
An information processing device having: The second image is an image including a character string indicating the pronunciation of the character string included in the first image.
The information processing device according to any one of claims 1 to 8. computer,
a first recognition unit that recognizes a first character string from a first image;
a second recognition unit that recognizes a second character string from a second image related to the first image;
an extraction unit that refers to a dictionary in which different character strings are associated in advance and extracts one or more third character strings related to the first character string;
a calculation unit that calculates, for each of the third character strings, a first degree of similarity indicating the degree of similarity to the second character string;
Evaluated using a first reliability indicating reliability of character recognition of the first character string , a second reliability indicating reliability of character recognition of the second character string , and the first similarity. When the reliability of the third character string with respect to the second character string is less than a threshold, the second character string is output as first information, and when the reliability is greater than or equal to the threshold, the first an output unit that outputs the third character string instead of the second character string as information ;
A program to function as computer,
a first recognition unit that recognizes a first character string from a first image;
a second recognition unit that recognizes a second character string from a second image related to the first image;
Referring to a dictionary in which different character strings are associated in advance, one or more third character strings related to the first character string are extracted , and one or more fourth character strings related to the second character string. an extraction unit that extracts
For each of the third character strings, calculate a first degree of similarity indicating the degree of similarity to the second character string , and for each fourth character string, calculate a second degree of similarity indicating the degree of similarity to the first character string. A calculation unit that calculates each ,
at least one of a first reliability level indicating the reliability level of character recognition of the first character string, and a second reliability level indicating the reliability level of character recognition of the second character string, and the first similarity level. , outputs first information based on at least one of the third character string and the first similarity , and is evaluated using the first reliability, the second reliability, and the second similarity. , when the reliability of the fourth character string with respect to the first character string is less than a threshold, output the first character string as second information, and when the reliability is greater than or equal to the threshold, output the first character string. an output unit that outputs the fourth character string instead of the first character string as 2 information ;
A program to function as

Images