KR20230029206A - Apparatus for constructing training data for artificial intelligence based text recognition - Google Patents

Abstract

Provided is a device for accurately and conveniently constructing massive training data for artificial intelligence-based text recognition. If a segmenting module draws a bounding box around each character for characters in the image of a scanned document, a user can correct the bounding box using a segmenting verification module. A cluster processing part performs cluster processing for grouping characters with high similarity into one cluster for each character image separated by the bounding box and provides a tool for correcting the clusters. A code input part provides a tool for recognizing a character in the clustered character image, matching the character with a text, and correcting the same. An examination and correction part provides a tool for examining and correcting a segment and a text for a document image having the text inputted thereinto.

Description

Apparatus for constructing training data for artificial intelligence based text recognition}

The present invention relates to an apparatus for constructing learning data for artificial intelligence character recognition, and more particularly, to an apparatus capable of accurately and conveniently constructing a vast amount of learning data for artificial intelligence character recognition.

Document to text conversion is in high demand. In particular, if documents such as old books are converted into text, it is of great help to research in the humanities and ancient documents.

It is most efficient to use an artificial intelligence character recognition method to automatically convert characters included in a document into text. Patent Registration No. 10-2264988 discloses a system for recognizing Chinese characters using artificial intelligence.

The process of recognizing characters such as Chinese characters included in a document is as follows. First, the character image is separated from the image file where the document is scanned, and meta information is obtained through the dynamic relationship analysis of the location of the character appearance. Recognizes separated character images by matching them with natural language characters. A knowledge base can be built based on recognized characters and meta information.

Patent Registration No. 10-1937398 discloses a method of separating and extracting Chinese character images from ancient document images. The separated and extracted character images can be recognized using a neural network.

However, in order to recognize a character using artificial intelligence, a neural network for character recognition must be trained in advance. Learning of the neural network is performed using training data including pairs of separately extracted character images and characters represented by the images.

In order to increase the accuracy of character recognition, it is necessary to build a large amount of training data extracted from actual documents. To this end, there is a high demand for tools to increase the accuracy and efficiency of building learning data.

The present invention has been made in view of these points, and an object of the present invention is to provide an apparatus for building learning data for artificial intelligence character recognition that can reduce errors while efficiently constructing learning data for artificial intelligence character recognition.

An apparatus for building learning data for artificial intelligence character recognition according to a preferred embodiment of the present invention includes a segmenting module for entering a bounding box one by one for each character in a scanned document image, and a tool capable of correcting the bounding box. a segment processing unit including a segmenting verification module provided; A cluster processing unit including a clustering module that performs a clustering process for grouping letters having a high degree of similarity into one cluster for each character image separated by the bounding box, and a cluster verification module that provides a tool capable of correcting the clusters. ; a code input unit for recognizing clustered character images, matching character texts, and providing a tool capable of correcting them; and an inspecting and correcting unit providing a tool for inspecting and correcting segments and text of a document image into which text is input.

The segmenting module may also output the reliability score when segmenting by artificial intelligence is performed. The segment verification module may display the color, thickness, background color, etc. of the bounding box differently from other segments for the segmenting result having low reliability.

The segment verification module is a vertical cut tool that separates two letters up and down at the clicked position by clicking between two letters arranged vertically, and a vertical cut tool that separates two letters arranged side by side at the clicked position. It provides a horizontal cut tool that separates two characters left and right, and a combine tool that selects two segments and merges them into one segment.

The clustering module extracts the feature vector of the word image and uses it as the feature vector of each letter to calculate the similarity between letters through the cosine similarity between feature vectors, and builds clusters of similar letters based on the similarity. .

When the clustering module performs clustering processing by artificial intelligence on text images, it can also output the reliability score. The cluster verification module may display the color, thickness, background color, border, etc. of text images with low reliability differently from other text images.

The cluster processing unit may provide a cluster optimization function that rearranges the clusters by reflecting the contents edited so far. The cluster optimization function may allow the cluster verification module to find and display clusters that are determined to be the same cluster for a certain character cluster, and allow the user to select the same cluster among them.

The cluster verification module performs a 'merging' operation that selects and merges the same letters as the first letter for the cluster remaining as a cluster of letters of a predetermined number or less, and when a cluster exists between the remaining letters although it is different from the first letter A tool can be provided to perform a 'create new' operation that combines them into a new cluster.

The code input unit includes a character recognition module that recognizes characters in a character image by performing a character recognition algorithm, and provides a tool capable of correcting recognized character text by cluster.

The inspection and proofreading unit displays a document image, segment marks applied to individual characters in the document image, and character text assigned to each character in the document image, and provides a tool capable of correcting them.

The inspection and proofreading unit may display input text text for letters belonging to a column or row to which the corresponding letter belongs when the user moves the mouse to a certain letter.

According to the present invention, it is possible to efficiently construct learning data for artificial intelligence character recognition because it is possible to easily correct character clustering and character recognition performed by artificial intelligence on scanned document images.

1 is a block diagram showing the configuration of an apparatus for building learning data for artificial intelligence character recognition according to a preferred embodiment of the present invention.
2 is a block diagram showing the configuration of a segment processing unit.
3 is a block diagram showing the configuration of a cluster processing unit.
4 is a screen example showing a segment processing result.
5 is an example screen for cluster verification.
6 is a diagram for explaining a cluster merging function.
7 is a diagram for explaining a 'merging' operation in which characters such as representative images are selected and merged in a 'remaining cluster' process.
FIG. 8 is a diagram for explaining the 'create new' operation of combining the remaining letters into a new cluster when there are clusters between the remaining characters, although different from the representative image, in the process of 'remaining clusters'.
9 is an example of a screen when text is input.
10 is an example of a screen when inspecting and correcting a document image.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings. Meanwhile, in the following description, the case of recognizing Chinese characters will be described as an example, but the present invention can also be applied to the recognition of other characters such as Hangul and the alphabet. In the following description, 'character' represents a character image included in a book, such as a Chinese character image or a Korean image, and 'character text' represents a code representing a character such as UNICODE on a computer.

As shown in FIG. 1, the learning data construction apparatus 100 for artificial intelligence character recognition of the present invention includes a segment processing unit 110, a cluster processing unit 120, a code input unit 130, and an inspection and correction unit 140. provide The segment processor 110 sets a bounding box letter by letter for each letter in the scanned document image D and provides a tool for the user to correct it. The clustering processing unit 120 performs clustering processing to group each character image included in the document into groups recognized as the same character, and provides a tool for the user to correct this. The code input unit 130 provides a tool capable of matching character texts by recognizing clustered character images, and correcting them. The inspection and proofreading unit 140 provides a tool for generating final learning data (T) by inspecting and correcting the result of segment and character text input with respect to the document image.

The segment processing unit 110 includes a segmenting module 111 and a segment verification module 112 as shown in FIG. 2 . When a user selects a target file for segment processing and requests segment processing, the segmenting module 111 performs segment processing by entering a bounding box for each character in a document image included in the corresponding file. Segment processing may be performed by a general segment method such as those disclosed in Patent Registration Nos. 10-1937398 and 10-1182173, and the present invention is not limited to a specific segmenting method. In one embodiment, in the segmenting module 111, an ancient document face-by-side image is converted into a heatmap while passing through a U-Net-based network. The segmenting module 111 separates the heatmap through a watershed algorithm to generate the position of the letter (bounding box). At this time, the heatmap expresses the probability that each pixel is located at the center of the letter.

When character segmentation by the segmenting module 111 is completed, the segment verification module 112 displays the document image segmented by the segmenting module 111 as shown in FIG. 4 and allows the user to modify it. .

As shown in FIG. 4 , the segment result is displayed in the form of a square box (bounding box) in the document image. When the segmenting module 111 performs segmentation by artificial intelligence such as a convolutional neural network, the reliability score may also be output. The segment verification module 112 may display the color, thickness, background color, etc. of the bounding box differently from other segments for the segmenting result having low reliability. In FIG. 4 , segments with high reliability are indicated by blue boxes and segments with relatively low reliability are indicated by yellow boxes. At the top center of FIG. 4, there is a portion indicated by a yellow box in which two letters arranged vertically are segmented into one. By displaying the segment result differently according to the degree of reliability, the user can more easily perform the segment calibration task.

The user may click the part where the segmentation is incorrect and correct the segmentation. The segment verification module 112 may provide, for example, the following tools to the user.

o Cross Cut: Split a segment into two segments horizontally at a specified location.

o Vertical Cut: Split a segment into two segments vertically at a specified location.

o Re-segmentation: Delete the current segment and create a new segment

o Merge: Merge two segments to create a single segment

o Undo, restore: Return or restore the work step to the previous work

o Zoom in/out: Zoom in/out the work screen

o Mini-map: Shows the current work as a whole

o Magnified Map: A larger view of the current segmented task

In the screen of FIG. 4, when clicking between two letters in the yellow area and selecting "vertical cut", the two letters are separated up and down and segmented. If two letters side by side are displayed as a single segment, you can use the "Cross to Crop" tool to separate them. Conversely, if a single letter is split into two segments, you can use the "Combine" tool to combine them into one segment. The segmented character images in the segment verification module 112 are stored.

The segmented document image verified in the segment verification module 112 may be configured as a JPEG+JSON file pair, and may be used as training data for training the neural network of the segmenting module 111 .

The cluster processing unit 120 includes a clustering module 121 and a cluster verification module 122 as shown in FIG. 3 .

When the segment processing is finished, the clustering module 121 performs a clustering process of grouping characters having a high degree of similarity into one cluster for each character image separated by a bounding box. The clustering process may be performed by a general image clustering method as disclosed in Patent Registration No. 10-1371657, Patent Registration No. 10-1486495, etc., and the present invention is not limited to a specific clustering method. In one embodiment, the cluster learning model may be configured based on ResNet to which Bottleneck is applied. In the early stage of building learning data with insufficient Chinese character data in old documents, after learning the model using Hangul and Chinese character open data, only the feature extraction unit was used to extract the feature vector of the Chinese character image and use it as a feature vector for each character. The similarity between letters can be calculated through the cosine similarity between vectors, and similar letter clusters can be constructed based on the similarity.

The cluster verification module 122 reads the file processed by clustering in the clustering module 121 and allows the user to verify and edit it. When cluster processing by artificial intelligence is performed on text images, the reliability score may also be output. The color, thickness, background color, border, etc. of text images with low confidence in clustering can be displayed differently from other text images.

5 shows an example of a cluster verification and editing screen. In the screen of FIG. 5, when the user clicks “next letter”, a list of clusters is displayed on the left window. In the cluster list, the representative character image and the number of clusters clustered in the corresponding character are displayed. When the user clicks one of the lists, the clustered character images for the corresponding character are displayed in the right window.

In the cluster displayed on the right window, the user clicks the letter to be excluded from the cluster. The clicked letter is marked with a marker indicating that it has been excluded from the cluster. In FIG. 5, the X mark indicates that the corresponding letter is excluded from the cluster. On the other hand, if you click the x-marked letter again, the x-mark disappears and the corresponding letter is included in the cluster.

As shown in FIG. 6, a plurality of identical characters may be displayed in the character cluster list. In this case, you can merge the clusters into one using the 'Merge Clusters' function. For example, the user selects one of the clusters to merge (①), presses the 'Merge Cluster' button (②), and selects another cluster to merge (③). If you select 'OK' in the message window asking whether it is correct to merge the two clusters, the two clusters are merged into one.

The user can perform the cluster optimization function in the middle when the cluster verification is completed to a certain extent. When the cluster optimization function is executed, the cluster verification module 122 rearranges the clusters by reflecting the contents edited so far. Depending on the embodiment, a 'find and merge' function may be provided for the rearranged clusters. The 'Find and Merge Cluster' function is a process of finding and merging overlapping letter clusters in an image file. It proceeds in order from the first letter to the most recent letter, selects the same cluster letters, and performs merging. For example, when the cluster verification module 122 finds and displays clusters determined to be the same cluster for a certain character cluster, the user repeats the process of selecting the same cluster from the first letter to the latest letter.

The above process may be performed on a cluster consisting of characters of a predetermined number or more. After going through this process, a 'remaining clustering' process may be performed on the remaining clusters of characters less than or equal to a predetermined number. In the 'remaining clustering' process, as shown in FIG. 7, the 'merging' operation selects and combines the same letters as the representative image (first letter), and as shown in FIG. If they exist, a 'create new' operation can be performed to merge them into a new cluster.

The cluster data of the character images verified in the cluster verification module 122 may consist of a JPEG+JSON file pair, and may be used as training data to train the neural network of the clustering module 121 .

When the clustering process is finished, the user can input character text for the clustered letters. The code input unit 130 may include a character recognition module that recognizes characters in a character image by performing a character recognition algorithm. The character recognition module uses a character recognition algorithm using a neural network to recognize character images and converts them into character text. The recognized character text is equally applied to all character images in the same cluster.

As shown in the screen example of FIG. 9 , the code input unit 130 provides a tool capable of correcting character text indicated by each character image for each cluster. In the cluster list window 91 on the left, a representative character image and character text recognized for the corresponding image are displayed. When there is an error in character recognition, the user selects a character image to be corrected in the cluster list window 91 and inputs the correct character text. The input text is collectively applied to all text images included in the same cluster. That is, character text correction is performed cluster by cluster. In the example of FIG. 9 , if a Chinese character image to be corrected is selected and then the Chinese character's single sound is input to the single sound input window 92, a list of Chinese characters with the corresponding single sound is displayed on the Chinese character string display window 93, and the user selects the desired Chinese character among them. Choose The character text of the selected Chinese character is applied to all character images belonging to the same cluster.

Depending on the embodiment, it is also possible to configure the clustering process and the code input process to be performed simultaneously. Also, depending on embodiments, the code input unit 130 may provide a tool capable of correcting a segmentation error or grouping error found during code input. For example, if other letters are included in the cluster on the screen of FIG. 9, after selecting the corresponding letter(s), press the Create New Cluster/Input (94) button to create a new cluster for the selected letter(s) and enter a code. can Alternatively, when a segmentation error or clustering error occurs, an error code is input, and the error is corrected later using the segment verification module 112 or the cluster verification module 122 for the letter image or cluster in which the error occurred. may be

In addition, depending on the embodiment, it may be configured so that a user can display Chinese characters unknown to the user and later input them by an expert. For example, if it is necessary to input letters separately from among the letter images in the cluster, such as unknown letters, after selecting these letter images on the screen of FIG. can be deleted from the cluster of and added to the list of characters to be entered as single characters.

Character images that cannot be entered into codes, such as characters that have segmentation errors, are mangled, or are difficult to recognize due to low resolution, special symbols, etc., can be excluded from input. For example, after selecting these character images on the screen of FIG. 9 , the 'exclude from input' button 95 is clicked to exclude the selected character images.

If character text is entered leaving only images of the same character for one cluster, the input character text for all character images included in the cluster is linked and stored. When text input for one cluster is finished, it moves on to the next cluster and the text text entry procedure is repeated.

The character image into which character text is input in the code input unit 130 may be composed of a JPEG+JSON file pair, and may be used as training data to train a neural network of a character recognition module.

When text input is completed for all clusters, the inspection and proofreading unit 140 provides a tool capable of inspecting and correcting segments and text for document images. As shown in FIG. 10 , the inspection and correction unit 140 displays a document image, segment marks applied to individual characters in the document image, and character text assigned to each character in the document image. In addition, the inspection and proofreading unit 140 provides a tool for the user to select a character and correct the bounding box or character text.

Depending on the embodiment, as shown in the screen example of FIG. 10 , when the mouse is moved to a certain letter, the input text of the letters belonging to the column or row to which the corresponding letter belongs is displayed. If there is an error in the input character text or an error in the segment, the user selects the corresponding segment and corrects the segment or the character text.

For example, as shown in FIG. 11, after deleting a segment with a segment error (①), performing segmentation again (②), the text of the corresponding letter image is input.

In the above, the present invention has been described with several examples, but even if all components constituting an embodiment of the present invention are combined or described as operating in combination, the present invention is not necessarily limited to these embodiments. Within the scope of the object of the present invention, all of the components may be selectively combined with one or more to operate. In addition, although all of the components may be implemented as a single independent piece of hardware, some or all of the components are selectively combined to perform some or all of the combined functions in one or a plurality of hardware. It may be implemented as a computer program having. Codes and code segments constituting the computer program may be easily inferred by a person skilled in the art. Such a computer program may implement an embodiment of the present invention by being stored in a computer or processor-readable storage medium, read and executed by a computer or processor.

Terms such as "comprise", "comprise" or "having" described above mean that the corresponding component may be present unless otherwise stated, and therefore do not exclude other components. It should be construed that it may further include other components.

The above description is merely an example of the technical idea of the present invention, and various modifications and variations can be made to those skilled in the art without departing from the essential characteristics of the present invention. Therefore, the embodiments disclosed in the present invention are not intended to limit the technical idea of the present invention, but to explain, and the scope of the technical idea of the present invention is not limited by these embodiments. The protection scope of the present invention should be construed according to the claims below, and all technical ideas within the equivalent range should be construed as being included in the scope of the present invention.

110 segment processing unit,
120 cluster processing unit;
130 code input unit,
140 Inspection and Proofreading Department.

Claims (10)

a segment processing unit including a segmenting module for entering a bounding box for each character in the scanned document image, and a segmenting verification module for providing a tool capable of correcting the bounding box;
A cluster processing unit including a clustering module that performs a clustering process for grouping letters having a high degree of similarity into one cluster for each character image separated by the bounding box, and a cluster verification module that provides a tool capable of correcting the clusters. ;
a code input unit for recognizing clustered character images, matching character texts, and providing a tool capable of correcting them;
an inspection and proofreading unit that provides a tool for inspecting and correcting segments and character text of document images;
Learning data construction device for artificial intelligence character recognition having a.
According to claim 1,
The segmenting module also outputs the reliability score when segmenting by artificial intelligence is performed,
The segment verification module displays the color, thickness, background color, etc. of the bounding box differently from other segments for the segmenting result with low reliability.
Learning data construction device for artificial intelligence character recognition.
The method of claim 1, wherein the segment verification module,
A vertical cutting tool that separates two letters up and down at the clicked position by clicking between two letters arranged vertically;
A horizontal cutting tool that separates two letters left and right at the clicked position by clicking between two letters arranged side by side;
Merge tool that selects two segments and merges them into one segment.
A device for building learning data for artificial intelligence character recognition that provides a.
According to claim 1,
The clustering module extracts the feature vector of the word image and uses it as the feature vector of each letter to calculate the similarity between letters through the cosine similarity between feature vectors, and constructs clusters of similar letters based on the similarity A device for building learning data for artificial intelligence character recognition.
According to claim 1,
When the clustering module performs clustering processing by artificial intelligence on the text images, the reliability score is also output,
The cluster verification module displays the color, thickness, background color, border, etc. of the text image having low reliability differently from other text images.
Learning data construction device for artificial intelligence character recognition.
According to claim 1,
The cluster processing unit provides a cluster optimization function that rearranges clusters by reflecting the contents edited so far,
In the cluster optimization function, the cluster verification module finds and displays clusters that are judged to be the same cluster for a certain character cluster, and allows the user to select the same cluster among them. Device for building learning data for artificial intelligence character recognition.
According to claim 6,
The cluster verification module performs a 'merging' operation in which letters same as the first letter are selected and combined for the remaining clusters of letters of a predetermined number or less, and when a cluster exists between the remaining letters although different from the first letter. An apparatus for building learning data for artificial intelligence character recognition, which provides a tool to perform a 'create' operation that combines them into a new cluster.
According to claim 1,
The code input unit includes a character recognition module that recognizes characters in a character image by performing a character recognition algorithm, and provides a tool for correcting the recognized character text by cluster. Learning data construction device for artificial intelligence character recognition.
According to claim 1,
The inspection and proofreading unit displays a document image, a segment display applied to individual characters in the document image, and character text assigned to each character in the document image, and provides a tool capable of correcting them. Learning data construction device for character recognition.
According to claim 9,
The learning data construction device for artificial intelligence character recognition, wherein the inspection and correction unit displays the input character text for the characters belonging to the column or row to which the character belongs when the user moves the mouse to a certain character.

Images