KR102627591B1 - Operating Method Of Apparatus For Extracting Document Information AND Apparatus Of Thereof - Google Patents

Abstract

The embodiment relates to a method of operating a device for extracting information from a document. A method of operating the device includes detecting at least one character area from a document image of a document; Recognizing letters represented in each of the detected letter areas based on the learned OCR model; Searching for a predefined definition word corresponding to the type of document from the recognized characters; Coordinates of the defined word in the document; and comparing the coordinates of words other than the defined words in the document; and inputting the characters recognized based on the comparison result into a digital document in a predefined format.

Description

Operating method of a device for extracting information from a document and the device {Operating Method Of Apparatus For Extracting Document Information AND Apparatus Of Thereof}

Embodiments relate to a method of operating a device and the device for extracting information from a document.

As digital transformation becomes a hot topic, there are increasing attempts to digitize analog data (paper documents) or raw data to facilitate information management and use it as input data for big data analysis.

Conventionally, in order to convert such data into digital data, a standardized document form was created and people manually typed it accordingly. Some have converted to digital documents by introducing OCR (optical character reader), but paper documents printed on low-quality printers, low-resolution image files, or noise such as creasing and slanting that occur during scanning and watermarks included in the documents , there is a problem that OCR recognition accuracy is reduced due to background color. In addition, since OCR converts the entire content of the document into letters and the relationship between the recognized letters is unknown, in order to create a document in a standardized format, a person must go through the process of manually editing only the necessary information after OCR.

If the content of the document, such as a business card, invoice, or receipt, is not long, the document format is fixed, the information to be entered in a standardized form is clear, and is always in the same place within the document (business card: name, contact information, email, receipt: date of use) , amount used, and location of use) can be automatically entered into a standardized form by recognizing a specific location, but the document format is not fixed and the technology for converting documents containing different information into digital documents has not been disclosed.

A related prior patent discloses the cloud OCR business card information management system of Korean Patent No. 10-2012-0111954.

The invention according to the embodiment seeks to provide a method of automatically converting documents written in different formats for one type into digital documents in a standardized format.

The invention according to the embodiment can provide a method of automatically converting documents written in different formats for one type into digital documents in a standardized format.

Accordingly, work time can be shortened and work productivity can be improved by processing work previously done by people through computers.

1 is a flow diagram of a method of operating a device for extracting information from a document, in an embodiment.
Figure 2 is a diagram for explaining a method of preprocessing a document image in an embodiment.
Figure 3 is a diagram for explaining a method of detecting a character area through labeling in an embodiment.
Figure 4 is a diagram for explaining learning of an OCR model in an embodiment.
Figure 5 is an example of characters in a document image being recognized in an embodiment.
Figure 6 is a diagram for explaining a method of recognizing a definition word with a space and recognizing letters corresponding to the definition word, in an embodiment.
7 is a diagram for explaining definition words and information on definition words in an embodiment.
8 is a diagram for explaining definition words and information on definition words in an embodiment.
Figure 9 is a diagram for explaining a method of recognizing a definition word and information about the definition based on coordinates in an embodiment.
Figure 10 is a block diagram for explaining the configuration of a device for extracting information from a document in an embodiment.

Hereinafter, embodiments will be described in detail with reference to the attached drawings. However, various changes can be made to the embodiments, so the scope of the patent application is not limited or limited by these embodiments. It should be understood that all changes, equivalents, or substitutes for the embodiments are included in the scope of rights.

The terms used in the examples are for descriptive purposes only and should not be construed as limiting. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as “comprise” or “have” are intended to designate the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, but are not intended to indicate the presence of one or more other features. It should be understood that this does not exclude in advance the possibility of the existence or addition of elements, numbers, steps, operations, components, parts, or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by a person of ordinary skill in the technical field to which the embodiments belong. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related technology, and unless explicitly defined in the present application, should not be interpreted in an ideal or excessively formal sense. No.

In addition, when describing with reference to the accompanying drawings, identical components will be assigned the same reference numerals regardless of the reference numerals, and overlapping descriptions thereof will be omitted. In describing the embodiments, if it is determined that detailed descriptions of related known technologies may unnecessarily obscure the gist of the embodiments, the detailed descriptions are omitted.

Additionally, in describing the components of the embodiment, terms such as first, second, A, B, (a), and (b) may be used. These terms are only used to distinguish the component from other components, and the nature, sequence, or order of the component is not limited by the term. When a component is described as being "connected," "coupled," or "connected" to another component, that component may be directly connected or connected to that other component, but there is no need for another component between each component. It should be understood that may be “connected,” “combined,” or “connected.”

Components included in one embodiment and components including common functions will be described using the same names in other embodiments. Unless stated to the contrary, the description given in one embodiment may be applied to other embodiments, and detailed description will be omitted to the extent of overlap.

1 is a flow diagram of a method of operating a device for extracting information from a document, in an embodiment.

In an embodiment, a device detects at least one character regions from a document image of a document.

The device can receive an image of a scanned document as input. Documents according to embodiments may include transcripts. The extension of the document image may include jpg, png, bmp, tiff, etc. The device can change the document image to a predetermined size in order to extract information from the document image. For example, a vertically long document can have more than 800 pixels in width and 1000 pixels in height, and a long document can have more than 1000 pixels in width and more than 800 pixels in height.

In an embodiment, OCR requires a clear boundary between text and the background to increase recognition accuracy, so in order to accurately detect the text area, the device may perform preprocessing on the scanned document image. Preprocessing may include, for example, removing noise, watermarks, background colors, and frames, and leaving only the text and white background of the document.

When detecting a character area, a part of the table constituting the document image is input to the OCR model, so there is a possibility of character misrecognition, so the device can delete the table.

According to the preprocessing process according to the embodiment, pixels showing brightness changes are detected by differentiating the document image with respect to the x-axis and y-axis, and pixels showing brightness changes are used to create at least one straight line that forms a table in the document image You can detect and delete the straight lines that make up the table. Long horizontal and vertical straight lines can be detected using the differential value of the pixel and the corresponding straight lines can be removed.

In an embodiment, the Canny Edge Detection algorithm may be utilized to detect the edge of the image.

Since the brightness value of the boundary of the image changes rapidly around the pixels of the image, pixels whose pixel values change rapidly are detected and differentiated for the X-axis and Y-axis of the image using Equation 1 above, respectively, and Equation 2 By calculating the size and angle of the detected pixel, only the longest horizontal and vertical straight lines can be detected.

The straight line (black pixel value: 0) corresponding to the table area detected in this way can be deleted (pixel value changed to 255) from the document image of the corresponding part.

Additionally, the preprocessing process may include a process of binarizing the document image. For example, you can split a document image into a white background and black text.

In embodiments, a device may binarize document images using an adaptive threshold algorithm. Binarization is performed to separate the background and text. Since the brightness and color of the background and document image are different depending on the form and scanning/output status, the entire image is not binarized using a fixed threshold, but the threshold is adjusted according to the input image status. This variable adaptive binarization algorithm can be applied.

Afterwards, after applying morphological filtering to the binarized document image, the preprocessing result can be obtained by assigning the pixel value of the document image to the common part of the filtered result and the pixel value of the document image.

The corresponding embodiment will be described in detail with reference to FIG. 2.

Figure 2 is a diagram for explaining a method of preprocessing a document image in an embodiment.

Figure 2(a) is an example of a document image from which the table has been removed, and the result shown in Figure 2(b) can be obtained by performing adaptive binarization on the document image of Figure 2(a).

The adaptive binarization algorithm refers to the values of surrounding pixels when determining the threshold for binarization. This algorithm uses a different threshold for each pixel, and the threshold is determined by subtracting a certain constant from the average brightness of the area around the block size (N

The threshold value of adaptive binarization varies depending on the block size and correction constant C value in Equation 3. blocksize is specified as an odd number so that a center point can exist, and in an embodiment, the quotient of 90 divided by the image width can be assigned as blocksize through experimentation. Depending on the size of the document image, if the blocksize is an even number, you can add 1 to specify the blocksize to be an odd number. The correction constant value C may be fixed to 2 in the embodiment.

The device applies a different threshold to each block of the image, and if the pixels inside the block are higher than the threshold, it assigns a pixel value of 255 to assign it to the background. Conversely, it assigns a pixel value of 255 to assign it to the text area. Assign as 0.

Figure 2(c) is the result of applying morphology filtering, and Figure 2(d) is the final preprocessing result.

When the original image is binarized, an aliasing effect may occur on the outer part of the letter, which may result in the loss of the letter's unique shape. Therefore, in order to maintain the original character shape of the document image, morphological filtering (dilation operation of the character part) is applied to the binarization result as shown in Figure 2(c), and the area where the pixel value of the original document image is below a certain standard (for example, 100) By finding common parts and assigning pixel values of the original document image to the common parts, the final preprocessing result as shown in Figure 2(d) can be generated.

Going back to Figure 1, the device can detect the letter area for the image for which preprocessing has been completed.

In an embodiment, Connected Component Labeling may be performed to detect text areas. Labeling is to label pixels that are adjacent to (pixel values other than 255) and non-background pixels with the same number (Label) in the document image for which text area preprocessing has been completed, and to label non-adjacent text pixels with different numbers.

In the example, 8-Connected Component Labeling was applied. In this method, if there are pixels other than the background in the pixel areas corresponding to the top, bottom, left, right, and diagonal lines based on one pixel (x, y), they are labeled with the same label. In addition, at least one character area can be detected by classifying labels classified in this way into adjacent labels based on distance.

After secondary labeling, word regions can be detected by merging adjacent labels using the gap between labels. At this time, if the width between labels is less than the threshold, they are merged into the same label. Otherwise, they are recognized as spaces. For example, the threshold for recognizing a space is that the size of the label is different for each document image, so if the gap between two labels is smaller than the pixel divided by half the average of the widths of the two labels, it can be recognized as one word.

The method for detecting the character area can be explained in detail with reference to FIG. 3.

Figure 3 is a diagram for explaining a method of detecting a character area through labeling in an embodiment.

Since Hangul is made up of a combination of consonants and vowels, multiple phoneme-level labels can be created even for one syllable letter. Since the labeling results are used as input to an OCR model that recognizes letters, labeling can be performed to generate syllable-level labeling so that one label is generated per syllable.

In an embodiment, phoneme-level labeling may be performed through labeling that first classifies pixels with the same pixel value as the pixel value of a specific pixel into the same label. Next, the spacing between each label can be calculated using the primary labeling result, and secondary labeling can be performed by merging labels where the spacing between labels is within a certain distance or overlapping.

The labeling results shown in Figure 3 show the first labeling and second labeling results. After performing primary labeling, the device may perform secondary labeling by merging adjacent labels to generate one label per syllable.

Next, in step 120, the device recognizes letters represented in each of the detected letter areas based on the learned OCR model.

In an embodiment, the learned OCR model corresponds to an MLP learned model with training data that normalizes images of letters and assigns pixel values and class labels to letters as input, and uses letters as output.

The OCR model will be explained in detail in Figure 4.

Figure 4 is a diagram for explaining learning of an OCR model in an embodiment.

Machine learning methods can be used to recognize letters. It corresponds to a model trained in advance for the embodiment, and letters can be recognized in real time through the OCR model generated after learning is completed.

In an embodiment, letters can be recognized using MLP (Multi-Layer Perceptron). As shown in Figure 4, MLP consists of an input layer, two hidden layers, and an output layer. The Input Layer is a structure in which the pixel value and class label of the character image to be learned are input as vectors and are transmitted to the Hidden Layer, and the output value of the Hidden Layer is transmitted to the Output Layer.

One learning vector input to the input layer consists of the pixel value of the letter image and the target value. The output value of the output layer is compared with the target value of the learning vector, and if the difference between the output value and the target value is greater than the tolerance, the weights are adjusted according to the learning rule and learning proceeds.

Training data can be images of black letters on a white background. In an embodiment, learning data can be constructed by converting document data for various fonts into images and using the labeling results of the document images. The training data can be generated by converting it into one image per character, normalizing the size (for example, 30 pixels wide and 30 pixels tall), and then assigning pixel values and class labels.

In the example, 11,257 classes were configured to recognize Korean, English, and some special characters, and the number of learning data was 100 for each class (11,172 Korean combinations, 52 English upper and lower case letters, and 23 special characters).

Table 1 describes the MLP learning process.

The value of each neuron is calculated through Equation 4. In Equation 4, represents the size of the input vector, represents the input value of the input vector (OCR learning data pixel value), and represents the weight.

The activation function in Equation 5 is a function that activates the output value of the neuron if the value is greater than the threshold, and deactivates the output value of the neuron otherwise. In the example, the Sigmoid Function was used.

Equation 6 expresses the learning rule for the jth weight of the th output layer neuron for the th learning vector.

In Equation 6, v_kj represents the jth weight of the th Output Layer, and η represents the learning rate. z_nk is the output value of the jth hidden neuron for the learning vector, and t_nk represents the nth learning vector and the kth target value. net_nk represents the value of the kth output layer neuron for the nth learning vector.

Equation (7) is the learning rule for the ith weight of the jth Hidden Layer neuron for the learning vector, and represents the size of the output layer.

Here, w_ji represents the ith weight of the jth hidden layer neuron, η is the learning rate, x_ni is the ith input value of the nth learning vector, t_nk is the target value of the nth learning vector, and O_nk is the nth learning vector. Indicates the output value of the kth Output Layer neuron.

Once learning is complete, an OCR model is created, and letters can be recognized in real time by inputting the document image of the area where the letter area is labeled in the document image into the model.

Figure 5 is an example of characters in a document image being recognized in an embodiment.

In an embodiment, the result of character recognition by applying the learned OCR model to a document image is shown. The secondary labeling results were used as input, and letters for each label can be output as results. Based on the detection results for the word area, you can check how many sub-labels the word label contains and then apply spacing to the recognition results.

However, types of documents with various formats, such as transcripts, have different formats for each school. For example, even if all letters in a transcript are recognized by OCR and created as a digital document, the OCR results must be manually edited by a person in order to document the necessary information (name, school, date of birth, major, subject name, grades, credits, etc.) There is a hassle to do this.

In an embodiment, documents in different formats for one type can be analyzed, and common information required from documents of that type can be defined and utilized in a standardized format. Accordingly, the embodiment may propose a method of automatically entering the OCR results into a defined form without re-editing them.

Returning to Figure 1, in step 130 the device searches for a predefined definition word corresponding to the type of document from the recognized characters.

In step 140 the device determines: the coordinates of the definition word in the document; And the two coordinates can be compared by recognizing the coordinates of words other than the defined words in the document.

In an embodiment, definition words as shown in Table 2 below can be defined using the analysis results for the transcript to automatically generate a standardized digital document by analyzing information commonly input for the type of document. .

Definition Words may not be detected as word areas because there are spaces between syllables of the word in the document image. Accordingly, in the embodiment, it is possible to check whether a label box in which a word area is labeled with a single syllable in a document image is merged with label boxes before and after it by referring to the definition word.

Please refer to Figure 6 for this matter.

Figure 6 is a diagram for explaining a method of recognizing a definition word with a space and recognizing letters corresponding to the definition word, in an embodiment.

As described above, each label box can be merged into one label box by checking whether label boxes that label the word area with one syllable in the document image are merged with the label boxes before and after.

_보다 작고 정의 단어의 라벨 박스 끝점 x좌표보다 우측에 존재하며, 정의 단어에 대한 라벨 박스의 끝점 x좌표와 주변 라벨박스 시작점 x좌표의 차이가

_인 라벨 박스를 찾아 해당 라벨 박스를 해당 정의 단어에 대한 정보로 인식할 수 있다.In an embodiment, the difference between the y-coordinate of the center point of the definition word label box and the y-coordinate of the center point of the surrounding label box is

It is smaller than _ and exists to the right of the x-coordinate of the end point of the label box of the definition word, and the difference between the x-coordinate of the end point of the label box for the definition word and the x-coordinate of the start point of the surrounding label box is

You can find the label box with _ and recognize it as information about the definition word.

_y는 y좌표의 임계치,

_x는 x좌표의 임계치를 나타낸다. P_h는 정의 단어 라벨 박스의 높이를 나타내며, P_w는 정의 단어 라벨 박스의 너비를 나타낸다. C는 상수 값을 나타내며 실시예에서는 실험을 통해 2로 결정될 수 있다.In equation 8:

_ y is the threshold value of the y coordinate,

_x represents the threshold value of the x-coordinate. P_h represents the height of the definition word label box, and P_w represents the width of the definition word label box. C represents a constant value and may be determined to be 2 through experiment in the embodiment.

7 is a diagram for explaining definition words and information on definition words in an embodiment.

In embodiments, a device may recognize the coordinates of definition words 710 to recognize the basic information in Table 2 for a transcript. Information 720 corresponding to the definition word 710 is disposed on the right side of the definition word 710, as shown. Accordingly, the label box that exists within a threshold to the right of the coordinates of the definition word label box can be recognized as information 720 corresponding to the definition word 710.

8 is a diagram for explaining definition words and information on definition words in an embodiment.

In an embodiment, the content may be recognized after recognizing the definition word corresponding to the basic information. In an embodiment, the y-coordinate of the bottom label box 901 among the basic information is recognized, and the label box corresponding to the definition word defined in the content that first appears on the left of the document image below based on the coordinates The coordinates of (902) can be recognized.

_y 이내인 라벨 박스 중 동일한 OCR 결과를 가진 라벨 박스가 있는지 찾고, 해당 좌표를 인식할 수 있다.For example, there are two or more “credits” and “grades” depending on the number of transcript columns, so the threshold value is based on the center point of the label box that first appears.

_ You can find whether there is a label box with the same OCR result among the label boxes within y and recognize the corresponding coordinates.

In one embodiment, the area 910 corresponding to the basic information and the area 920 corresponding to the content are designated as areas of interest based on the coordinates of the definition word, respectively, so that the contents of the document can be recognized in units of areas of interest. In an embodiment, when there are multiple label boxes corresponding to “credit” and “grade,” an area of interest can be designated for each. For example, the coordinates of the word on the right of the two words can be assigned as the end point of the ROI, and the starting point of the first ROI can be based on the leftmost pixel, which is the starting pixel of the document image.

Figure 9 is a diagram for explaining a method of recognizing a definition word and information about the definition based on coordinates in an embodiment.

_x 이내인 라벨 박스들을 찾아 학점, 성적 내용을 먼저 인식할 수 있다.In an embodiment, after specifying an area of interest, subject names, credits, and grades may be recognized. For this purpose, the x-coordinate of the center point of the label box corresponding to the definition word in the region of interest and the threshold of Equation 8

_ You can first recognize the credits and grades by finding the label boxes within x .

_y 이내인 라벨 박스를 찾아 과목명으로 인식할 수 있다.In an embodiment, after finding the label boxes corresponding to credits and grades and belonging to the same ROI, the y-coordinate of the center point of each label box and the threshold

_ You can find the label box within y and recognize it as the subject name.

In an embodiment, if there is a label box that does not correspond to credits, grades, or subject names and contains the word "semester" among the label boxes, an area of interest is created for each semester and the credits, grades, and subject names within the area of interest are created. You can match.

In step 150, the device inputs the characters recognized based on the comparison result into a digital document in a predefined format.

In an embodiment, a dictionary is used to correct errors included in OCR in the contents of a document or to automatically correct parts that require normalization among recognized contents (e.g., normalizing completion classifications that differ depending on the school in the transcript). It can be defined in .

The types of errors are Substitution (recognized as a different character), Missing (character not recognized), Addition (insertion: a new missing character is added), Combination (two characters are combined into one), and decomposition ( Decomposition: One character is broken into two), etc., and it is possible to analyze the types of errors that frequently occur among characters recognized through the OCR model learned in the embodiment and create a dictionary to correct them.

In an embodiment, the dictionary may be created in .csv format, and in the correction step according to the embodiment, the device may read the dictionary and correct or normalize the characters according to the rules.

Table 3 above shows the types of dictionaries created for examples of transcripts.

In an embodiment, the OCR result is compared with the character correction list of the dictionary, and if a word corresponding to the left side of the character correction list is found, it can be changed to the word on the right.

In an embodiment, the completion classification recognition list may be referred to in order to recognize different completion classification marks for each transcript issued depending on the school and normalize them into two types: “liberal arts” and “major.” For example, if the school name on the transcript matches the school name in the completion classification recognition list, and the letter corresponding to the subject name in the content recognition result matches the second letter, it will be corrected to a normalized completion classification according to the predefined digital document format. You can.

In an embodiment, a combination of the longest common sequence and the edit distance algorithm can be used for text comparison. The school name and major on the transcript are the same as the automatic completion classification recognition list, the similarity of the longest common search (LCS) is more than 80% by comparing the subject name and the text of the subject name in the automatic completion classification list, and the editing distance is string/ If it is less than 3, the course division indicated in the list can be assigned.

In embodiments, a device may generate a digital document by inputting recognized characters into digital form in a predefined format.

Figure 10 is a block diagram for explaining the configuration of a device for extracting information from a document in an embodiment.

The device 1000 according to the embodiment may be configured to include a memory 1010 and a processor 1020, and may include a program executed by the processor 1020. In an embodiment, the program may include a method of operating the device described in FIGS. 1 to 9.

In an embodiment, device 1000 detects at least one character regions from a document image of a document.

The device 1000 can receive an image of a scanned document. Documents according to embodiments may include transcripts. The extension of the document image may include jpg, png, bmp, tiff, etc. The device 1000 may change the document image to a predetermined size in order to extract information from the document image. For example, a vertically long document can have more than 800 pixels in width and 1000 pixels in height, and a long document can have more than 1000 pixels in width and more than 800 pixels in height.

In an embodiment, OCR requires a clear boundary between text and the background to increase recognition accuracy, so in order to accurately detect the text area, the device 1000 may perform preprocessing on the scanned document image. Preprocessing may include, for example, removing noise, watermarks, background colors, and frames, and leaving only the text and white background of the document.

When detecting a character area, a part of the table constituting the document image is input to the OCR model, so there is a possibility of character misrecognition, so the device can delete the table.

According to the preprocessing process according to the embodiment, pixels showing brightness changes are detected by differentiating the document image with respect to the x-axis and y-axis, and pixels showing brightness changes are used to create at least one straight line that forms a table in the document image You can detect and delete the straight lines that make up the table. Long horizontal and vertical straight lines can be detected using the differential value of the pixel and the corresponding straight lines can be removed.

The straight line (black pixel value: 0) corresponding to the table area detected in this way can be deleted (pixel value changed to 255) from the document image of the corresponding part.

Additionally, the preprocessing process may include a process of binarizing the document image. For example, you can split a document image into a white background and black text.

In an embodiment, device 1000 may binarize a document image using an adaptive threshold algorithm. Binarization is performed to separate the background and text. Since the brightness and color of the background and document image are different depending on the form and scanning/output status, the entire image is not binarized using a fixed threshold, but the threshold is adjusted according to the input image status. This variable adaptive binarization algorithm can be applied.

Afterwards, after applying morphological filtering to the binarized document image, the preprocessing result can be obtained by assigning the pixel value of the document image to the common part of the filtered result and the pixel value of the document image.

The device 1000 can detect a character area in an image for which preprocessing has been completed.

In an embodiment, Connected Component Labeling may be performed to detect text areas. Labeling is to label pixels that are adjacent to (pixel values other than 255) and non-background pixels with the same number (Label) in the document image for which text area preprocessing has been completed, and to label non-adjacent text pixels with different numbers.

In the example, 8-Connected Component Labeling was applied. In this method, if there are pixels other than the background in the pixel areas corresponding to the top, bottom, left, right, and diagonal lines based on one pixel (x, y), they are labeled with the same label. In addition, at least one character area can be detected by classifying labels classified in this way into adjacent labels based on distance.

In an embodiment, the device 1000 may detect a word region by merging adjacent labels using the gap between labels after secondary labeling. At this time, if the width between labels is less than the threshold, they are merged into the same label. Otherwise, they are recognized as spaces. For example, the threshold for recognizing a space is that the size of the label is different for each document image, so if the gap between two labels is smaller than the pixel divided by half the average of the widths of the two labels, it can be recognized as one word.

The device 1000 recognizes letters represented in each of the detected letter areas based on the learned OCR model.

In an embodiment, the learned OCR model corresponds to an MLP learned model with training data that normalizes images of letters and assigns pixel values and class labels to letters as input, and uses letters as output.

The device 1000 searches for a predefined definition word corresponding to the type of document from the recognized characters.

The device 1000 includes coordinates of definition words in a document; And the two coordinates can be compared by recognizing the coordinates of words other than the defined words in the document.

In an embodiment, definition words may be defined using the results of the analysis of the transcript to automatically generate a standardized digital document by analyzing commonly input information about the type of document.

Definition Words may not be detected as word areas because there are spaces between syllables of the word in the document image. Accordingly, in the embodiment, for a label box in which a word area is labeled with a single syllable in a document image, whether or not it is merged with label boxes before and after it can be confirmed by referring to the definition word.

In one embodiment, the device 1000 may recognize the contents of the document in units of interest areas by designating each corresponding area as an area of interest according to the content based on the coordinates of the definition word.

The device 1000 inputs the characters recognized based on the comparison result into a digital document in a predefined format.

In an embodiment, a dictionary is used to correct errors included in OCR in the contents of a document or to automatically correct parts that require normalization among recognized contents (e.g., normalizing completion classifications that differ depending on the school in the transcript). It can be defined in .

The types of errors are Substitution (recognized as a different character), Missing (character not recognized), Addition (insertion: a new missing character is added), Combination (two characters are combined into one), and decomposition ( Decomposition: One character is broken into two), etc., and a dictionary can be created to analyze and correct the types of errors that frequently occur among characters recognized through the OCR model learned in the embodiment.

The embodiment relates to a technology for converting documents such as birth certificates into standardized digital documents. In more detail, computer vision technology is applied to image files to remove noise and watermarks, detect text areas, and then machine We can propose RPA technology that recognizes letters by applying learning-based OCR (Optical Character Recognition) and converts them into digital documents in a standardized format defined in advance based on specific recognized words.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may contain program instructions, data files, data structures, etc., singly or in combination. Program instructions recorded on the medium may be specially designed and configured for the embodiment or may be known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -Includes optical media (magneto-optical media) and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing unit to operate as desired, or may be processed independently or collectively. You can command the device. Software and/or data may be used on any type of machine, component, physical device, virtual equipment, computer storage medium or device to be interpreted by or to provide instructions or data to a processing device. , or may be permanently or temporarily embodied in a transmitted signal wave. Software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

Although the embodiments have been described with limited drawings as described above, those skilled in the art can apply various technical modifications and variations based on the above. For example, the described techniques are performed in a different order than the described method, and/or components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or other components are used. Alternatively, appropriate results may be achieved even if substituted or substituted by an equivalent.

Therefore, other implementations, other embodiments, and equivalents of the claims also fall within the scope of the following claims.

Claims (29)

In a method of operating a device for extracting information from a document,
detecting at least one character area from a document image of the document;
Recognizing letters represented in each of the detected letter areas based on the learned OCR model;
searching for a predefined definition word corresponding to the type of the document from the recognized characters;
coordinates of the defined word in the document; and comparing coordinates of words other than the definition word in the document; and
Inputting the recognized characters into a digital document in a predefined format based on the comparison result
Including,
The step of searching for a predefined definition word corresponding to the type of the document from the recognized letters,
Detecting the definition word containing a space by referring to the predefined definition word and determining whether to merge the label box labeled with one syllable with the syllable corresponding to the label box existing before and after it; and
Based on the coordinates of the label box based on the definition word, designate the region of interest corresponding to the basic information and the region of interest corresponding to the content, based on the x-coordinate of the center point of the label box corresponding to the definition word within each region of interest. Steps to recognize multiple label boxes
Including,
How the device works.
According to paragraph 1,
Detecting at least one character area from the document image includes:
classifying a first pixel having a pixel value in the document image and at least one pixel having the same pixel value as the pixel value of the first pixel among adjacent pixels of the first pixel to the same label; and
Detecting at least one character area by classifying adjacent labels based on distance from the classified label
Including,
How the device works.
According to paragraph 2,
Detecting a word area based on the gap between the at least one letter area
Containing more,
How the device works.
According to paragraph 1,
The learned OCR model is,
An MLP-trained model that uses training data that normalizes images of letters and assigns pixel values and class labels to the letters as input, and uses the letters as output.
How the device works.
According to paragraph 1,
The above documents include transcripts,
How the device works.
According to paragraph 1,
The step of searching for a predefined definition word corresponding to the type of the document is,
Checking whether a first letter labeled with one syllable among the recognized letters is labeled with a letter located before or after the first letter;
merging the first letter and letters located before and after the first letter into one word based on whether or not the label is labeled; and
Recognizing the definition word through the merged word
Including,
How the device works.
According to paragraph 1,
coordinates of the defined word in the document; And the step of comparing the coordinates of words other than the definition word in the document,
Recognizing the coordinates of each of the recognized letters based on their positions within the document image
Including,
How the device works.
According to paragraph 1,
coordinates of the defined word in the document; And the step of comparing the coordinates of words other than the definition word in the document,
generating a region of interest based on the coordinates of the defined word; and
Recognizing at least one word within a threshold among the remaining words based on the x coordinate corresponding to the center point of the region of interest.
Including,
How the device works.
According to clause 8,
The step of inputting the recognized characters into a digital document in a predefined format is,
Matching the definition word and at least one word within the threshold
Containing more,
How the device works.
According to paragraph 1,
The step of inputting the recognized characters into a digital document in a predefined format is,
Inputting the definition word and the remaining words into a predetermined position in the digital document
Including,
How the device works.
According to paragraph 1,
The step of inputting the recognized characters into a digital document in a predefined format is,
Detecting whether a recognition error occurs among the recognized characters by referring to a dictionary in which recognition errors of the recognized characters and corrections for the recognition errors are defined; and
Correcting the detected recognition error by referring to the dictionary
Including,
How the device works.
According to paragraph 1,
further comprising removing noise from the document image to detect at least one character region from the document image of the document,
How the device works.
According to clause 12,
The step of removing noise from the document image is,
Differentiating the document image with respect to the x-axis and y-axis to detect pixels in which brightness changes appear;
detecting at least one straight line constituting a table in the document image using the detected pixels; and
Deleting the at least one straight line
Including,
How the device works.
According to clause 12,
The step of removing noise from the document image is,
Binarizing the document image;
extracting a common portion of the binarized document image and pixel values of the document image; and
Assigning a pixel value of the document image to the common portion
Including,
How the device works.
A computer program stored in a computer-readable medium in combination with hardware to execute the method of any one of claims 1 to 14.
In a device for extracting information from a document,
One or more processors;
Memory; and
Comprising one or more programs stored in the memory and configured to be executed by the one or more processors,
The above program is,
detecting at least one character area from a document image of the document;
Recognizing letters represented in each of the detected letter areas based on the learned OCR model;
searching for a predefined definition word corresponding to the type of the document from the recognized characters;
coordinates of the defined word in the document; and comparing coordinates of words other than the definition word in the document; and
Inputting the recognized characters into a digital document in a predefined format based on the comparison result
Including,
The step of searching for a predefined definition word corresponding to the type of the document from the recognized letters,
Detecting the definition word containing a space by referring to the predefined definition word and determining whether to merge the label box labeled with one syllable with the syllable corresponding to the label box existing before and after it; and
Based on the coordinates of the label box based on the definition word, designate the region of interest corresponding to the basic information and the region of interest corresponding to the content, based on the x-coordinate of the center point of the label box corresponding to the definition word within each region of interest. Steps to recognize multiple label boxes
Including,
Device.
According to clause 16,
Detecting at least one character area from the document image includes:
classifying a first pixel having a pixel value in the document image and at least one pixel having the same pixel value as the pixel value of the first pixel among adjacent pixels of the first pixel to the same label; and
Detecting at least one character area by classifying adjacent labels based on distance from the classified label
Including,
Device.
According to clause 17,
Detecting a word area based on the gap between the at least one letter area
Containing more,
Device.
According to clause 16,
The learned OCR model is,
An MLP-trained model that uses training data that normalizes images of letters and assigns pixel values and class labels to the letters as input, and uses the letters as output.
Device.
According to clause 16,
The above documents include transcripts,
Device.
According to clause 16,
The step of searching for a predefined definition word corresponding to the type of the document is,
Checking whether a first letter labeled with one syllable among the recognized letters is labeled with a letter located before or after the first letter;
merging the first letter and letters located before and after the first letter into one word based on whether or not the label is labeled; and
Recognizing the definition word through the merged word
Including,
Device.
According to clause 16,
coordinates of the defined word in the document; And the step of comparing the coordinates of words other than the definition word in the document,
Recognizing the coordinates of each of the recognized letters based on their positions within the document image
Including,
Device.
According to clause 16,
coordinates of the defined word in the document; And the step of comparing the coordinates of words other than the definition word in the document,
generating a region of interest based on the coordinates of the defined word; and
Recognizing at least one word within a threshold among the remaining words based on the x coordinate corresponding to the center point of the region of interest.
Including,
Device.
According to clause 23,
The step of inputting the recognized characters into a digital document in a predefined format is,
Matching the definition word and at least one word within the threshold
Containing more,
Device.
According to clause 16,
The step of inputting the recognized characters into a digital document in a predefined format is,
Inputting the definition word and the remaining words into a predetermined position in the digital document
Including,
Device.
According to clause 16,
The step of inputting the recognized characters into a digital document in a predefined format is,
Detecting whether a recognition error occurs among the recognized characters by referring to a dictionary in which recognition errors of the recognized characters and corrections for the recognition errors are defined; and
Correcting the detected recognition error by referring to the dictionary
Including,
Device.
According to clause 16,
further comprising removing noise from the document image to detect at least one character region from the document image of the document,
Device.
According to clause 27,
The step of removing noise from the document image is,
Differentiating the document image with respect to the x-axis and y-axis to detect pixels in which brightness changes appear;
detecting at least one straight line constituting a table in the document image using the detected pixels; and
Deleting the at least one straight line
Including,
Device.
According to clause 27,
The step of removing noise from the document image is,
Binarizing the document image;
extracting a common portion of the binarized document image and pixel values of the document image; and
Assigning a pixel value of the document image to the common portion
Including,
Device.

Images