KR20200063342A - Method to update character recognition model - Google Patents

Abstract

Disclosed is a method of updating a character recognition model. An operation stored in a computer program for processing the method comprises the operations of: classifying each of one or more character images included in a document image into a character through a character recognition model which includes one or more pre-learned network functions so as to determine a character included in the character images; determining whether the character determined according to classification results of the character images based on the classification is a suitable character; re-classifying, when the character determined according to the classification results of the character images is not suitable, the character images, and re-determining the character included in the character images; generating first learning data in which the character images are used as an input of first learning data and the character determined according to re-classification results is used as a label of the first learning data; and generating a character recognition module updated based on a first learning data set including the one or more first learning data.

Description

How to update the character recognition model{METHOD TO UPDATE CHARACTER RECOGNITION MODEL}

The present disclosure relates to a method of updating a character recognition model, and more particularly, to a method of updating a character recognition model based on error learning data.

Pharmacies, insurance companies, hospitals, etc., automatically recognize patient information, prescription details, and supply/receipt information on prescriptions using optical recognition modules such as OCR devices for prescriptions.

However, since the person does not directly check and work, the recognition through the OCR device is likely to cause a recognition error between numbers or characters with high similarity. In the case of a pharmacy, if a prescription OCR device recognition error occurs, the patient may be prescribed the wrong medicine, and the insurer may pay the wrong amount to the insured.

Accordingly, there is an increasing demand for a method for raising the recognition rate of OCR devices. In response to these demands, in particular, research is being conducted on ways to increase awareness of OCR solutions using deep learning.

Republic of Korea Patent Registration No. 10-0717734 discloses an input error prevention device for an optical mark or text reader.

The present disclosure has been devised in correspondence with the background art described above, and is intended to provide a method for updating a character recognition model.

Disclosed is a computer program stored in a computer readable storage medium in accordance with one embodiment of the present disclosure for realizing the above-described problems. The computer program, when executed on one or more processors of the computing device, performs the following operations for updating the character recognition model, and the operations are: one or more network functions pre-trained for each one or more character images included in the document image Determining a character included in the character image by classifying each character through a character recognition model including a character; Determining whether a character determined as a result of the classification of the character image based on the classification is a suitable character; Re-determining a character included in the character image by re-classifying the character image when the character determined as the result of the classification of the character image is not suitable; Generating the first learning data using the text image as the input of the first learning data, and using the text determined as a result of the reclassification as a label of the first learning data; And generating an updated character recognition model based on the first training data set including one or more first training data.

In an alternative embodiment of computer program operations to perform the following operations for updating the character recognition model, the operation of determining whether a character determined as a result of the classification of the character image based on the classification is a suitable character is the character When the reliability of the classification result for the image is equal to or less than a predetermined value, an operation of determining that the character determined as the classification result of the character image is not suitable may be included.

In an alternative embodiment of computer program operations to perform the following operations for updating the character recognition model, the operation of determining whether a character determined as a result of the classification of the character image based on the classification is a suitable character is the classification If the character determined as a result is a character that is not included in the entire character classification class, or when the character determined as the result of the classification corresponds to a character whose frequency of use for each character of the domain of the document image is less than or equal to a certain value, the character image And determining that the letter determined as the result of the classification is not suitable.

In an alternative embodiment of computer program operations to perform the following operations for updating the character recognition model, the operation of determining whether a character determined as a result of the classification of the character image based on the classification is a suitable character is the document Extracting one or more recognition words by grouping characters determined as a result of classification of one or more character images included in the image into semantic units; And a letter determined as a result of classifying at least one letter image among one or more letter images that are the basis of the recognized word, based on a result of comparing each of the one or more recognized words with one or more reference words stored in memory. And determining.

In an alternative embodiment of computer program operations to perform the following operations for updating the character recognition model, the operation of determining whether a character determined as a result of the classification of the character image based on the classification is a suitable character is the document Extracting one or more recognition words by grouping characters determined as a result of classification of one or more character images included in the image into semantic units; And determining whether the classification result of the character image is an appropriate character through context analysis of the recognition word on the document image.

In an alternative embodiment of computer program operations to perform the following operations for updating the character recognition model, if the character determined as the result of the classification of the character image is not suitable, the character image is reclassified to classify the character image The operation of re-determining the characters included in the operation may include re-classifying the character images such that the classification result of the character images is included in one or more character classes stored in the memory.

In an alternative embodiment of computer program operations to perform the following operations for updating the character recognition model, if the character determined as the result of the classification of the character image is not suitable, the character image is reclassified to classify the character image The operation of re-determining the characters included in the operation may include: extracting a recognition word based on the character image for which the classification result is not suitable; And one or more letters included in the recognition word and one or more reference words stored in the memory, based on the reference word having the highest matching ratio of each of the reference words, the classification result included in the recognition word. May include reclassifying an inappropriate text image.

In an alternative embodiment of computer program operations to perform the following operations for updating the character recognition model, if the character determined as the result of the classification of the character image is not suitable, the character image is reclassified to classify the character image Re-determining the characters included in the operation comprises: receiving manual correction information for the character image from a user computing device; And re-classifying the text image based on the manual correction information.

In an alternative embodiment of computer program operations to perform the following operations for updating the character recognition model, when the character determined as the result of the classification of the character image is not suitable, displaying the non-conformance information in the character image It may further include.

In an alternative embodiment of computer program operations to perform the following operations for updating the character recognition model, the first set of training data includes a predetermined number of first training data collected after training of the character recognition model. Or, it may include first learning data collected during a predetermined period.

In an alternative embodiment of computer program operations to perform the following operations for updating the character recognition model, generating an updated character recognition model based on the first training data set comprising one or more first training data. The method may include classifying the text image of the first learning data into one text character as an input of the text recognition model; Calculating an error by comparing a letter determined as a result of the learning classification based on classification of the first learning data and a letter determined as the result of the re-classification, which is a label of the first learning data; And generating an updated character recognition model by updating the weight of the character recognition model based on the error.

In an alternative embodiment of computer program operations to perform the following operations for updating the character recognition model, generating an updated character recognition model based on the first training data set comprising one or more first training data. A new character recognition model is trained on the basis of the second learning data and the first learning data of at least some of the second learning data sets that are the basis for learning the pre-trained character recognition model, and thereby updating the updated character recognition model. It may include an action to generate.

In an alternative embodiment of computer program operations to perform the following operations for updating the character recognition model, the updated character recognition model may share at least some of the weights of the pre-trained character recognition model.

In an alternative embodiment of computer program operations to perform the following operations for updating the character recognition model, the updated character recognition model is a new character recognition that shares a layer of at least a portion of the pre-trained character recognition model. It can be a model.

In an alternative embodiment of computer program operations to perform the following operations for updating the character recognition model, the operation of classifying the character image included in the character image using the character recognition model and the updated character recognition model ; And ensemble the classification result of the character recognition model and the classification result of the updated character recognition model to generate a final classification result for characters included in the character image.

In an alternative embodiment of computer program operations to perform the following operations for updating the character recognition model, the operation of classifying the character image included in the document image into the first character through the character recognition model; If the first character corresponds to a character classification class determined based on the first learning data, determining to classify the character image using the updated character recognition model; And classifying the text image using the updated text recognition model to generate a final classification result for text included in the text image.

Disclosed is a method for updating a character recognition model according to an embodiment of the present disclosure. The method may include determining a character included in the character image by classifying each character image included in the document image into each character through a character recognition model including one or more pre-trained network functions; Determining whether a letter determined as a result of the classification of the letter image based on the classification is a suitable letter; Re-determining a character included in the character image by re-classifying the character image when the character determined as the result of the classification of the character image is not suitable; Generating the first learning data using the text image as the input of the first learning data and the text determined as a result of the reclassification as a label of the first learning data; And generating an updated character recognition model based on the first training data set including one or more first training data.

A computing device is disclosed in accordance with one embodiment of the present disclosure. The computing device includes a processor comprising one or more cores; And a memory, wherein the processor classifies each of the one or more character images included in the document image into each character through a character recognition model including one or more pre-trained network functions to identify characters included in the character image. Determines whether or not the character determined as a result of the classification of the character image based on the classification is a suitable character, and when the character determined as the result of the classification of the character image is not suitable, classifies the character image and reclassifies the character The first learning data is generated by re-determining the characters included in the image, using the character image as the input of the first learning data, and using the character determined as the result of the re-classification as a label of the first learning data, and An updated character recognition model may be generated based on the first training data set including one or more first training data.

The present disclosure may provide a method of updating a character recognition model.

1 is a block diagram of a computing device performing an operation for updating a character recognition model according to an embodiment of the present disclosure.
2 is a diagram illustrating an example of a method of generating learning data for updating a character recognition model according to an embodiment of the present disclosure.
3 is a flowchart of a method for updating a character recognition model according to an embodiment of the present disclosure.
4 is a block diagram illustrating logic for implementing a method of updating a character recognition model according to an embodiment of the present disclosure.
5 is a block diagram of a computing device according to an embodiment of the present disclosure.

Various embodiments are now described with reference to the drawings, and like reference numbers throughout the drawings are used to denote similar elements. In this specification, various descriptions are presented to provide an understanding of the present disclosure. However, it is clear that these embodiments can be practiced without these specific details.

As used herein, the terms “component”, “module”, “system” and the like refer to computer-related entities, hardware, firmware, software, a combination of software and hardware, or execution of software. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, a thread of execution, a program, and/or a computer. For example, both the application and the computing device running on the computing device can be components. One or more components can reside within a processor and/or thread of execution, and one component can be localized within one computer, or distributed between two or more computers. In addition, these components can execute from various computer readable media having various data structures stored therein. Components are, for example, signals having one or more data packets (e.g., data from one component interacting with another component in a local system, distributed system, and/or data via a network such as the Internet with other systems via signals and/or signals). ) Through local and/or remote processes.

In addition, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or”. That is, unless specified otherwise or unclear in context, "X uses A or B" is intended to mean one of the natural inclusive substitutions. That is, X uses A; X uses B; Or, if X uses both A and B, "X uses A or B" can be applied in either of these cases. It should also be understood that the term “and/or” as used herein refers to and includes all possible combinations of one or more of the listed related items.

Also, the terms “comprises” and/or “comprising” should be understood as meaning that the corresponding features and/or components are present. It should be understood, however, that the terms “comprises” and/or “comprising” do not exclude the presence or addition of one or more other features, components, and/or groups thereof. In addition, unless otherwise specified or contextually unclear as indicating a singular form, the singular in the specification and claims should generally be construed to mean "one or more."

Descriptions of the presented embodiments are provided to enable any person skilled in the art to use or practice the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art of the present disclosure, and the general principles defined herein can be applied to other embodiments without departing from the scope of the present disclosure. Thus, the present disclosure should not be limited to the embodiments presented herein, but should be interpreted in the broadest scope consistent with the principles and novel features presented herein.

In one embodiment of the present disclosure, the server may include other configurations for performing the server environment of the server. The server can include any type of device. The server is a digital device, and may be a digital device equipped with a processor and having computing power, such as a laptop computer, a notebook computer, a desktop computer, a web pad, and a mobile phone. The server may be a web server that processes services. The above-described types of servers are only examples, and the present disclosure is not limited thereto.

Throughout this specification, models, neural networks, network functions, and neural networks may be used in the same sense. A neural network may consist of a set of interconnected computational units, which may generally be referred to as "nodes." These “nodes” may also be referred to as “neurons”. The neural network comprises at least one node. Nodes (or neurons) constituting neural networks may be interconnected by one or more “links”.

In this specification, the network function may include one or more neural networks, and in this case, the output of the network function may be an ensemble of the output of the one or more neural networks. The model can perform data processing with other models. The models can be interconnected in series or in parallel with other models.

1 is a block diagram of a computing device performing an operation for updating a character recognition model according to an embodiment of the present disclosure.

The configuration of the computing device 100 illustrated in FIG. 1 is merely an example for simplicity. In one embodiment of the present disclosure, the computing device 100 may include other configurations for performing the computing environment of the computing device 100.

The computing device 100 for a method of updating a character recognition model according to an embodiment of the present disclosure may be used to update a character recognition model based on recognition error data of the character recognition model.

The computing device 100 for a method of updating a character recognition model according to an embodiment of the present disclosure may include a network unit 110, a processor 120, and a memory 130.

The network unit 110 may transmit and receive data, etc. for performing a method of updating a character recognition model according to an embodiment of the present disclosure to other computing devices, servers, and the like. The network unit 110 may receive information on a recognition error of the text image from another computing device. The network unit 110 may receive a pre-stored terminology for determining whether a character image is recognized or not, from a server. In addition, the network unit 110 may enable communication between a plurality of computing devices so that learning of a model is distributed in each of the plurality of computing devices. The network unit 110 may enable communication between a plurality of computing devices so that character recognition of images using a model can be distributed.

The processor 120 may be composed of one or more cores, a central processing unit (CPU) of a computing device, a general purpose graphics processing unit (GPGPU), and a tensor processing unit (TPU). unit), and a processor for data analysis and deep learning. The processor 120 may read a computer program stored in the memory 130 to perform a method of updating a character recognition model according to an embodiment of the present disclosure. According to an embodiment of the present disclosure, the processor 120 may perform calculation for learning a neural network.

At least one of the CPU, GPGPU, and TPU of the processor 120 may process training of the model. Further, the computer program executed in the computing device according to an embodiment of the present disclosure may be a CPU, GPGPU, or TPU executable program.

In one embodiment of the present disclosure, the computing device 100 may distribute and process a model using at least one of a CPU, GPGPU, and TPU. In addition, in one embodiment of the present disclosure, the computing device 100 may distribute and process a model together with other computing devices.

Hereinafter, a method of preprocessing a document image by the processor 120 will be described.

The processor 120 may classify each of the one or more character images included in the document image into each character through a character recognition model including one or more pre-trained network functions to determine characters included in the character image. The document image may be an image of a document including one or more recognition target characters. The document image may include one or more text images. The letter image may be an image including one letter. For example, the letter image may be an image that includes one letter of'A'. The processor 120 may receive a document image through the network unit 110 or a document image through a camera unit (not shown). For example, when the character recognition model is used in a pharmacy, the document image may be a prescription, or when used by an insurance company, the document image may be a statement of medical expenses. The description of the document image and the text image described above is merely an example, and the present disclosure is not limited thereto.

The processor 120 may perform image pre-processing on the received document image. Image pre-processing may include skew correction, line and noise removal, and character area detection.

The skew correction may mean that, when the document image is inclined, the text recognition model rotates the document image in a forward direction to facilitate recognition of the text. The processor 120 may recognize a line having a threshold length or more included in the document image. The processor 120 may correct the inclination of the document image based on a line having a threshold length or more. The processor 120 may compare a line having a threshold length or more and an edge of the document image itself. The processor 120 may compare the edges of the document image itself with a line having a threshold value equal to or greater than a predetermined number. This is to prevent comparing the edges of the document image and the diagonal display in the table except for the table in the document. The processor 120 may determine that the document image is in a forward direction when a line having a threshold length or more and an edge of the document image are parallel. The processor 120 may determine that the document image is inclined when the line having a threshold length or more and the edge of the document image are not parallel, and may adjust the document image to be parallel. For example, when a table in a document is compared to an edge of a document image, the tilt of the document image may be corrected when the table is positioned not parallel to the edge.

Line and noise removal may refer to pre-processing to exclude lines that are not an object to be recognized except letters in the document image. The processor 120 may exclude lines that are not recognized, except for letters, from the text image. The processor 120 may perform a binarization process on a text image in order to remove noise on a document image from which lines have been removed. The noise may refer to a case in which a character included in the character image is blurred, and a part of the table is shaded. The binarization process may mean processing as 1 when a sharpness is greater than or equal to a predetermined threshold, and processing as 0 when a sharpness is less than a predetermined threshold. The processor 120 may perform pre-processing to calculate the sharpness for each pixel included in the document image, to display darker when the sharpness is greater than or equal to the threshold, and to delete it from the document image when the sharpness is less than the threshold. For example, the sharpness of a pixel with respect to a shaded part or a smeared part included in a document image may be lower than that of a letter part. The processor 120 determines that the sharpness of the shaded portion or the smeared portion is less than a threshold value, and the pixels of the corresponding portion are treated as 0, and can be excluded from the document image. The specific description of the above-mentioned pre-processing for removing noise and noise is only an example, and the present disclosure is not limited thereto.

Text area detection may mean character segmentation of a document image. The processor 120 may perform segmentation for each character. The processor 120 may recognize one or more letter images included in the document image.

Throughout this specification, neural networks, network functions, and neural networks may be used in the same sense. A deep neural network (DNN) may mean a neural network including a plurality of hidden layers in addition to an input layer and an output layer. Deep neural networks can be used to identify potential latent structures in data. In other words, it is possible to grasp the potential structure of photos, text, video, voice, and music (for example, what objects are in the picture, what the content and emotions of the text are, what the content and emotions of the speech, etc.) are. . Deep neural networks include convolutional neural networks (CNNs), recurrent neural networks (RNNs), restricted boltzmann machines (RBMs), and deep belief networks (DBNs). , Q network, U network, Siam network, and the like.

The convolutional neural network is a kind of deep neural network, and includes a neural network including a convolutional layer. Convolutional neural networks are a type of multilayer perceptorns designed to use minimal preprocessing. The CNN may consist of one or several convolutional layers and artificial neural network layers combined with it. CNN may additionally utilize weights and pooling layers. Thanks to this structure, CNN can fully utilize the input data of the two-dimensional structure. Convolutional neural networks can be used to recognize objects in an image. Convolutional neural networks can process image data as a matrix with dimensions. For example, in the case of red-green-blue (RGB)-encoded image data, each of R, G, and B colors may be represented as a two-dimensional (for example, two-dimensional image) matrix. That is, the color value of each pixel of the image data can be a component of the matrix, and the size of the matrix can be the same as the size of the image. Therefore, the image data can be represented by three two-dimensional matrices (three-dimensional data array).

A convolutional process (input/output of the convolutional layer) can be performed by moving the convolutional filter in the convolutional neural network and multiplying the convolutional filter and matrix components at each position of the image. The convolutional filter may be configured as an n*n matrix. The convolutional filter may generally consist of a fixed type filter smaller than the total number of pixels in the image. That is, when an m*m image is input to a convolutional layer (for example, a convolutional layer having a size of n*n of a convolutional filter), a matrix representing n*n pixels including each pixel of the image It can be multiplied by this convolutional filter (ie, the product of each component of the matrix). A component matching the convolutional filter in the image may be extracted by multiplying with the convolutional filter. For example, a 3*3 convolutional filter for extracting a vertical line component from an image may be configured as [[0,1,0], [0,1,0], [0,1,0]] Can be. When a 3*3 convolutional filter for extracting vertical components from the image is applied to the input image, vertical components that match the convolutional filter from the image may be extracted and output. The convolutional layer may apply a convolutional filter to each matrix for each channel representing an image (ie, R, G, and B colors in the case of R, G, and B coded images). The convolutional layer can apply a convolutional filter to the input image to extract features matching the convolutional filter from the input image. The filter value of the convolutional filter (that is, the value of each component of the matrix) may be updated by back propagation in the learning process of the convolutional neural network.

A subsampling layer is connected to the output of the convolutional layer, thereby simplifying the output of the convolutional layer to reduce memory usage and computation. For example, if the output of the convolutional layer is input to a pooling layer having a 2*2 max pooling filter, the maximum value included in each patch is output for each 2*2 patch in each pixel of the image to compress the image. You can. The aforementioned pooling may be a method of outputting a minimum value in a patch or outputting an average value of patches, and any pooling method may be included in the present disclosure.

The convolutional neural network may include one or more convolutional layers and sub-sampling layers. The convolutional neural network can extract features from an image by repeatedly performing a convolutional process and a subsampling process (for example, the Max pooling described above). Through the iterative convolutional and subsampling process, the neural network can extract global features of the image.

The output of the convolutional layer or subsampling layer may be input to a fully connected layer. A fully connected layer is a layer in which all neurons in one layer are connected to all neurons in a neighboring layer. The full connected layer may refer to a structure in which all nodes of each layer are connected to all nodes of another layer in a neural network.

In one embodiment of the present disclosure, in order to perform segmentation of image data, the neural network may include a deconvolutional neural network (DCNN). The deconvolutional neural network performs an operation similar to that of calculating the convolutional neural network in the reverse direction. The deconvolutional neural network may output features extracted from the convolutional neural network as a feature map related to original data. The description of the specific configuration for the convolutional neural network is discussed in more detail in US patent US9870768B2, which is incorporated herein by reference in its entirety.

Hereinafter, a method in which the processor 120 classifies each of the one or more character images into each character through a character recognition model to determine the characters included in the character image.

The processor 120 may extract features of the character image through the character recognition model, and extract scores for each of one or more classes. The class may be a letter that the letter image means. For example, each of the one or more classes may be A, B, C, D, or the like. The processor 120 may classify the text image into text based on the score value for each class of the text image. As a result of calculating the character image through the character recognition model, the processor 120 may classify the character image into a class having the highest score value, and determine a character included in the character image.

Referring to FIG. 2, when the document image 200 is a detailed statement of medical expenses, the processor 120 may classify one or more letter images of each of the one or more letter images included in the detailed statement of medical expenses into a single letter through a letter recognition model. have. As a result of the processor 120 operating on the character image 212 through the character recognition model, when the score value of the'plate' class is 0.8 and the score value of the other classes is less than 0.3, the character image 212 is' Edition'. The processor 120 may determine that the letter image 212 includes the letter'plate'. The detailed description of the above-described letter classification is only an example, and the present disclosure is not limited thereto.

Hereinafter, a method for determining whether a character determined by the result of the classification of the character image based on the classification is a suitable character by the processor 120 will be described.

According to an embodiment of the present disclosure, the processor 120 may determine that the character determined as the result of the classification of the character image is not suitable when the reliability of the character determined as the result of the classification for the character image is equal to or less than a predetermined value. . When the reliability is less than or equal to a predetermined value, it may mean that the score of the class that is the basis of the classification of the text image is equal to or less than the predetermined value. The processor 120 may calculate a character image through a character recognition model, and output score values for each of one or more classes. The processor 120 may classify the character image based on the class having the highest score value among the one or more classes. The processor 120 may determine the reliability of the classification result of the corresponding letter image based on the score value of the class with the highest score value. When the score of the class with the highest score value is less than or equal to a predetermined value, the processor 120 may determine that the reliability of the letter determined as a result of classification for the character image is equal to or less than the predetermined value. When the score of the class with the highest score value is less than or equal to a predetermined value, the processor 120 may determine that the letter determined as a result of classifying the character images is not suitable.

For example, referring to FIG. 2, the result of the processor 120 calculating and outputting the character image 212 through the character recognition model is that the score value of the'pin' class is 0.6 and that of the'plate' class. The score value may be 0.5, and the score value of the'open' class may be 0.5. The processor 120 may determine the letter included in the letter image 212 as the'pin' because the score value for the'pin' class has the highest value as 0.6. When the predetermined reliability value is 0.7, since the score value for the character'pin' class determined to be included in the character image 212 is 0.6, the processor 120 has the reliability of the classification result for the character image 212. It can be judged to be below a predetermined value. The processor 120 may determine that the character'pin' determined as a result of the classification of the character image 212 is not suitable. The specific description of the method for determining whether the classification result of the above-described character image is suitable is merely an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, when the character determined as the result of the classification is a character not included in the entire character classification class, the processor 120 may determine that the character determined as the result of the classification of the character image is not suitable. have. The entire character classification class may include one or more classes for predetermined characters. The processor 120 may calculate a character image through a character recognition model, and output score values for each of one or more classes. The processor 120 may not determine a score value having the largest value among score values for each of the one or more classes, or if the score values for all classes are lower than a predetermined threshold, characters included in the text image It can be decided that it is a different character that is not included in the whole character classification class. The processor 120 may determine that the character determined as the result of the classification of the character image is not suitable because the character determined as the result of the classification of the character image is another character not included in the entire classification class. For example, the total character classification class is'A','I','C', and the result of the operation of the character recognition model for the character image is that the score value of the'A' class is 0.2 and the score value of the'B' class is When the score value of the class 0.3 and'c' is 0.3, the processor 120 may determine that the letters determined as the result of classifying the letter images are not included in the entire class (in this example, A, B, C). have. The processor 120 may determine that the letters determined as a result of the classification are not suitable. The specific description of the method for determining whether the classification result of the above-described character image is suitable is merely an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, when the letter determined as the result of the classification corresponds to a letter whose frequency of use for each letter of the domain of the document image is equal to or less than a predetermined value, the processor 120 determines that the letter is determined as the classification result of the letter image. You may decide that the letters are not suitable. The domain of the document image may mean characteristics of a document included in the document. The domain of the document image may mean a field in which the document image is used. For example, the domain of the document image may be a hospital, law firm, trade, publishing company, or the like. Each of the one or more domains may have a term frequently used depending on the characteristics of the domain. For each domain, based on the characteristics of the domain, the frequency of use for each letter may be stored. The frequency of use for each letter for each domain may be stored in the memory 130, and the processor 120 may receive the use frequency for each letter for each domain from the server. The processor 120 may check the frequency of use of the letters determined as a result of classifying the letter recognition model, based on the domain of the document image. When the frequency of use of the letters determined as a result of the classification of the letter recognition model is less than or equal to a certain value, the processor 120 may determine that the letters determined as the result of the classification of the letter images are not suitable. For example, when the domain of the document image is a prescription, since the frequency of use for'methylphenidate' is high, the frequency of use for the letter'til' may be high. When the domain of the document image is a coffee order, the frequency of use of the letter'til' may be low. If the domain of the document image is a coffee order, and the letter determined as the result of the classification of the one letter image included in the document image is'til', the processor 120 determines that the letter determined as the classification result of the letter image is not suitable. Can decide. The specific description of the method for determining whether the classification result of the above-described character image is suitable is merely an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, the processor 120 compares a word based on the classification result of the text image with a word stored in the memory 130, and stores the word based on the classification result of the text image in the memory 130 If not, it may be determined that the character determined as the result of the classification of the character image is not suitable. The processor 120 may extract one or more recognition words by grouping characters determined as a result of classification of each of the one or more character images included in the document image into semantic units. The recognition word may be a word included in a letter recognition model document image. The recognition word may be a grouping of letters in a semantic unit. For example, the processor 120 may group characters on the basis of spacing, or may group characters on a noun basis. For example, the recognition word may be a single word such as'examination fee', or a phrase in which a plurality of words such as'blood sugar measurement' are combined. The detailed description of the above-mentioned recognition words is only an example, and the present disclosure is not limited thereto. The processor 120 compares each of the one or more recognition words with one or more reference words stored in memory, and in the case of a recognition word without a matching reference word, at least one letter of one or more letter images that are the basis of the recognition word It can be determined that the letter determined as a result of classifying the image is not suitable. The reference word may be a word pre-stored in the memory 130. Since the words used for each domain may be different, words used for each domain corresponding to the document image may be stored in the memory 130. The processor 120 may compare the recognized word with the reference word stored in the memory 130 with respect to the domain of the document image. When the reference word matching the recognized word is not stored in the memory 130, the processor 120 may not match the letter determined as a result of classification of at least one letter image among one or more letter images that are the basis of the recognized word. You can decide.

For example, referring to FIG. 2, the document image 200 may include one or more character images. The domain of the document image 200 may be a medical bill detailing statement or an insurance company. The processor 120 groups letters determined as a result of classification of each of the one or more text images 212 and 214 included in the document cost calculation statement, which is the document image 200, into a semantic unit and recognizes the word 240 as a'pando magjeong' 'Can be extracted. The processor 120 may compare the recognized word 240 with reference words 252 and 254 of the reference word table 250 stored in the memory 130 based on the domain of the document image 200. In the processor 120, since the domain of the document image 200 is a medical fee detail statement, the recognition word 240 is a reference word 252 and 254 of the reference word table 250 corresponding to the medical fee detail statement domain stored in the memory 130 ). The processor 120 determines that there is no reference word matching the recognition word 240,'pan dog correction', and a character determined as a result of classification of at least one character image among one or more character images that are the basis of the recognition word Can be determined to be unsuitable. The specific description of the method for determining whether the classification result of the above-described character image is suitable is merely an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, the processor 120 extracts a recognition word based on a letter determined as a result of classifying a letter image, extracts a category of the recognition word from the document image, and recognizes the recognition word in the memory 130. If there is no reference word that is a sub-structure of the category to be matched, it may be determined that the classification result of the character image is not suitable. The processor 120 may extract one or more recognition words by grouping characters determined as a result of classification of each of the one or more character images included in the document image into semantic units. The processor 120 may determine whether the classification result of the character image is a suitable character through context analysis of the recognition word on the document image. Hereinafter, it will be described with reference to FIG. 2. The processor 120 may determine a recognition word of one of the one or more recognition words as a target recognition word. The object recognition word may be one recognition word that is the object of context analysis. The processor 120 may include a group of letter images (210 in FIG. 2) that are the basis of the target recognition word in the document image and one or more recognition words positioned vertically or horizontally (in FIG. 2) Recognition words matching the category name stored in the memory 130 among the recognition words'internal medicine' based on the group 220 and the recognition words'name' based on the group 230 of the letter image (refer to FIG. 2, The recognition word'internal medicine' based on the group 220 is determined as the upper category of the target recognition word (refer to FIG. 2, the target recognition word 240 based on the group 210 of the letter image). Can be. Based on the domain of the document image, the processor 120 may compare the category name stored in the memory 130 with the target recognition word and one or more recognition words positioned vertically or horizontally for the domain. Referring to FIG. 2, for example, the domain of the document image 200 may be a detailed statement of medical expenses, and the processor 120 may include detailed medical fees in one or more recognized words and memory 130 positioned vertically or horizontally with the target recognized word. You can compare one or more category names stored for the statement. The processor 120 may include one or more reference words (referred to in FIG. 2, pantomag tablets 252, Hartmann's sum, Ricky Pan tablets, etc.) stored as a sub-structure of the category name (refer to FIG. 2, internal medicine 254). And whether or not the target recognition word (refer to FIG. 2, the recognition word 240 panmagagjeong) is matched. When there is no reference word matching the target recognition word, the processor 120 may determine that the target recognition word does not match the context of the document image. The processor 120 may determine that the classification result of at least one letter image among the one or more letter images that are the basis of the target recognition word is not suitable. The specific description of the method for determining whether the classification result of the above-described character image is suitable is merely an example, and the present disclosure is not limited thereto.

Hereinafter, a method for re-determining the characters included in the character image by reclassifying the character image when the processor 120 determines that the character determined as the result of the classification of the character image is not suitable. The re-classification of the text image may refer to an operation of correcting the text image to display a correct recognition result.

According to an embodiment of the present disclosure, the processor 120 may re-determine a character so that the result of the classification of the character image is included in one or more character classification classes stored in the memory 130. The processor 120 may re-determine a character to be included in the character classification class stored in the memory 130 when the character determined as a result of the classification of the character image is a character not included in the entire character classification class. The specific method of re-determining the characters to be included in the character classification class will be described later.

According to an embodiment of the present disclosure, the processor 120 may re-determine a letter based on the recognized word and the reference word having the highest matching ratio. The processor 120 may extract a recognition word based on the letter image for which the classification result is not suitable. The processor 120 based on the reference word having the highest matching ratio of one or more letters included in each of the reference words among the one or more letters included in the recognition word and one or more reference words stored in the memory 130, the recognition The character image in which the classification result included in the word is not suitable may be re-classified. The processor 120 may compare the recognition word with reference words for a domain of the document image from which the recognition word is extracted from among the reference words in the memory 130. The processor 120 compares one or more letters included in the recognized word and one or more letters included in each of the one or more reference words stored in the memory 130 to match the matching rate of the reference word containing the same letter the most. You can decide with this highest reference word. The processor 120 may determine the reference word having the highest matching ratio based on the arrangement of one or more letters included in the recognized word and the arrangement of one or more letters included in the reference word. For example, referring to FIG. 2, the processor 120 stores the recognition word 240 pandomagjeong in the memory 130, the domain of the recognition word 240 pandomagjeong, a reference word table stored for the details of medical expenses It can be compared based on (250). Since the word arrangement and the same word ratio of the recognition word 240 and the reference word pantomagjeong 252 are the highest, the processor 120 may re-classify the character image based on the pantomagjeong 252. . The processor 120 may re-classify a letter image based on letters different from letters included in the reference word having the highest matching ratio among the one or more letters included in the recognized word. The processor 120 may re-classify a letter image into letters of a reference word with respect to the letter images that are the basis of the different letters. The processor 120 may re-classify the character image based on the difference between the letter'degree' of the recognition word 240 and the reference word'toe' of the reference word pantomagjeong 252. The processor 120 may re-classify the text image 214 that is the basis of the different letters'degrees' as'sat'. The specific description regarding the re-classification of the character image described above is merely an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, the processor 120 may receive manual correction information for the text image from a user computing device. The processor 120 may re-classify the text image based on the manual correction information. The manual correction information may be error information received from a user using a character recognition model. For example, when the hospital checks the details of the medical fee details through the letter recognition model, the hospital staff can manually correct the letters that are incorrectly recognized and input them to the user computing device. The processor 120 may re-classify the corresponding letter image based on the correction manually input by the hospital staff. The specific description regarding the manual correction described above is merely an example, and the present disclosure is not limited thereto.

The processor 120 may display non-conformance information on the character image when the character determined as the result of the classification of the character image is not suitable. The processor 120 may display the non-conformance information on the text image, and notify the user terminal. The processor 120 may receive correction information for the character image in which the non-conformance information is displayed from the user terminal through the network unit 110. The processor 120 may re-classify the text image based on the correction information.

Hereinafter, a description will be given of a method for the processor 120 to generate learning data, which is a basis for updating a character recognition model.

The processor 120 may generate the first learning data using a character image as an input of the first learning data and a character determined as a result of the reclassification as a label of the first learning data. The processor 120 may generate first learning data based on the character image that is the basis of the character when the character determined as the classification result is not suitable. For example, referring to FIG. 2, the first learning data using the text image 260 as the input of the first learning data, and the character 270 determined as the result of the reclassification as the label of the first learning data Can generate The characters determined as the result of the reclassification may be characters included in the entire character classification class of the character recognition model, or may not be characters included in the entire character classification class of the character recognition model. The first training data set may include a predetermined number of first training data collected after training of the character recognition model, or may include first training data collected during a predetermined period. According to an embodiment of the present disclosure, the processor 120 may collect a predetermined number of first learning data and generate an updated character recognition model based on the first learning data set. According to an embodiment of the present disclosure, the processor 120 may collect the first learning data for a predetermined period, and generate an updated character recognition model based on the first learning data set.

Hereinafter, a method of generating the updated character recognition model by the processor 120 will be described.

The processor 120 may generate an updated character recognition model based on the first learning data set including one or more first learning data. The processor 120 may classify the character image of the first learning data as one character by using the character recognition model as an input. The processor 120 may calculate an error by comparing a letter determined as a result of the learning classification based on the classification of the first learning data and a letter determined as the result of the re-classification, which is a label of the first learning data. The processor 120 may update the weight of the character recognition model based on the error to generate an updated character recognition model.

The updated character recognition model training data may be second training data of at least a portion of the first training data set and the second training data set on which the pre-trained character recognition model training is based. At least a part of the second training data set may be second training data corresponding to a predetermined ratio of the second training data set, or may be all of the second training data included in the second training data set.

According to an embodiment of the present disclosure, the processor 120 may perform learning by setting the initial weight of the updated character recognition model as the final weight of the existing character recognition model. The processor 120 may perform learning on a new character recognition model based on at least some of the second training data and the first training data among the second training data sets that are the basis of the pre-trained character recognition model training. . If the final weight of the existing character recognition model is set as the initial weight of the updated character recognition model, the updated character recognition model is the first, even if the entire second training data set is not included in the training data set of the updated character recognition model. It may not be overfitting only for the training data, and may have generalized performance for classes other than the character classes included in the first training data.

According to an embodiment of the present disclosure, the processor 120 may perform learning by setting the initial weight of the updated character recognition model to share at least a portion of the final weight of the existing character recognition model. The processor 120 may perform learning by setting at least some nodes of the updated character recognition model to be the same as those of the existing character recognition model. The layer may include one or more nodes. The processor 120 may perform learning by setting at least some layers of the updated character recognition model to be the same as those of the existing character recognition model. The processor 120 may perform learning by setting the remaining layers, except for a predetermined number of layers, based on the last layer of the updated character recognition model, the same as the layer of the existing character recognition model. The predetermined number of layers may be one, or may be two or more. The layer corresponding to the front portion of the character recognition model is mainly used to extract features from the character image, and the layer for classifying the character image may be a layer corresponding to the rear portion of the character recognition model. Therefore, the layer corresponding to the front part of the updated character recognition model may be set to be the same as the layer of the existing character recognition model to perform learning.

According to an embodiment of the present disclosure, the processor 120 may perform learning by setting an initial weight of the updated character recognition model at random. When the initial weight of the updated character recognition model is randomly set, the processor 120 may use the first training data set and the entire second training data set as training data.

Hereinafter, a method of classifying the character image using the updated character recognition model by the processor 120 will be described.

The processor 120 may classify the character image through the ensemble of the existing character recognition model and the updated character recognition model. The operation through the ensemble may be a method of deriving the final output based on the output of a plurality of models (ie, a model including one or more network functions).

According to an embodiment of the present disclosure, the processor 120 may classify a character image by ensemble the existing character recognition model and the updated character recognition model in parallel. The processor 120 may classify the text image included in the document image using the text recognition model and the updated text recognition model. The processor 120 may ensemble the classification result of the character recognition model and the classification result of the updated character recognition model to generate a final classification result for characters included in the character image.

According to an embodiment of the present disclosure, the updated character recognition model may be a model trained based on the second training data and the first training data set corresponding to at least a portion of the second training data set. The updated character recognition model may be a model learned by setting an initial weight equal to the final weight of the existing character recognition model. The processor 120 may classify the character images included in the document image as inputs to the character recognition model and the updated character recognition model, respectively. When the existing character recognition model classifies the character image as a character not included in the character classification class, the processor 120 places a greater weight on the classification result of the updated character recognition model than the classification result of the character recognition model. The final classification result for the text included in the text image may be generated. This is because the recognition result of the updated character recognition model may be determined to be more accurate for the newly learned character classification class. For example, the character classification class of the existing character recognition model is “A, B, D,” and the character classification class of the updated character recognition model is “A, B, D, D, D,” and the character image is written. When the result classified through the recognition model is output as letters other than “A, B, C,” and the result of classifying the text image through the updated character recognition model is output as La letters, the processor 120 It is possible to place a greater weight on the output result of the updated character recognition model than the output result of the existing character recognition model. For example, the processor 120 may calculate the weight by putting 0.5 weight on the output of the existing character recognition model and 0.7 weight on the output of the updated character recognition model, so that the final classification result may be determined to be'L'. When the existing character recognition model classifies the character image as a character included in the character classification class, the processor 120 places a larger weight on the classification result of the character recognition model than the updated character recognition model classification result. The final classification result for the text included in the image may be generated. This is because the operation of the characters corresponding to the existing character classification class may be determined to be more accurate than the operation of the updated character recognition model. For example, when the existing character recognition model and the updated character recognition model have a character classification class as in the above example, the result of classifying the character image through the character recognition model is output as a character, and the character image is updated. When the result classified through the recognized character recognition model is output as a letter or the character, the processor 120 may place a weight greater than the output result of the updated character recognition model in the output result of the existing character recognition model. The processor 120 may calculate the weight of the letter'a', which is the result of classification through the letter recognition model, as 0.8, and the weight of the letter'I', which is the result of classification through the updated letter recognition model, of 0.5. . For example, as a result of classifying through the letter recognition model, the score value of the letter'A' is 0.6, the weight value based on the ensemble is 0.8, and the score value of the letter'I', which is the result of classification through the updated letter recognition model, is If it is 0.9 and the weight value based on the ensemble is 0.5, the processor 120 has a total operation value for the letter'a' of 0.48 and a total operation value for the letter'b' of 0.45, so the final classification result is'a'. You can decide. The detailed description of the above-described method for classifying the letter image is only an example, and the present disclosure is not limited thereto. According to an embodiment of the present disclosure, the processor 120 may ensemble an existing character recognition model and two or more character recognition models in parallel, and classify a character image through voting. The processor 120 may determine a final classification result as a classification result for the character images that receive the most votes among the classification results of each of the three or more character recognition models. That is, in the present disclosure, an ensemble method using a parallel model may be simple voting, an ensemble method in which weights for each model and classification are applied to the output of each model, and may include any known ensemble method. Character images may be classified using the existing character recognition model, the first updated character recognition model, and the second updated character recognition model. The initial weight of the updated character recognition model may be set randomly. For example, the first updated character recognition model may be a model trained based on the first learning data set and the second learning data set. The second updated character recognition model may be a model trained based on the third training data set and the first training data set, or in at least a portion of the third training data set, the second training data set, and the first training data set. It may be a model trained on the basis. The third training data set may be a set of training data generated during a predetermined period or a predetermined number of training data after the first training data set is generated. For example, if the character classification result of the existing character recognition model is character Po, the classification result of the first updated character recognition model is F, and the classification result of the second updated character recognition model is Po, more tables The final classification result can be determined by the'po' received. The detailed description of the above-described method for generating the text image is only an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, the processor 120 may classify a character image by configuring an existing character recognition model and an updated character recognition model in series. The processor 120 may classify the text image included in the document image as the first text through the text recognition model. When the first character corresponds to a character classification class determined based on the first learning data, the processor 120 may determine to re-classify the character image using the updated character recognition model. The processor 120 may classify the text image using the updated text recognition model to generate a final classification result for the text included in the text image. For example, the character image is classified as the first character'plate' through the character recognition model, and there is a'plate' on the label of the first learning data, so the'plate' is in the character classification class of the updated character recognition model. If present, the image of the character may be classified through the updated character recognition model, and the classification result may be determined as the final classification result of the serial ensemble model. The detailed description of the above-described method for classifying text images is only an example, and the present disclosure is not limited thereto.

The memory 130 may store a computer program for performing a method of updating a character recognition model according to an embodiment of the present disclosure, and the stored computer program may be read and driven by the processor 120.

The memory 130 according to embodiments of the present disclosure may store a program for the operation of the processor 120, and re-classify data input/output (eg, a document image, a text image, a text image) It can be stored temporarily or permanently. The memory 130 may store data related to a display and sound. The memory 130 is a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (for example, SD or XD memory, etc.), RAM (Random Access Memory, RAM), Static Random Access Memory (SRAM), Read-Only Memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Programmable Read-Only Memory (PROM), magnetic memory, magnetic It may include at least one type of storage medium among disks and optical disks.

3 is a flowchart of a method for updating a character recognition model according to an embodiment of the present disclosure.

The computing device 100 classifies each of the one or more character images included in the document image into each character through a character recognition model including one or more pre-trained network functions to determine the characters included in the character image (310) can do. The document image may be an image of a document including one or more recognition target characters. The document image may include one or more text images. The letter image may be an image including one letter.

The computing device 100 may determine whether a character determined as a result of the classification of the character image based on the classification is a suitable character (320).

According to an embodiment of the present disclosure, when the reliability of the classification result for the character image is equal to or less than a predetermined value, the computing device 100 may determine that the character determined as the classification result of the character image is not suitable. When the reliability is less than or equal to a predetermined value, it may mean that the score of the class that is the basis of the classification of the text image is equal to or less than the predetermined value.

According to an embodiment of the present disclosure, the computing device 100 may determine whether a character determined as a result of the classification is a character not included in the entire character classification class, or a character determined as a result of the classification is character-by-character in the domain of the document image. When the frequency of use corresponds to a letter having a predetermined value or less, it may be determined that the letter determined as the result of the classification of the letter image is not suitable. The entire character classification class may include one or more classes for predetermined characters. The domain of the document image may mean characteristics of a document included in the document. The domain of the document image may mean a field in which the document image is used.

According to an embodiment of the present disclosure, the computing device 100 may extract one or more recognition words by grouping characters determined as a result of classification of each of the one or more character images included in the document image into semantic units. The computing device 100 is based on a result of comparing each of the one or more recognized words with one or more reference words stored in a memory, and is determined as a classification result of at least one letter image among one or more letter images that are the basis of the recognized word Can be determined to be unsuitable. The recognition word may be a word included in a letter recognition model document image. The recognition word may be a grouping of letters in a semantic unit. The reference word may be a word pre-stored in the computing device 100.

According to an embodiment of the present disclosure, the computing device 100 may extract one or more recognition words by grouping characters determined as a result of classification of each of the one or more character images included in the document image into semantic units. The computing device 100 may determine whether the classification result of the character image is a suitable character through context analysis of the recognition word on the document image. The computing device 100 extracts a recognition word based on a letter determined as a result of classifying a letter image, extracts a category of the recognition word from the document image, and sub-structures the category matching the recognition word in the computing device 100 If there is no phosphorus reference word, it may be determined that the classification result of the letter image is not suitable.

If the character determined as the result of the classification of the character image is not suitable, the computing device 100 may re-classify the character image and re-determine the character included in the character image (330).

According to an embodiment of the present disclosure, the computing device 100 may re-classify the character image so that the result of the classification of the character image is included in one or more character classes stored in memory. The re-classification of the text image may refer to an operation of correcting the text image to display a correct recognition result.

According to an embodiment of the present disclosure, the computing device 100 may extract a recognition word based on the character image for which the classification result is not suitable. The computing device 100 based on the reference word having the highest matching ratio of one or more letters included in each of the reference words among the one or more letters included in the recognition word and one or more reference words stored in the memory, to the recognition word It is possible to re-classify the character images in which the included classification results are not suitable. The computing device 100 compares one or more letters included in the recognized word with one or more letters included in each of the one or more reference words stored in the computing device 100, and compares the reference word containing the same letter the most. It can be determined by the reference word with the highest matching ratio.

According to an embodiment of the present disclosure, the computing device 100 may receive manual correction information for the text image from a user computing device. The computing device 100 may re-classify the text image based on the manual correction information.

The computing device 100 may display non-conformance information on the character image when the character determined as the result of the classification of the character image is not suitable.

The computing device 100 may generate the first learning data (340) using the character image as the input of the first learning data and the character determined as a result of the reclassification as the label of the first learning data. The first training data set may include a predetermined number of first training data collected after training of the character recognition model, or may include first training data collected during a predetermined period.

The computing device 100 may generate 350 an updated character recognition model based on the first learning data set including one or more first learning data.

According to an embodiment of the present disclosure, the computing device 100 may classify the character image of the first learning data as one character by using the character recognition model as an input. The computing device 100 may calculate an error by comparing a letter determined as a result of the learning classification based on the classification of the first learning data and a letter determined as the result of the re-classification, which is a label of the first learning data. The computing device 100 may generate an updated character recognition model by updating the weight of the character recognition model based on the error.

According to an embodiment of the present disclosure, the computing device 100 is based on the second learning data and the first learning data set of at least some of the second learning data sets that are the basis of learning the pre-trained character recognition model. By learning a new character recognition model, an updated character recognition model can be generated.

According to an embodiment of the present disclosure, the weight of the pre-trained character recognition model may be shared at least partially. The computing device 100 may perform learning by setting the initial weight of the updated character recognition model as the final weight of the existing character recognition model. The computing device 100 may perform learning by setting at least some nodes of the updated character recognition model to be the same as those of the existing character recognition model. The computing device 100 may perform learning by setting at least some layers of the updated character recognition model to be the same as those of the existing character recognition model. The computing device 100 may perform learning by setting the remaining layers except for a predetermined number of layers based on the last layer of the updated character recognition model, the same as the layer of the existing character recognition model.

The computing device 100 may classify the text image included in the document image using the text recognition model and the updated text recognition model. The computing device 100 may ensemble the classification result of the character recognition model and the classification result of the updated character recognition model to generate a final classification result for characters included in the character image.

The computing device 100 may classify the text image included in the document image as the first text through the text recognition model. When the first character corresponds to a character classification class determined based on the first learning data, the computing device 100 may determine to classify the character image using the updated character recognition model. The computing device 100 may classify the text image using the updated text recognition model to generate a final classification result for the text included in the text image.

4 is a block diagram illustrating logic for implementing a method of updating a character recognition model according to an embodiment of the present disclosure.

According to an embodiment of the present disclosure, a method of updating a character recognition model may be implemented by the following logic.

In the method of updating a character recognition model according to an embodiment of the present disclosure, each character is classified into each character through a character recognition model including one or more network functions that are pre-trained in each of the one or more character images included in the document image. Logic 410 for determining the characters included in the image; A logic 420 for determining whether a character determined as a result of the classification of the character image based on the classification is a suitable character; A logic 430 for re-determining the characters included in the character image by re-classifying the character images when the characters determined as the result of the classification of the character images are not suitable; A logic 440 for generating the first learning data, wherein the text image is the input of the first learning data, and the text determined as a result of the reclassification is a label of the first learning data; And a logic 450 for generating an updated character recognition model based on the first learning data set including one or more first learning data.

In an alternative embodiment of the method of updating the character recognition model, the logic 420 for determining whether a character determined as a result of the classification of the character image based on the classification is a suitable character,

In an alternative embodiment of the method for updating the character recognition model, the logic 420 for determining whether a character determined as a result of the classification of the character image based on the classification is a suitable character, the reliability of the classification result for the character image When is less than or equal to a predetermined value, the character determined as the result of the classification of the character image may be implemented by logic to determine that it is not suitable.

In an alternative embodiment of the method for updating the character recognition model, the logic 420 for determining whether the character determined as the result of the classification of the character image based on the classification is a suitable character, the character determined as the classification result is the entire character Characters determined as the result of the classification of the character image are suitable when the characters are not included in the classification class, or if the character determined as the result of the classification corresponds to a character whose frequency of use for each letter of the domain of the document image is less than or equal to a certain value. It can be implemented by logic to decide not to.

In an alternative embodiment of the method for updating the character recognition model, the logic 420 for determining whether a character determined as a result of the classification of the character image based on the classification is a suitable character, one or more characters included in the document image Logic for extracting one or more recognition words by grouping the characters determined as the result of each classification of the images into semantic units; And a letter determined as a result of classifying at least one letter image among one or more letter images that are the basis of the recognized word, based on a result of comparing each of the one or more recognized words with one or more reference words stored in memory. It can be implemented by logic to determine.

In an alternative embodiment of the method for updating the character recognition model, the logic 420 for determining whether a character determined as a result of the classification of the character image based on the classification is a suitable character, one or more characters included in the document image Logic for extracting one or more recognition words by grouping the characters determined as the result of each classification of the images into semantic units; And logic for determining whether a classification result of the character image is a suitable character through context analysis of the recognition word on the document image.

In an alternative embodiment of the method for updating the character recognition model, if the character determined as the result of the classification of the character image is not suitable, logic for re-classifying the character image and re-determining the characters included in the character image ( 430) may be implemented by logic to re-classify the character image so that the classification result of the character image is included in one or more character classes stored in memory.

In an alternative embodiment of the method for updating the character recognition model, if the character determined as the result of the classification of the character image is not suitable, logic for re-classifying the character image and re-determining the characters included in the character image ( 430), logic for extracting a recognition word based on the character image for which the classification result is not suitable; And one or more letters included in the recognition word and one or more reference words stored in the memory, based on the reference word having the highest matching ratio of each of the reference words, the classification result included in the recognition word. Can be implemented by logic to re-classify unsuitable text images.

In an alternative embodiment of the method for updating the character recognition model, if the character determined as the result of the classification of the character image is not suitable, logic for re-classifying the character image and re-determining the characters included in the character image ( 430) logic for receiving manual correction information for the text image from a user computing device; And logic for re-classifying the text image based on the manual correction information.

In an alternative embodiment of the method for updating the character recognition model, if the character determined as the result of the classification of the character image is not suitable, it may be additionally implemented by logic for displaying non-conformance information on the character image.

In an alternative embodiment of the method for updating the character recognition model, the logic 450 for generating an updated character recognition model based on the first training data set including one or more first training data includes: the first training data Logic for classifying the text image of the text into one text by inputting the text recognition model; Logic for calculating an error by comparing a letter determined as a result of the learning classification based on classification of the first learning data and a letter determined as the result of the re-classification, which is a label of the first learning data; It can be implemented by logic for generating an updated character recognition model by updating the weight of the character recognition model based on the error.

In an alternative embodiment of the method for updating a character recognition model, the logic 450 for generating an updated character recognition model based on a first training data set including one or more first training data includes: the pre-trained character Based on logic for generating an updated character recognition model by training a new character recognition model based on at least some of the second training data sets and the first training data set, which are the basis of the recognition model training. Can be implemented.

In an alternative embodiment of a method for updating a character recognition model, logic for classifying a character image included in the document image using the character recognition model and the updated character recognition model; And logic for ensembles the classification result of the character recognition model and the classification result of the updated character recognition model to generate a final classification result for the characters included in the character image.

In an alternative embodiment of a method for updating a character recognition model, logic for classifying a character image included in the document image into a first character through the character recognition model; Logic for determining that the character image is to be classified using the updated character recognition model when the first character corresponds to a character classification class determined based on the first learning data; And a logic for classifying the text image using the updated text recognition model to generate a final classification result for text included in the text image.

The logic for implementing the update method of the character recognition module according to an embodiment of the present disclosure may be implemented by a module, circuit, or means for implementing the update method of the character recognition module.

5 is a block diagram of a computing device according to an embodiment of the present disclosure.

5 shows a simplified and general schematic diagram of an exemplary computing environment in which embodiments of the present disclosure can be implemented.

Although the present disclosure has been described above in general with respect to computer-executable instructions that can be executed on one or more computers, those skilled in the art will appreciate that the present disclosure can be implemented in combination with other program modules and/or as a combination of hardware and software. will be.

Generally, program modules include routines, programs, components, data structures, etc. that perform particular tasks or implement particular abstract data types. In addition, those of ordinary skill in the art may find that the methods of the present disclosure include single-processor or multiprocessor computer systems, minicomputers, mainframe computers, as well as personal computers, handheld computing devices, microprocessor-based or programmable consumer electronics, and the like (each of which is). It will be appreciated that it may be implemented with other computer system configurations, including those that may operate in conjunction with one or more associated devices.

The described embodiments of the present disclosure can also be practiced in distributed computing environments where certain tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

Computers typically include a variety of computer readable media. Any computer-accessible medium can be a computer-readable medium. Computer-readable media includes volatile and non-volatile media, transitory and non-transitory media, removable and non-removable media. By way of example, and not limitation, computer readable media may include computer readable storage media and computer readable transmission media. Computer-readable storage media are volatile and non-volatile media, temporary and non-transitory media, removable and non-removable, implemented in any method or technology for storing information such as computer readable instructions, data structures, program modules or other data Includes media. Computer storage media includes RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital video disk (DVD) or other optical disk storage devices, magnetic cassettes, magnetic tapes, magnetic disk storage devices or other magnetic storage devices, Or any other medium that can be accessed by a computer and used to store desired information.

Computer readable transmission media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal, such as a carrier wave or other transport mechanism, and Includes all information delivery media. The term modulated data signal means a signal in which one or more of the characteristics of the signal are set or changed to encode information in the signal. By way of example, and not limitation, computer readable transmission media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media. Combinations of any of the above described media are also intended to be included within the scope of computer readable transmission media.

An exemplary environment 1100 is shown that implements various aspects of the present disclosure, including a computer 1102, and the computer 1102 includes a processing device 1104, a system memory 1106, and a system bus 1108. do. System bus 1108 connects system components, including, but not limited to, system memory 1106 to processing device 1104. The processing device 1104 can be any of a variety of commercial processors. Dual processor and other multiprocessor architectures may also be used as the processing unit 1104.

The system bus 1108 can be any of several types of bus structures that can be further interconnected to a memory bus, peripheral bus, and local bus using any of a variety of commercial bus architectures. System memory 1106 includes read-only memory (ROM) 1110 and random access memory (RAM) 1112. The basic input/output system (BIOS) is stored in a non-volatile memory 1110 such as ROM, EPROM, EEPROM, etc. This BIOS is a basic to help transfer information between components in the computer 1102 at the same time as during startup. Contains routines. The RAM 1112 may also include high-speed RAM, such as static RAM for caching data.

Computer 1102 is also an internal hard disk drive (HDD) 1114 (eg, EIDE, SATA)-this internal hard disk drive 1114 can also be configured for external use within a suitable chassis (not shown). Yes, magnetic floppy disk drive (FDD) 1116 (e.g., for reading from or writing to removable diskette 1118), and optical disk drive 1120 (e.g., CD-ROM Disc 1122, or for reading from or writing to other high-capacity optical media such as DVDs). The hard disk drive 1114, the magnetic disk drive 1116, and the optical disk drive 1120 are the system bus 1108 by the hard disk drive interface 1124, the magnetic disk drive interface 1126, and the optical drive interface 1128, respectively. ). The interface 1124 for implementing an external drive includes at least one or both of Universal Serial Bus (USB) and IEEE 1394 interface technologies.

These drives and their associated computer-readable media provide non-volatile storage of data, data structures, computer-executable instructions, and the like. In the case of computer 1102, drives and media correspond to storing any data in a suitable digital format. Although the above description of computer readable media refers to HDDs, removable magnetic disks, and removable optical media such as CDs or DVDs, those of ordinary skill in the art can use a zip drive, magnetic cassette, flash memory card, cartridge, etc. It will be appreciated that other types of media readable by a computer, etc., may also be used in the exemplary operating environment and any such media may include computer-executable instructions for performing the methods of the present disclosure.

A number of program modules may be stored in the drive and RAM 1112, including the operating system 1130, one or more application programs 1132, other program modules 1134, and program data 1136. All or part of the operating system, applications, modules, and/or data may also be cached in RAM 1112. It will be appreciated that the present disclosure can be implemented in various commercially available operating systems or combinations of operating systems.

The user may input commands and information to the computer 1102 through one or more wired/wireless input devices, for example, pointing devices such as a keyboard 1138 and a mouse 1140. Other input devices (not shown) may include a microphone, IR remote control, joystick, game pad, stylus pen, touch screen, etc. These and other input devices are often connected to the processing unit 1104 through an input device interface 1142 connected to the system bus 1108, but the parallel port, IEEE 1394 serial port, game port, USB port, IR interface, And other interfaces.

The monitor 1144 or other type of display device is also connected to the system bus 1108 through an interface such as a video adapter 1146. In addition to the monitor 1144, the computer generally includes other peripheral output devices (not shown), such as speakers, printers, and the like.

The computer 1102 can operate in a networked environment using logical connections to one or more remote computers, such as remote computer(s) 1148 via wired and/or wireless communication. The remote computer(s) 1148 may be a workstation, computing device computer, router, personal computer, portable computer, microprocessor-based entertainment device, peer device, or other conventional network node, typically in computer 1102. It includes many or all of the components described with respect to, but for simplicity, only the memory storage device 1150 is shown. The illustrated logical connections include wired/wireless connections to a local area network (LAN) 1152 and/or a larger network, such as a wide area network (WAN) 1154. Such LAN and WAN networking environments are common in offices and companies, and facilitate enterprise-wide computer networks, such as intranets, all of which can be connected to computer networks around the world, such as the Internet.

When used in a LAN networking environment, the computer 1102 is connected to the local network 1152 via a wired and/or wireless communication network interface or adapter 1156. The adapter 1156 may facilitate wired or wireless communication to the LAN 1152, which also includes a wireless access point installed therein to communicate with the wireless adapter 1156. When used in a WAN networking environment, the computer 1102 may include a modem 1158, connect to a communication computing device on the WAN 1154, or establish communication over the WAN 1154, such as through the Internet. Have other means. The modem 1158, which may be an internal or external and wired or wireless device, is connected to the system bus 1108 via a serial port interface 1142. In a networked environment, program modules described for the computer 1102 or portions thereof may be stored in the remote memory/storage device 1150. It will be appreciated that the illustrated network connection is exemplary and other means of establishing a communication link between computers can be used.

The computer 1102 is associated with any wireless device or entity that is deployed and operates in wireless communication, such as a printer, scanner, desktop and/or portable computer, a portable data assistant (PDA), communication satellite, or wireless detectable tag. It operates to communicate with any equipment or place and telephone. This includes at least Wi-Fi and Bluetooth wireless technology. Accordingly, the communication may be a predefined structure as in a conventional network or simply ad hoc communication between at least two devices.

Wi-Fi (Wireless Fidelity) enables a connection to the Internet or the like without a wired connection. Wi-Fi is a wireless technology such as a cell phone that allows a computer, for example, a computer to transmit and receive data indoors and outdoors, ie anywhere within the base station's coverage area. Wi-Fi networks use a wireless technology called IEEE 802.11 (a,b,g, etc.) to provide a secure, reliable and high-speed wireless connection. Wi-Fi can be used to connect computers to each other, to the Internet, and to a wired network (using IEEE 802.3 or Ethernet). Wi-Fi networks can operate in unlicensed 2.4 and 5 GHz radio bands, for example, at 11 Mbps (802.11a) or 54 Mbps (802.11b) data rates, or in products that include both bands (dual band). have.

Those of ordinary skill in the art of the present disclosure will understand that information and signals can be represented using any of a variety of different technologies and techniques. For example, data, instructions, instructions, information, signals, bits, symbols and chips that can be referenced in the above description are voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields Or particles, or any combination thereof.

Those of ordinary skill in the art of the present disclosure may use various examples of logical blocks, modules, processors, means, circuits and algorithm steps described in connection with the embodiments disclosed herein in electronic hardware, (convenience For this, it will be understood that it may be implemented by various forms of program or design code (referred to herein as “software”) or a combination of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Those skilled in the art of the present disclosure may implement the functions described in various ways for each particular application, but such implementation decisions should not be interpreted as being outside the scope of the present disclosure.

Various embodiments presented herein can be implemented as a method, apparatus, or article of manufacture using standard programming and/or engineering techniques. The term “manufactured article” includes a computer program, carrier, or media accessible from any computer-readable device. For example, computer-readable media include magnetic storage devices (eg, hard disks, floppy disks, magnetic strips, etc.), optical disks (eg, CDs, DVDs, etc.), smart cards, and flash memory Devices (eg, EEPROMs, cards, sticks, key drives, etc.), but are not limited to these. In addition, various storage media presented herein include one or more devices and/or other machine-readable media for storing information.

It is understood that the specific order or hierarchy of steps in the processes presented is an example of exemplary approaches. Based on design priorities, it is understood that within the scope of the present disclosure, a specific order or hierarchy of steps in the processes may be rearranged. The accompanying method claims provide elements of the various steps in a sample order, but are not meant to be limited to the specific order or hierarchy presented.

Descriptions of the presented embodiments are provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art of the present disclosure. The general principles defined herein can be applied to other embodiments without departing from the scope of the present disclosure. Thus, the present disclosure should not be limited to the embodiments presented herein, but should be interpreted in the broadest scope consistent with the principles and novel features presented herein.

Claims (17)

A computer program stored on a computer-readable storage medium, the computer program, when executed on one or more processors, performs the following operations for updating the character recognition model, the operations being:
Determining a letter included in the letter image by classifying each of the one or more letter images included in the document image into each letter through a letter recognition model including one or more pre-trained network functions;
Determining whether a character determined as a result of the classification of the character image based on the classification is a suitable character;
Re-determining a character included in the character image by re-classifying the character image when the character determined as the result of the classification of the character image is not suitable;
Generating the first learning data using the text image as the input of the first learning data, and using the text determined as a result of the reclassification as a label of the first learning data; And
Generating an updated character recognition model based on the first set of learning data including one or more first learning data;
Containing,
A computer program stored on a computer readable storage medium.
According to claim 1,
The operation of determining whether a character determined as a result of the classification of the character image based on the classification is a suitable character,
If the reliability of the classification result for the character image is equal to or less than a predetermined value, determining that the character determined as the classification result of the character image is not suitable;
Containing,
A computer program stored on a computer readable storage medium.
According to claim 1,
The operation of determining whether a character determined as a result of the classification of the character image based on the classification is a suitable character,
If the character determined as the result of the classification is a character that is not included in the entire character classification class, or when the character determined as the result of the classification corresponds to a character whose frequency of use for each letter of the domain of the document image is less than or equal to a certain value, the character image Determining that the letter determined as the result of the classification is not suitable;
Containing,
A computer program stored on a computer readable storage medium.
According to claim 1,
The operation of determining whether a character determined as a result of the classification of the character image based on the classification is a suitable character,
Extracting one or more recognition words by grouping characters determined as a result of classification of each of the one or more character images included in the document image into semantic units; And
Based on a result of comparing each of the one or more recognized words with one or more reference words stored in a memory, it is determined that a letter determined as a result of classifying at least one letter image among one or more letter images that are the basis of the recognized word is not suitable. To do;
Containing,
A computer program stored on a computer readable storage medium.
According to claim 1,
The operation of determining whether a character determined as a result of the classification of the character image based on the classification is a suitable character,
Extracting one or more recognition words by grouping characters determined as a result of classification of each of the one or more character images included in the document image into semantic units; And
Determining whether a classification result of the character image is a suitable character through context analysis of the recognition word on the document image;
Containing,
A computer program stored on a computer readable storage medium.
According to claim 1,
When the character determined as the result of the classification of the character image is not suitable, the operation of re-classifying the character image and re-determining the characters included in the character image may include:
Re-classifying the text image so that the result of the classification of the text image is included in one or more text classes stored in memory;
Containing,
A computer program stored on a computer readable storage medium.
According to claim 1,
When the character determined as the result of the classification of the character image is not suitable, the operation of re-classifying the character image and re-determining the characters included in the character image may include:
Extracting a recognition word based on a letter image in which the classification result is not suitable; And
The classification result included in the recognition word is based on the reference word having the highest matching ratio of one or more letters included in the reference word among one or more letters included in the recognition word and one or more reference words stored in the memory. Reclassifying inappropriate text images;
Containing,
A computer program stored on a computer readable storage medium.
According to claim 1,
When the character determined as the result of the classification of the character image is not suitable, the operation of re-classifying the character image and re-determining the characters included in the character image may include:
Receiving manual correction information for the text image from a user computing device; And
Re-classifying the text image based on the manual correction information;
Containing,
A computer program stored on a computer readable storage medium.
According to claim 1,
Displaying inappropriate information in the character image when the character determined as the result of the classification of the character image is not suitable;
Containing more,
A computer program stored on a computer readable storage medium.
According to claim 1,
The first training data set,
Including a predetermined number of first learning data collected after learning the character recognition model, or including first learning data collected during a predetermined period,
A computer program stored on a computer readable storage medium.
According to claim 1,
The operation of generating an updated character recognition model based on the first training data set including one or more first training data may include:
Classifying the character image of the first learning data into one character as an input of the character recognition model;
Calculating an error by comparing a letter determined as a result of the learning classification based on classification of the first learning data and a letter determined as the result of the re-classification, which is a label of the first learning data; And
Generating an updated character recognition model by updating the weight of the character recognition model based on the error;
Containing,
A computer program stored on a computer readable storage medium.
According to claim 1,
The operation of generating an updated character recognition model based on the first training data set including one or more first training data may include:
A new character recognition model is trained based on at least some of the second training data and the first training data set, which are the basis of the pre-trained character recognition model training, to generate an updated character recognition model. To do;
Containing,
A computer program stored on a computer readable storage medium.
According to claim 1,
The updated character recognition model,
At least partially sharing the weight of the pre-trained character recognition model,
A computer program stored on a computer readable storage medium.
According to claim 1,
Classifying a character image included in the document image using the character recognition model and the updated character recognition model; And
Ensemble the classification result of the character recognition model and the classification result of the updated character recognition model to generate a final classification result for characters included in the character image;
Containing more,
A computer program stored on a computer readable storage medium.
According to claim 1,
Classifying a text image included in the document image as a first text through the text recognition model;
If the first character corresponds to a character classification class determined based on the first learning data, determining to classify the character image using the updated character recognition model; And
Classifying the text image using the updated text recognition model to generate a final classification result for text included in the text image;
Containing more,
A computer program stored on a computer readable storage medium.
In the method for updating the character recognition model,
Determining each character included in the character image by classifying each one or more character images included in the document image into each character through a character recognition model including one or more pre-trained network functions;
Determining whether a letter determined as a result of the classification of the letter image based on the classification is a suitable letter;
Re-determining a character included in the character image by re-classifying the character image when the character determined as the result of the classification of the character image is not suitable;
Generating the first learning data using the text image as the input of the first learning data and the text determined as a result of the reclassification as a label of the first learning data; And
Generating an updated character recognition model based on the first set of learning data including one or more first learning data;
Containing,
Method for updating the character recognition model.
As a computing device for updating the character recognition model,
A processor comprising one or more cores; And
Memory;
Including,
The processor,
Each character image included in the document image is classified into each character through a character recognition model including one or more pre-trained network functions to determine characters included in the character image,
It is determined whether a letter determined as a result of the classification of the letter image based on the classification is a suitable letter,
If the character determined as the result of the classification of the character image is not suitable, the character image is reclassified to re-determine the characters included in the character image,
The first learning data is generated using the text image as the input of the first learning data, and the text determined as a result of the reclassification as a label of the first learning data, and
Generating an updated character recognition model based on a first set of learning data comprising one or more first learning data,
Computing device for updating the character recognition model.

Images