KR102073644B1 - Text recognition apparatus included in image, text recognition method and recording media storing with program for the performing method - Google Patents

Abstract

Disclosed is a text recognition apparatus. The apparatus for recognizing text includes one or more processors, memory, and storage. The memory loads computer programs executed by the processor. Storage stores computer programs and data. The computer program includes a feature extraction module for extracting feature values from each of a plurality of pixels included in the image, and a plurality of recognition components constituting text included in an image among the plurality of components using the extracted feature values. A component determining module for determining and a text completion module for combining the plurality of recognition components determined for each pixel to complete the text.
As a result, the text recognition accuracy and the recognition speed are improved in the image in which letters and expressions are mixed.

Description

Text recognition apparatus included in image, text recognition method and recording media storing with program for the performing method}

The present invention relates to a recording medium on which a text recognition apparatus, a text recognition method, and a program for executing the text recognition method included in an image are recorded. More particularly, the present invention relates to a text recognition apparatus, a text recognition method, and a text without performing segmentation on an image. A recording medium having recorded thereon a program for executing a recognition method.

As IT technology advances, there are various devices that provide video such as TVs, computers, smart phones, tablets, wearable devices, and virtual reality devices. In addition, various studies are being conducted to recognize text existing in an image while providing an image and to use the recognized text.

Recognition of text is performed by the electronic device analyzing the image file. An example of the prior art for recognizing text is region-based recognition. Region-based recognition performs segmentation for segmenting an image for each character, and determines whether text exists in each divided region, and if text exists in the region, determines whether it is Korean or English. Furthermore, optical character recognition (OCR) is performed in accordance with the language of the text in the divided region to recognize the text in the image.

In order to recognize text included in an image through region-based recognition, segmentation must be performed on the image. Therefore, the area-based recognition has a disadvantage in that it takes a long time to recognize the text on the image because the area is divided by the character as the number of characters included in the image. In addition, since the text recognition is possible only by grasping the language included in the area in advance, there is a problem of recognizing the text through several steps.

Furthermore, when Korean, mathematical, and English are mixed and present on an image, it is difficult to precisely perform segmentation because characteristics such as height or ligature are different for each letter.

That is, the text recognition method according to the prior art has a problem that the accuracy and speed of the text recognition falls.

Korean Patent Publication No. 2009-0035541

SUMMARY OF THE INVENTION The present invention has been made to solve the above-described problem, and a method, apparatus and text recognition method for recognizing text provided by recognizing text in an image more quickly and accurately by recognizing text included in an image without performing segmentation. A recording medium is provided which records a program for performing the program.

According to an embodiment of the present invention for solving the above problems, at least one processor; A memory for loading a computer program executed by the processor; And a storage for storing the computer program and data, wherein the computer program comprises: a feature extraction module for extracting feature values from each of the plurality of pixels included in the image; each of the plurality of pixels using the extracted feature values A component determination module for determining a recognition component constituting text included in the image among a plurality of components, and a text completion module for combining the recognition components determined for each pixel to complete the text. Is provided. As a result, the text recognition accuracy and the recognition speed are improved in the image with mixed letters and expressions.

The feature extraction module extracts the feature value by applying a first filter to each of the plurality of pixels included in the image, and the component determination module extracts the feature value extracted according to the application of the first filter. Applying a second filter with respect to the pixel to calculate the probability distribution that each of the plurality of components can be present for each pixel, and by using the calculated probability distribution of the plurality of components as a recognition component of each pixel You can decide.

The component determining module is configured to determine, as a group of pixels, at least one pixel adjacent to each other within a predetermined distance of the plurality of pixels and having the determined recognition components equal to each other after the recognition component is determined for each of the plurality of pixels. The recognition component may be determined to exist in the group of pixels.

The component determination module calculates a center coordinate of each recognition component based on a shape of each recognition component determined for each pixel, and determines center pixels based on the calculated center coordinates, wherein the plurality of pixels Pixels whose distances from the respective center pixels are less than or equal to a reference distance level and whose probability that the determined component exists exceed a reference probability level may be determined as a group of pixels in which each of the recognition components exists.

The computer program may further include a training module for training at least one of the feature extraction module, the component determination module, and the text completion module.

The training module may generate a test image and correct answer data.

When the training module transmits the test image to the feature extraction module, the feature extraction module applies a first filter to the test image to extract a test feature value, and the component determination module is configured to apply the second feature to the test feature value. A test probability distribution is calculated and applied to a training module by applying a filter, and the training module compares the correct answer data with the calculated test probability distribution, and the correct answer data and the test probability distribution are predetermined based on a comparison result. If the difference is more than the level, at least one of the first filter and the second filter may be adjusted.

The component may include at least one of a character, a component of an expression, and a space.

According to an embodiment of the present invention for solving the above problems, a text recognition method performed by a text recognition apparatus, the method comprising: extracting a feature value from each of a plurality of pixels in an image; Determining a recognition component constituting text included in the image among a plurality of components in each of the plurality of pixels using the extracted feature values; And combining the recognition components determined for each pixel to complete the text.

Extracting a feature value from each of the plurality of pixels in the image includes extracting the feature value by applying a first filter to each of the plurality of pixels included in the image, wherein the extracted feature value Determining a recognition component constituting the text included in the image among the plurality of components in each of the plurality of pixels by using a second filter, applying a second filter to the feature value extracted in accordance with the application of the first filter Calculating a probability distribution in which each of the plurality of components may exist for each pixel; And determining a selected component as a recognition component of each pixel by using the calculated probability distribution among the plurality of components.

Determining a selected component as the recognition component of each pixel by using the calculated probability distribution among the plurality of components, after the recognition component is determined for each of the plurality of pixels, a predetermined one of the plurality of pixels Determining at least one pixel that is adjacent to each other within a distance and that the determined recognition components are equal to each other as a group of pixels; And determining that the recognition component exists in the group of pixels.

The determining of the at least one pixel as a group of pixels may include calculating a center coordinate of each recognition component based on a shape of each recognition component determined for each pixel; Determining center pixels based on the calculated center coordinates; And determining, as a group of pixels in which each of the recognition components exists, pixels in which the distance from each of the plurality of pixels is less than or equal to a reference distance level and the probability that the determined component is present exceeds a reference probability level. can do.

The method may further include training the text recognition device.

Training the recognition of the text may include generating a test image and correct answer data.

The training of recognition of the text may include: transmitting, by the training module, the generated test image to the feature extraction module; Extracting a test feature value by applying the first filter to the test image; Calculating, by the component determining module, a test probability distribution by applying a second filter to the test feature value and delivering the test probability distribution to the training module; Comparing the correct answer data with the calculated test probability distribution; And adjusting at least one of the first filter and the second filter when the correct answer data and the test probability distribution differ by more than a predetermined level based on the comparison result.

The component may include at least one of a character, a component of an expression, and a space.

According to one embodiment of the present invention for solving the above problems of the present invention, there is provided a computer-readable recording medium on which a program for executing a text recognition method is recorded.

According to an embodiment of the present invention, text recognition accuracy and recognition speed are improved in an image in which letters and expressions are mixed.

Furthermore, according to another embodiment of the present invention, it is possible to correctly distinguish and recognize text including Korean and mathematical expressions in an image.

In addition, according to another embodiment of the present invention, through the machine learning performed on the text recognition device it is possible to recognize the text faster and more accurately.

1 illustrates an example of use of a user terminal according to an embodiment of the present invention.
2 is a block diagram of a system using text recognition included in an image according to an embodiment of the present invention.
3 is a block diagram of a text recognition apparatus according to an embodiment of the present invention.
4 illustrates a module included in a text recognition program according to an embodiment of the present invention.
5 illustrates an example of extracting a feature included in an image according to an embodiment of the present invention.
FIG. 6 illustrates a digital representation of an input object, a mask, and an output object when extracting a feature included in an image according to an exemplary embodiment of the present invention.
7 illustrates an example of extracting a probability distribution according to an embodiment of the present invention.
8 illustrates an example of calculating a center pixel according to an embodiment of the present invention and determining pixels at a predetermined distance from the center pixel as a group of pixels.
9 is a flowchart illustrating a method of recognizing text included in an image according to an embodiment of the present invention.
10 illustrates a text recognition program including a training module according to another embodiment of the present invention.
11 is a flowchart illustrating a method of recognizing text included in an image according to another exemplary embodiment of the present invention.
12 is a block diagram of a text recognition device according to another embodiment of the present invention.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. Advantages and features of the present invention, and methods for achieving them will be apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but can be implemented in various forms, and only the embodiments are intended to complete the disclosure of the present invention, and the general knowledge in the art to which the present invention pertains. It is provided to fully convey the scope of the invention to those skilled in the art, and the present invention is defined only by the scope of the claims. Like reference numerals refer to like elements throughout.

Unless otherwise defined, all terms used in the present specification (including technical and scientific terms) may be used in a sense that can be commonly understood by those skilled in the art. In addition, terms that are defined in a commonly used dictionary are not ideally or excessively interpreted unless they are specifically defined clearly. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. In this specification, the singular also includes the plural unless specifically stated otherwise in the phrase.

In the present specification, text refers to interchangeable information that can be written and interpreted by a person, and means any one of letters, numbers, symbols, figures, and pictures, and a combination thereof. In particular, according to an embodiment of the present invention, the text may be a result of image recognition by a text recognition apparatus to be described later. Hereinafter, the idea of the present invention described by the embodiments may be applied to a device that recognizes text included in an image and uses the same. The text recognition apparatus according to the present invention may be applied to an external device such as a computer, a portable terminal, a smartphone, a server, a wearable device, and various electronic devices.

1 illustrates an example of use of a user terminal according to an embodiment of the present invention.

Referring to FIG. 1, the user terminal 100 photographs the paper 10 on which text is written according to a user's manipulation and generates an image 101. The generated image 101 may be displayed on a display unit (not shown) of the user terminal 100. The user terminal 100 may transmit the image 101 to an external device by a user's additional manipulation. Here, the external device is a computing device capable of communicating with the user terminal 100 through a network. According to an embodiment, the external device may be a server device capable of recognizing text on an image, but embodiments of the present invention are not limited thereto, and the external device may be a terminal device such as a desktop, a laptop, a smartphone, a tablet, and the like. It may be.

The image 101 is not limited to being generated by directly photographing the user terminal 100. The image 101 may be received from the outside or may be generated and stored in the user terminal 100 according to a user's manipulation.

The external device recognizes the text in the received image 101 and transmits the recognized text to another terminal device. When the external device receives the information 103 corresponding to the recognized text from another terminal device, the external device transfers the received information 103 to the user terminal 100 again. The received information 103 may be displayed on the display unit of the user terminal 100.

As one example, the text included in the paper 10 may include a problem. In addition, the information 103 corresponding to the recognized text may include a correct answer to the problem and a description thereof.

2 is a block diagram of a system using text recognition included in an image according to an embodiment of the present invention.

Hereinafter, the configuration and operation of a system using text recognition included in an image according to an embodiment of the present invention will be described in detail with reference to FIG. 2. For convenience of explanation, a system using text recognition included in an image is also referred to as a system hereinafter.

Referring to FIG. 2, the system may include a user terminal 100, a text recognition apparatus 200, and a correct answer provider terminal 300. The user terminal 100, the text recognition apparatus 200, and the correct answer provider terminal 300 are composed of electronic devices capable of transmitting and receiving data to and from each other. To this end, for example, the user terminal 100, the text recognition device 200 or the correct answer provider terminal 300 is a server, a fixed computer device such as a desktop PC, laptops, smartphones, tablet PCs, portable terminals, wearable devices such as mobile computers It may consist of any of the devices.

The user terminal 100 transmits an image including the text to the text recognition apparatus 200. In particular, according to an embodiment of the present invention, the text recognition apparatus 200 may be the external device described above in the description of FIG. 1.

The text recognition apparatus 200 recognizes the text included in the image and transmits the recognized text to the correct answer provider terminal 300 or stores the text in a database.

The correct answer provider terminal 300 receives the recognized text from the text recognition apparatus 200. The user of the correct answer provider terminal 300 inputs information corresponding to the recognized text to the correct answer provider terminal 300 with reference to the recognized text. If the recognized text is a math problem as an example, the corresponding information may be a correct answer to the math problem. The correct answer provider terminal 300 transmits data including the input information to the text recognition apparatus 200. In particular, according to an embodiment of the present invention, the answer provider terminal 300 may be another terminal device described above in the description of FIG.

The text recognition apparatus 200 transmits the information related to the recognized text stored as data and / or the information received from the correct answer provider terminal 300 to the user terminal 100. As an example, when the recognized text is a mathematical problem, the text recognizing apparatus 200 searches whether the stored data has the same problem. If the answer to the same problem is included as data, the text recognition apparatus 200 may transmit the searched solution to the user terminal 100. In addition, the text recognizing apparatus 200 may transmit the related information to the user terminal 100 as related information when a similar or related problem is detected.

The user terminal 100 provides the user with information received from the text recognition apparatus 200.

Although not shown in FIG. 2, the system may further include an intermediary server for mediating data transmission, a server for data synchronization, etc. in addition to the user terminal 100, the text recognition apparatus 200, and the correct answer provider terminal 300. have.

In addition, a plurality of user terminals and a correct answer provider terminal may be provided so that multiple users may transmit images and multiple users may provide corresponding information.

The user terminal 100 and the correct answer provider terminal 300 are divided into terminal devices according to roles for convenience of explanation, and the same terminal devices may perform functions by different roles from each other as necessary.

3 is a block diagram of a text recognition apparatus according to an embodiment of the present invention.

Referring to FIG. 3, the text recognition apparatus 200 receives an image including text as described above with reference to FIG. 2. The text recognition apparatus 200 recognizes text in the received image. The text recognition apparatus 200 transmits and stores the text recognized in the image to the outside. The text recognition apparatus 200 receives information corresponding to the recognized text from the outside or searches for information previously stored in the text recognition apparatus 200 and transmits the information to the apparatus providing the image.

Referring to FIG. 3, the text recognition apparatus 200 includes a processor 201, a network interface 203, a memory 205, and a storage 207.

The processor 201 controls the overall operation of the text recognition apparatus 200. The processor 201 includes a central processing unit (CPU), a micro processor unit (MPU), a micro controller unit (MCU), or any type of processor well known in the art. In addition, the processor 201 performs operations on at least one application or program for executing a method according to embodiments of the present invention. The text recognition apparatus 200 may include at least one processor.

The network interface 203 communicates with the outside of the text recognition apparatus 200. The network interface 203 allows the text recognition apparatus 200 to communicate with external devices and to transmit and receive data. Types of data transmitted and received are various, such as a voice, an image, a text, a video, and the like, and are not limited to the above examples.

The network interface 203 may include a connection for wired communication to communicate with an external device. The connection unit may transmit / receive signals / data according to standards of high definition multimedia interface (HDMI), HDMI-consumer electronics control (USB-CEC), USB, component, etc., and at least one corresponding to each standard The above connector or terminal is included. The text recognition apparatus 200 may perform wired communication with a plurality of servers through a wired local area network (LAN).

The type of communication supported by the network interface 203 is not limited to wired communication. The network interface 203 may include an RF circuit for transmitting and receiving RF (radio frequency) signals for performing wireless communication, and may include Wi-fi, Bluetooth, Zigbee, Ultra-Wide Band (UWM), and Wireless USB. , NFC may be configured to perform at least one communication of Near Field Communication (NFC).

The memory 205 stores various data, commands and / or information. At least a part of a program may be loaded into the memory 205 from the storage 207. The memory 205 is composed of volatile memory capable of reading and writing and having a high read or write speed. As one example, the memory 205 may be provided with any one of RAM, DRAM, or SRAM.

The storage 207 is a module for storing information and data received from the outside. The storage 207 should retain data even when the power supplied to the text recognition device 200 is cut off, and may be provided as a non-volatile memory that can be written to reflect changes. The storage 207 may be a hard disk, a removable disk, such as a read only memory (ROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EPROM), flash memory, or the like, or any form well known in the art. And a computer readable recording medium.

The storage 207 may store a text recognition program 210 and data 212. The data 212 may include text recognized in an image and records exchanged with an external device. As an example, the data 212 may further include various problems, keywords of the problem, difficulty of the problem, a subject, a grade of learning, and the like as related information. As another example, the data 212 may include a description of the problem, information about the answer provider who wrote the commentary, and the like.

Hereinafter, methods according to embodiments of the present invention will be described as being performed by the text recognition apparatus 200.

Hereinafter, embodiments of the present invention will be described in detail with reference to the above description with reference to FIGS. 1 to 3.

4 illustrates a module included in a text recognition program according to an embodiment of the present invention.

Referring to FIG. 4, the text recognition program 210 according to an embodiment of the present invention includes a feature extraction module 211, a component determination module 213, and a text completion module 215. The text recognition program 210 may further include a storage module 217 and a control module 219. Hereinafter, the functions of each component of the text recognition program 210 may be implemented by being executed by the processor 201 referred to in FIG. 3 as a module unit. That is, the operations performed by the respective modules, such as the feature extraction module 211, the component determination module 213, the text completion module 215, the storage module 217, and the control module 219, which will be described later, may be performed by a processor ( By 201) means a function implemented by executing each module.

The feature extraction module 211 is configured to perform convolution on the received image to extract feature values. By convolution, at least a part of the received image is multiplied by a weight of a mask to image the output to output an output having a smaller size than the received image. An image can contain various components, such as numbers, letters, symbols, and margins. In this case, the mask is a software tool for extracting feature values from each component included in the image, and is also referred to as a window or a kernel.

The feature extraction module 211 outputs output data by applying a first filter to the input data. Output data on which the first filter is applied is also referred to as a feature value. Hereinafter, a process in which the first filter is applied will be referred to as first filtering.

According to the first filtering, the output of the feature value may be performed in one or more steps. As described above, while one or more steps are performed, different masks may be applied to each step. The feature extraction module 211 performs first filtering on all pixels of the input data without performing letter segmentation on the image. Masks can be nested in multiple layers to more accurately extract feature values.

In this case, the number of layers of the mask may correspond to the number of features to be extracted from each component of the image. The feature value may mean a value corresponding to a feature of each component that can be included in an image. For example, the feature of each component included in the image means a feature of a straight shape, a feature of a round shape, a feature of a curved shape, and the like. Masks of various layers are provided in the feature extraction module 211 to extract features of various shapes as feature values. The feature is not limited to the above-described content, and there may be various features for recognizing letters in the image. The larger the number of hierarchies of masks, the more various feature values are extracted. The more various feature values are extracted, the more accurately the components of the text present in the image are determined.

Hereinafter, for convenience of description, different weights of the plurality of masks used in the first filtering may be referred to as first weights. For example, adjusting the first weight means modifying at least some of different weights of the plurality of masks.

Hereinafter, the operation of the feature extraction module 211 will be described in more detail with reference to FIGS. 5 and 6. 5 illustrates an example of extracting a feature included in an image according to an embodiment of the present invention. 6 illustrates an example of extracting a feature included in an image according to an embodiment of the present invention. In particular, in FIG. 6, a feature included in an image, a mask for extracting the feature, and a machine language processing process of the extracted output data are shown as an example.

Referring to FIG. 5, the first output data 502 is generated by applying the first mask 501 to the input data 500 which is an image, and the second mask 503 is applied to the first output data 502. The process of generating the second output data 504 is shown. The input data 500 includes more pixels than the first output data 502, and the first output data 502 includes more pixels than the second output data 504. The first mask 501 and the second mask 503 may have different sizes.

Referring to FIG. 6, input data 601, a first mask 601, and a first output data 602, in which a machine language is input to each pixel as an example, are described in more detail with reference to FIG. 5. The numbers in each pixel are merely examples provided for convenience of explanation, and the spirit of the present invention is not limited to the input numbers and the size of the pixels.

The first mask 501 and the second mask 503 may be provided with a plurality of layers each having various weights in order to extract various feature values. The number of output data increases by the number of layers of masks in each stage. The method may further include extracting feature values according to the size of the input data 500 to be processed. When the step is added, the third and fourth masks having a plurality of layers may be additionally provided in the feature extraction module 211.

The component determination module 213 calculates a probability that a specific component may exist for each pixel by using the feature values extracted by the feature extraction module 211, and based on the calculation result, the component determination module 213 calculates a component that is a component of text included in the image. Determined by the recognition component. The component determination module 213 is also called a classifier. The component determination module 213 uses a neural network. Components include, for example, consonants and vowels in Korean, modifiers, numbers, alphabets, and the like. Space is a state in which there is no letter, and according to an embodiment of the present invention, it may be treated as one of the components of the letter. The component determination module 213 calculates a probability that each component exists in the pixel by applying a second filter having a second weight value to the feature values extracted for each pixel. For example, when the text ab is included in the image, the component determination module 213 may calculate a high probability that the component 'a' is positioned in the pixels where a is located on the image. In addition, the component determination module 213 may calculate a high probability that the component 'b' is positioned in the pixels where b is positioned. Furthermore, the component determination module 213 may refer to the feature values of the surrounding pixels in the blank pixels existing inside the letters a and b, thereby calculating a high probability of the presence of the components 'a' and 'b'. have. At this time, the component determination module 213 may calculate a low probability that other components are located in the pixels where a and b are located. In addition, the component determination module 213 may calculate a high probability that a blank component is positioned in pixels where a and b are not located. Next, the component determination module 213 determines the recognition component using the calculated result.

Hereinafter, the detailed operation of the component determination module 213 will be described in more detail with reference to FIGS. 7 and 8.

7 illustrates an example of extracting a probability distribution according to an embodiment of the present invention.

Referring to FIG. 7, an example in which the component determination module 213 calculates a probability distribution using extracted feature values is illustrated. The output data used by the component determination module 213 is final output data output from the feature extraction module 211. As described above, the step of extracting the feature value from the input data is not limited to two steps to which the first mask 501 and the second mask 503 of FIG. 5 are applied. The output data 504 is assumed to be final output data.

The component determination module 213 calculates a probability that each component exists in each pixel of the image by using the feature value included in the second output data 504. To this end, a second filter having a second weight value set may be applied to the input second output data 504. The component determination module 213 generates a layer for each component. Subsequently, the component determination module 213 generates a probability distribution 700 that illustrates the probability that each component exists in each pixel.

Looking at the extracted probability distribution 700, 'space', 'a', 'b', '2', 'c', '4', 'd', '+', '=', Pixels with high probability that each component is present. The component determination module 213 calculates probability distributions for other components that do not actually exist in the image, such as 'c', 'e', 'f', '<', etc., but the illustration is omitted. In other words, the component determination module 213 calculates the probability of existence by generating layers for all other components, instead of calculating the probability only for the components existing in the image. In the drawings, for convenience, it is calculated that the probability that the corresponding component is located only where the specific component is actually located is not limited thereto. That is, the probability that the corresponding component exists and the probability that the other component exists in the pixel in which the specific component is located may exist. For example, a specific pixel located adjacent to the component 'a' may have a 70% probability of having 'a' present and a 40% probability of having 'e' present.

The component determination module 213 may determine, based on the calculation result, the component having the highest probability of being present in a specific pixel as a recognition component that is a component recognized in the pixel.

8 illustrates an example of calculating a center pixel according to an embodiment of the present invention and determining pixels at a predetermined distance from the center pixel as a group of pixels.

The component determination module 213 may determine that one component is located in the group of pixels 1001, 1011, and 1021. To this end, the component determination module 213 calculates a center coordinate of the recognition component based on the determined shape of the recognition component, and determines the center pixels 1000, 1010, and 1020 where the center coordinate is located. In addition, the component determination module 213 may include pixels in which the distance between the center pixels 1000, 1010, and 1020 is equal to or less than a predetermined reference distance level, and the probability that the determined recognition component exists exceeds a predetermined reference probability level. 1001, 1011, and 1021. The reference distance level may be a distance value preset in order for the component determination module 213 to determine a pixel including the recognition component from the calculated center pixel. In addition, the reference probability level may be a probability value that serves as a reference when the component determination module 213 determines a component present in a group of pixels as a recognition component.

The component determination module 213 is within a reference distance level from the center pixel, and for the pixels whose probability that a component such as a recognition component determined at the center pixel exists exceeds the reference probability level, the probability that another component exists is calculated even if the group is calculated. Can be determined in pixels. The component determination module 213 determines a component, such as a recognition component of a center pixel, as a recognition component for a group of pixels.

This is true even if the group of pixels has a higher probability that there is a different component from the recognition component of the center pixel. For example, the component determination module 213 may calculate a probability that 'a' exists in a specific pixel as 50% based on the calculated feature value, and calculate a probability that 'e' exists as 65%. If the particular pixel is within a predetermined reference distance level from the center pixel 1000 of the 'a' component, and the probability that 'a' exists also exceeds the reference probability level, the component determination module 213 has a probability that 'e' exists. Although 'a' is calculated to be higher than the probability of existence, the 'a' component may be determined as a recognition component in the corresponding pixel. The component determination module 213 transfers the recognition component determined for each pixel to the text completion module 215.

That is, the feature extraction module 211 and the component determination module 213 according to an embodiment of the present invention extract feature values for the entire image and calculate a probability that a component exists using the extracted feature values. According to the present invention, after performing segmentation for each letter included in the image, a process of determining whether a letter included in each segment generated according to the segmentation is Korean, English, or mathematical is omitted. According to an embodiment of the present invention, it is possible to directly extract and recognize the letters included in the entire image, so that fewer judgments are performed compared to the prior art in which judgment on the letters is individually performed after the segmentation and thus included in the image. Recognize text faster and more accurately

The text completion module 215 combines the recognition components determined by the component determination module 213 to complete the text. The text completion module 215 combines the recognition components that are not yet combined to complete the text. For example, the text completion module 215 combines these when the recognition component is a Hangul alphabet (a, ㅏ, ㅇ), thereby completing the letter 'gang'. Further, the text completion module 215 combines the recognition components which are equations (signs, fraction display, sigma, combination display, exponent, etc.) to complete the actual equations included in the image. Then, the text completion module 215 combines the completed letters and expressions into a single line. In this process, if it is determined that the characters included in a line are obvious coming based on the context before and after, it may be corrected automatically or may be corrected by asking the user for confirmation. To this end, the text recognition apparatus 200 may include a display and output an interface for obtaining a user's confirmation and correcting a mistake.

The operation of the text completion module 215 is not limited to the above description, and is implemented to combine the determined recognition components in various ways.

The storage module 217 stores the information generated according to the execution of the text recognition program 210 and transmitted / received to the outside as data 212 in the storage 207 of the text recognition apparatus 200.

The control module 219 controls the overall operation of each module of the text recognition program 210.

9 is a flowchart illustrating a method of recognizing text included in an image according to an embodiment of the present invention.

First, in operation S500, the text recognition apparatus 200 receives an image from the user terminal 100. The image may be generated by directly photographing by the user terminal 100, may be received by the user terminal 100 from the outside through the Internet, or may be generated by a user operating the user terminal 100. .

In operation S501, the feature extraction module 211 extracts a feature value from an image. The feature extraction module 211 extracts a feature value by applying a first filter of a first weight value to all pixels of the image.

In operation S502, the component determining module 213 determines the recognition component based on the extracted feature value. The component determination module 213 may calculate the probability that each component exists for each pixel by using the extracted feature value, and determine the component having the highest probability of being located in the pixel based on the calculated probability as the recognition component.

Finally, in operation S503, the text completion module 215 combines the determined recognition components to complete the text.

10 illustrates a text recognition program including a training module according to another embodiment of the present invention.

Another text recognition program 210 according to another embodiment of the present invention may further include a training module 1100 for learning the accuracy of text recognition. In the present embodiment, the text recognition program 210 learns to recognize text more accurately through machine learning using the training module 1100.

The training module 1100 learns at least one of the feature extraction module 211, the component determination module 213, and the text completion module 215. The training module 1100 generates learning data for learning, and provides at least some of them to the feature extraction module 211. The training data includes, for example, test images and correct answer data. The correct answer data may include a probability distribution value for each component of the text included in the test image as the correct probability distribution, and include the text included in the test image as the correct answer text. The training module 1100 renders an image including equations and Korean characters, adds noise in a random form to the image, and generates correct answer data for components existing in corresponding coordinates for each coordinate in the rendered image. The correct answer data includes the correct answer probability distribution. As another example, the training module 1100 may receive from the outside without generating the training data.

The feature extraction module 211 and the component determination module 213 extract a test feature value by applying a first filter having a first weight to a test image included in the received training data, and extract a test feature value from the extracted test feature value. A probability distribution may be calculated by applying a second filter having a weight. The probability distribution calculated from the test image will be referred to as a test probability distribution below.

The training module 1100 may compare the test probability distribution and the correct answer probability distribution and train the feature extraction module 211 and the component determination module 213 based on the comparison result. As an example, the training module 1100 calculates an error level by comparing a test probability distribution and a correct answer probability distribution. Further, the training module 1100 may adjust the first weight value and / or the second weight value based on the calculated error level. To this end, a back propagation algorithm may be applied to the text recognition program 210 including the training module 1100.

Further, the text completion module 215 may complete the text from the extracted test probability distribution. The training module 1100 may compare the text completed in the text completion module 215 with the correct answer text included in the correct answer data, and train the text completion module 215 based on the comparison result.

However, the process of training the feature extraction module 211, the component determination module 213 and the text completion module 215 by the training module 1100 according to the present invention is not limited to the above description. Various known machine learning techniques can be applied to the present invention.

11 is a flowchart illustrating a method of recognizing text included in an image to which a learning process is added according to another embodiment of the present invention.

The learning process is performed in operations S600 to S602.

First, in operation S600, the training module 1100 receives training data including a test image and correct answer data. According to another example, the training module 1100 may directly generate learning data. The training module 1100 transfers the test image to the feature extraction module 211.

In operation S601, the feature extraction module 211 extracts a feature value from the test image, and the component determination module 213 extracts a test probability distribution using the extracted feature value.

Then, in S602, the training module 1100 compares the test probability distribution and the correct answer data. Then, at least one of the first weight of the first filter of the feature extraction module 211 and the second weight of the second filter of the component determination module 213 is adjusted based on the comparison result.

After the above learning process is completed, the text recognition apparatus 200 recognizes the text in the previously described image through operations S1103 to S1106.

First, in operation S603, the text recognition apparatus 200 receives an image.

In operation S604, the feature extraction module 211 extracts a feature value by applying the first filter having the first weight value adjusted above to the received image.

Then, in operation S605, the component determination module 213 calculates the probability distribution by applying the second filter having the adjusted second weight to the extracted feature value. Then, the component determination module 213 determines the recognition component for each pixel using the calculated probability distribution.

Finally, in operation S606, the text completion module 215 combines the determined recognition components to complete the text.

12 is a block diagram of a text recognition device according to another embodiment of the present invention.

Referring to FIG. 12, the storage 207 in the present embodiment may further include a slicer 1201.

The slicer 1201 is a program stored in the storage 207, and its function is implemented by being executed by the processor 201 referred to in FIG. 3. That is, the operation of the slicer 1201 to be described later refers to a function implemented by being executed by the processor 201. When the image 1200 is transmitted from the network interface 203, the slicer 1201 determines the size of the received image 1200. The slicer 1201 generates image fragments by dividing the image 1200 into an appropriate size when the received image 1200 has a predetermined size or more. The slicer 1201 transfers the pieces of the image to the text recognition program 210. That is, the slicer 1201 cuts the image 1200 and provides the text recognition program 210 to the text recognition program 210 so that the text recognition program 210 divides and recognizes the image 120 that is too large at once.

As an additional embodiment, the slicer 1201 may determine whether a letter is located at a boundary where the image 1200 is cut off. If it is determined that the letters are partially cut and the letters are located, the slicer 1201 may appropriately adjust the size of the cut so that the letters included in the image 1200 are not cut in the middle.

The text recognition program 210 recognizes the text 1210 in the image fragment and transmits the recognized text to the slicer 1201. The method for the text recognition program 210 to recognize the text 1210 in the image 1200 has been described above.

 The slicer 1201 collects the recognized texts from the image fragments provided by the text recognition program 210. The slicer 1201 stores the text 1210, which is the result of the aggregation, in the storage as data 212 or transmits the data to the network interface 203. The text 1210 transmitted to the network interface 203 is output to the outside.

According to the present exemplary embodiment, even if an image 1200 having an excessively large size is received, the text 1210 may be recognized by processing without delay.

100 User Terminal
200 Text Recognition Device
201 processor
203 network interface
205 memory
207 storage
210 text recognition program
212 data
300 correct answer providers

Claims (17)

One or more processors;
A memory for loading a computer program executed by the processor;
And
Storage for storing said computer program and data,
The computer program,
Feature extraction module for extracting a feature value by applying a first filter to each of the plurality of pixels included in the image,
A probability distribution in which a plurality of components may exist for each of the plurality of pixels may be calculated by applying a second filter to the extracted feature value, and the selected component is selected using the calculated probability distribution among the plurality of components. A component determination module for determining the recognition component of each pixel constituting the text included in the text, and
And a text completion module for completing the text by combining the recognition components determined for each pixel.
The component determination module,
After the recognition component is determined for each of the plurality of pixels, a center coordinate of each of the recognition components is calculated based on the shape of each of the recognition components determined for each pixel, and the center pixel is based on the calculated center coordinates. The center pixel and the distance of the plurality of pixels is less than the reference distance level, adjacent to each other within a predetermined distance,
Characterized in that the pixels of the probability that the determined recognition component is greater than the reference probability level is determined as a group of pixels that the determined recognition components are the same, and it is determined that the recognition component is present in the group of pixels, the text Recognition device. delete delete delete The method of claim 1,
The computer program further comprises a training module for training at least one of the feature extraction module, the component determination module, and the text completion module. The method of claim 5,
The training module is a text recognition device, characterized in that for generating a test image and the correct answer data. The method of claim 6,
When the training module delivers the test image to the feature extraction module,
The feature extraction module extracts a test feature value by applying a first filter to the test image, and the component determination module calculates a test probability distribution by applying a second filter to the test feature value and delivers it to the training module. ,
The training module compares the correct answer data with the calculated test probability distribution and, if the correct answer data and the test probability distribution differ by more than a predetermined level, based on a comparison result, at least one of the first filter and the second filter. Text Recognition Device. The method of claim 1,
And the component includes at least one of a character, a component of an expression, and a space. In the text recognition method performed by the text recognition apparatus,
Extracting feature values by applying a first filter to each of the plurality of pixels in the image;
A probability distribution in which a plurality of components may exist for each of the plurality of pixels may be calculated by applying a second filter to the extracted feature value, and the selected component is selected using the calculated probability distribution among the plurality of components. Determining as a recognition component of each pixel constituting text included in the text; And
Comprising a combination of the recognition components determined for each pixel to complete the text,
Determining the recognition component,
Calculating a center coordinate of each recognition component based on a shape of each recognition component determined for each pixel after the recognition component is determined for each of the plurality of pixels;
Determining center pixels based on the calculated center coordinates;
The determined recognition components of the plurality of pixels have a distance equal to or less than a reference distance level and adjacent to each other within a predetermined distance, and the probability that the determined recognition component exists exceeds a reference probability level is equal to each other. Determining with a group of pixels; And
Determining that the recognition component is present in the group of pixels;
Text recognition method. delete delete delete delete The method of claim 9,
Training the recognition of the text by the text recognition device;
The training may include generating a test image and correct answer data by the training. The method of claim 14,
Training the recognition of the text,
Extracting a test feature value by applying a first filter to the generated test image;
Calculating a test probability distribution by applying a second filter to the test feature value;
Comparing the correct answer data with the calculated test probability distribution; And
And adjusting at least one of the first filter and the second filter if the correct answer data and the test probability distribution differ by more than a predetermined level based on the comparison result. The method of claim 9,
And the component comprises at least one of a character, a component of an expression, and a space. A computer-readable recording medium having recorded thereon a program for executing the text recognition method according to any one of claims 9 and 14 to 16.

Images